Risk

Caspar Hare presents a compelling argument for “taking the sugar” in cases of opaque sweetening: you have no reason to take the unsweetened option, and you have some reason to take the sweetened one. I argue that this argument fails—there is a perfectly good sense in which you do have a reason to take the unsweetened option. I suggest a way to amend Hare's argument to overcome this objection. I then argue that, although the improved version fares better, there is (...) still room to resist Hare's argument—in a way that raises interesting questions about rational agency. In short, rationality is not about doing what one has the most reason to do; rather, it is about aiming to do what there is most reason to do. (shrink)

In situations where we ignore everything but the space of possibilities, we ought to suspend judgment—that is, remain agnostic—about which of these possibilities is the case. This means that we cannot sum our degrees of belief in different possibilities, something that has been formalised as an axiom of non-additivity. Consistent with this way of representing our ignorance, I defend a doxastic norm that recommends that we should nevertheless follow a certain additivity of possibilities: even if we cannot sum degrees of (...) belief in different possibilities, we should be more confident in larger groups of possibilities. It is thus shown that, in the type of situation considered (in so-called “classical ignorance”, i.e. “behind a thin veil of ignorance”), it is epistemically rational for advocates of suspending judgment to endorse this comparative confidence, while on the other hand it is shown that, even in classical ignorance, no stronger belief—such as a precise uniform probability distribution—is warranted. (shrink)

Fanatics claim that we must give up guaranteed goods in pursuit of extremely improbable Utopia. Recently, Wilkinson has defended Fanaticism by arguing that nonfanatics must violate at least one plausible rational requirement. We reject Fanaticism. We show that by taking stakes-sensitive risk attitudes seriously, we can resist the core premises in Wilkinson’s argument.

This thesis investigates the question of whether we can make transformative decisions rationally. The first chapter introduces and explores the nature of transformative experiences: what are they, and how do they bring about such drastic change? I argue that there is an tension between Paul’s (2014) characterisation of transformative experience and arguments that transformative experiences are imaginable. I propose a broader characterisation of transformative experiences on which transformation isn’t driven by experiential acquaintance. From Chapter 2 onwards, the thesis focuses on (...) whether we can rationally choose to have transformative experiences. Transformative experiences pose a problem to Expected Utility Theory, our standard theory of rationality. I introduce Expected Utility Theory and the problem posed by transformative experience before surveying prominent responses to the problem of transformative experience. Some responses are more promising given my characterisation of transformative experiences, such as Kind’s (2020a) Imaginative Scaffolding proposal, though I ultimately offer reasons to reject the available responses. Chapters 4 and 5 are devoted to developing and discussing Pettigrew’s (2015; 2019) solution to the problem of transformative experience: the Fine-Graining Response and the Aggregate Utility Solution. I argue that many transformative experiences involve constitutive risk and that this poses a problem for Pettigrew’s solution. The final substantive chapter of the thesis attempts to provide an alternative diagnosis of the problem of transformative experience for rational decision-making. I argue that if we are realists about decision-theoretic attitudes, the problem can be construed as a problem about self-knowledge of desire. On this proposal, the limitations of third-personal evidence are evident: whilst it can help us recognise what is desirable, it cannot settle the question of what we, ourselves, desire. (shrink)

Many philosophers accept that, as well as having a right that others not harm us, we also have a right that others not subject us to a risk of harm. And yet, when we attempt to spell out precisely what this ‘right against risk imposition’ involves, we encounter a series of notorious puzzles. Existing attempts to deal with these puzzles have tended to focus on the nature of rights – but I propose an approach that focusses instead on the nature (...) of risk. The key move is to distinguish two different ways in which to conceptualise the risk that a given action presents – one of which is linked to the notion of probability and the other to the notion of normalcy. (shrink)

Rowe has recently argued that pure risk of harm cannot itself be a harm. I respond to Rowe and argue that given an appropriate understanding of objective probabilities, pure objective risk of harm can itself be a harm.

Many people are worried about the harmful effects of climate change but nevertheless enjoy some activities that contribute to the emission of greenhouse gas (driving, flying, eating meat, etc.), the main cause of climate change. How should such people make choices between engaging in and refraining from enjoyable greenhouse-gas-emitting activities? In this chapter, we look at the answer provided by decision theory. Some scholars think that the right answer is given by interactive decision theory, or game theory; and moreover think (...) that since private climate decisions are instances of the prisoner’s dilemma, one rationally should engage in these activities provided that one enjoys them. Others think that the right answer is given by expected utility theory, the best-known version of individual decision theory under risk and uncertainty. In this chapter, we review these different answers, with a special focus on the latter answer and the debate it has generated. (shrink)

I defend a weak version of the Pigou-Dalton principle for chances. The principle says that it is better to increase the survival chance of a person who is more likely to die rather than a person who is less likely to die, assuming that the two people do not differ in any other morally relevant respect. The principle justifies plausible moral judgements that standard ex post views, such as prioritarianism and rank-dependent egalitarianism, cannot accommodate. However, the principle can be justified (...) by the same reasoning that has recently been used to defend the core axiom of ex post prioritarianism and egalitarianism, namely, Pigou-Dalton for well-being. The arguably biggest challenge for proponents of Pigou-Dalton for chances is that it violates State Dominance for social prospects. However, I argue that we have independent reason for rejecting State Dominance for social prospects, since it prevents a social planner from properly respecting people's preferences. (shrink)

In a recent paper in this journal Duncan Pritchard responds to an objection to the modal account of risk pressed by Ebert, Smith and Durbach ( 2020 ). In this paper, I expand upon the objection and argue that it still stands. I go on to consider a more general question raised by this exchange – whether risk is ‘objective’, or whether it is something that varies from one perspective to another.

Many philosophers assume that, when making moral decisions under uncertainty, we should choose the option that has the greatest expected moral value, regardless of how risky it is. But their arguments for maximizing expected moral value do not support it over rival, risk-averse approaches. In this paper, I present a novel argument for maximizing expected value: when we think about larger series of decisions that each decision is a part of, all but the most risk-averse agents would prefer that we (...) consistently choose the option with the highest expected value. To the extent that what we choose on a given occasion should be guided by the entire series of choices we prefer, then on each occasion, we should choose the option with the highest expected moral value. (shrink)

Many healthcare practices expose people to risks of harmful outcomes. However, the major theories of moral philosophy struggle to assess whether, when and why it is ethically justifiable to expose individuals to risks, as opposed to actually harming them. Sven Ove Hansson has proposed an approach to the ethical assessment of risk imposition that encourages attention to factors including questions of justice in the distribution of advantage and risk, people’s acceptance or otherwise of risks, and the scope individuals have to (...) influence the practices that generate risk. This paper investigates the ethical justifiability of preventive healthcare practices that expose people to risks including overdiagnosis. We applied Hansson’s framework to three such practices: an ‘ideal’ breast screening service, a commercial personal genome testing service, and a guideline that lowers the diagnostic threshold for hypertension. The framework was challenging to apply, not least because healthcare has unclear boundaries and involves highly complex practices. Nonetheless, the framework encouraged attention to issues that would be widely recognised as morally pertinent. Our assessment supports the view that at least some preventive healthcare practices that impose risks including that of overdiagnosis are not ethically justifiable. Further work is however needed to develop and/or test refined assessment criteria and guidance for applying them. (shrink)

The notion of risk plays a central role in economics, finance, health, psychology, law and elsewhere, and is prevalent in managing challenges and resources in day-to-day life. In recent work, Duncan Pritchard (2015, 2016) has argued against the orthodox probabilistic conception of risk on which the risk of a hypothetical scenario is determined by how probable it is, and in favour of a modal conception on which the risk of a hypothetical scenario is determined by how modally close it is. (...) In this article, we use Pritchard’s discussion as a springboard for a more wide-ranging discussion of the notion of risk. We introduce three different conceptions of risk: the standard probabilistic conception, Pritchard’s modal conception, and a normalcy conception that is new (though it has some precursors in the psychological literature on risk perception). Ultimately, we argue that the modal conception is ill-suited to the roles that a notion of risk is required to play and explore the prospects for a form of pluralism about risk, embracing both the probabilistic and the normalcy conceptions. (shrink)

We argue that two long-term goals of AI research stand in tension with one another. The first involves creating AI that is safe, where this is understood as solving the problem of value alignment. The second involves creating artificial general intelligence, meaning AI that operates at or beyond human capacity across all or many intellectual domains. Our argument focuses on the human capacity to make what we call “existential choices”, choices that transform who we are as persons, including transforming what (...) we most deeply value or desire. It is a capacity for a kind of value misalignment, in that the values held prior to making such choices can be significantly different from (misaligned with) the values held after making them. Because of the connection to existentialist philosophers who highlight these choices, we call the resulting form of risk “existentialist risk.” It is, roughly, the risk that results from AI taking an active role in authoring its own values rather than passively going along with the values given to it. On our view, human-like intelligence requires a human-like capacity for value misalignment, which is in tension with the possibility of guaranteeing value alignment between AI and humans. (shrink)

In response to the worry that autonomous generally intelligent artificial agents may at some point take over control of human affairs a common suggestion is that we should “solve the alignment problem” for such agents. We show that current discourse around this suggestion often uses a particular framing of artificial intelligence (AI) alignment as binary, a natural kind, mainly a technical‐scientific problem, realistically achievable, or clearly operationalizable. Each of these assumptions may not actually be true. We further argue that this (...) “Manhattan project framing” of AI alignment may bias societal discourse and decision‐making towards faster AI development and deployment than is responsible. (shrink)

It is estimated that five hundred billion dollars are spent on philanthropy every year. How should we spend those resources to do the most good? One possible answer, based on expected-value reasoning, is that we should spend those resources “fanatically” on interventions that can possibly produce enormous benefit, but with minuscule chance of success. This paper develops a new kind of knowledge-first decision theory that implies that we should not spend those resources fanatically. As such, this paper would be of (...) interest to anyone who wants to explore a fresh reason against acting fanatically, and it would be of interest to anyone looking for a template on how to develop a knowledge-first decision theory of their own. (shrink)

Does the potential creation of a time machine present an existential risk to our current timeline? Time travel is theoretically possible under general relativity, and there is steady progress (similar to Moore's Law) in developing ideas about how to create time machines with decreasing effort. While time travel may seem like a remote possibility due to its dependence on space travel to black holes, there is a concept of a quantum time machine (suggested by Deutsch in 1991 and further developed (...) by Lloyd in 2011) that would require much less energy to send a few bits back in time and could still cause paradoxes. An artificial superintelligence (ASI) might soon create such a device, especially in scenarios where multiple ASIs compete for control with each other. However, the catastrophic implications of time machines are largely unexplored in scientific research. Two common, yet contradictory, perspectives exist: either time machines are impossible, rendering risk moot, or Novikov’s self-consistency principle eliminates any chance of timeline alteration. I argue that Novikov’s principle itself can precipitate various catastrophes, for instance, by annihilating civilizations capable of building time machines. This paper classifies the catastrophic risks associated with potential future time machines. The aim is not to declare time machines inherently risky, but to emphasize the necessity of risk assessment given ongoing theoretical advancements in their construction. (shrink)

Those who worry about a superintelligent AI destroying humanity often appeal to the instrumental convergence thesis—the claim that even if we don’t know what a superintelligence’s ultimate goals will be, we can expect it to pursue various instrumental goals which are useful for achieving most ends. In this paper, we argue that one of these proposed goals is mistaken. We argue that instrumental goal preservation—the claim that a rational agent will tend to preserve its goals—is false on the basis of (...) the timing problem: an agent which abandons or otherwise changes its goal does not thereby fail to take a required means for achieving a goal it has. Our argument draws on the distinction between means-rationality (adopting suitable means to achieve an end) and ends-rationality (choosing one’s ends based on reasons). Because proponents of the instrumental convergence thesis are concerned with means-rationality, we argue, they cannot avoid the timing problem. After defending our argument against several objections, we conclude by considering the implications our argument has for the rest of the instrumental convergence thesis and for AI safety more generally. (shrink)

Many people believe that it is better to extend the length of a happy life than to create a new happy life, even if the total welfare is the same in both cases. Despite the popularity of this view, one would be hard-pressed to find a fully compelling justification for it in the literature. This paper develops a novel account of why and when extension is better than replacement that applies not just to persons but also to non-human animals and (...) humanity as a whole. (shrink)

Creating systems that are aligned with our goals is seen as a leading approach to create safe and beneficial AI in both leading AI companies and the academic field of AI safety. We defend the view that misaligned AGI – future, generally intelligent (robotic) AI agents – poses catastrophic risks. At the same time, we support the view that aligned AGI creates a substantial risk of catastrophic misuse by humans. While both risks are severe and stand in tension with one (...) another, we show that – in principle – there is room for alignment approaches which do not increase misuse risk. We then investigate how the tradeoff between misalignment and misuse looks empirically for different technical approaches to AI alignment. Here, we argue that many current alignment techniques and foreseeable improvements thereof plausibly increase risks of catastrophic misuse. Since the impacts of AI depend on the social context, we close by discussing important social factors and suggest that to reduce the risk of a misuse catastrophe due to aligned AGI, techniques such as robustness, AI control methods and especially good governance seem essential. (shrink)

Researchers worried about catastrophic risks from advanced AI have argued that we should expect sufficiently capable AI agents to pursue power over humanity because power is a convergent instrumental goal, something that is useful for a wide range of final goals. Others have recently expressed skepticism of these claims. This paper aims to formalize the concepts of instrumental convergence and power-seeking in an abstract, decision-theoretic framework, and to assess the claim that power is a convergent instrumental goal. I conclude that (...) this claim contains at least an element of truth, but might turn out to have limited predictive utility, since an agent’s options cannot always be ranked in terms of power in the absence of substantive information about the agent’s final goals. However, the fact of instrumental convergence is more predictive for agents who have a good shot at attaining absolute or near-absolute power. (shrink)

This report explores interactions between large-scale social simulations and catastrophic risks. It offers a tour of the surrounding theoretical terrain and brings together disparate literatures that bear on the topic.

Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems (...) capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance. (shrink)

Capitalism is defined as the economic structure in which decisions over production are largely made by or on behalf of individuals in virtue of their private property ownership, subject to the incentives and constraints of market competition. In this paper, I will argue that considerations of long-term welfare, such as those developed by Greaves and MacAskill (2021), support anticapitalism in a weak sense (reducing the extent to which the economy is capitalistic) and perhaps support anticapitalism in a stronger sense (establishing (...) an alternative economic structure in which capitalism is not predominant). I hope to encourage longtermists to give anticapitalism serious consideration, and to encourage anticapitalists to pursue criticisms of capitalism’s efficiency as well as its injustices. (shrink)

What moral reasons, if any, do we have to prevent the extinction of humanity? In “Unfinished Business,” Jonathan Knutzen argues that certain further developments in culture would make our history more “collectively meaningful” and that premature extinction would be bad because it would close off that possibility. Here, I critically examine this proposal. I argue that if collective meaningfulness is analogous to individual meaningfulness, then our meaning-based reasons to prevent the extinction of humanity are substantially different from the reasons discussed (...) by Knutzen. (shrink)

Expected value maximization gives plausible guidance for moral decision‐making under uncertainty in many situations. But it has unappetizing implications in ‘Pascalian’ situations involving tiny probabilities of extreme outcomes. This paper shows, first, that under realistic levels of ‘background uncertainty’ about sources of value independent of one's present choice, a widely accepted and apparently innocuous principle—stochastic dominance—requires that prospects be ranked by the expected value of their consequences in most ordinary choice situations. But second, this implication does not hold when differences (...) in expected value are driven by tiny probabilities of extreme outcomes. Stochastic dominance therefore lets us draw a surprisingly principled line between ‘ordinary’ and ‘Pascalian’ situations, providing a powerful justification for de facto expected value maximization in the former context while permitting deviations in the latter. Drawing this distinction is incompatible with an in‐principle commitment to maximizing expected value, but does not require too much departure from decision‐theoretic orthodoxy: it is compatible, for instance, with the view that moral agents must maximize the expectation of a utility function that is an increasing function of moral value. (shrink)

Over one million annual deaths have been attributed to bacterial antimicrobial resistance. Although antibiotics have saved countless other lives, overuse and misuse of antibiotics increases this global threat. Developing new antibiotics and retraining clinicians can be undermined by patients who pressure clinicians to prescribe unnecessary antibiotics. So we validated two low-cost, scalable interventions for improving antibiotic decisions in an online randomized control trial and a pre-registered replication (N = 985). Both first-person vignette experiments found that an infographic and text message (...) caused intermediate to large improvements in antibiotic decisions compared to active controls — even when controlling for a dozen confounds. Notably, these educational interventions also reduced trust in a clinician who offered antibiotics for an upper respiratory infection. These data show how low-cost adjustments to existing communication systems can teach people to reduce the existential risks of antibiotic resistance and how inappropriate antibiotic prescribing can erode trust in health systems. (shrink)

Abstract: Longtermist Effective Altruists (EAs) aim to mitigate the risk of existential catastrophes. In this paper, I have three goals. First, I identify a catastrophic risk that EAs have completely ignored. I call it religious catastrophe: the threat that (as Christians and Muslims have warned for centuries) billions of people stand in danger of going to hell for all eternity. Second, I argue that, even by secular EA lights, religious catastrophe is at least as bad and at least as probable (...) and, therefore, at least as important as many of the standard EA catastrophic risks (e.g., catastrophic climate change, nuclear winter). Third, I present the following dilemma for secular EAs: either adopt religious catastrophe as an EA cause or ignore religious catastrophe but also ignore catastrophic risks whose mitigation has a similar or lower expected value (i.e., most or all of them). Business as usual—ignoring religious catastrophe while championing the typical EA causes—is not an option consistent with longtermist EA principles. (shrink)

Rawls famously argues that the parties in the original position would agree upon the two principles of justice. Among other things, these principles guarantee equal political liberty—that is, democracy—as a requirement of justice. We argue on the contrary that the parties have reason to reject this requirement. As we show, by Rawls’ own lights, the parties would be greatly concerned to mitigate existential risk. But it is doubtful whether democracy always minimizes such risk. Indeed, no one currently knows which political (...) systems would. Consequently, the parties—and we ourselves—have reason to reject democracy as a requirement of justice in favor of political experimentalism, a general approach to political justice which rules in at least some non-democratic political systems which might minimize existential risk. (shrink)

> Future people count. There could be a lot of them. We can make their lives better. > > -- William MacAskill, What We Owe The Future > > [Longtermism is] quite possibly the most dangerous secular belief system in the world today. > > -- Émile P. Torres, Against Longtermism Philosophers,1 2 psychologists,3 4 politicians5 and even some tech billionaires6 have sounded the alarm about artificial intelligence (AI) and the dangers it may pose to the long-term future of humanity. (...) Some believe it poses an existential risk (X-Risk)7 to our species, potentially causing our extinction or bringing about the collapse of human civilisation as we know it. The above quote from philosopher Will MacAskill captures the key tenets of ‘longtermism’, an ethical standpoint that places the onus on current generations to prevent AI-related—and other—X-Risks for the sake of people living in the future.1 2 Developing from an adjacent social movement commonly associated with utilitarian philosophy, ‘effective altruism’, longtermism has amassed following of its own. Its supporters argue that preventing X-Risks is at least as morally significant as addressing current challenges like global poverty. However, critics are concerned that such a distant-future focus will sideline efforts to tackle the many pressing moral issues facing humanity now.8 9 Indeed, according to ‘strong’ longtermism,10 future needs arguably should take precedence over present ones. In essence, the claim is that there is greater expected utility to allocating available resources to prevent human extinction in the future than there is to focusing on present lives, since doing so stands to benefit the incalculably large number of people in later generations who will far outweigh existing populations.1 Taken to the extreme, this view suggests it would be morally permissible, or even required, to actively neglect, harm or destroy large swathes of humanity as …. (shrink)

‘Who argues what’ bears a certain relevance in relation to what is being argued. We are entitled to know those personal circumstances which play a significant role in relation to the argument one supports, so that we can take those circumstances into consideration when evaluating their argument. This is why journals have conflict of interest declarations, and why we value reflexivity in the social sciences. We also often perform double-blind peer review. We recognise that the evaluation of certain statements of (...) fact or value can, and sometimes ought to, be independent of any reference to their authors. Jecker et al argue that we have reasons to prioritise justice concerns over the existential risk (X-Risk) posed by artificial intelligence (AI) as we transition towards more AI-centred societies, despite the warnings about X-Risk that come from the powerful voice of tech leaders, effective altruists and longtermists. Those powerful voices, Jecker et al argue, often have conflicts of interest and are not demographically or epistemically representative. Hence, we should deprioritise their concerns. I believe the opposite may be true. Even though warnings about X-Risk come from sources and in forms we may disagree with, we have reasons to …. (shrink)

Recent claims that artificial intelligence (AI) systems demonstrate “emergent abilities” have fueled excitement but also fear grounded in the prospect that such systems may enable a wider range of parties to make unprecedented advances in areas that include the development of chemical or biological weapons. Ambiguity surrounding the term “emergent abilities” has added avoidable uncertainty to a topic that has the potential to destabilize the strategic landscape, including the perception of key parties about the viability of nonproliferation efforts. To avert (...) these problems in the future, scientists, developers, policymakers, and other parties should take credible steps to strengthen the health of the scientific ecosystem around AI. (shrink)

In this public communication, AI policy theorist Demetrius Floudas introduces a novel era classification for the AI epoch and reveals the hidden dangers of AGI, predicting the potential obsolescence of humanity. In retort, he proposes a provocative International Control Treaty. -/- According to this scheme, the age of AI will unfold in three distinct phases, introduced here for the first time. An AGI Control & non-Proliferation Treaty may be humanity’s only safeguard. This piece aims to provide a publicly accessible exposé (...) of the most pertinent current issues in the governance, ethics and social impact of Artificial Intelligence. (shrink)

The ongoing conflict in Ukraine constitutes a poignant reminder of the enduring relevance and potential devastation associated with nuclear weapons. For decades, the possibility of such catastrophic conflict has not seemed so imminent as in the current world affairs. -/- This contribution presents a comprehensive analysis of nuclear strategy for the 21st century. By examining the evolving geostrategic landscape the talk illuminates key concepts such as nuclear posture, credible deterrence, first & second strike capabilities, flexible response, EMP , variable yield, (...) counterforce & countervalue, limited war and escalation dominance, thereby providing a nuanced understanding of the delicate balance of power in the nuclear arena in modern conflicts. Furthermore, it concentrates on the era that started with the war in Ukraine, offering insights into how recent geopolitical developments influence nuclear doctrines. -/- Through conceivable real-world scenarios, we gain perspective on the multifaceted challenges posed by newer technologies and potential escalation dynamics involving global nuclear actors. -/- Such scenarios include: -/- 1. A non-weaponised or sub-critical test within Russia to intimidate opponents into understanding Russia’s resolve and capabilities. Prior norms have restricted testing since 2000, apart from North Korea. In case of a full surface test, Russia could detonate in uninhabited areas within its own vast territory with minimal fallout. While clarifying military strength, demonstrations risk hardening resistance without incentivising surrender. -/- 2. Coercive atomic threat to compel Ukraine or its supporters to make concessions. By leveraging the spectre of nuclear warfare, Russia could expect to achieve its objectives through intimidation and fear. However, the effectiveness of such a strategy is questionable, as it may provoke a strong international response and further galvanise support for Ukraine. -/- 3. Demonstration detonation over/under the Black Sea to signal resolve without targeting Ukraine itself; possibly amidst negotiations to gain diplomatic leverage, whilst limiting human casualties to zero or causing a mild, non-destructive tsunami. -/- 4. Ukraine could decide to launch a pre-emptive strike on Russian nuclear facilities or weapons, fearing an imminent attack. Such an action would have a high risk of failure and might even provoke a nuclear retaliation from Russia. It could also lead to a broader conflict, as neighbouring countries might feel compelled to intervene. -/- 5. Limited tactical strike against Ukrainian military or infrastructure to weaken capacity and morale. Russia’s 2000+ non-strategic arsenal could target positions using short-range missiles with 0.3 – 5 kiloton yields. Objectives like crippling Kharkiv defences might create no-go zones hindering advances. However, Ukraine might maintain drone and cyber offensive capabilities regardless. -/- 6. Electromagnetic Pulse attack designed to disable critical infrastructure in Ukraine and NATO countries without radioactive fallout. This aims to hinder the war effort and economy without provoking the same conventional military response as a blast. However, an EMP attack affecting NATO shall still be considered an act of war warranting retaliation. -/- 7. A weapon could be deployed accidentally or without proper authorisation. This could occur due to a technical malfunction, miscommunication, human error, or the actions of rogue elements. The consequences of such an event could be catastrophic, leading to a rapid escalation of the conflict and potentially triggering a global crisis. -/- 8. Sub-strategic offensive targeting NATO in the European theatre, so as to fracture allied resolve, against arms routes in Poland, Baltic stockpiles, British naval bases, American troop concentrations in Germany etc. Putin may perceive defeat or deposition as an existential threat, viewing such an option as self-preservation despite adverse consequences. -/- . (shrink)

Since the release of ChatGPT, there has been a lot of debate about whether AI systems pose an existential risk to humanity. This paper develops a general framework for thinking about the existential risk of AI systems. We analyze a two-premise argument that AI systems pose a threat to humanity. Premise one: AI systems will become extremely powerful. Premise two: if AI systems become extremely powerful, they will destroy humanity. We use these two premises to construct a taxonomy of ‘survival (...) stories’, in which humanity survives into the far future. In each survival story, one of the two premises fails. Either scientific barriers prevent AI systems from becoming extremely powerful; or humanity bans research into AI systems, thereby preventing them from becoming extremely powerful; or extremely powerful AI systems do not destroy humanity, because their goals prevent them from doing so; or extremely powerful AI systems do not destroy humanity, because we can reliably detect and disable systems that have the goal of doing so. We argue that different survival stories face different challenges. We also argue that different survival stories motivate different responses to the threats from AI. Finally, we use our taxonomy to produce rough estimates of ‘P(doom)’, the probability that humanity will be destroyed by AI. (shrink)

Many of us think human extinction would be a very bad thing, and that we have moral reasons to prevent it. But there is disagreement over what would make extinction so bad, and thus over what grounds these moral reasons. Recently, several theorists have argued that our reasons to prevent extinction stem not just from the value of the welfare of future lives, but also from certain additional values relating to the existence of humanity itself (for example, humanity’s “final” value, (...) or the value of humanity in itself). In this paper, I argue against these “additional value” views. Despite their initial appeal, these views will inevitably face conflicts between the additional values to which they appeal, and the value of the welfare of future lives. And, I argue, the views cannot plausibly resolve these conflicts. In contrast, these conflicts do not arise for a rival view, on which our reasons to prevent extinction stem just from the value of the welfare of future lives. I conclude that this gives us reason to prefer the latter view, despite the greater initial plausibility of additional value views. (shrink)

The precautionary principle is invoked in a number of important personal and policy decision contexts. Peterson shows that certain ways of making the principle precise are inconsistent with other criteria of decision-making. Some object that the results do not apply to cases of deep uncertainty or value incommensurability which are alleged to be in the principle’s wheelhouse. First, I show that Peterson’s impossibility results can be generalized considerably to cover cases of both deep uncertainty and incommensurability. Second, I contrast an (...) alternative way of giving voice to the precautionary impulse. (shrink)

Longtermists have recently argued that it is overwhelmingly important to do what we can to mitigate existential risks to humanity. I consider three mistakes that are often made in calculating the value of existential risk mitigation. I show how correcting these mistakes pushes the value of existential risk mitigation substantially below leading estimates, potentially low enough to threaten the normative case for existential risk mitigation. I use this discussion to draw four positive lessons for the study of existential risk. -/- (...) . (shrink)

I defend the claim that life-suspending technologies can constitute a catastrophic and existential security factor for risks structurally similar to those related to climate change. The gist of the argument is that, under certain conditions, life-suspending technologies such as cryonics can provide self-interested actors with incentives to efficiently tackle such risks—in particular, they provide reasons to overcome certain manifestations of generational egoism, a risk factor of several catastrophic and existential risks. Provided we have reasons to decrease catastrophic and existential risks (...) such as climate change, we also have a (defeasible) reason for investing in developing and making life-suspending technologies (more) widespread. (shrink)

The ongoing evolution of advanced AI systems will have profound, enduring, and significant impacts on human existence that must not be overlooked. These impacts range from empowering humanity to achieve unprecedented transcendence to potentially causing catastrophic threats to our existence. To proactively and preventively mitigate these potential threats, it is crucial to establish clear redlines to prevent AI-induced existential risks by constraining and regulating advanced AI and their related AI actors. This paper explores different concepts of AI existential risk, connects (...) the enactment of AI red lines to broader efforts addressing AI's impacts, constructs a theoretical framework for analyzing the direct impacts of AI existential risk, and upon that proposes a set of exemplary AI red lines. By contemplating AI existential risks and formulating these red lines, we aim to foster a deeper and systematic understanding of the potential dangers associated with advanced AI and the importance of proactive risk management. We hope this work will contribute to the strengthening and refinement of a comprehensive AI redline system for preventing humanity from AI existential risks. (shrink)

Recent existential-risk thinkers have noted that the analysis of the fine-tuning argument for God’s existence, and the analysis of certain forms of existential risk, employ similar types of reasoning. This paper argues that insofar as the “many worlds objection” undermines the inference to God’s existence from universal fine-tuning, then a similar many worlds objection undermines the inference that the historic risk of global nuclear catastrophe has been low from the lack of such a catastrophe has occurred in our world. A (...) version of the fine-tuning argument applied to nuclear risk, The Nuclear Fine-Tuning Argument, utilizes the set of nuclear close calls to show that 1) conventional explanations fail to adequately explain how we have survived thus far and 2) the existence of many worlds provides an adequate explanation. This is because, if there are many worlds, observers are disproportionately more likely to reflect upon a world that hasn’t had a global nuclear catastrophe than upon one that has had a global nuclear catastrophe. This selection bias results from the catastrophic nature of such an event. This argument extends generally to all global catastrophic risks that both A) have been historic threats and B) would result in a significantly lower global population. (shrink)

Is the overall value of a world just the sum of values contributed by each value-bearing entity in that world? Additively separable axiologies (like total utilitarianism, prioritarianism, and critical level views) say 'yes', but non-additive axiologies (like average utilitarianism, rank-discounted utilitarianism, and variable value views) say 'no'. This distinction appears to be practically important: among other things, additive axiologies generally assign great importance to large changes in population size, and therefore tend to strongly prioritize the long-term survival of humanity over (...) the interests of the present generation. Non-additive axiologies, on the other hand, need not assign great importance to large changes in population size. We show, however, that when there is a large enough `background population' unaffected by our choices, a wide range of non-additive axiologies converge in their implications with additive axiologies—for instance, average utilitarianism converges with critical-level utilitarianism and various egalitarian theories converge with prioritarianism. We further argue that real-world background populations may be large enough to make these limit results practically significant. This means that arguments from the scale of potential future populations for the astronomical importance of avoiding existential catastrophe, and other arguments in practical ethics that seem to presuppose additive separability, may succeed in practice whether or not we accept additive separability as a basic axiological principle. (shrink)

Climate change is often described as an existential risk to the human species, but this terminology has generally been avoided in the climate-justice literature in analytic philosophy. I investigate the source of this disconnect and explore the prospects for incorporating the idea of climate change as an existential risk into debates about climate justice. The concept of existential risk does not feature prominently in these discussions, I suggest, because assumptions that structure ‘ideal’ accounts of climate justice ensure that the prospect (...) of climate change as an extinction-level threat does not arise. Given persistent noncompliance with mitigation duties, however, we have reason to revisit these assumptions. I argue that the most promising way for theories of climate justice to account for the significance of existential risk is to look to the practices of protest and resistance in which a concern about extinction or global catastrophe is frequently given expression. (shrink)

The notion of existential catastrophe is increasingly appealed to in discussion of risk management around emerging technologies, but it is not completely clear what this notion amounts to. Here, I provide an opinionated survey of the space of plausibly useful definitions of existential catastrophe. Inter alia, I discuss: whether to define existential catastrophe in ex post or ex ante terms, whether an ex ante definition should be in terms of loss of expected value or loss of potential, and what kind (...) of probabilities should be involved in any appeal to expected value. (shrink)

Longtermism is the view that the most urgent global priorities, and those to which we should devote the largest portion of our resources, are those that focus on (i) ensuring a long future for humanity, and perhaps sentient or intelligent life more generally, and (ii) improving the quality of the lives that inhabit that long future. While it is by no means the only one, the argument most commonly given for this conclusion is that these interventions have greater expected goodness (...) per unit of resource devoted to them than each of the other available interventions, including those that focus on the health and well-being of the current population. In this paper, I argue that, even if we grant the consequentialist ethics upon which this argument depends, and even if we grant one of the axiologies that are typically paired with that ethics to give the argument, we are not morally required to choose an option that maximises expected utility; indeed, we might not even be permitted to do so. Instead, I will argue, if the argument's consequentialism is correct, we should choose using a decision theory that is sensitive to risk, and allows us to give greater weight to worse-case outcomes than expected utility theory does. And, I will show, such decision theories do not always recommend longtermist interventions. Indeed, sometimes, they recommend exactly the opposite: sometimes, they recommend hastening human extinction. Many, though not all, will take this as a reductio of the consequentialism or the axiology of the argument. I remain agnostic on the conclusion we should draw. (shrink)

In this paper, we argue that our moral concern for future well-being should reduce over time due to important practical considerations about how humans interact with spacetime. After surveying several of these considerations (around equality, special duties, existential contingency, and overlapping moral concern) we develop a set of core principles that can both explain their moral significance and highlight why this is inherently bound up with our relationship with spacetime. These relate to the equitable distribution of (1) moral concern in (...) a universe that we can only causally affect in one temporal direction; (2) access to the benefits from using spacetime as a resource; and (3) the burdens of care given (1) and (2). We conclude by considering the practical implications of our argument and find that, while it is often assumed that a preference for present over future well-being weakens the case for existential risk mitigation, this likely is not the case. (shrink)

This paper examines a recent argument in favour of strong precautionary action—possibly including working to hasten human extinction—on the basis of a decision-theoretic view that accommodates the risk-attitudes of all affected while giving more weight to the more risk-averse attitudes. First, we dispute the need to take into account other people’s attitudes towards risk at all. Second we argue that a version of the non-identity problem undermines the case for doing so in the context of future people. Lastly, we suggest (...) that we should not work to hasten human extinction, even if significant risk aversion is warranted. (shrink)

In an effort to inform the discussion surrounding existential risks from AI, we formulate Extinction-level Goodhart’s Law as “Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity”, and we aim to understand which formal models are suitable for investigating this hypothesis. Note that we remain agnostic as to whether Extinction-level Goodhart’s Law holds or not. As our key contribution, we identify a set of conditions that are necessary for a model that aims to be (...) informative for evaluating specific arguments for Extinction-level Goodhart’s Law. Since each of the conditions seems to significantly contribute to the complexity of the resulting model, formally evaluating the hypothesis might be exceedingly difficult. This raises the possibility that whether the risk of extinction from artificial intelligence is real or not, the underlying dynamics might be invisible to current scientific methods. (shrink)