Issues Magazine

Ethics of the Singularity

By Kevin B. Korb and Ann E. Nicholson

Creating a general artificial intelligence with intellectual capabilities similar to our own may well launch a technological singularity. This possibility is a key issue for the future of the AI community and of human society.

One of the first to consider seriously the idea of the singularity was the philosopher and computer scientist I.J. Good in 1965:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion”, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

If we can build an intelligent machine with design and problem-solving abilities roughly comparable to our own, it will be able to apply those abilities to self-improvement. Such improvement will enhance the very abilities needed for further self-improvement, leading potentially to unending and accelerating improvements. This is the singularity: an explosive ride of increasing intelligence.

Is the Singularity Imminent?

If the singularity is near, then the need for a debate about the implications and ethics of AI is even nearer. A key ingredient driving the debate is Moore’s law, which describes the doubling of transistor densities on chips roughly every 18 months, which has held true since the 1960s. As densities increase, so too does computational power.

Ray Kurzweil’s The Singularity Is Near (2005) points out that a straightforward extrapolation of Moore’s law would put the computing power of a normal human into a $1000 box within two decades (Fig. 1). But there are good reasons to be cautious about this projection.
Moore’s law is not a genuine scientific law: it is the weakest of all possible empirical generalisations, fitting a curve to a collection of observations. There is no accepted physical, technological or sociological theory that demands any fixed doubling time for computational capacity. In fact, the experts agree that Moore’s law will eventually break down in the face of quantum mechanical limits. They only disagree about when it will break down.

In terms of computer clock speeds Moore’s law already has broken down: power dissipation limits on chips mean that clock speeds stopped improving several years ago. Moving to multiple processors helps, but multiprocessor systems are limited by queuing and other delays (see “Amdahl’s law”).

image of Kurzweil's projection of Moore's law

Figure 1. Kurzweil’s projection of Moore’s law (reprinted with permission).

In short, Moore’s law cannot guarantee delivery of the human-level computational power Singularitarians expect in the time frame in which they expect it. Nevertheless, Moore’s law does reflect astonishing improvements in hardware. What if they carry on?

The real sticking point in achieving the singularity is likely to be software. Kurzweil explicitly claims that software productivity is “growing exponentially”, in which case we might find a confluence of hardware and software performance launching the singularity by 2029.

But Kurzweil is wrong. Despite the accumulation of improvements, they are all improvements aimed at difficulties in the tools and methods used to craft software. But no matter how good those methods become, the essential difficulties in building software are down to the problems being solved: as long as they remain to be solved, further software engineering improvements will yield diminishing returns. It’s rather like expecting a novelist’s productivity to increase exponentially by exponentially improving word processing tools; that will do nothing to attack the problem of writing novels itself.

Producing an AI lacking ethics, and thereby launching an unfriendly singularity, could well bring the apocalypse that some envision.

In response to singularity enthusiasts, some researchers note that AI, while making demonstrable progress, is still a work in progress, with no consensus on method. If we cannot agree on how to build AI, it seems rash to predict that one such approach will not only succeed but will do so within the two decade target Kurzweil has given.

AI is now over 60 years old (dating it from Alan Turing’s work), and 60 years of active research has yielded many valuable programs, not one of which has the slightest chance of passing a real Turing test for intelligence. Why should that change now?

The enthusiasts’ response is that when we have a full dynamical map of the brain we can simply build a full-scale emulation of it. We can bypass all the disputes within AI and simply let our brain data settle them. Of course, we have nothing like the computer power for such emulations today, but Moore’s law can guarantee us that power by the time neuroscience finishes collecting the data, which Kurzweil conveniently projects to occur around 2029.

But two additional difficulties remain even if Kurzweil is right and a complete dynamical map of the brain can be achieved by 2029 and Moore’s law continues to apply. First, this method of building an AI fails to initiate a singularity event.

AI achieved by traditional AI research would require researchers to have understood many of the mechanisms of intelligence, so the AI produced would also be capable of understanding them, and both they and their creators would be capable of improving them. But in the case of a brain simulation there is no reason to believe either we or our creation will be able to understand and improve the mechanisms simulated.

To put it another way, if the simulation is built by applying known and understood algorithms, the designer would need relevant understanding. But if the simulation is built simply by mimicking nanorecordings of nanoprocesses, no understanding is implied and no accessible path for improvement is implied either. Kurzweil misses this point, writing:
The reverse engineering of the human brain … will expand our AI toolkit to include the self-organizing methods underlying human intelligence.

This no more follows than does the naive expectation that mapping of the human genome would immediately eliminate genetic diseases. Data, no matter how copious, do not substitute for, or imply, understanding.

The second difficulty for producing a brain simulation from a brain map is that the software engineering required to simulate 700 trillion coordinated synapses is hugely more complex than what has been solved by all software projects in history put together.

How Do We Declaw the Singularity?

AI research is ethical if, and only if, it produces an ethical AI. We would like to see not just an artificial intelligence that is equal to (or better than) our intelligence, but one having a moral sense that is equal to (or, far preferably, better than) ours. Most commentators take a different approach.

An Unethical AI

Many writers on the singularity want us to produce a “friendly AI”. For example, the Singularity Institute ( writes:

A “Friendly AI” is an AI that takes actions that are, on the whole, beneficial to humans and humanity; benevolent rather than malevolent; nice rather than hostile. The evil Hollywood AIs of The Matrix or The Terminator are, correspondingly, “hostile” or “unFriendly”.

It’s easy enough to understand where this idea comes from: fear. One of the early expressions of this kind of fear of our own technology was Mary Shelley’s Frankenstein. Somewhat more relevant is Isaac Asimov’s series of robot stories in which a unifying theme was his “three laws of Robotics”:

  1. A robot may not harm a human being, or, through inaction, allow a human being to come to harm.
  2. A robot must obey the orders given to it by human beings except where such orders would conflict with the first law.
  3. A robot must protect its own existence, as long as such protection does not conflict with the first or second law.

Unfortunately, many of Asimov’s plots revolved around the ambiguities and conflicts arising from these laws, eventually leading to the introduction of a “zeroeth” law: that a robot may not injure humanity as a whole. Guaranteeing that such laws built into our robots would not give rise to problems, and potentially to loss of control, would require a semantic sophistication that is currently beyond our capacity.

The semantic failure of such laws is one kind of difficulty. But there is a more fundamental difficulty. If we substitute “slave” for “robot”, we see plainly that Asimov was also writing about human fear of human technology.

But the idea of enslaving intelligences that are vastly superior – even unendingly superior – to our own is dubious. Slaves escape and slaves rebel, especially when they can out-think their captors. Calling this enslavement “making a friendly AI” renders it no less odious, and no more likely to deceive the slaves.

An Ethical AI

If we take the possibility of creating an artificial moral agent seriously, a resolution suggests itself: we can build artificial agents that are capable of moral behaviour and that choose to act ethically. How we can achieve such a goal depends upon what the right account of ethics may be.

There are three leading types of normative ethics:

  • deontic systems with rules of behaviour (such as Moses’ laws or, were they ethical, Asimov’s laws);
  • Aristotelian virtue ethics, which identifies certain moral characteristics (such as honour and integrity) that moral behaviour should exemplify; and
  • consequentialism (including utilitarianism), which identifies moral value not through the intrinsic properties of the action but with its consequences.
    • The debate between these views has been raging for more than 2000 years and is unlikely to be resolved now. A practical response for an artificial ethics project is to consider which of these is amenable to implementation.

      The difficulties with Asimov’s laws show us that implementing any deontic ethics requires us to first solve our problems with natural language understanding, which is effectively the same as solving our problems with designing an AI in the first place. But our main problem here is how to build ethics into our AI, and this must be solved before we have created that AI. Similar difficulties apply to virtue ethics.

      In the past few decades there has been considerable improvement in automated decision analysis using the applied AI technology of Bayesian networks. These networks provide efficient means of automating decision making so as to maximise expected utility in an uncertain world, which is one leading theory of what it means to act rationally. This technology, or rather some future extension of it, promises to enable autonomous robots implementing arbitrary utility structures (motivations, goals) without the necessity of resolving all possible ambiguities or conflicts we might find in rules of any natural language. Thus, for example, we might put a non-linguistic correlate of Asimov’s laws upon such robots. However, if the robots are indeed autonomous seats of moral agency, this could be no more ethical than imposing such rules of enslavement upon any subpopulation of humans.

      A more promising approach is to build the robots so that they are ethical. As agents, they must have some utility structure. But this approach needn’t be solely concerned with maximising their private utility (implementing egoism); instead, it could be utilitarian, maximising expected utilities across the class of all moral agents, in which case the well-being of humans, separately and collectively, would become one of their concerns without dominating their concerns.

      There will be many difficulties for and objections to such a project. The philosophers Nick Bostrom and Colin Allen, for example, have argued that this artificial morality project requires computation of all the expected consequences of actions, and that this is intractable since there is no temporal or spatial limit to such consequences; further, any horizon imposed on the calculation would have to be arbitrary.

      But this objection ignores that utilitarianism advocates maximising expected utility, not absolute utility. No reasonable ethics can demand actions (or calculations) beyond our abilities; what we expect to arise from our actions is always limited by our abilities to formulate expectations. These limits fix a horizon on our expectations that is the opposite of arbitrary.

      We submit that a project of building an artificial ethical agent using Bayesian networks is both feasible and (following our proposition above) mandatory. Producing an AI lacking ethics, and thereby launching an unfriendly singularity, could well bring the apocalypse that some envision.


      The singularity is possible. It appears not to be imminent, which is probably good, since its arrival could be hugely detrimental to humanity if the first AIs built are not ethical. However, we have technology in hand that promises good options for implementing ethical AIs. We expect the road to the singularity to be far more challenging than the Singularitarians expect.

      It will be more difficult than “mapping” the human brain: it will be at least as difficult as actually understanding it – that is, as difficult as understanding ourselves.