5 min read

Man builds an AGI and then either destroys all of humanity or colonises $4\times 10^{20}$ stars.

This book was really, really good. I’ve been interested in the idea of AI safety for a while now after reading LessWrong and this book was just full of so much good information and new perspectives.


Past developments and present capabilities

Growth models and big history

  • A few hundred thousand years ago, it took on the order of a million years to increase civilisations ability to support an extra one million citizens living at subsistence level.
  • Then it took 2 centuries.
  • Now it takes 90 minutes.
  • Another revolution on the scale of the industrial revolution could mean the world economy doubling every 2 weeks.

Great expectations

  • Sometimes a problem that initially looks hopelessly complicated (creating superintelligence) turns out to have a surprisingly simple solutions (but most of the time, this isn’t the case).

Seasons of hope and despair

No highlights.

State of the art

  • Donald Knuth: “AI has by now succeeded in doing essentially everything that requires ‘thinking’ but has failed to do most of what people and animals do ‘without thinking’”.
  • An AI-Complete problem: a problem whose difficulty is roughly equivalent to creating a generally intelligent AI.
  • Copernican Revolution – astronomy seemed hopelessly complicated when we thought we were at the center of the earth, until a paradigm shift occurred. Could be be tackling AI in the wrong way?
  • 2010 Flash Crash: a trillion dollars were wiped off the market due to the complex and unexpected interactions of individually simple trading components.

Opinions about the future of machine intelligence

No highlights.

Paths to superintelligence

Artificial intelligence

  • We know that blind evolutionary processes can produce human-level general intelligence, since it’s happened at least once.
  • But we don’t have to rely on it being “blind”, a specially crafted approach could get us there a lot quicker.
  • This would be computationally unfeasible to carry out, but it does provide us with an upper bound on the difficulty.

Whole brain emulation

  • We learnt to fly after ditching nature and running careful experiments in wind tubes.
  • We learnt combustion after observing naturally occurring fires.
  • What will the path to superintelligence be? Artificial or natural?
  • The WBE approach looks to create superintelligence by simulating human brains.

Biological cognition

  • This approach is about augmenting our existing biological cognition, like if there was a magical smart drug that everyone could take and it would dramatically boost their IQ.
  • This is not revolutionary, the author drank coffee and chewed nicotine gum to help write the book.
  • There are already low-hanging fruit with biological cognition, such as fixing iodine deficiency in pregnant woman and infants: Iodine and Adult IQ Meta-analysis.
  • It’s not likely that there will be a magical smart drug however due to how delicately balanced and complex the human brain is. It’s much more likely that a careful approach will need to be taken rather than flooding the brain with a certain type of chemical.
  • Another approach is embryo selection.
  • Embryo selection is already used in practice to select against genetic diseases that the parents of the embryos have, such as Huntington’s disease.
  • In this new approach, embryos would be selected according to genetic markers of intelligence.
  • 10 generations of selecting 1 out of 10 embryos could lead to an IQ boost of 130 points.
  • Techniques could be used to get sperm and egg cells directly from the embryos instead which would prevent the need for the selected embryos to grow up.
  • This is ignoring the ethical objections, though perception could change. 18% of Americans said they wouldn’t use IVF and then 53% they would once the first IVF baby was born.
  • Other ideas: vats full of cortical tissue, large animal brains enhanced with human genes.

Brain-computer interfaces

  • Brain-computer interfaces would be complex to set up, and to get them to a level that had seamless integration of information in conciousness would likely be an AI-complete problem.
  • It’s not about how fast information

Networks and organizations


Forms of superintelligence

Speed superintelligence

Collective superintelligence

Quality superintelligence

Direct and indirect reach

Sources of advantage for digital intelligence

The kinetics of an intelligence explosion

Timing and speed of the takeoff


Optimization power and explosivity

Decisive strategic advantage

Will the frontrunner get a decisive strategic advantage?

How large will the successful project be

From decisive strategic advantage to singleton

Cognitive superpowers

Functionalities and superpowers

An AI takeover scenario

The superintelligent will

The relation between intelligence and motivation

Instrumental convergence

Is the default outcome doom?

Existential catastrophe as the default outcome of an intelligence explosion?

The treacherous turn

Malignant failure modes

The control problem

Two agency problems

Capability control methods

Motivation selection methods


Oracles, genies, sovereigns, tools


Genies and sovereigns



Multipolar scenarios

Of horses and men

Life in an algorithmic economy

Post-transition formation of a singleton?

Acquiring values

The value loading problem

Evolutionary selection

Reinforcement learning

Associative value accretion

Motivational scaffolding

Value learning

Emulation modulation

Institution design


Choosing the criteria for choosing

The need for indirect normativity

Coherent extrapolated volition

Morality models

Do What I Mean

Component list

Getting close enough

The strategic picture

Science and technology strategy

Pathways and enablers


Crunch time

Philosophy with a deadline

What is to be done?

Will the best in human nature please stand up


Further Reading?

date: 2021-10-01 21:11
finished: true
rating: 10
- '@?notes'
- '@?superintelligence'
- '@?book'
- '@?public'
- '@?safe-to-post-online'
title: Superintelligence