Man builds an AGI and then either destroys all of humanity or colonises $4\times 10^{20}$ stars.
This book was really, really good. I’ve been interested in the idea of AI safety for a while now after reading LessWrong and this book was just full of so much good information and new perspectives.
Flashcards
Past developments and present capabilities
Growth models and big history
- A few hundred thousand years ago, it took on the order of a million years to increase civilisations ability to support an extra one million citizens living at subsistence level.
- Then it took 2 centuries.
- Now it takes 90 minutes.
- Another revolution on the scale of the industrial revolution could mean the world economy doubling every 2 weeks.
Great expectations
- Sometimes a problem that initially looks hopelessly complicated (creating superintelligence) turns out to have a surprisingly simple solutions (but most of the time, this isn’t the case).
Seasons of hope and despair
No highlights.
State of the art
- Donald Knuth: “AI has by now succeeded in doing essentially everything that requires ‘thinking’ but has failed to do most of what people and animals do ‘without thinking’”.
- An AI-Complete problem: a problem whose difficulty is roughly equivalent to creating a generally intelligent AI.
- Copernican Revolution – astronomy seemed hopelessly complicated when we thought we were at the center of the earth, until a paradigm shift occurred. Could be be tackling AI in the wrong way?
- 2010 Flash Crash: a trillion dollars were wiped off the market due to the complex and unexpected interactions of individually simple trading components.
Opinions about the future of machine intelligence
No highlights.
Paths to superintelligence
Artificial intelligence
- We know that blind evolutionary processes can produce human-level general intelligence, since it’s happened at least once.
- But we don’t have to rely on it being “blind”, a specially crafted approach could get us there a lot quicker.
- This would be computationally unfeasible to carry out, but it does provide us with an upper bound on the difficulty.
Whole brain emulation
- We learnt to fly after ditching nature and running careful experiments in wind tubes.
- We learnt combustion after observing naturally occurring fires.
- What will the path to superintelligence be? Artificial or natural?
- The WBE approach looks to create superintelligence by simulating human brains.
Biological cognition
- This approach is about augmenting our existing biological cognition, like if there was a magical smart drug that everyone could take and it would dramatically boost their IQ.
- This is not revolutionary, the author drank coffee and chewed nicotine gum to help write the book.
- There are already low-hanging fruit with biological cognition, such as fixing iodine deficiency in pregnant woman and infants: Iodine and Adult IQ Meta-analysis.
- It’s not likely that there will be a magical smart drug however due to how delicately balanced and complex the human brain is. It’s much more likely that a careful approach will need to be taken rather than flooding the brain with a certain type of chemical.
- Another approach is embryo selection.
- Embryo selection is already used in practice to select against genetic diseases that the parents of the embryos have, such as Huntington’s disease.
- In this new approach, embryos would be selected according to genetic markers of intelligence.
- 10 generations of selecting 1 out of 10 embryos could lead to an IQ boost of 130 points.
- Techniques could be used to get sperm and egg cells directly from the embryos instead which would prevent the need for the selected embryos to grow up.
- This is ignoring the ethical objections, though perception could change. 18% of Americans said they wouldn’t use IVF and then 53% they would once the first IVF baby was born.
- Other ideas: vats full of cortical tissue, large animal brains enhanced with human genes.
Brain-computer interfaces
- Brain-computer interfaces would be complex to set up, and to get them to a level that had seamless integration of information in conciousness would likely be an AI-complete problem.
- It’s not about how fast information
Networks and organizations
Summary
Forms of superintelligence
Speed superintelligence
Collective superintelligence
Quality superintelligence
Direct and indirect reach
Sources of advantage for digital intelligence
The kinetics of an intelligence explosion
Timing and speed of the takeoff
Recalcitrance
Optimization power and explosivity
Decisive strategic advantage
Will the frontrunner get a decisive strategic advantage?
How large will the successful project be
From decisive strategic advantage to singleton
Cognitive superpowers
Functionalities and superpowers
An AI takeover scenario
The superintelligent will
The relation between intelligence and motivation
Instrumental convergence
Is the default outcome doom?
Existential catastrophe as the default outcome of an intelligence explosion?
The treacherous turn
Malignant failure modes
The control problem
Two agency problems
Capability control methods
Motivation selection methods
Synopsis
Oracles, genies, sovereigns, tools
Oracles
Genies and sovereigns
Tool-AIs
Comparison
Multipolar scenarios
Of horses and men
Life in an algorithmic economy
Post-transition formation of a singleton?
Acquiring values
The value loading problem
Evolutionary selection
Reinforcement learning
Associative value accretion
Motivational scaffolding
Value learning
Emulation modulation
Institution design
Synopsis
Choosing the criteria for choosing
The need for indirect normativity
Coherent extrapolated volition
Morality models
Do What I Mean
Component list
Getting close enough
The strategic picture
Science and technology strategy
Pathways and enablers
Collaboration
Crunch time
Philosophy with a deadline
What is to be done?
Will the best in human nature please stand up
Afterword
Further Reading?
Backlinks
Metadata
date: 2021-10-01 21:11
finished: true
rating: 10
tags:
- '@?notes'
- '@?superintelligence'
- '@?book'
- '@?public'
- '@?safe-to-post-online'
title: Superintelligence