AI theorist Ben Goertzel has posted an article in which he declares his rejection of the Singularity Institute for Artificial Intelligence's claim that "progressing toward advanced AGI without a design for "provably non-dangerous AGI"...is highly likely to lead to an involuntary end for the human race." Goertzel calls this their "Scary Idea" and attempts to show that the fear is largely overstated.
He breaks the SIAI argument down to four primary points:
- If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans (as opposed to, e.g., utterly ignoring us, and being willing to repurpose our molecules for its own ends) are very low
- Human value is fragile as well as complex, so if you create an AGI with a roughly-human-like value system, then this may not be good enough, and it is likely to rapidly diverge into something with little or no respect for human values
- "Hard takeoffs" (in which AGIs recursively self-improve and massively increase their intelligence) are fairly likely once AGI reaches a certain level of intelligence; and humans will have little hope of stopping these events
- A hard takeoff, unless it starts from an AGI designed in a "provably Friendly" way, is highly likely to lead to an AGI system that doesn't respect the rights of humans to exist
Taking these points into consideration, Goertzel pieces together what he feels is the SIAI's argument:
If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.
Goertzel then expresses his particular concerns with this argument, including SIAI's Eliezer Yudkowsky's suggestion that we can get human values into an AGI system, what he calls Coherent Extrapolated Volition:
...I think this is a very science-fictional and incredibly infeasible idea (though a great SF notion). I've discussed it and proposed some possibly more realistic alternatives in a previous blog post (e.g. a notion called Coherent Aggregated Volition). But my proposed alternatives aren't guaranteed-to-succeed nor neatly formalized.
But setting those worries aside, is the computation-theoretic version of provably safe AI even possible? Could one design an AGI system and prove in advance that, given certain reasonable assumptions about physics and its environment, it would never veer too far from its initial goal (e.g. a formalized version of the goal of treating humans safely, or whatever)?
I very much doubt one can do so, except via designing a fictitious AGI that can't really be implemented because it uses infeasibly much computational resources. My GOLEM design, sketched in this article, seems to me a possible path to a provably safe AGI -- but it's too computationally wasteful to be practically feasible.
Oooh, it looks like we have the makings of a great debate, here. I'll be interested to see if the SIAI retorts and how they address Goertzel's concerns.
- Neurodiversity vs. Cognitive Liberty - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.13 - November 8th, 2009 [November 8th, 2009]
- Limits to the biolibertarian impulse - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.15 - November 8th, 2009 [November 8th, 2009]
- Neurodiversity vs. Cognitive Liberty, Round II - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.17 - November 8th, 2009 [November 8th, 2009]
- Cognitive liberty and right to one's mind - November 8th, 2009 [November 8th, 2009]
- TED Talks: Henry Markram builds a brain in a supercomputer - November 8th, 2009 [November 8th, 2009]
- And Now, for Something Completely Different: Doomsday! - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.19 - November 8th, 2009 [November 8th, 2009]
- Oklahoma and abortion - some fittingly harsh reflections - November 8th, 2009 [November 8th, 2009]
- Pigliucci on science and the scope of skeptical inquiry - November 8th, 2009 [November 8th, 2009]
- Remembering Mac Tonnies - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.24 - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.26 - November 8th, 2009 [November 8th, 2009]
- The Bright Side of Nuclear Armament - November 8th, 2009 [November 8th, 2009]
- Grieving chimps - November 8th, 2009 [November 8th, 2009]
- Elephant prosthetic - November 8th, 2009 [November 8th, 2009]
- Mass produced artificial skin to replace animal testing - November 8th, 2009 [November 8th, 2009]
- Dog gets osseointegrated prosthetic - November 8th, 2009 [November 8th, 2009]
- NASA Shuttle-derived Sidemount Heavy Launch Vehicle Concept - November 8th, 2009 [November 8th, 2009]
- Link dump for 2009.02.02 - November 8th, 2009 [November 8th, 2009]
- Link dump for 2009.11.04 - November 8th, 2009 [November 8th, 2009]
- Link dump for 2009.11.05 - November 8th, 2009 [November 8th, 2009]
- IEET's Biopolitics of Popular Culture Seminar - November 8th, 2009 [November 8th, 2009]
- Einstein and Millikan should have done a Kurzweil - November 8th, 2009 [November 8th, 2009]
- Affective Death Spirals - November 8th, 2009 [November 8th, 2009]
- Cure aging or give a small number of disabled people jobs as janitors? - November 8th, 2009 [November 8th, 2009]
- Would unary notation prevent scope insensitivity? - November 8th, 2009 [November 8th, 2009]
- Cure aging or give a small number of disabled people jobs as janitors - unary version. - November 8th, 2009 [November 8th, 2009]
- At SENS4, Cambridge, UK - November 8th, 2009 [November 8th, 2009]
- SENS4 overview and review - how evolution complicates SENS, and why we must try harder - November 8th, 2009 [November 8th, 2009]
- SENS4 top 10 photos - November 8th, 2009 [November 8th, 2009]
- My AI research for this year - November 8th, 2009 [November 8th, 2009]
- My AI research: Formal Logic - November 8th, 2009 [November 8th, 2009]
- My AI research: Category theory and institution theory - November 8th, 2009 [November 8th, 2009]
- My AI research: The Semantic Web - November 8th, 2009 [November 8th, 2009]
- My AI research: Features and Flaws of Logical representation - November 8th, 2009 [November 8th, 2009]
- My AI research: Graphical models and probabilistic logics - November 8th, 2009 [November 8th, 2009]
- Hughes and More engage Italian Catholicism: Image caption competition - November 8th, 2009 [November 8th, 2009]
- Surprisingly good solutions, falling in love and life in a materialistic universe - November 8th, 2009 [November 8th, 2009]
- What do you get when you cross slightly evolved, status seeking monkeys with the scientific method? - November 8th, 2009 [November 8th, 2009]
- Seeking the optimal philanthropic strategy: Global Warming or AI risk? - November 8th, 2009 [November 8th, 2009]
- Machine Learning - harbinger of the future of AI? - November 8th, 2009 [November 8th, 2009]
- At the Singularity Summit in NYC - November 8th, 2009 [November 8th, 2009]
- Katja Grace: world-dominating superintelligence is "unlikely" - November 8th, 2009 [November 8th, 2009]
- Normal Human Heroes on "Nightmare futures" - November 8th, 2009 [November 8th, 2009]
- Anissimov on Intelligence Enhancement - November 8th, 2009 [November 8th, 2009]
- Yudkowsky on "Value is fragile" - November 8th, 2009 [November 8th, 2009]
- Response to Pearce - November 8th, 2009 [November 8th, 2009]
- Creative thinking lets you believe whatever you want - December 13th, 2009 [December 13th, 2009]
- Let’s get metaphysical: How our ongoing existence could appear increasingly absurd - December 13th, 2009 [December 13th, 2009]
- Linda MacDonald Glenn guest blogging in November and December - December 13th, 2009 [December 13th, 2009]
- Link dump for 2009.11.15 - December 13th, 2009 [December 13th, 2009]
- Call 1-800-New-Organ, by 2020? - December 13th, 2009 [December 13th, 2009]
- IBM's claim to have simulated a cat's brain grossly overstated - December 13th, 2009 [December 13th, 2009]
- John Hodgman pulls off Fermi Paradox schtick - December 13th, 2009 [December 13th, 2009]
- Deus Sex Machina - December 13th, 2009 [December 13th, 2009]
- How Americans spent themselves into ruin... but saved the world - December 13th, 2009 [December 13th, 2009]
- I am my own grandpa (or grandma)? - December 13th, 2009 [December 13th, 2009]
- Link dump for 2009.11.29 - December 13th, 2009 [December 13th, 2009]
- The art of Tomas Saraceno - December 13th, 2009 [December 13th, 2009]
- Link dump: 2009.12.05 - December 13th, 2009 [December 13th, 2009]
- The Harmonic Convergence of Science, Sight, & Sound - December 13th, 2009 [December 13th, 2009]
- Working on my website - December 13th, 2009 [December 13th, 2009]
- Transhumanism, personal immortality and the prospect of technologically enabled utopia - December 13th, 2009 [December 13th, 2009]
- RokoMijic.com is up - December 13th, 2009 [December 13th, 2009]
- Why the Fuss About Intelligence? - December 13th, 2009 [December 13th, 2009]
- Initiation ceremony - December 13th, 2009 [December 13th, 2009]
- Birthing Gods - December 13th, 2009 [December 13th, 2009]
- 11 core rationalist skills - from LessWrong - December 13th, 2009 [December 13th, 2009]
- The best of the guests - December 13th, 2009 [December 13th, 2009]
- The best of Sentient Developments: 2009 - December 13th, 2009 [December 13th, 2009]
- Link dump: 2009.12.15 - December 15th, 2009 [December 15th, 2009]
- The Utopia Force - December 22nd, 2009 [December 22nd, 2009]
- Avatar: The good, the bad and ugly - December 23rd, 2009 [December 23rd, 2009]
- Singularity Institute launches "2010 Singularity Research Challenge" - December 24th, 2009 [December 24th, 2009]
- Transhumanism as a "nonissue" - December 24th, 2009 [December 24th, 2009]
- Hanson on "Meh, Transhumanism" - December 25th, 2009 [December 25th, 2009]
- Merry Newtonmas from Transhuman Goodness - December 25th, 2009 [December 25th, 2009]