I just met someone on facebook who convinced me to re-iterate the need for careful thought about human cognitive biases. A classic from the Yudkowsky school of thought - "Affective death spirals" that I probably linked to before:
Many, many, many are the flaws in human reasoning which lead us to overestimate how well our beloved theory explains the facts. The phlogiston theory of chemistry could explain just about anything, so long as it didn't have to predict it in advance. And the more phenomena you use your favored theory to explain, the truer your favored theory seems - has it not been confirmed by these many observations? As the theory seems truer, you will be more likely to question evidence that conflicts with it. As the favored theory seems more general, you will seek to use it in more explanations.
If you know anyone who believes that Belgium secretly controls the US banking system, or that they can use an invisible blue spirit force to detect available parking spaces, that's probably how they got started.
(Just keep an eye out, and you'll observe much that seems to confirm this theory...)
This positive feedback cycle of credulity and confirmation is indeed fearsome, and responsible for much error, both in science and in everyday life.
But it's nothing compared to the death spiral that begins with a charge of positive affect - a thought that feels really good.
A new political system that can save the world. A great leader, strong and noble and wise. An amazing tonic that can cure upset stomachs and cancer.
Heck, why not go for all three? A great cause needs a great leader. A great leader should be able to brew up a magical tonic or two.
The halo effect is that any perceived positive characteristic (such as attractiveness or strength) increases perception of any other positive characteristic (such as intelligence or courage). Even when it makes no sense, or less than no sense.Positive characteristics enhance perception of every other positive characteristic? That sounds a lot like how a fissioning uranium atom sends out neutrons that fission other uranium atoms.
Weak positive affect is subcritical; it doesn't spiral out of control. An attractive person seems more honest, which, perhaps, makes them seem more attractive; but the effective neutron multiplication factor is less than 1. Metaphorically speaking. The resonance confuses things a little, but then dies out.
With intense positive affect attached to the Great Thingy, the resonance touches everywhere. A believing Communist sees the wisdom of Marx in every hamburger bought at McDonalds; in every promotion they're denied that would have gone to them in a true worker's paradise; in every election that doesn't go to their taste, in every newspaper article "slanted in the wrong direction". Every time they use the Great Idea to interpret another event, the Great Idea is confirmed all the more. It feels better - positive reinforcement - and of course, when something feels good, that, alas, makes us want to believe it all the more.
- Neurodiversity vs. Cognitive Liberty - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.13 - November 8th, 2009 [November 8th, 2009]
- Limits to the biolibertarian impulse - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.15 - November 8th, 2009 [November 8th, 2009]
- Neurodiversity vs. Cognitive Liberty, Round II - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.17 - November 8th, 2009 [November 8th, 2009]
- Cognitive liberty and right to one's mind - November 8th, 2009 [November 8th, 2009]
- TED Talks: Henry Markram builds a brain in a supercomputer - November 8th, 2009 [November 8th, 2009]
- And Now, for Something Completely Different: Doomsday! - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.19 - November 8th, 2009 [November 8th, 2009]
- Oklahoma and abortion - some fittingly harsh reflections - November 8th, 2009 [November 8th, 2009]
- Pigliucci on science and the scope of skeptical inquiry - November 8th, 2009 [November 8th, 2009]
- Remembering Mac Tonnies - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.24 - November 8th, 2009 [November 8th, 2009]
- Link dump: 2009.10.26 - November 8th, 2009 [November 8th, 2009]
- The Bright Side of Nuclear Armament - November 8th, 2009 [November 8th, 2009]
- Grieving chimps - November 8th, 2009 [November 8th, 2009]
- Elephant prosthetic - November 8th, 2009 [November 8th, 2009]
- Mass produced artificial skin to replace animal testing - November 8th, 2009 [November 8th, 2009]
- Dog gets osseointegrated prosthetic - November 8th, 2009 [November 8th, 2009]
- NASA Shuttle-derived Sidemount Heavy Launch Vehicle Concept - November 8th, 2009 [November 8th, 2009]
- Link dump for 2009.02.02 - November 8th, 2009 [November 8th, 2009]
- Link dump for 2009.11.04 - November 8th, 2009 [November 8th, 2009]
- Link dump for 2009.11.05 - November 8th, 2009 [November 8th, 2009]
- IEET's Biopolitics of Popular Culture Seminar - November 8th, 2009 [November 8th, 2009]
- Einstein and Millikan should have done a Kurzweil - November 8th, 2009 [November 8th, 2009]
- Cure aging or give a small number of disabled people jobs as janitors? - November 8th, 2009 [November 8th, 2009]
- Would unary notation prevent scope insensitivity? - November 8th, 2009 [November 8th, 2009]
- Cure aging or give a small number of disabled people jobs as janitors - unary version. - November 8th, 2009 [November 8th, 2009]
- At SENS4, Cambridge, UK - November 8th, 2009 [November 8th, 2009]
- SENS4 overview and review - how evolution complicates SENS, and why we must try harder - November 8th, 2009 [November 8th, 2009]
- SENS4 top 10 photos - November 8th, 2009 [November 8th, 2009]
- My AI research for this year - November 8th, 2009 [November 8th, 2009]
- My AI research: Formal Logic - November 8th, 2009 [November 8th, 2009]
- My AI research: Category theory and institution theory - November 8th, 2009 [November 8th, 2009]
- My AI research: The Semantic Web - November 8th, 2009 [November 8th, 2009]
- My AI research: Features and Flaws of Logical representation - November 8th, 2009 [November 8th, 2009]
- My AI research: Graphical models and probabilistic logics - November 8th, 2009 [November 8th, 2009]
- Hughes and More engage Italian Catholicism: Image caption competition - November 8th, 2009 [November 8th, 2009]
- Surprisingly good solutions, falling in love and life in a materialistic universe - November 8th, 2009 [November 8th, 2009]
- What do you get when you cross slightly evolved, status seeking monkeys with the scientific method? - November 8th, 2009 [November 8th, 2009]
- Seeking the optimal philanthropic strategy: Global Warming or AI risk? - November 8th, 2009 [November 8th, 2009]
- Machine Learning - harbinger of the future of AI? - November 8th, 2009 [November 8th, 2009]
- At the Singularity Summit in NYC - November 8th, 2009 [November 8th, 2009]
- Katja Grace: world-dominating superintelligence is "unlikely" - November 8th, 2009 [November 8th, 2009]
- Normal Human Heroes on "Nightmare futures" - November 8th, 2009 [November 8th, 2009]
- Anissimov on Intelligence Enhancement - November 8th, 2009 [November 8th, 2009]
- Yudkowsky on "Value is fragile" - November 8th, 2009 [November 8th, 2009]
- Response to Pearce - November 8th, 2009 [November 8th, 2009]
- Creative thinking lets you believe whatever you want - December 13th, 2009 [December 13th, 2009]
- Let’s get metaphysical: How our ongoing existence could appear increasingly absurd - December 13th, 2009 [December 13th, 2009]
- Linda MacDonald Glenn guest blogging in November and December - December 13th, 2009 [December 13th, 2009]
- Link dump for 2009.11.15 - December 13th, 2009 [December 13th, 2009]
- Call 1-800-New-Organ, by 2020? - December 13th, 2009 [December 13th, 2009]
- IBM's claim to have simulated a cat's brain grossly overstated - December 13th, 2009 [December 13th, 2009]
- John Hodgman pulls off Fermi Paradox schtick - December 13th, 2009 [December 13th, 2009]
- Deus Sex Machina - December 13th, 2009 [December 13th, 2009]
- How Americans spent themselves into ruin... but saved the world - December 13th, 2009 [December 13th, 2009]
- I am my own grandpa (or grandma)? - December 13th, 2009 [December 13th, 2009]
- Link dump for 2009.11.29 - December 13th, 2009 [December 13th, 2009]
- The art of Tomas Saraceno - December 13th, 2009 [December 13th, 2009]
- Link dump: 2009.12.05 - December 13th, 2009 [December 13th, 2009]
- The Harmonic Convergence of Science, Sight, & Sound - December 13th, 2009 [December 13th, 2009]
- Working on my website - December 13th, 2009 [December 13th, 2009]
- Transhumanism, personal immortality and the prospect of technologically enabled utopia - December 13th, 2009 [December 13th, 2009]
- RokoMijic.com is up - December 13th, 2009 [December 13th, 2009]
- Why the Fuss About Intelligence? - December 13th, 2009 [December 13th, 2009]
- Initiation ceremony - December 13th, 2009 [December 13th, 2009]
- Birthing Gods - December 13th, 2009 [December 13th, 2009]
- 11 core rationalist skills - from LessWrong - December 13th, 2009 [December 13th, 2009]
- The best of the guests - December 13th, 2009 [December 13th, 2009]
- The best of Sentient Developments: 2009 - December 13th, 2009 [December 13th, 2009]
- Link dump: 2009.12.15 - December 15th, 2009 [December 15th, 2009]
- The Utopia Force - December 22nd, 2009 [December 22nd, 2009]
- Avatar: The good, the bad and ugly - December 23rd, 2009 [December 23rd, 2009]
- Singularity Institute launches "2010 Singularity Research Challenge" - December 24th, 2009 [December 24th, 2009]
- Transhumanism as a "nonissue" - December 24th, 2009 [December 24th, 2009]
- Hanson on "Meh, Transhumanism" - December 25th, 2009 [December 25th, 2009]
- Merry Newtonmas from Transhuman Goodness - December 25th, 2009 [December 25th, 2009]
- After proud knowledge - December 26th, 2009 [December 26th, 2009]