{"id":31565,"date":"2011-02-20T16:43:00","date_gmt":"2011-02-20T16:43:00","guid":{"rendered":"http:\/\/euvolution.com\/futurist-transhuman-news-blog\/anders-sandberg-why-we-should-fear-the-paperclipper\/"},"modified":"2011-02-20T16:43:00","modified_gmt":"2011-02-20T16:43:00","slug":"anders-sandberg-why-we-should-fear-the-paperclipper","status":"publish","type":"post","link":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/transhuman\/anders-sandberg-why-we-should-fear-the-paperclipper.php","title":{"rendered":"Anders Sandberg: Why we should fear the paperclipper"},"content":{"rendered":"
<\/a><\/div>

Most people in the singularity community are familiar with the nightmarish \"paperclip\" scenario, but it's worth reviewing. Anders Sandberg summarizes the problem:<\/p>

A programmer has constructed an artificial intelligence based on an architecture similar to Marcus Hutter's AIXI<\/a> model...This AI will maximize the reward given by a utility function the programmer has given it. Just as a test, he connects it to a 3D printer and sets the utility function to give reward proportional to the number of manufactured paper-clips.<\/i><\/p>

At first nothing seems to happen: the AI zooms through various possibilities. It notices that smarter systems generally can make more paper-clips, so making itself smarter will likely increase the number of paper-clips that will eventually be made. It does so. It considers how it can make paper-clips using the 3D printer, estimating the number of possible paper-clips. It notes that if it could get more raw materials it could make more paper-clips. It hence figures out a plan to manufacture devices that will make it much smarter, prevent interference with its plan, and will turn all of Earth (and later the universe) into paper-clips. It does so.<\/p>

Only paper-clips remain.<\/p><\/blockquote>

In the article, Why we should fear the paperclipper<\/a>, Sandberg goes on to address a number of objections, including:<\/p>