Deep in the bowels of the Smithsonian National Museum of American History in Washington DC sits a large metal cabinet the size of a walk-in wardrobe. The cabinet houses a remarkable computer the front is covered in dials, switches and gauges and inside it is filled with potentiometers controlled by small electric motors. Behind one of the cabinet doors is a 20 x 20 array of light sensitive cells, a kind of artificial eye.
This is the Perceptron Mark I, a simplified electronic version of a biological neuron. It was designed by the American psychologist Frank Rosenblatt at Cornell University in the late 1950s who taught it to recognize simple shapes such as triangles.
Rosenblatts work is now widely recognized as the foundation of modern artificial intelligence but at the time it was controversial. Despite the original success, researchers were unable to build on this, not least because more complex pattern recognition required vastly more computational power than was available at the time. This insatiable appetite prevented further study of artificial neurons and the networks they create.
Todays deep learning machines also eat power, lots of it. And that raises an interesting question about how much they will need in future. Is this appetite sustainable as the goals of AI become more ambitious?
Today we get an answer thanks to the work of Neil Thompson at the Massachusetts Institute of Technology in Cambridge and several colleagues. This team has measured the improved performance of deep learning systems in recent years and show that how it depends on increases in computing power.
By extrapolating this trend, they say that future advances will soon become unfeasible. Progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable, say Thompson and colleagues, echoing the problems that emerged for Rosenblatt in the 1960s.
The teams approach is relatively straightforward. They analyzed over 1000 papers on deep learning to understand how learning performance scales with computational power. The answer is that the correlation is clear and dramatic.
In 2009, for example, deep learning was too demanding for the computer processors of the time. The turning point seems to have been when deep learning was ported to GPUs, initially yielding a 5 15 speed-up, they say.
This provided the horsepower for a neural network called AlexNet, which famously triumphed in a 2012 image recognition challenge where it wiped out the opposition. The victory created huge and sustained interest in deep neural networks that continues to this day.
But while deep learning performance increased by 35x between 2012 and 2019, the computational power behind it increased by an order of magnitude each year. Indeed, Thompson and co say this and other evidence suggests the computational power for deep learning has increased 9 orders of magnitude faster than the performance.
So how much computational power will be required in future? Thompson and co say that error rate for image recognition is currently 11.5 percent using 10^14 gigaflops of computational power at a cost of millions of dollars (ie 10^6 dollars).
They say achieving an error rate of just 1 per cent will require 10^28 gigaflops. And extrapolating at the current rate, this will cost 10^20 dollars. By comparison, the total amount of money in the world right now is measured in trillions ie 10^12 dollars.
Whats more, the environmental cost of such a calculation will be enormous, an increase in the amount of carbon produced of 14 orders of magnitude. Progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable, conclude Thompson and colleagues.
The future isnt entirely bleak, however. Thompson and cos extrapolations assume that future deep learning systems will use the same kinds of computers that are available today.
But various new approaches offer much more efficient computation. For example, in some tasks the human brain can outperform the best supercomputers while running on little more than a bowl of porridge. Neuromorphic computing attempts to copy this. And quantum computing promises orders of magnitude more computing power with relatively little increase in power consumption.
Another option is to abandon deep learning entirely and concentrate on other forms of machine learning that are less power hungry.
Of course, there is no guarantee that these new techniques and technologies will work. But if they dont, its hard to see how artificial intelligence will get much better than it is now.
Curiously, something like this happened after the Perceptron Mark I first appeared, a period that lasted for decades and is now known as the AI winter. The Smithsonian doesnt currently have it on display, but it is surely marks a lesson worth remembering.
Ref: arxiv.org/abs/2007.05558 : The Computational Limits of Deep Learning
View original post here:
The Computational Limits of Deep Learning Are Closer Than You Think - Discover Magazine
- The Quantum Computer Revolution Is Closer Than You May Think - National Review - May 3rd, 2017 [May 3rd, 2017]
- Time Crystals Could be the Key to the First Quantum Computer - TrendinTech - May 3rd, 2017 [May 3rd, 2017]
- quantum computing - WIRED UK - May 3rd, 2017 [May 3rd, 2017]
- Chinese scientists build world's first quantum computing machine - India Today - May 3rd, 2017 [May 3rd, 2017]
- Here's How We Can Achieve Mass-Produced Quantum Computers - ScienceAlert - June 6th, 2017 [June 6th, 2017]
- D-Wave partners with U of T to move quantum computing along - Financial Post - June 6th, 2017 [June 6th, 2017]
- Team develops first blockchain that can't be hacked by quantum computer - Siliconrepublic.com - June 6th, 2017 [June 6th, 2017]
- Telstra just wants a quantum computer to offer as-a-service - ZDNet - June 6th, 2017 [June 6th, 2017]
- Research collaborative pursues advanced quantum computing - Phys.Org - June 6th, 2017 [June 6th, 2017]
- Quantum Computing Market Forecast 2017-2022 | Market ... - June 6th, 2017 [June 6th, 2017]
- Quantum Computing Is Real, and D-Wave Just Open ... - WIRED - June 7th, 2017 [June 7th, 2017]
- FinDEVr London: Preparing for the Dark Side of Quantum Computing - GlobeNewswire (press release) - June 9th, 2017 [June 9th, 2017]
- Purdue, Microsoft to Collaborate on Quantum Computer - Photonics.com - June 9th, 2017 [June 9th, 2017]
- Scientists May Have Found a Way to Combat Quantum Computer Blockchain Hacking - Futurism - June 9th, 2017 [June 9th, 2017]
- Microsoft and Purdue work on scalable topological quantum computer - Next Big Future - June 12th, 2017 [June 12th, 2017]
- HYPRES Expands Efforts in Quantum Computing with Launch of European Subsidiary SeeQC - Business Wire (press release) - June 12th, 2017 [June 12th, 2017]
- From the Abacus to Supercomputers to Quantum Computers - Duke Today - June 13th, 2017 [June 13th, 2017]
- Accenture, Biogen, 1QBit Launch Quantum Computing App to ... - HIT Consultant - June 14th, 2017 [June 14th, 2017]
- The US and China "Quantum Computing Arms Race" Will Change Long-Held Dynamics in Commerce, Intelligence ... - PR Newswire (press release) - June 14th, 2017 [June 14th, 2017]
- Quantum Computing Technologies markets will reach $10.7 billion by 2024 - PR Newswire (press release) - June 14th, 2017 [June 14th, 2017]
- A Hybrid of Quantum Computing and Machine Learning Is Spawning New Ventures - IEEE Spectrum - June 14th, 2017 [June 14th, 2017]
- KPN CISO details Quantum computing attack dangers - Mobile World Live - June 16th, 2017 [June 16th, 2017]
- Get ahead in quantum computing AND attract Goldman Sachs - eFinancialCareers - June 16th, 2017 [June 16th, 2017]
- Accenture, 1QBit partner for drug discovery through quantum ... - ZDNet - June 16th, 2017 [June 16th, 2017]
- Toward optical quantum computing - MIT News - June 17th, 2017 [June 17th, 2017]
- Quantum computing, the machines of tomorrow | The Japan Times - The Japan Times - June 17th, 2017 [June 17th, 2017]
- Its time to decide how quantum computing will help your ... - June 18th, 2017 [June 18th, 2017]
- Israel Enters Quantum Computer Race, Placing Encryption at Ever-Greater Risk - Sputnik International - June 20th, 2017 [June 20th, 2017]
- Prototype device enables photon-photon interactions at room ... - Phys.Org - June 20th, 2017 [June 20th, 2017]
- Dow and 1QBit Announce Collaboration Agreement on Quantum Computing - Business Wire (press release) - June 21st, 2017 [June 21st, 2017]
- Imperfect crystals may be perfect storage method for quantum computing - Digital Trends - June 21st, 2017 [June 21st, 2017]
- Dow Chemical, 1QBit Ink Quantum Computing Development Deal - Zacks.com - June 22nd, 2017 [June 22nd, 2017]
- Google on track for quantum computer breakthrough by end of 2017 - New Scientist - June 22nd, 2017 [June 22nd, 2017]
- USC to lead project to build super-speedy quantum computers - USC News - June 24th, 2017 [June 24th, 2017]
- The Quantum Computer Factory That's Taking on Google and IBM ... - WIRED - June 24th, 2017 [June 24th, 2017]
- The weird science of quantum computing, communications and encryption - C4ISR & Networks - June 27th, 2017 [June 27th, 2017]
- Multi-coloured photons in 100 dimensions may make quantum ... - Cosmos - June 30th, 2017 [June 30th, 2017]
- Global Quantum Computing Market Growth at a CAGR of 35.12 ... - PR Newswire (press release) - June 30th, 2017 [June 30th, 2017]
- Qudits: The Real Future of Quantum Computing? - IEEE Spectrum - IEEE Spectrum - June 30th, 2017 [June 30th, 2017]
- New method could enable more stable and scalable quantum ... - Phys.Org - June 30th, 2017 [June 30th, 2017]
- Quantum computers are about to get real | Science News - Science News Magazine - June 30th, 2017 [June 30th, 2017]
- Quantum Computing - Scientific American - June 30th, 2017 [June 30th, 2017]
- Australia's ambitious plan to win the quantum race - ZDNet - July 3rd, 2017 [July 3rd, 2017]
- How quantum mechanics can change computing - The Conversation - The Conversation US - August 24th, 2017 [August 24th, 2017]
- UNSW joins with government and business to keep quantum computing technology in Australia - The Australian Financial Review - August 24th, 2017 [August 24th, 2017]
- UNSW launches Australia's first hardware quantum computing company with investments from federal and NSW ... - OpenGov Asia - August 24th, 2017 [August 24th, 2017]
- Finns chill out quantum computers with qubit refrigerator to cut out errors - ZDNet - August 24th, 2017 [August 24th, 2017]
- Hype and cash are muddying public understanding of quantum ... - The Conversation AU - August 24th, 2017 [August 24th, 2017]
- IEEE Approves Standards Project for Quantum Computing ... - insideHPC - August 24th, 2017 [August 24th, 2017]
- Silicon Quantum Computing launched to commercialise UNSW ... - ZDNet - August 24th, 2017 [August 24th, 2017]
- The Era of Quantum Computing Is Here. Outlook: Cloudy ... - January 30th, 2018 [January 30th, 2018]
- The Era of Quantum Computing Is Here. Outlook: Cloudy | WIRED - February 6th, 2018 [February 6th, 2018]
- Quantum computing in the NISQ era and beyond - February 6th, 2018 [February 6th, 2018]
- What is quantum computing? - Definition from WhatIs.com - February 6th, 2018 [February 6th, 2018]
- Quantum computers - WIRED UK - February 19th, 2018 [February 19th, 2018]
- Is Quantum Computing an Existential Threat to Blockchain ... - February 21st, 2018 [February 21st, 2018]
- What is Quantum Computing? Webopedia Definition - March 25th, 2018 [March 25th, 2018]
- Quantum Computing Explained - WIRED UK - April 15th, 2018 [April 15th, 2018]
- Quantum computing: A simple introduction - Explain that Stuff - June 2nd, 2018 [June 2nd, 2018]
- What are quantum computers and how do they work? WIRED ... - June 22nd, 2018 [June 22nd, 2018]
- How Quantum Computers Work - July 22nd, 2018 [July 22nd, 2018]
- The reality of quantum computing could be just three years ... - September 12th, 2018 [September 12th, 2018]
- The 3 Types of Quantum Computers and Their Applications - November 24th, 2018 [November 24th, 2018]
- Quantum Computing - VLAB - January 27th, 2019 [January 27th, 2019]
- Quantum Computing | Centre for Quantum Computation and ... - January 27th, 2019 [January 27th, 2019]
- Microsofts quantum computing network takes a giant leap ... - March 7th, 2019 [March 7th, 2019]
- IBM hits quantum computing milestone, may see 'Quantum ... - March 7th, 2019 [March 7th, 2019]
- Quantum technology - Wikipedia - March 13th, 2019 [March 13th, 2019]
- Quantum Computing | D-Wave Systems - April 18th, 2019 [April 18th, 2019]
- Microsoft will open-source parts of Q#, the programming ... - May 7th, 2019 [May 7th, 2019]
- What Is Quantum Computing? The Complete WIRED Guide | WIRED - May 8th, 2019 [May 8th, 2019]
- The five pillars of Edge Computing -- and what is Edge computing anyway? - Information Age - October 1st, 2019 [October 1st, 2019]
- Moore's Law Is Dying. This Brain-Inspired Analogue Chip Is a Glimpse of What's Next - Singularity Hub - October 1st, 2019 [October 1st, 2019]
- Experts Gather at Fermilab for International Workshop on Cryogenic Electronics for Quantum Systems - Quantaneo, the Quantum Computing Source - October 1st, 2019 [October 1st, 2019]
- Princeton announces initiative to propel innovations in quantum science and technology - Princeton University - October 1st, 2019 [October 1st, 2019]
- Detecting Environmental 'Noise' That Can Damage The Quantum State of Qubits - In Compliance - October 1st, 2019 [October 1st, 2019]
- Quantum Computing beginning talks with clients on its quantum asset allocation application - Proactive Investors USA & Canada - October 1st, 2019 [October 1st, 2019]
- What is quantum computing? The next era of computational evolution, explained - Digital Trends - October 1st, 2019 [October 1st, 2019]
- IT sees the Emergence of Quantum Computing as a Looming Threat to Keeping Valuable Information Confidential - Quantaneo, the Quantum Computing Source - October 23rd, 2019 [October 23rd, 2019]
- More wrong answers get quantum computers to find the right one - Futurity: Research News - October 23rd, 2019 [October 23rd, 2019]