Businesses still dont have a clear understanding of what to expect when it comes to the ROI of AI. Many believe that AI is just like any other software solution: the returns should, in theory, be immediate. But this is not the case. In addition, business leaders are often duped into thinking the path to ROI is a lot smoother than it is when it comes to AI because AI vendors tend to exaggerate the results their software generates.
In reality, identifying a metric to reliably measure the impact AI is having at a business is very hard.
In this article, we delve deeper into how business leaders should think about identifying ROI metrics that might help them understand the return they could generate from AI projects. To do this, we explore insights from interviews with three experts who were on our AI in Industry podcast this past month.
Special thanks to our three interviewees:
You can listen to our full playlist of episodes in our AI ROI playlist from the AI in Industry podcast. This article is based in large part on all three of these interviews:
Subscribe to the AI in Industry podcast wherever you get your podcasts:
We begin our analysis with a discussion of how to measure the ROI of AI.
AI projects inherently involve a level of uncertainty and experimentation before they can be deemed successful. In a small number of AI use-cases, identifying a measurable metric for projected returns may be relatively simple. For instance, in predictive maintenance applications for the manufacturing sector, businesses can link the returns directly to a reduction in maintenance costs or reduction in machinery downtimes.
But in other applications, such as improving customer experiences in banking, identifying a small number of reliable metrics to measure success is far more challenging.
Unless businesses have a clear understanding of the returns, they stand to risk losing on their AI investments. One way to ensure an AI project has a measurable metric is to choose a specific business problem where there already exists a non-AI solution and results are being measured and tracked.
Jan Kautz, VP of Learning and Perception Research at NVIDIA, who we interviewed for our previous podcast series on getting started with AI, seemed to agree that developing an AI solution for an existing business problem might be easier when it comes to measuring success rather than developing a completely new AI use-case with no precedent:
The danger of doing something completely new in AI is that you dont actually know if what you are doing is actually correct because you have nothing to compare it to. I would suggest banks to pick an area where they already have an existing system in place so that you can compare what the results of the AI system are and know if you are at least getting better results than the existing system
Business leaders also need to understand that in order to deploy an AI project across an organization, they not only require data scientists, but also data engineers. Data scientists are those that develop machine learning algorithms for a particular capability.
Data engineers usually undertake the task of implementing the solution across the enterprise. This might involve identifying if the existing data infrastructure is set up in a sustainable way that will allow AI systems to function smoothly over time and across the organization or that the devops process is capable of sustaining AI projects.
Narayanan believes most successful AI projects that can show positive results will involve data scientists working in collaboration with data engineers. Input from these employees is critical to understanding what a measurable metric of return might because they have the deepest understanding of what the AI system can do.
But these employees usually lack the insight to connect technical benefits to the overall business gains, which needs to come from the subject-matter experts in the domain into which AI is being applied.
Business leaders need to take into account both these perspectives to truly understand what benefits they are likely to get from their AI projects today. This will also help them accurately analyze what they want these AI benefits to look like in the future and tweak their systems towards that eventuality.
According to Martin, in order to successfully realize returns from AI projects, businesses need to figure out how to test their initial assumptions, experiment with AI systems, and identify use-cases as quickly as possible.
Testing whether these initial pilot projects have been successful means measuring the performance of the AI system in the task that it is being applied to.
Measuring success in these initial projects can even go wrong in ways that are not related to the technical challenges involved with AI. For instance, if a business implements an AI customer service software and only a few users are introduced to it because of ineffective marketing campaigns, measuring the returns of the AI system become even more challenging.
This is because the AI system might have been designed perfectly, but the pilot test might not have been accurately representative of whether any returns gained will actually lead to gains when deployed across the organization.
According to Martin, its critical for business leaders to understand that pilot test projects must not be run at scale across the enterprise. Enacting a large project, such as completely overhauling a fraud detection system at a bank, should only be done after careful analysis of the results from several experimental pilot projects. This is in line with Andrew Ngs advice to shoot for first AI projects with 6-12 month timeframes, not massive multi-year roll-outs.
Leaders need to think about this in phases, where the first step is to identify which small AI projects can potentially help the business gain knowledge about working with data and AI capabilities.
This doesnt mean that the smaller AI projects dont need to result in any success metrics. Rather, it means that in some cases, the pilot that shows the most immediate returns may not be the ideal first step for enterprise-wide adoption given a companys goals and long-term AI strategy. Leaders should focus on AI being a long-term skillset that is attained in incremental steps.
In order to measure a specific return, businesses also need to establish what kind of budgets they need for AI projects.
Unlike simple software automation, where costs are much easier to calculate, predicting the budgetary requirements for AI projects is more complex. Martin added that this was one of the more common AI-related questions that business leaders ask him. He said:
If a business leader is looking to answer questions like how much budget an AI project might require before starting the project, the best advice I can give businesses is to first ask how much budget can they realistically allocate to AI projects and then plan around that figure. AI projects are not easy to budget for because you dont know whats going to work and what is not; it involves a lot of experimentation. A business might not be able to ascertain how many such experiments they might need to run before finding a valuable use-case.
Martin stresses the fact that businesses need to think about AI from a long-term strategic perspective. They will have to make a decision on whether they are an AI company or not. Being an AI company means there will be a period of constant experimentation with uncertain results that could sometimes even take 6 months of experimentation to yield any noticeable results.
A recent article on MIT Sloan Business Review states that new ways of working and new management strategies (what might be called change management) are among the largest factors keeping most AI initiatives from generating ROI. Our own research arrives at the same conclusion.
Theres also no guarantee that an AI project will not go above budget, given the aforementioned uncertainty and experimentation involved. Butthis will give the data science team leaders an idea of how many experiments they might be able to conduct realistically and which ones they might need to prioritize.
One big challenge that many businesses might be grappling with when it comes to AI likely lies in ensuring that every dollar that goes into AI projects sees a significant return. Getting a return as soon as possible is the ideal business scenario.
Narayanan spoke about what misconception business leaders might have about measuring returns from AI :
Most of our knowledge around what AI can do for business stems from well-marketed examples in the news media. We find that most of these use-cases have been business problems that are well defined in nature. For example, we have seen reports of AI software beating the best human chess players or Beating humans in Alpha Go These are problems that have definitive end points. But when it comes to the most common business problems in fortune 500 companies. These do not have definite outcome.
What Narayanan seems to be articulating is that businesses might ask questions such as, Is our next product launch going to succeed? These questions are significantly more open-ended than a board game with definitive results. The term success might mean different things to different people or teams within an organization.
It might be hard to frame a clear question with a definitive answer for business problems. These questions at best might indicate that their problem statements can be extremely hazy and complex.
It might be impossible for any firm to look at a bucket of data and report how much business value might be gained from leveraging that data given the right kind of algorithms. This might be hard to digest for business leaders, but they need to expect uncertainty when it comes to AI.
This is not a traditional business mindset in many industries. According to Narayanan:
This is a cultural shift in the way of thinking of about how data might be critical for AI success. Leaders need to think about how AI can solve a business problem at scale for the enterprise, that is aligned with their business objectives as a whole while being highly sustainable.
In this section, we put forth a list of frameworks that business leaders can follow in order to maximize the possibility of gaining positive returns from their AI projects and effectively measuring it as such.
Traditionally in business, the term ROI usually corresponds to short-term financial gain, often in terms of improved revenue. AI is a broad technology and sticking to this traditional method of defining ROI might not be the best place to start for businesses. For instance, AI might very well be used to potentially increase revenue in an application.
However, AI can also be used to reduce costs, improve customer experience, or increase the productivity of a specific team within the business. The first step to understanding AI ROI might be to associate the returns with any types of positive business outcome, not necessarily financial gains, including leveling up a teams AI-related skillset.
Carmona said that in his experience, there have been several instances in which businesses have needed to invest funds in an AI project as it is being built due to budgetary constraints.
At the same time, business leaders might be looking for immediate returns on their AI investments. According to Carmona, balancing these two factors (uncertainty in AI projects and gaining returns fast) is something business leaders have to figure out before starting AI projects of any kind.
He spoke about a particular framework used by Microsoft (called the Agile AI framework) to find a balance between the two. We detail the steps involved in this framework below with insights from the interview:
Narayanan stated that one of the critical things for business leaders to understand about measuring the returns of AI projects is to first frame the business question that AI is being applied to in a way that is specific.
For instance, leaving aside the technical concerns, businesses first need to ask questions such as Is AI being used to solve a problem in the rate of growth of the organization, or is it being used to improve the efficiency of a business process or to improve customer experiences?
He went on to give an example of a firm that he claimed worked with Fractal Analytics in the past to explain this concept better:
About 18 months before getting into AI projects the client we were working with brought in a visionary leader who said im not supporting any initiative that cant show progress in 6 weeks.. He seemed to enforce this constraint even though he understood that there are a number of initiatives that are transformative long term engagements at enterprise level. This allowed the company to be more rapid defining areas to work and ruthless about what success and progress means and therefore establish a codified approach to measurement.
In a 12 month period they executed 30-40 different initiatives that they called Minimum Viable Proposition (MVPs). They identified 5-6 which had the potential to become transformative at enterprise level and this year they are taking these to deploy on an organizational scale.
According to Narayanan, the client gleaned the following three insights from this process:
See the rest here:
Unlocking the ROI of Artificial Intelligence Key Considerations for Business Leaders - Emerj
- Researchers create innovative verification techniques to increase security in artificial intelligence and image processing - European Research Council - April 29th, 2024 [April 29th, 2024]
- Who in Europe is investing the most in artificial intelligence? - Euronews - April 29th, 2024 [April 29th, 2024]
- 4 Artificial Intelligence (AI) Stocks Members of Congress Can't Stop Buying (and Nvidia Isn't 1 of Them!) - The Motley Fool - April 29th, 2024 [April 29th, 2024]
- The Standard Names Porter Orr Second Vice President of Artificial Intelligence Strategy and Development - NRToday.com - April 29th, 2024 [April 29th, 2024]
- Meet Nvidia CEO Jensen Huang, the man behind the $2 trillion company powering today's artificial intelligence - CBS News - April 29th, 2024 [April 29th, 2024]
- This Warren Buffett Dividend King Stock Just Invested $1.1 Billion Into Artificial Intelligence (AI) - The Motley Fool - April 29th, 2024 [April 29th, 2024]
- Joe Rogan Reveals What Could Be 'Game Over for the Human Race' - Newsweek - April 29th, 2024 [April 29th, 2024]
- LegalTech Artificial Intelligence Market to Soar to $6.4 Billion by 2028: Global Long-term Forecast to 2033 - Strategic ... - Yahoo Finance UK - April 29th, 2024 [April 29th, 2024]
- Artificial Intelligence Tokens Stumble Again! Is The AI Hype Over? - Coinpedia Fintech News - April 29th, 2024 [April 29th, 2024]
- CISA unveils guidelines for AI and critical infrastructure - FedScoop - April 29th, 2024 [April 29th, 2024]
- IBM's Webinar: Cybersecurity In The Era Of Artificial Intelligence Keynotes - AiThority - April 29th, 2024 [April 29th, 2024]
- Being and becoming a good doctor in the age of artificial intelligence - Firstpost - April 29th, 2024 [April 29th, 2024]
- Pope to take part in G7 summit in June to talk Artificial Intelligence - Crux Now - April 29th, 2024 [April 29th, 2024]
- Pope Francis to attend G7 summit to speak on artificial intelligence - Catholic News Agency - April 29th, 2024 [April 29th, 2024]
- 1 Unstoppable Artificial Intelligence (AI) Stock to Buy and Hold Forever - Yahoo Finance - April 29th, 2024 [April 29th, 2024]
- Pope to bring his call for ethical artificial intelligence to G7 summit in June in southern Italy - The Associated Press - April 29th, 2024 [April 29th, 2024]
- Elon Musk Called Tesla an Artificial Intelligence (AI) Robotics Company. Does That Make It a Buy? - The Motley Fool - April 29th, 2024 [April 29th, 2024]
- Emulating neurodegeneration and aging in artificial intelligence systems - Tech Xplore - April 29th, 2024 [April 29th, 2024]
- Pope Francis to participate in G7 session on AI - Vatican News - English - April 29th, 2024 [April 29th, 2024]
- A Once-in-a-Generation Investment Opportunity: 1 Bill Ackman Artificial Intelligence (AI) Stock to Buy Hand Over Fist ... - Yahoo Finance - April 29th, 2024 [April 29th, 2024]
- Tesla Stock Investors Cheer Progress on Its Artificial Intelligence (AI) Future - Yahoo Finance - April 29th, 2024 [April 29th, 2024]
- In Race to Build A.I., Tech Plans a Big Plumbing Upgrade - The New York Times - April 29th, 2024 [April 29th, 2024]
- A Baltimore-area teacher is accused of using AI to make his boss appear racist - NPR - April 29th, 2024 [April 29th, 2024]
- 3 Top Artificial Intelligence (AI) Stocks That Billionaires Jim Simons, Ray Dalio, and Israel Englander Are Buying - The Motley Fool - April 29th, 2024 [April 29th, 2024]
- The iPad Pro can give us a big surprise with its artificial intelligence chips. - Softonic EN - April 29th, 2024 [April 29th, 2024]
- This Artificial Intelligence (AI) Stock Could Soar 70%, According to Wall Street. Time to Buy? - Yahoo Finance - April 29th, 2024 [April 29th, 2024]
- Billionaire Bill Ackman Owns 8 Stocks -- and This Hypergrowth Artificial Intelligence (AI) Stock Isn't One of Them - The Motley Fool - April 29th, 2024 [April 29th, 2024]
- Artificial Intelligence Has Come for Our...Beauty Pageants? - Glamour - April 29th, 2024 [April 29th, 2024]
- 3 Stocks to Grab Now to Ride the Artificial Intelligence Chip Boom to Riches - InvestorPlace - April 29th, 2024 [April 29th, 2024]
- Is ASML's Big Sell-off a Warning Sign to Artificial Intelligence (AI) Investors? - The Motley Fool - April 29th, 2024 [April 29th, 2024]
- 3 Artificial Intelligence (AI) Stocks to Buy With $1,150 and Hold for Decades - Yahoo Finance - March 22nd, 2024 [March 22nd, 2024]
- This combines the power of artificial intelligence with human insights - finews.com - March 22nd, 2024 [March 22nd, 2024]
- Cohere Targets $5 Billion Valuation for ChatGPT Rival - PYMNTS.com - March 22nd, 2024 [March 22nd, 2024]
- IMF: Artificial Intelligence (AI) Will Transform 40% of Jobs. Can Investors Capitalize? - Yahoo Finance - March 22nd, 2024 [March 22nd, 2024]
- Cathie Wood Is Selling These 2 Artificial Intelligence (AI) Stocks. Should You? - Yahoo Finance - March 22nd, 2024 [March 22nd, 2024]
- iShares Robotics and Artificial Intelligence Multisector ETF (NYSEARCA:IRBO) Shares Acquired by Creative Financial ... - Defense World - March 22nd, 2024 [March 22nd, 2024]
- Inaugural Plenary Meeting of States Endorsing the Political Declaration on Responsible Military Use of Artificial ... - Department of State - March 22nd, 2024 [March 22nd, 2024]
- AMD Fell Today -- Is This a Chance to Buy the Artificial Intelligence (AI) Stock? - Yahoo Finance - March 22nd, 2024 [March 22nd, 2024]
- Artificial Intelligence (AI) Stocks Are Red-Hot, but Here's 1 to Avoid (for Now) - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- A Once-in-a-Generation Investment Opportunity: 1 Artificial Intelligence (AI) Growth Stock to Buy and Hold Forever - Yahoo Finance - March 22nd, 2024 [March 22nd, 2024]
- 2 Artificial Intelligence (AI) Stocks That Could Go Parabolic - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- Artificial intelligence will radically improve health care, but only if managed carefully - The Hill - March 22nd, 2024 [March 22nd, 2024]
- Tennessee Makes A.I. an Outlaw to Protect Its Country Music and More - The New York Times - March 22nd, 2024 [March 22nd, 2024]
- When Using Artificial Intelligence In Pharma R&D, Start With Identifying Problem To Solve - HBW Insight - March 22nd, 2024 [March 22nd, 2024]
- Why Super Micro Computer, Advanced Micro Devices, and Other Artificial Intelligence (AI) Stocks Tumbled on Tuesday - Yahoo Finance - March 22nd, 2024 [March 22nd, 2024]
- 3 Billionaires Are Selling Artificial Intelligence (AI) Stock Nvidia and Buying These 10 AI Stocks Instead - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- Nvidia Just Bought 5 Artificial Intelligence (AI) Stocks. These 2 Stand Out the Most. - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- UN adopts first global artificial intelligence resolution - ARMENPRESS - March 22nd, 2024 [March 22nd, 2024]
- This New Artificial Intelligence (AI) Chip Is a Massive Game Changer for Nvidia Stock - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- Nvidia Led the First Phase of Artificial Intelligence (AI), but These 2 Growth Stocks Will Lead the Next Phases ... - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- Meet Wall Street's Newest Stock-Split Stock, Along With the Artificial Intelligence (AI) Stock Likeliest to Follow in Its ... - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- Amid Rumors of a Deal With Rivian, Apple Acquired This Artificial Intelligence (AI) Start-Up Instead - Yahoo Finance - March 22nd, 2024 [March 22nd, 2024]
- UN adopts first global artificial intelligence resolution - CGTN - March 22nd, 2024 [March 22nd, 2024]
- 1 No-Brainer Artificial Intelligence (AI) Stock to Buy With $25 and Hold for 10 Years - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- 2 Artificial Intelligence (AI) Stocks That Could Make You a Millionaire - The Motley Fool - March 22nd, 2024 [March 22nd, 2024]
- Is Intel the Best Artificial Intelligence (AI) Semiconductor Stock to Buy Before It Skyrockets? - The Motley Fool - February 20th, 2024 [February 20th, 2024]
- The greater social implications of artificial intelligence - New Day NW - KING5.com - February 20th, 2024 [February 20th, 2024]
- AI comes to the world of beauty as eyelash robot uses artificial intelligence to place fake lashes - Fox News - February 20th, 2024 [February 20th, 2024]
- Beyer Appointed To Bipartisan Task Force On Artificial Intelligence - Falls Church News Press - February 20th, 2024 [February 20th, 2024]
- Why Did Nvidia Invest in These 5 Artificial Intelligence (AI) Stocks? Should You Buy Them, Too? - The Motley Fool - February 20th, 2024 [February 20th, 2024]
- Three key themes on artificial intelligence - Research Information - February 20th, 2024 [February 20th, 2024]
- How Artificial Intelligence is transforming consumerism - WFLA - February 20th, 2024 [February 20th, 2024]
- SAP names Philipp Herzig as chief artificial intelligence officer - CIO - February 20th, 2024 [February 20th, 2024]
- Unleashing the Power of Artificial Intelligence: Transforming Web-Based Applications for Enhanced Efficiency and User ... - Financialbuzz.com - February 20th, 2024 [February 20th, 2024]
- Koch Industries continues to accelerate its artificial intelligence initiative - The Business Journals - February 20th, 2024 [February 20th, 2024]
- Forget Nvidia: These 3 Artificial Intelligence (AI) Stocks Can Be the Next Stock-Split Stocks - The Motley Fool - February 20th, 2024 [February 20th, 2024]
- Artificial Intelligence for small business focus of upcoming JWCC Lunch and Learn on Feb. 28 Muddy River News - Muddy River News - February 20th, 2024 [February 20th, 2024]
- Opponents Highlight the Environmental Impact of Artificial Intelligence - News-Press Now - February 20th, 2024 [February 20th, 2024]
- Generative AI's environmental costs are soaring and mostly secret - Nature.com - February 20th, 2024 [February 20th, 2024]
- Chapter Summary: Genesis of Artificial Intelligence and a Scientific Revolution: 1950-1979 - EIN News - February 20th, 2024 [February 20th, 2024]
- What would Thomas Aquinas make of Artificial Intelligence? - ACI Africa - February 20th, 2024 [February 20th, 2024]
- ChatGPT Predicted Bitcoin Price Will "Skyrocket" - Cryptonews - February 20th, 2024 [February 20th, 2024]
- Worried About an Artificial Intelligence (AI) Stock Bubble? Consider This Billionaire Investor's Advice. - Yahoo Finance - February 20th, 2024 [February 20th, 2024]
- This Super Artificial Intelligence (AI) Stock Could Be at the Beginning of a Terrific Bull Run - Yahoo Finance - February 20th, 2024 [February 20th, 2024]
- 5 Artificial Intelligence (AI) Stocks That Could Make You a Millionaire - Yahoo Finance - February 20th, 2024 [February 20th, 2024]
- Will AI replace Colorado teachers? Here's what experts say. - The Colorado Sun - February 20th, 2024 [February 20th, 2024]
- Nvidia Could Be About to Counter a Big Artificial Intelligence (AI) Threat With This Move - Yahoo Finance - February 20th, 2024 [February 20th, 2024]
- The Healthiest U.S. Pharma Companies: Ranked by RealRate's Impressive Artificial Intelligence - Medium - February 20th, 2024 [February 20th, 2024]
- Down 84%, Is This Artificial Intelligence (AI) Stock a Buy After an Earnings Pop? - The Motley Fool - February 20th, 2024 [February 20th, 2024]
- 'AI for Humans' may be the most entertaining way to learn about artificial intelligence - Fast Company - February 20th, 2024 [February 20th, 2024]