How fast is real-time? According to Robert Millers 1968 seminal study on human-computer interaction, we perceive a response time of 100 milliseconds or less as instantaneous. This is much quicker than the average human reaction time, which one online test clocks at 273 milliseconds.
Nowadays, customer-facing applications need to respond in as close to real-time as possible to win our attention and meet standard requirements. For example, 100 milliseconds is a golden rule for credit card processors to meet SLAs. Especially in our new remote-first economy, other areas like media, health care, retail and manufacturing need to take full advantage of real-time data processing to enhance digital business and stay relevant. Yet, leveling up with real-time requires some technical nuances that might be challenging to implement with the current developer skills gap.
According to Manish Devgan, chief product officer at Hazelcast, the problem with most data management techniques is that data isnt being processed in-stream at the edge. Instead, data gets stored and thrown into a data lake where it accumulates, unfulfilled. To make matters more complicated, decentralized data management means organizations may be overseeing many different data lakes, adding to the potential for unused data and increased latency.
I recently met with Devgan to learn how in-memory stream processing goes beyond real-time data. According to Devgan, the secret sauce is processing incoming data in-stream while tying it to persistent storage to create context. By doing so, companies could unlock more innovative real-time experiences, he said.
Real-time presents many opportunities for business operations and end-user customer engagements. This is especially true in the financial sector. For example, Devgan explains how one bank was able to use real-time data at the moment of customer interaction at their ATMs. The system was able to fetch a history of the user account and extend an offer for a low line of credit, increasing their total loans by 400%. Capturing value at the moment of user intent is a golden opportunity, said Devgan.
Capital One is another example of a financial institution leveraging advanced technology to improve the product experience. Theyre one of the leading banks in terms of technology, said Devgan. Capital Ones threat detection system calls a machine learning model in real-time to detect and prevent fraud. Only by acting on incoming data in the moment of interaction can preventative responses like these be signaled.
Other scenarios are also beginning to incorporate increased real-time analysis. This split-second preemptive decision-making is important for many other use cases, said Devgan. Physical retail might loop into customer loyalty programs at point-of-sale systems to increase in-store spending. Or, automobile manufacturers may want to use real-time processing to detect a problem with their robotic arms sooner rather than later.
When an event comes in, it usually has very little information. But when you look up a customer profile and connect it to a greater context, you can direct further actions. Correlating data in motion with data at rest is where the magic happens, Devgan. This distinction uncovers the difference between real-time data and stream processing.
The real-time label tends to get thrown around a lot, and its quantified differently for different groups. Real-time might be five minutes for some applications but 10 milliseconds for others. Real-time could simply mean that a system accesses data at rest very quickly, delivering perceivably real-time results.
On the other hand, stream processing incorporates data at rest and data in motion. It computes incoming, in-stream data and correlates it with data at rest. You dont let the data land you process it as it comes in, Devgan said.
For years, enterprises have been collecting and storing exorbitant amounts of data. A survey by Seagate and IDC reveals that 43% of data collected is largely unleveraged and other studies estimate the amount of this dark data is over 50%. And, not acting upon data effectively makes it just another pile on top of the growing mountain of technical debt.
There is a significant storage expense for holding on to too much data, whether on-premises or in the cloud. But, the paradigm is shifting a bit, said Devgan. Organizations dont want to hold onto everything. The attitude is beginning to shift toward garbage in, garbage out.
So, what sort of technology does stream processing require? Processing data at the edge requires some fundamental changes to design thinking. For one, since data must be computed in-stream, theres more emphasis on in-memory processing. Devgan explained this as pushing compute around instead of pushing data around. To enable this, Devgan pointed to open-source projects like Jet that provide powerful distributed batch and stream processing.
SQL has a storied history of acting upon data at rest. Now, Devgan said, he noticed the reemergence of SQL in a stream processing context. Though the industry is still defining how the SQL dialect will conform in a streaming world, hes bullish on the prospect of SQL guiding this area. Streaming SQL will be the lingua franca for streaming, he insisted. Were also in an early era of determining how to bridge the stateful and serverless worlds, Devgan added.
But regardless of computing location, it will come down to the net customer experience outcomes. People want to build their business applications and are less worried about the operational management of the system, said Devgan. Whether its building real-time offers for e-commerce or implementing fraud detection, it will come down to the numbers. There is also a role for stream processing in health care to optimize physical resources and enhance patient experiences with real-time data. In all these scenarios, rapid access to data and processing at the edge will determine much of their success.
View post:
How Stream Processing Goes Beyond Real-Time - DevOps.com
- This Is My New Golden Rule for Renting Vacation Homes (I Was Doing It All Wrong) - Yahoo Life - April 12th, 2024 [April 12th, 2024]
- Misguided culture warriors should heed the golden rule [column] - LNP | LancasterOnline - April 12th, 2024 [April 12th, 2024]
- Michael L. Fischler: Anger and The Golden Rule - The Union Leader - April 10th, 2024 [April 10th, 2024]
- Dog trainer reveals her golden rule for recall training (we can't believe how simple it is!) - Yahoo Life - April 10th, 2024 [April 10th, 2024]
- Fr. Aristides Palaynes and the Golden Rule Community - Greek Orthodox Archdiocese of America - Greek Orthodox Archdiocese of America - April 10th, 2024 [April 10th, 2024]
- How Lulu still looks fabulous at 75, from her skincare 'golden rule' to refusing to take lifts - as star revea - Daily Mail - April 10th, 2024 [April 10th, 2024]
- Ken Henry: We are breaking the 'golden rule' of economic policy - ABC News - April 10th, 2024 [April 10th, 2024]
- The Golden Rule and the Free Market - Foundation for Economic Education - March 6th, 2024 [March 6th, 2024]
- Courtney B. Vance and Angela Bassett share their golden rule for successful parenting - The Times of India - March 6th, 2024 [March 6th, 2024]
- Live music at The Golden Rule Coffee House on Friday, March 8 - Redwood Falls Gazette - February 29th, 2024 [February 29th, 2024]
- Blake Lively and Ryan Reynolds have always followed one relationship rule - Marie Claire UK - February 29th, 2024 [February 29th, 2024]
- The Golden Rule - A Way of Life - County 10 News - January 27th, 2024 [January 27th, 2024]
- Dear Annie: Nurses Golden Rule might be the answer to relationship problems - MLive.com - January 27th, 2024 [January 27th, 2024]
- Aitana's golden rule and the pending account she has with her most loyal audience - WECB - January 27th, 2024 [January 27th, 2024]
- Guest columnist David Hernndez: Climate, refugees, and the golden rule - GazetteNET - January 27th, 2024 [January 27th, 2024]
- A golden rule that should have been followed in North - The Sun Chronicle - January 27th, 2024 [January 27th, 2024]
- The Golden Rule Refined | | news-journal.com - Longview News-Journal - January 27th, 2024 [January 27th, 2024]
- Film About The Golden Rule Released in Bali - EIN News - January 27th, 2024 [January 27th, 2024]
- Heres a Golden Rule Jeff Bezos Seems to Have Forgotten: Never Let Your Ego Get in the Way of Doing Business - The Good Men Project - January 27th, 2024 [January 27th, 2024]
- Universalists to consider the mandate of the Golden Rule - Ashland Daily Press - November 20th, 2023 [November 20th, 2023]
- 6-Year-Old Boy Dies a Month After Adult Neighbor Allegedly Beats ... - PEOPLE - November 20th, 2023 [November 20th, 2023]
- COLUMN: Celebrate Thanksgiving year-round with 'Thanks-living ... - Andalusia Star-News - November 20th, 2023 [November 20th, 2023]
- Silence is craven, not golden - The Gazette - November 20th, 2023 [November 20th, 2023]
- Sellars CEO Named to Wisconsin 'Titan 100' - Industrial Distribution - November 20th, 2023 [November 20th, 2023]
- The most valuable decluttering lessons I have learned | - Homes & Gardens - November 20th, 2023 [November 20th, 2023]
- COMMENTARY| Bethel: Too much hate | Opinion ... - Bennington Banner - November 20th, 2023 [November 20th, 2023]
- Cooking with Love Nello's Continues Impress Diners - St. Albert Gazette - November 20th, 2023 [November 20th, 2023]
- What Happened to James Garner? Inside the 'Maverick' Star's ... - Yahoo Entertainment - November 20th, 2023 [November 20th, 2023]
- Successful trial of coating that converts UV into PAR boosts sales - hortidaily.com - November 20th, 2023 [November 20th, 2023]
- Debt rules will affect the most vulnerable, EU trade union chief warns - EURACTIV - November 20th, 2023 [November 20th, 2023]
- Guest Opinion | Bill Paparian: Pasadena City Hall Believes in the ... - Pasadena Now - May 18th, 2023 [May 18th, 2023]
- Golden ratio in venation patterns of dragonfly wings | Scientific Reports - Nature.com - May 18th, 2023 [May 18th, 2023]
- The Golden Rule Of Data Gathering And New Expectations Of Value ... - The Drum - May 18th, 2023 [May 18th, 2023]
- Glass gets away with breaking the golden rule - The Irish News - May 18th, 2023 [May 18th, 2023]
- Red Horse Recruiter Ranks First in Class - 125fw.ang.af.mil - May 18th, 2023 [May 18th, 2023]
- These 19 Birmingham BBQ joints are the ultimate Memorial Day ... - Bham Now - May 18th, 2023 [May 18th, 2023]
- Mountain West Technologies wins Casper Area Chamber of ... - Cap City News - May 18th, 2023 [May 18th, 2023]
- The Unseen Engine of South Florida's Booming Vacation Rental Market: Estaga - Yahoo Finance - May 18th, 2023 [May 18th, 2023]
- Simon Cowell to break important BGT rule this weekend, ITV confirms - South Wales Argus - May 18th, 2023 [May 18th, 2023]
- Starting out on the St. Croix: What High School Skiing Taught Jessie ... - fasterskier.com - May 18th, 2023 [May 18th, 2023]
- These are the weirdest and least understood pickleball rules and ... - msnNOW - May 18th, 2023 [May 18th, 2023]
- Media Source Not Showing in OBS? Here's How to Fix It - MUO - MakeUseOf - May 18th, 2023 [May 18th, 2023]
- Youngkin avoids talk of diversity, politics at VMI graduation - Cardinal News - May 18th, 2023 [May 18th, 2023]
- Mother's Day roots: the opposition to war | READER COMMENTARY - Baltimore Sun - May 18th, 2023 [May 18th, 2023]
- Rep. Gallagher says US needs to take off 'golden blindfolds' and 'open our eyes' to China risk - Fox News - May 18th, 2023 [May 18th, 2023]
- Open Heaven 18 May 2023: The Gift of Love - ELANHUB MEDIA - May 18th, 2023 [May 18th, 2023]
- 10 Best Quotes About Family In The Fast & Furious Franchise - CBR - Comic Book Resources - May 18th, 2023 [May 18th, 2023]
- Life in the Fast Lane - New DOE Rule Changes Push LNG Projects ... - RBN Energy - May 18th, 2023 [May 18th, 2023]
- Dont Say That Word- Days After Golden Advice From Wrexhams Ryan Reynolds, NFL Icon JJ Watt Reminds Wife of Unspoken Rule After Burnley Investment -... - May 18th, 2023 [May 18th, 2023]
- 14 Handy Tricks To Get Better Sleep While Backpacking - The Trek - May 18th, 2023 [May 18th, 2023]
- Investors overlooking the golden rule as they flock to DIY apps - Stuff - April 29th, 2023 [April 29th, 2023]
- 7 money 'rules' you can actually break, according to financial experts - MarketWatch - April 29th, 2023 [April 29th, 2023]
- Dorsman: MP Motorsport betting on caution ahead of inevitable ... - Formula 2 - April 29th, 2023 [April 29th, 2023]
- Boomer explains why millennials are having a hard time at work: 'It's ... - Upworthy - April 29th, 2023 [April 29th, 2023]
- AFA.net - Basic Fairness in Women's Sports - American Family Association - April 29th, 2023 [April 29th, 2023]
- How To Prep Air Fryer Vegetables So They Don't Escape The Basket - Tasting Table - April 29th, 2023 [April 29th, 2023]
- Pansexual Flag: Here's what the Pride Flag's colors mean and more - USA TODAY - April 29th, 2023 [April 29th, 2023]
- What is the "Rules-Based-Order"? - CounterPunch.org - CounterPunch - April 29th, 2023 [April 29th, 2023]
- Hudson WWII vet turns 100, decides to quit the bocce ball team - Suncoast News - April 29th, 2023 [April 29th, 2023]
- Kansas City reportedly let Meta developer break diversity rules, then ... - KCUR - April 29th, 2023 [April 29th, 2023]
- From the Right: Drawing the line between trans rights and parental ... - The Malibu Times - April 29th, 2023 [April 29th, 2023]
- I was 40,000 in debt but got myself out of it here is the rule I swear by and how it changed my life... - The Sun - April 29th, 2023 [April 29th, 2023]
- OBITUARY - Mary Anderson - Berthoud Weekly SurveyorBerthoud ... - BerthoudSurveyor.com - April 29th, 2023 [April 29th, 2023]
- 8 ways to improve your written communications - Smartbrief - April 29th, 2023 [April 29th, 2023]
- Draymond Green: Draymond wont be moved by the Draymond rule - Eurohoops - April 29th, 2023 [April 29th, 2023]
- Dont Make These 4 Nursing Home Abuse Claim Mistakes in Las ... - Legal Reader - April 29th, 2023 [April 29th, 2023]
- What Food to Pack for Family Hikes - Outside - April 29th, 2023 [April 29th, 2023]
- Lawyer Discernment Is Critical In The World Of AI - JD Supra - April 29th, 2023 [April 29th, 2023]
- 12 blockbuster movies for summer 2023 ranked: we can't wait! - What To Watch - April 29th, 2023 [April 29th, 2023]
- Meghan broke 'golden rule' months before royal exit and has done so ever since - Express - April 20th, 2023 [April 20th, 2023]
- We Are All Connected: Loving One Another in Different Religions - Graphic - April 20th, 2023 [April 20th, 2023]
- Grammar Girl AP style tips on the Oxford comma, headlines and more - PR Daily - April 20th, 2023 [April 20th, 2023]
- How to prevent HVAC havoc this summer - KCAU 9 - April 20th, 2023 [April 20th, 2023]
- Take the right care of your yard | News | hometownnewsbrevard.com - Hometown News - April 20th, 2023 [April 20th, 2023]
- Britain's Got Talent: Simon Cowell speaks out after new judge Bruno Tonioli breaks rule on first appearance - LADbible - April 20th, 2023 [April 20th, 2023]
- After experimenting with directives, ISSF reinstates the Tokyo Olympics old-golden rule book for Paris 2024 - Free Press Journal - April 20th, 2023 [April 20th, 2023]
- Want to stay married? Gogglebox's Steph and Dom reveal golden ... - Daily Mail - April 20th, 2023 [April 20th, 2023]
- Overcoming Spiritual Violence and Embracing Peace with Rev. Dr ... - Bear World Magazine - April 20th, 2023 [April 20th, 2023]
- Good News, Part 3 of Resurrection as the Good News | Good News ... - Patheos - April 20th, 2023 [April 20th, 2023]
- What to Do After an Auto Accident: A Step-by-Step Guide - BBN Times - April 20th, 2023 [April 20th, 2023]