Lately, we've seen many "x-Ops" management practices appear on the scene, all derivatives from DevOps, which seeks to coordinate the output of developers and operations teams into a smooth, consistent and rapid flow of software releases. Another emerging practice, DataOps, seeks to achieve a similarly smooth, consistent and rapid flow of data through enterprises. Like many things these days, DataOps is spilling over from the large Internet companies, who process petabytes and exabytes of information on a daily basis.
Such an uninhibited data flow is increasingly vital to enterprises seeking to become more data-driven and scale artificial intelligence and machine learning to the point where these technologies can have strategic impact.
Awareness of DataOps is high. A recent survey of 300 companies by 451 Research finds 72 percent have active DataOps efforts underway, and the remaining 28 percent are planning to do so over the coming year. A majority, 86 percent, are increasing their spend on DataOps projects to over the next 12 months. Most of this spending will go to analytics, self-service data access, data virtualization, and data preparation efforts.
In the report, 451 Research analyst Matt Aslett defines DataOps as "The alignment of people, processes and technology to enable more agile and automated approaches to data management."
The catch is "most enterprises are unprepared, often because of behavioral norms -- like territorial data hoarding -- and because they lag in their technical capabilities -- often stuck with cumbersome extract, transform, and load (ETL) and master data management (MDM) systems," according to Andy Palmer and a team of co-authors in their latest report,Getting DataOps Right, published by O'Reilly. Across most enterprises, data is siloed, disconnected, and generally inaccessible. There is also an abundance of data that is completely undiscovered, of which decision-makers are not even aware.
Here are some of Palmer's recommendations for building and shaping a well-functioning DataOps ecosystem:
Keep it open: The ecosystem in DataOps should resemble DevOps ecosystems in which there are many best-of-breed free and open source software and proprietary tools that are expected to interoperate via APIs." This also includes carefully evaluating and selecting from the raft of tools that have been developed by the large internet companies.
Automate it all:The collection, ingestion, organizing, storage and surfacing of massive amounts of data at as close to a near-real-time pace as possible has become almost impossible for humans to manage. Let the machines do it, Palmer urges. Areas ripe for automaton include "operations, repeatability, automated testing, and release of data." Look to the ways DevOps is facilitating the automation of the software build, test, and release process, he points out.
Process data in both batch and streaming modes. While DataOps is about real-time delivery of data, there's still a place -- and reason -- for batch mode as well. "The success of Kafka and similar design patterns has validated that a healthy next-generation data ecosystem includes the ability to simultaneously process data from source to consumption in both batch and streaming modes," Palmer points out.
Track data lineage: Trust in the data is the single most important element in a data-driven enterprise, and it simply may cease to function without it. That's why well-thought-out data governance and a metadata (data about data) layer is important. "A focus on data lineage and processing tracking across the data ecosystem results in reproducibility going up and confidence in data increasing," says Palmer.
Have layered interfaces. Everyone touches data in different ways. "Some power users need to access data in its raw form, whereas others just want to get responses to inquiries that are well formulated," Palmer says. That's why a layered set of services and design patterns is required for the different personas of users. Palmer says there are three approaches to meeting these multilayered requirements:
Business leaders are increasingly leaning on their technology leaders and teams to transform their organizations into data-driven digital entities that can react to events and opportunities almost instantaneously. The best way to accomplish this -- especially with the meager budgets and limited support that gets thrown out with this mandate -- is to align the way data flows from source to storage.
Go here to read the rest:
Artificial intelligence requires trusted data, and a healthy DataOps ecosystem - ZDNet
- Green with Envy | How to Spot an Eco-Snob | Part III - November 8th, 2009 [November 8th, 2009]
- EcoLogo - November 8th, 2009 [November 8th, 2009]
- 5 Ways to Green Your Exercise Routine - November 8th, 2009 [November 8th, 2009]
- Seed Bombs - November 8th, 2009 [November 8th, 2009]
- Guerrilla gardening - November 8th, 2009 [November 8th, 2009]
- Green Your Morning Routine - November 8th, 2009 [November 8th, 2009]
- Environmental Benefits of Telecommuting - November 8th, 2009 [November 8th, 2009]
- Safeway Sponsors Portland Community Cleanup - November 8th, 2009 [November 8th, 2009]
- Electric Vehicle Race - November 8th, 2009 [November 8th, 2009]
- Portland Bridge Pedal 2009 - November 8th, 2009 [November 8th, 2009]
- E-waste in Oregon - November 8th, 2009 [November 8th, 2009]
- Bike Sharing in Portland - November 8th, 2009 [November 8th, 2009]
- Bucks for the Bay Challenge - November 8th, 2009 [November 8th, 2009]
- Drive to Make a Difference with MyMPG - November 8th, 2009 [November 8th, 2009]
- Bathroom Sprayers - Green your Toilet Routine - November 8th, 2009 [November 8th, 2009]
- Ubuntu OS can Save Energy - November 8th, 2009 [November 8th, 2009]
- Green Metropolis, David Owen - November 8th, 2009 [November 8th, 2009]
- Sustainable Pens: GLO Pens - November 8th, 2009 [November 8th, 2009]
- International Day of Climate Action - November 8th, 2009 [November 8th, 2009]
- Donate to Oregon Toxics Alliance - November 8th, 2009 [November 8th, 2009]
- Biomass Energy Generation Myths - November 8th, 2009 [November 8th, 2009]
- Crude The Real Price of Oil | Playing in Portland - November 8th, 2009 [November 8th, 2009]
- Pictures From 350 Climate Day in Portland - November 8th, 2009 [November 8th, 2009]
- Arcimoto Electric Vehicles in Oregon - November 8th, 2009 [November 8th, 2009]
- Urban Rooftop Wind Turbines - November 8th, 2009 [November 8th, 2009]
- Chromium 6 Emissions from ESCO in Portland - December 13th, 2009 [December 13th, 2009]
- Food Inc. Review - December 19th, 2009 [December 19th, 2009]
- Making Maps with Google Earth and Google Maps by Shane Bradt of the University of New Hampshire Cooperative Extension - March 23rd, 2010 [March 23rd, 2010]
- Demonstration of Miradi 3.1 by Nick Salafsky of Foundations of Success - March 23rd, 2010 [March 23rd, 2010]
- Advanced Mashups – KML and the Mapping API by Cary Chadwick of the University of Connecticut Center for Land Use Education and Research - March 23rd, 2010 [March 23rd, 2010]
- Demonstration of InVEST by Heather Tallis of the Natural Capital Project - March 23rd, 2010 [March 23rd, 2010]
- GIS Maps Online by Emily Wilson of the University of Connecticut Center for Land Use Education and Research - March 23rd, 2010 [March 23rd, 2010]
- From ArcGIS to Web Maps: Simple Techniques for Publishing GIS Maps Online by Emily Wilson of the University of Connecticut Center for Land Use Education and Research - March 25th, 2010 [March 25th, 2010]
- Demonstration of Marine InVEST by Anne Guerry of the Natural Capital Project - March 31st, 2010 [March 31st, 2010]
- Eliminate and Decrease Styrofoam - March 31st, 2010 [March 31st, 2010]
- Portland Plans to Spend $600 million on Master Bike Plan - April 2nd, 2010 [April 2nd, 2010]
- (Webinar in Spanish) Demostración sobre Vista 2.5 de NatureServe en línea (Webinar) por Ian Varley, Carmen Josse, y Alexandra Sanchez de Lozada de NatureServe. - April 6th, 2010 [April 6th, 2010]
- Using and Adding Your Content to Google Ocean by Charlotte Vick, Google Content Manager of Mission Blue - April 13th, 2010 [April 13th, 2010]
- End Paper Receipts - May 1st, 2010 [May 1st, 2010]
- Demonstration of CanVis by Chris Haynes of NOAA Coastal Services Center - May 6th, 2010 [May 6th, 2010]
- Demonstration of HD.gov Web Portal by Jeff Adkins from NOAA Coastal Services Center - May 13th, 2010 [May 13th, 2010]
- Demonstration of Ecosystem Assessment and Reporting Tool by Steve Schill of The Nature Conservancy - May 13th, 2010 [May 13th, 2010]
- Demonstration of Version 2.0 of the Multipurpose Marine Cadastre by Adam Bode and Brian Smith of NOAA Coastal Services Center - May 17th, 2010 [May 17th, 2010]
- CRUDE Filmmakers Subpoenaed by Chevron - May 22nd, 2010 [May 22nd, 2010]
- Demonstration of the Digital Coast Coastal Inundation Toolkit by Steph Beard, Jodie Sprayberry and Billy Brooks of NOAA Coastal Services Center - May 25th, 2010 [May 25th, 2010]
- Presentation on the Creating Resilient Communities EBM Tool Demonstration Project by Jocelyn Hittle of PlaceMatters - June 10th, 2010 [June 10th, 2010]
- Presentation on Economic Data Needed for EBM by Linwood Pendleton of Duke University - October 11th, 2010 [October 11th, 2010]
- Recycling Water - October 16th, 2010 [October 16th, 2010]
- ODOT Partners with Oregon Toxics Alliance to Reduce Pesticides - October 17th, 2010 [October 17th, 2010]
- Goats Hired to Mow Portland Lot - October 17th, 2010 [October 17th, 2010]
- A World of Health: Connecting People, Place, and Planet - October 17th, 2010 [October 17th, 2010]
- Alternative Recycling Options - October 17th, 2010 [October 17th, 2010]
- No More Bullying the Bull Trout - October 17th, 2010 [October 17th, 2010]
- 1000+ EV Charging Stations Slated for Oregon I-5 Corridor - October 17th, 2010 [October 17th, 2010]
- The Vertical Farm Concept - October 17th, 2010 [October 17th, 2010]
- Blog Action Day 2010 | Water - October 17th, 2010 [October 17th, 2010]
- Eco Districts - October 24th, 2010 [October 24th, 2010]
- Will The Nissan Leaf Thrive? - October 24th, 2010 [October 24th, 2010]
- A Green Railroad - October 24th, 2010 [October 24th, 2010]
- Biomass is not Oregon's clean-energy future as currently promoted - October 24th, 2010 [October 24th, 2010]
- Electrified Parking Spaces - October 24th, 2010 [October 24th, 2010]
- Tree Planting - October 24th, 2010 [October 24th, 2010]
- Three Tips to Reduce Your Carbon Footprint and Live Longer. - October 24th, 2010 [October 24th, 2010]
- Biomass is not Oregon’s clean-energy future as currently promoted - October 31st, 2010 [October 31st, 2010]
- Rail~Volution - October 31st, 2010 [October 31st, 2010]
- Green Streets Initiative - October 31st, 2010 [October 31st, 2010]
- Mayor Kitty Piercy and Envision Eugene - November 7th, 2010 [November 7th, 2010]
- The Willamette River Transit Bridge - November 13th, 2010 [November 13th, 2010]
- Collaborative Learning and Land Use Tools to Support Community Based Ecosystem Management by Chris Feurt of the Wells National Estuarine Research Reserve - November 14th, 2010 [November 14th, 2010]
- Portland Federal Building Begins Green Makeover - November 14th, 2010 [November 14th, 2010]
- Vestas’ New HQ in Portland Shoots for LEED Platinum - November 14th, 2010 [November 14th, 2010]
- College Degrees to Get You in the Environmental Field - November 14th, 2010 [November 14th, 2010]
- Demonstration of openNSPECT, an Open Source Version of the Nonpoint-Source Pollution and Erosion Comparison Tool by Dave Eslinger of NOAA Coastal Services Center - February 14th, 2011 [February 14th, 2011]
- Demonstration of EMDS by Keith Reynolds of the US Forest Service - February 14th, 2011 [February 14th, 2011]
- Demonstration of Habitat Priority Planner by Chrissa Waite and Danielle Bamford of NOAA Coastal Services Center - February 14th, 2011 [February 14th, 2011]
- Presentation on the Coastal Adaptation to Sea Level Rise Tool (COAST) by Sam Merrill of the New England Environmental Finance Center - February 14th, 2011 [February 14th, 2011]
- Presentation on the Coastal and Marine Ecological Classification Standard by Kathy Goodin of NatureServe - February 14th, 2011 [February 14th, 2011]
- Demonstration of Coral Reef Scenario Evaluation Tool (CORSET) by Jessica Melbourne-Thomas of the University of Tasmania - February 14th, 2011 [February 14th, 2011]
- Demonstration of Multi-scale Integrated Models of Ecosystem Services (MIMES) by Roel Boumans and David McNally of AFORDable Futures LLC - February 14th, 2011 [February 14th, 2011]
- Creating Life in the Desert - February 14th, 2011 [February 14th, 2011]