Monthly Archives: June 2017

Reasons people are migrating to Spark. Image: Databricks

Hadoop and MapReduce, the parallel programming paradigm and API originally behind Hadoop, used to be synonymous. Nowadays when we talk about Hadoop, we mostly talk about an ecosystem of tools built around the common file system layer of HDFS, and programmed via Spark.

Spark is the new Hadoop. One of the defining trends of this time, confirmed by both practitioners in the field and surveys, is the en masse move to Spark for Hadoop users. Spark is itself an ecosystem of sorts, offering options for SQL-based access to data, streaming, and machine learning.

People are migrating to Spark for a number of reasons, including easier programming paradigm. Easier than MapReduce does not necessarily mean easy though, and there are a number of gotchas when programming and deploying Spark applications.

So why are people migrating to Spark? The top reason seems to be performance: 91 percent of 1615 people from over 900 organizations participating in the Databricks Apache Spark Survey 2016 cited this as their reason for using Spark. But there's more. Advanced analytics and ease of programming are almost equally important, cited by 82 percent and 76 percent of respondents.

All industry sources we have spoken to over the last months point to the same direction: programming against Spark's API is easier than using MapReduce, so MapReduce is seen as a legacy API at this point. Vendors will continue to offer support for it as long as there are clients using it, but practically all new development is Spark-based.

Not everyone using Spark has the same responsibilities or skills. Image: Databricks

As Ash Munshi, Pepperdata CEO puts it: "Spark offers a unified framework and SQL access, which means you can do advanced analytics, and that's where the big bucks are. Plus it's easier to program: gives you a nice abstraction layer, so you don't need to worry about all the details you have to manage when working with MapReduce. Programming at a higher level means it's easier for people to understand the down and dirty details and to deploy their apps."

Great. What's the problem then? Munshi points out that the flip side of Spark abstraction, especially when running in Hadoop's YARN environment which does not make it too easy to extract metadata, is that a lot of the execution details are hidden. This means it's hard to pinpoint which lines of code cause something to happen in this complex distributed system, and it's also hard to tune performance.

Having a complex distributed system in which programs are run also means you have be aware of not just your own application's execution and performance, but also of the broader execution environment. Pepperdata calls this the cluster weather problem: the need to know the context in which an application is running. A common issue in cluster deployment for example is inconsistency in run times because of transient workloads.

Pepperdata is not the only one that has taken note. A few months back Alpine Data also pinpointed the same issue, albeit with a slightly different framing. Alpine Data pointed to the fact that Spark is extremely sensitive to how jobs are configured and resourced, requiring data scientists to have a deep understanding of both Spark and the configuration and utilization of the Hadoop cluster being used.

Failure to correctly resource Spark jobs will frequently lead to failures due to out of memory errors, leading to inefficient and time-consuming, trial-and-error resourcing experiments. This requirement significantly limits the utility of Spark, and impacts its utilization beyond deeply skilled data scientists, according to Alpine Data.

This is based on hard-earned experience, as Alpine Data co-founder & CPO Steven Hillion explained. At some point one of Alpine Data's clients was using Alpine Data Science platform (ADSP) to do some very large scale processing on consumer data: billions of rows and thousands of variables. ADSP uses Spark under the hood for data crunching jobs, but the problem was that these jobs would either take forever or break.

The reason was that the tuning of Spark parameters in the cluster was not right. People using ADSP in that case were data scientists, not data engineers. They were proficient in finding the right models to process data and extracting insights out of them, but not necessarily in deploying them at scale.

The result was that data scientists would get on the phone with ADSP engineers to help them diagnose the issues and propose configurations. As this would obviously not scale, Alpine Data came up with the idea of building the logic their engineers applied in this process into ADSP. Alpine Data says it worked, enabling clients to build workflows within days and deploy them within hours without any manual intervention.

So the next step was to bundle this as part of ADSP and start shipping it, which Alpine Labs did in Fall 2016. This was presented in Spark Summit East 2017, and Hillion says the response has been "almost overwhelming. In Boston we had a long line of people coming to ask about this".

Hillion emphasized that their approach is procedural, not based on ML. This may sound strange, considering their ML expertise. Alpine Labs however says this is not a static configuration, but works by determining the correct resourcing and configuration for the Spark job at run-time is based on the size and dimensionality of the input data, the complexity of the Spark job, and the availability of resources on the Hadoop cluster.

"You can think of it as a sort of equation if you will, in a simplistic way, one that expresses how we tune parameters" says Hillion. "Tuning these parameters comes through experience, so in a way we are training the model using our own data. I would not call it machine learning, but then again we are learning something from machines."

Pepperdata now also offers a solution for Spark automation with last week's release of Pepperdata Code Analyzer for Apache Spark (PCAAS), but addressing a different audience with a different strategy. Data scientists make for 23 percent of all Spark users, but data engineers and architects combined make for a total of 63 percent of all Spark users. This is the audience Pepperdata aims at with PCAAS.

Architects are the people who design (big data) systems, and data engineers are the ones who work with data scientists to take their analyses to production. Munshi says PCAAS aims to give them the ability to take running Spark applications, analyze them to see what is going on and then tie that back to specific lines of code.

The thinking there is that by being able to understand more about CPU utilization, garbage collection or I/O related to their applications, engineers and architects should be able to optimize applications. PCAAS boasts the ability to do part of the debugging, by isolating suspicious blocks of code and prompting engineers to look into them.

PCAAS aims to help decipher cluster weather as well, making it possible to understand whether run time inconsistencies should be attributed to a specific application or to the workload at the time of execution. Munshi also points out the fact that YARN heavily uses static scheduling, while using more dynamic approaches could result in better hardware utilization.

Better hardware utilization is clearly a top concern in terms of ROI, but in order to understand how this relates to PCAAS and why Pepperdata claims to be able to overcome YARN's limitations we need to see where PCAAS sits in Pepperdata's product suite. PCAAS is Pepperdata's latest addition to a line of products including the Application Profiler, the Cluster Analyzer, the Capacity Optimizer, and the Policy Enforcer.

The latter three are about collecting telemetry data, while the former two are about intervening in real-time, says Munshi. Pepperdata's overarching ambition is to bridge the gap between Dev and Ops, and Munshi believes that PCAAS is a step in that direction: a tool Ops can give to Devs to self-diagnose issues, resulting in better interaction and more rapid iteration cycles.

Interestingly, Hillion also agrees that there is a clear division between proprietary algorithms for tuning ML jobs and the information that a Spark cluster can provide to inform these algorithms. There are differences as well as similarities in Alpine Labs and Pepperdata offerings though.

To begin with, both offerings are not stand-alone. Spark auto-tuning is part of ADSP, while PCAAS relies on telemetry data provided by other Pepperdata solutions. So if you are only interested in automating parts of your Spark cluster tuning or application profiling, tough luck.

When discussing with Hillion, we pointed out the fact that not everyone interested in Spark auto tuning will necessarily want to subscribe to ADSP in its entirety, so perhaps making this capability available as a stand-alone product would make sense. Hillion alluded that the part of their solution that is about getting Spark cluster metadata from YARN may be open sourced, while the auto-tuning capabilities may be sold separately at some point.

Alpine Labs is worried about giving away too much of their IP, however this concern may be holding them back from commercial success. When facing a similar situation, not every organization reacts in the same way. Case in point: Metamarkets built Druid and then open sourced it. Why? "We built it because we needed it, and we open sourced it because if we had not, something else would have replaced it."

The AI lock-in loop: great investment begets greater results begetting greater investment. Image: Azeem Azhar / Schibsted

In all fairness though, for Metamarkets Druid is just infrastructure, not core business, while for Alpine Labs ADSP is their bread and butter. As for Pepperdata, they are toying with the idea of giving free access to PCAAS for non-production clusters to get a foothold in organizations. The reasoning is tested and true: get engineers to know and love a tool, and the tool will eventually spread and find its way in IT budgets.

Either way, if you are among those who would benefit from having such automation capabilities for your Spark deployment, for the time being you don't have much of a choice. You will have to either pay a premium and commit to a platform, or wait until such capabilities eventually trickle down.

The bigger picture however is clear: automation is finding an increasingly central role in big data. Big data platforms can be the substrate on which automation applications are developed, but it can also work the other way round: automation can help alleviate big data pain points.

Remember the AI lock in the loop? First mover advantage may prove significant here, as sitting on top of million telemetry data points can do wonders for your product. This is exactly the position Pepperdata is in, and it intends to leverage it to apply Deep Learning to add predictive maintenance capabilities as well as monetize it in other ways.

Whether Pepperdata manages to execute on that strategy and how others will respond is another issue, but at this point it looks like a strategy that has more chances of addressing the needs for big data automation services.

Read more:

Spark gets automation: Analyzing code and tuning clusters in production - ZDNet

Tech guru and venture capitalist Marc Andreessen said Tuesday that automation, like the technologyembedded in self-driving cars, will not take away available jobs but will actually createmore.

Its a Luddite fallacy. Its a recurring panic, Andreessen said during a conference hosted by Recode, specifically referring to the oft-professed concernthat automationis bad for the country and the world. This happens every 25 or 50 years people get all amped up about machines are going to take all the jobs and it never happens.

Andreessen used particular historical references to help corroborate his contentions.

People, especially horse and carriage workers, for example, were very worried about the ascension of cars around a century or so ago.

The manufacturing of vehicles not only created more jobs directly, according to Andreessen, but also spurred innovation in other areas, such as paved road construction. This led to the idea of suburbs and consumer establishments like hotels, restaurants and movie theaters.

The jobs that were created by the automobile on the second, third, and fourth order effects were 100 times, 1000 times, the number of jobs that blacksmiths had, he continued. (RELATED: Tech Tycoon: Silicon Valley Is Extremely Liberal, Doesnt Understand Rest of America)

Likely due to the public outcry overcomputerization and its effects on society, Andreessen defended technology. He said the three main sectors that are eating the economy and are currently undergoing a price crisis (construction, healthcare, and education) have been slow to progress technologically. In other words, Andreessen claims that the parts of American society that are struggling the most are the ones that technology hasnt changed or improved.

I think the opportunity and the challenge for the tech industry and Silicon Valley and all of us to go to figure out how to have a much bigger impact in the slow growth sectors of the economy, Andresssen explained.

Andreessen, along with his fellow speaker Reid Hoffman, co-founder of LinkedIn, cited several other potential benefits of driverless cars, including reduced roadway deaths caused by human error, less traffic congestion, and more leisure time during transit. (RELATED: Distracted Driving Is A Huge Problem, And Autonomous Cars Could Help)

However,the National Bureau of Economic Researchpublisheda study in March that found industrial robots had a significantly negative impact on U.S. employment and wages for many local labor markets between 1990 and 2007.

Ryan Hagemann, director of technology policy at the think tank, the Niskanen Center, said he finds this relatively surprising, while also adding that the analysis doesnt appear to factor inall of the potential effectsof advanced technology.

Were more likely to see humans working withand not competing against robots in many of the industry jobs imperiled by automation, Hagemman explained to The Daily Caller News Foundation. Because the authors model treats the labor market as one of competition between human labor and automated labor, it doesnt seem to account for potential productivity gains through cooperation between the two.(RELATED: Google Exec: I Am A Job Elimination Denier When It Comes To Robots)

Others, like Hagemann, argue the tangible and intangible benefits of automation, robotics, and advanced technology in general, may not transpire right away or be as easily perceptive.

The endless search for new and better ways of doing things drives human learning and, ultimately, prosperity in every senseeconomic, social, and cultural, Adam Thierer, senior research fellow at the Mercatus Center, wrote in his book Permissionless Innovation.The pessimistic critics of technological progress and permissionless innovation have many laments, but they typically fail to consult the historical record to determine how much better off we are than our ancestors.

Follow Eric on Twitter

Send tips to [emailprotected].

Content created by The Daily Caller News Foundation is available without charge to any eligible news publisher that can provide a large audience. For licensing opportunities of our original content, please contact [emailprotected].

More:

Andreessen: Automation, Driverless Car Will Create More Jobs | The ... - The Daily Caller

In January I wrote about three major shifts on the horizon for accountantscollaboration, automation, and triple-entry accounting. Recently, automation in particularhas created a lot of industry chatter, and I would like to explain what exactly the impact will be on your firm ... specifically, your workflow.

For the record, in our space we have seen:

So, lets explore automation more deeplyand look at what it means to you, your practice, and your workflow.

The Age of Automation

Today, automation rests in the hands of the major online accounting software providers. With their access to enormous amounts of data from more than 3 million online subscribers (2.2 million from Intuit QuickBooks Online and 1 million from Xero), they are aggregating their customer data to build algorithms to auto-categorize, auto-configure, auto-alert, and auto-complete many of the tasks within their products.

This is changing your day-to-day by simplifying data entry and tasks. Specifically, client services, such as tax, bookkeeping, and payroll, are most impacted, and soonmany of these underlying data-entry tasks in these client services will be eliminated altogether.

Extending the Ecosystem

A constantly growing ecosystem of products surrounding the online general ledgers is making it easier to extend the workflows for your accounting system. Working with specific vertical industries or within your practice is becoming more seamless.

In some cases, additional workflows are being embedded inside general ledger software, such as the extension of QuickBooks Online with time tracking (viaTSheets) and AR/AP management (viaBill.com). In other cases, applications and their subsequent workflows are being stitched together by savvy practitioners using programs like Zapier and Workato.

In the medium-term future, you will be able to use broader APIs, notification proliferation, and easy-to-configure connectors to create cross-application custom workflows. These sophisticated workflow objects will manage, coordinate, and complete a set of predetermined tasks without you needing to lift a finger.

Your Workflow, Your Way

With the advent of online practice management applications and lightweight offerings like QuickBooks Online Accountant Practice Management, many firms are already documenting, deploying, following, and completing the necessary workflows that support their practice and complete the work for their clients.

An essential starting point for all practices is to identify key processes to document and standardize across the team. A central place that outlines how work is done will ensure quality, consistency, and efficient delivery. To fully take advantage of workflow automation, this must be done today.

Automation by the large players like Intuit, Xero, and Sagewill make your client work and other workflows easier by auto-completing or eliminating steps through learned, repeatable behavior. The reach of your existing workflows will be expanded with greater insights, real-time information, proactive vs. reactive actions, and more control over what can be done across other industry and small business applications. This is all evolving right now.

But what practice automation can you leverage today? Within your accounting application like Xero, you can map your clientschart of accounts to one common ledger to automate your report production via a standardized set of reporting templates. Intuit's recent release of auto-categorization will auto-recognize the vendor and present recommended categorization for the transaction based on similar crowdsourced behavior.

As for your small business clients, TSheets and its GPS functionality can track when and where people clock in and out. Hubdoc can automate the process of fetching documents and parsing bank statements and utility bills. Or vehicle mileage can be auto-captured for clients withExpensify's mileage tracking feature.

The list of apps, activities, and automation is growing every day. It is a matter of learning what your core apps are capable of doing and leveraging the technology to simplify the workflow of your firm.

We are already experiencing the effects of automation in the accounting industry, and this is only going to shift further and more rapidly. The challenge for you and your practice is to embrace this now, get a jump on your competitors, and make technology do as much heavy lifting as it is capable of.

Read the rest here:

The Age of Automation and the Impact on Your Workflow - Accountingweb.com

Automation In The Oil Industry: What's Next For One Of The Big Players Forbes Automation is everywhere; even industries where some may least expect it, such as oil and gas. According to Craig Clark, VP of Finance at National Oilwell Varco (IntelliServ), the last two years for the oil and gas industry have been difficult and as a ...

Read more here:

Automation In The Oil Industry: What's Next For One Of The Big Players - Forbes