The 10 Biggest Big Data News Stories Of 2018

Big Data, Big News

Whether complying with new data protection regulations or cleaning up after a data security breach or data management failure, businesses and organizations everywhere in 2018 continued to wrestle with a host of big data challenges.

Developers of big data and business analytics software, meanwhile, continued to push the technology envelope and develop ever-more sophisticated products to help customers meet those big data challenges.

From industry mergers and acquisitions, to the growing use of artificial intelligence within big data technology, to new data regulations and data security hacks, big data was in the news in 2018. Here are 10 stories that particularly caught our attention this year.

10. Big Data And Business Analytics Market Continues To Grow At A Double-Digit Pace

Big data and business analytics continued to be one of the most dynamic and fastest growing segments of the IT industry in 2018.

The market for big data and business analytics solutions is expected to finish out 2018 with total worldwide revenue of $166 billion, up 11.7 percent from 2017 sales, according to market forecaster IDC.

IDC is predicting that worldwide spending for big data and business analytics technology will reach $260 billion in 2022, an 11.9 percent compound annual growth rate over the 2017-2022 period.

IDC also concluded that the industries making the biggest investments in big data and business analytics solutions through 2022 are banking, discrete manufacturing, process manufacturing, professional services, and federal/central government. Those five industries will account for nearly half ($81 billion) of worldwide big data/business analytics spending this year, growing to $129 billion in 2022. But the retail industry will record the fastest five-year CAGR at 13.5 percent.

9. Big Companies Augment Big Data Capabilities Through Acquisitions

Big data technology, either as standalone software tools or built into broader products, has been hot in recent years. In 2018 there were a number of noteworthy cases where leading IT companies – some big data/business analytics vendors themselves, some not – made key acquisitions to strengthen their technology portfolios.

Data analytics software vendor bought Podium Data, a developer of an enterprise data management platform, in a move to expand its product portfolio into a more complete big data system. Hewlett Packard Enterprise acquired big-data-as-a-service startup BlueData in a bid to inject artificial intelligence and big data management capabilities into its software-defined architecture. And Oracle had its eye on adding to the machine learning functionality of the Oracle Cloud with its purchase of DataScience.com.

Other notable acquisitions in the big data arena included Google's buyout of Cask Data and its tools for building big data analytics services on Hadoop, IBM's acquisition of Armanta and its data aggregation and analytics software for financial service companies, and Salesforce's purchase of marketing intelligence and analytics platform developer Datorama.

8. Big Data Software Developers Offer Self-Service Data Preparation Capabilities

Traditionally, business analytics software and tools used to prepare data for analysis have been separate entities, with the latter almost exclusively used by data management professionals. But there has been a move by developers of big data and business analytics software to offer simpler data preparation tools that business workers can use to collect, prepare and format data on their own for business analytics tasks without waiting for the IT department.

Earlier in 2018 Tableau introduced Tableau Prep for self-service data preparation, making it easier for users to combine, shape and clean data for analysis. In November the company expanded on that technology with Tableau Prep Conductor, add-on software for automating and managing data flows for self-service data preparation at scale. "We see a future where data prep is so easy anyone can do it," said chief product officer Francois Ajenstat at Tableau's annual conference.

In July Qlik acquired Podium Data, a developer of "data catalog" data management software that helps prepare and deliver analytics-ready data to business users. Other business analytics software developers, including Birst, Hitachi Ventana and Sisense, already offer combined data preparation and business analytics capabilities.

7. Snowflake Computing Leads The Rush To Cloud Data Warehousing

Businesses and organizations have been steadily adopting the concept of moving their data warehouse operations to the cloud in recent years. That trend seemed to accelerate in 2018 with Snowflake Computing, the provider of cloud-native data warehousing services, leading the way.

Cloud-based data warehousing is forecast to be a $20 billion market in 2020, up from $14 billion in 2017. While industry behemoths like Amazon Web Services and Oracle are competing in that market, San Mateo, Calif,-based Snowflake Computing has emerged as the cloud data warehouse mindshare leader.

Snowflake, founded in 2012, went live with its cloud data warehouse services in 2015 and it's been on an upward trajectory ever since. While the privately held company doesn't disclose sales, the roster of its 1,000-plus customers includes Netflix, Office Depot, Ebates and Yamaha.

Some of the aura around Snowflake is due to the astounding amount of venture capital the company has raised – including $263 million in January and $450 million in October (for a total of $923 million since its founding) – and a pre-money valuation of $3.5 billion.

6. Amazon’s Move Off Of Oracle Database Not Without Glitches

Amazon now says it plans to complete that transition by the end of 2019.

But the move has not been entirely smooth. Amazon experienced system failures during “Prime Day” in July and a report later concluded that the decision to move from the Oracle database led to the web site outages and delays in shipping packages.

Amazon’s move away from Oracle has also been the source of some trash talking between Oracle and AWS executives. At Oracle OpenWorld in October Oracle CTO Larry Ellison claimed that Amazon’s cloud system could not protect data as well as the Oracle database. But a few weeks later AWS CEO Andy Jassy taunted Ellison in a Tweet noting that Amazon’s consumer business had just turned off its Oracle data warehouse.

5. Marriott Breach Exposes Personal Data On Up To 500 Million Guests

Each year seems to have a major data security breach incident (such as with retailer Target in 2013 and consumer reporting agency Equifax in 2017) in which data from a huge number of consumers are affected.

This year hotel chain Marriott International and its Starwood network was the victim of a big data heist in which personal data on as many as 500 million guests was stolen. The compromised information included sensitive passport numbers and payment card information, along with names, addresses, birthdates, email addresses and phone numbers.

Along with Marriott hotel guests the security breech compromised data of guests who stayed at other Starwood properties including Sheraton, W and Westin hotels.

The incident took on a sinister dimension in December when published reports said U.S. government investigators suspected that Chinese state hackers were behind the Marriott security breach.

4. Businesses Scramble To Comply With GDPR Data Protection Rules

The need to protect data, especially sensitive customer and consumer information, was pushed to the top of the agenda of nearly every business and organization in 2018 by the European Union’s General Data Protection Regulation — requirements that some called the most important change in data privacy regulation in 20 years.

The GDPR, which became effective in May, included requirements for safely handling the transfer of data across borders, the consent of subjects for data processing, data breach notifications and the anonymization of collected data to protect privacy.

While the law was intended to apply to European Union countries, companies that did business in Europe — which is just about everyone — had to comply.

The GDPR was something of a boon for solution providers who provided their clients with technology and services to help them comply with the new requirements.

3. Facebook Faces Blowback Over Data Privacy Failures

Data security lapses are, sadly, nothing new, as evidenced by the 2017 security breach at Equifax that exposed sensitive data on 146 million consumers.

But in 2018 Facebook became, well, the face of corporate America's seeming inability to secure data entrusted to it by millions of users. At times it even seemed that Facebook was more concerned with monetizing all that data rather than protecting it.

Facebook found itself in the midst of a firestorm in March when published reports said that Cambridge Analytica, a London-based political consulting firm, had harvested data from Facebook users and used the information to target them and their friends. Initial reports said data on 50 million users was used, a number later increased to 87 million. News of the incident was swift and angry. Compounding the problem was the slow response from Facebook and founder/CEO Mark Zuckerberg to address the incident.

Facebook's disclosure in September that a code vulnerability in its "View As" feature allowed attackers to potentially take over nearly 50 million user accounts only reinforced the view that Facebook was more concerned with its growth and bottom line than with protecting user data.

2. Big Data Software Developers Jump On The AI Bandwagon

Companies in nearly every segment of the IT industry jumped on the artificial intelligence bandwagon in 2018, finding ways to incorporate AI technology into their products to make them "smarter" and improve their performance.

But the big data segment of the industry did more than embrace AI this year – it was practically redefined by it. Both startups and established developers of business intelligence and data management software leveraged AI and machine learning to take their products to the next level.

Developers of business intelligence software added AI and machine learning technology to their products to boost their predictive analysis and self-service capabilities. With data volumes growing and data speeds accelerating, AI helps sort and process that data to avoid data overload and make insights available in real-time.

SAS, for example, debuted a new release of its Viya business analysis software with new machine learning capabilities for developing more accurate analytical models. Microsoft added AI features to its Power BI software for image recognition and text analysis. And Qlik added machine learning capabilities to the cognitive engine within the November release of its Qlik Sense software to provide what it calls "augmented analytics."

AI technology companies were also on the shopping list of some big data companies. Tableau, for example, acquired AI startup Empirical Systems in a move to expand the statistical analysis capabilities of its software.

AI and machine learning are also seen as a way to help overcome the industry shortage of skilled data analysts and data scientists by automating many data processing and preparation tasks.

1. Cloudera And Hortonworks Unveil Merger Deal

In October rival big data platform companies Cloudera and Hortonworks announced a plan to merge in a deal that will create a new big data industry behemoth with a broad portfolio of data management and related software and a combined equity value of $5.2 billion.

The merger won't be easy, however. The companies have a number of overlapping products and technologies and rationalizing the combined company's product lines alone will be a challenge.

The new company, operating under the Cloudera name, also hopes to realize more than $125 million in annualized savings through sales optimization and R&D. The fact is, despite growing sales, both Cloudera and Hortonworks were racking up significant losses: $114.8 million in the nine months ended Sept. 30 for Hortonworks and $111.0 million for the nine months ended Oct. 31 for Cloudera.

Still, with $720 million in annual sales and more than 2,500 customers, the new Cloudera and its portfolio of data management, data warehousing, IoT and edge computing, machine learning/AI and hybrid cloud products will clearly be a leader in the big data arena.