IDC Digital Universe Study: Issues, Opportunities From The Data Explosion

The amount of data being stored is more than doubling every two years, and could grow by 50 times by 2020, showing the need for new ways to manage data and derive valued from it, according to the fifth annual IDC survey on the state of the digital data universe.

IDC Tuesday released its annual survey, which found that the amount of data created and replicated is expected to top 1.8 zettabytes, or 1.8 billion TBs, in 2011, up from just over 1 zettabyte in 2010.

The findings of the 2011 IDC Digital Universe Study, which has been sponsored by EMC for five years, can be found at EMC's website.

The new survey has serious implications for those who store and manage data.

For instance, IDC said, 75 percent of the information being created today is coming from individuals who write text, take photos, or upload videos and music. Information about such individuals is actually growing faster than data created by them.

However, IDC said, enterprises will at some point be responsible for some part of the liability of 80 percent of that data.

IDC also found that only about half of data that should be secured in some form is actually protected, and that less than one-third of all data has some sort of security protecting it.

The growth of digital data is faster than the growth in storage capacity to store that data, IDC said. However, much of that data, such as digital TV signals, is transient in nature and is not stored.

Chuck Hollis, vice president of global marketing CTO at EMC, said the IDC Digital Universe Study shows how difficult it will be for customers and channel partners to manage the growth of data going forward.

For instance, Hollis said, by 2020 there will likely be ten times as many virtual servers than physical servers deployed, and that enterprise data centers will likely be managing 50 more times data at the end of this decade than they currently manage. During the same time period, Hollis said, IDC and EMC estimate that the number of IT personnel will grow by only 50 percent.

"We'll see 50 times as much data managed by enterprise data centers, and ten times more physical servers than virtual servers, but only 50 percent more IT people," he said. "If you are an IT person, the good news is that you are going to be popular. But the bad news is, you will be very busy."

The IDC Digital Universe Study has several implications for businesses, consumers, and solution providers, Hollis said.

For instance, the use of cloud computing technologies has made it easier for businesses and consumers to more efficiently store and manage data. "But you can also look at the cause and effect of the cloud," he said. "When you make it easier to manage data, people do what? They store more. The cloud is taking the friction out of storage."

The fast growth of data, and the fact that 80 percent of it is touches enterprise data centers at some point, points to the need to increase awareness of security, government risk, and compliance issues, Hollis said.

"Much of this data is landing inside enterprise data centers," he said. "This brings an explicit, and sometimes an implicit, level of responsibility for managing it. We can't say, 'I'm not responsible for it. It's not on my watch.'"

NEXT: How To Handle All That Data

Amidst all the concerns about the explosion in data is good news. Hollis said that IDC is touting the increased ability of companies to get value from that data, especially "big data," or data which scales to multiple petabytes of capacity and is created or collected, is stored, and is collaborative in real time.

"Big Data is different from enterprise storage," Hollis said. "It's built differently, and managed differently. When I asked one big data customers what he wants from his storage vendor, he said a conveyor belt. With big data, you have to look at how to draw value from the data. Big data people know their data, and know the value of the data and what it costs."

For solution providers, the IDC Digital Universe Study offers a guide for areas in which they can develop new skills to meet changing customer requirements, Hollis said.

"Solution providers will be helping customers plan and manage growth beyond just selling more stuff," he said. "If one TB is on the floor today, there will be 50 more there by the end of the decade. The opportunity is there to talk to customers about how to limit growth, and to talk to them about how to derive value from all that data as it comes in."