Erin McCarthy, Consultative Analyst, Customer Intelligence,
Aug 18, 2012
(Continued from Part 1) Every human on the planet will feel some effect of big data in the coming quarter-century.
Where Does Big Data Come From?
- Estimates range from 150 – 300 billion per day. That equates to 2.8 million per second and about 90 trillion per year.
- 901 million Active Users (3/12). Estimated to be 1 billion by year’s end.
- Average Active User has 90 pieces of content per month.
- 300 million+ photos uploaded to Facebook per day (3/12).
- An average of 3.2 billion Likes and Comments generated per day (Q1 2012).
- 50% of Active Users log on to Facebook in any given day.
- Average Active User has 137 friends.
- People spend over 740 billion minutes per month on Facebook.
- 140 Million Active Users (5/12).
- 400 Million Tweets per Day (6/12).
- 1 Million transactions per hour.
- Walmart’s database is estimated to be 2.5+ petabytes.
- Google processes about 24 petabytes of data per day.
- BBC's iPlayer uses 7 petabytes of bandwidth each month.
- Imgur transfers about 4 petabytes of data per month.
- Netflix uses 1 petabyte to store the videos for streaming.
- AT&T transfers about 30 petabytes of data through its networks each day.
- The German Climate Computing Centre (DKRZ) has storage for 60 petabytes of climate data.
- The Internet Archive contains about 5.8 petabytes of data as of December 2010.
- World of Warcraft uses 1.3 petabytes of storage to maintain its game.
- Steam (a digital gaming service developed by Valve) delivers 30 petabytes of content monthly.
- Avatar is reported to have taken over 1 petabyte of storage to render the 3D CGI effects.
- IBM was reported to have built the largest storage array ever, with a capacity of 120 petabytes.
- CERN amassed 200 petabytes of data from 800 trillion collisions looking for the Higgs boson.
- Cray has begun construction of the Blue Waters Supercomputer, which will have a capacity of 500 petabytes, making it the largest storage array ever if realized.
Where Is It Going?
The Great Data Rush of 2013?
Almost everyone sees the potential in Big Data and is rushing to figure out ways to leverage technology to harness the power of Big Data. There will likely be significant lag times between making the investment in developing a Big Data enterprise and the full realization of ROI for that investment. Much of the upfront investment will take a leap of faith. But, like the lure of going west for land and gold, the potential rewards of the new data frontier are great. IBM says information is "becoming the petroleum of the 21st century," fueling business decisions across a variety of industries moving forward.
Why Do We Need Big Data?
Big Data is a train barreling down the tacks; it is unstoppable. The question is not if you will leverage Big Data, it is when and to what extent you choose to do so. One of the first challenges is deciding what to capture and why to capture it. Do we capture it simply because we can or do we capture it to fill a need? There will be a gap between what is possible to capture and store versus what is practical to actually put to use. Data in and of itself is useless; even if it is captured and stored, you have accomplished nothing of value until you actually DO something with the data. It has to be analyzed and turned into insight and knowledge. Then those insights and knowledge have to be communicated and turned into strategies. And finally, the strategies have to be executed, measured, analyzed, and improved upon.
How Do We Get There?
This is perhaps the biggest challenge of the coming quarter-century. Almost everyone agrees that harnessing big data will take top-down support and empowerment. At the core of creating a big data culture will be embracing and fostering an environment of testing, learning and experimentation. It will require an investment in people, technology and tools. Advancements are happening at breakneck speed in areas such as data visualization – one of the key ways to recognize patterns and turn seemingly overwhelming amounts of data into insights and knowledge. But at the end of the day, the data and the tools are only as good as your people. The crux of success will be finding, hiring, training, and supporting a new generation of ultra-engaged, imaginative, and creative data and business intelligence analysts who understand how to leverage the data to achieve results. These analysts will be the pioneers of the Age of Data.
As technology continues to advance, we are limited only by our imagination of what we capture, how we turn it into insights, and how we use it to improve our lives.