The Value of Business Analytics: Identifying the Path to Profitability

MODERN-DAY MAGICIANS

Today, we face the opposite of Gosset’s challenge. While we still sometimes have to deal with fewer observations than we would like, the growing challenge is working out how to deal with the massive amounts of information we do have. As Moore’s law suggests, we have seen the number of transistors we can fit inexpensively on an integrated circuit double roughly every two years. Although it is not an exact relationship, we have seen roughly exponential growth in processing power since the early 1950s.

Importantly, processing power is not the only area that has grown by leaps and bounds. Kryder’s law suggests that disk storage density doubles annually, a pattern that has largely held true since the mid-1990s, when it was first suggested. And Butters’s law suggests that the amount of data carried by a single optical fiber doubles every nine months.

Combined, these create our future. We have the storage capacity to track ever-increasing amounts of information, often referred to as “big data.” We have the bandwidth to support the transfer of this information as needed. And, we have the processing power to extract insight from this data.

Because of this, we are drowning in data. For the first time ever, we have more data than we have storage capacity. In 2008, International Data Corporation (IDC) estimated that the amount of data being generated exceeded our total aggregate storage. Organiza­tions such as eBay have repositories in the petabytes, and there are no signs ...