Big Data Sucks...

No, I’m not saying that big data is good for nothing. Nor am I referring to the privacy issues it creates, although they are very real and dangerous. What I’m saying is that the current mythology of big data is sucking the life blood out of the heart of real, meaningful business computing.

Both the business world and the computing industry have been chasing down a marketing rabbit hole. The story goes that if only businesses knew every last detail about the lives and loves, the outer behaviors and inner landscapes of every potential customer—scratch that, every potential person (take a look at “Bump Tracker: Nine Months of Big Data”)—their marketing machines and customer retention teams would achieve world domination! Ever increasing quantities of data are harvested, often surreptitiously, in pursuit of this goal. The problem is that the IT budget, ever limited, is being diverted into this boundless data junkyard to the detriment of real business computing and actual innovation. “Bump Tracker” doesn’t just focus on the big data marketing madness that accompanies today’s pregnancy. Author, Nathalia Holt, also reveals the medical advances enabled by genomic big data, as well as their ethical implications and life-changing consequences. Where is the real innovation?

But, let’s get back to real business computing. Traditionally, business computing has focused on capturing, processing and storing business transactions. But, what are business transactions? Fundamentally, business is a social mechanism to exchange goods and money in a regulated and (hopefully) equitable manner. Business transactions are the legally binding agreements between the parties involved reflecting the physical—and increasingly digital—reality of what has been exchanged, who was involved, when it occurred, and so on. Until the middle of the last century, these transactions were simple enough and few enough to be tracked manually in paper ledgers. The introduction of computers, most notably the IBM 360 mainframe, enabled the automation of business transactions. It also kick-started the process of increasing their complexity and number to the stage that, today, it would be impossible to run a business of any size without computer automation. Furthermore, many business transactions now exist only in the applications and databases of the computer world.

Over the past decade or two, the scope and scale of business has exploded dramatically. Mobile consumers expect instant gratification, and so the rhythm of business has moved to 24/7, real-time. Marshall McLuhan’s global village has become real, and so business is totally dispersed geographically, spanning time zones with alacrity. The physical computing infrastructure, from processors to networks and everything in between, has become so inexpensive that businesses have come to believe they can store and manipulate every byte of data over every conceivable time period for any imaginable use.

It is this breathtaking vision that has given rise to the myth of big data. The mistaken belief is that we can mine insight from a slagheap of data, given sufficient commodity hardware and a simple framework to write and execute parallel programs. What seems to have been forgotten is that the lifeblood of business lies not in these tailings but in the precious ore of business transactions carefully crafted in complex applications and comprehensive databases. It is to these business transactions and the databases they demand that I want to turn my attention next.

Add new comment