Forget “Big Data” – What about “Small Data”?

Posted by Ross Coundon on September 24, 2014

No doubt you’ve seen the internet meme: “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…” And to be fair, there’s an element of truth in that. There’s massive hype on the subject with so many internet blogs, websites, companies, products and seminars on the subject. Googling Big Data returns “about 46,100,000 results” (Does anyone even look beyond page 1?) However, I’d wager that there aren’t too many companies that are embarking on Big Data projects and those that are are bleeding edge and experiencing all the frustrations that come with that territory.

There are certain industries that “do” Big Data and do it well but for many it’s an enticing prospect but difficult to quantify the benefits, or even the objectives. Big Data used to be a somewhat colloquial term to refer to datasets that were so large that the traditional database technologies were no longer sufficient to be able to process or make sense of them. Now, it’s the name of an industry and an awful lot of marketecture. Various different technologies have sprung up over the last few years, some of which are beginning to mature and are seeing some adoption but I think for most, there are more fundamental areas to address before Big Data is even a consideration.

The story goes like this – we’re all generating so much data with our smartphones, our computers, our tablets, our wearables that in order to make sense of it all companies need to adopt new, specialist technology to store and process it. This will give competitive advantage because you will have access to so much data you can very quickly create reports, models and hypotheses about your business and your customers, in real-time.

This sounds fantastic, just imagine the power; you’re Tom Cruise in The Minority Report waving at a 3D projected pie chart and making operational decisions on the fly and trouncing the competition. You’re dreaming up complex and prescient models on-the-fly – you’re part data scientist part operations director. Except, I just don’t see this happening any time soon. So many companies we work with are wrestling with the fundamentals of “Small Data”. They’re looking to understand the data from their mobile workforces, trying to answer questions such as, “How can I find out where my mobile workers are right now?”, “How do I make economical appointments that still delight my customers?”, “How do I react to emergencies in cost effective way?”, “How can I improve my customer service?”, “How can I get the right information to my workforce at the right time?”, “How can I reduce the cost to serve whilst maintaining or even improving my reputation?”

The answers to these questions are not, and never will be, absolute and will differ based on the business and the current situation. This isn’t Big Data, this is just data, and the panacea for so many businesses we work with is to turn this data into information and make powerful decisions based on it. Granted, there are more streams of data now than at any time in the past, but it’s not what the industry terms “Big”, it’s just data. Determine how to answer these questions and you will get 95% of the way to massively improving your operation. Only once you have that in hand can Big Data offer you additional insight, but don’t major in the minors, fix the Small 95%.