This might hurt, but cast your mind back to 2013 and the victory of Oracle Team USA over Team New Zealand. We were so close, and had it not been for a few unlucky wind shifts, that trophy would now be tucked safely inside a cabinet in the Westhaven Marina. But there was more than just luck in this, right? We also know that money bought success for Oracle. What did that money actually buy? Data.
Oracle installed more than 300 sensors gathering intelligence on over 3000 variables at around 10 times per second on their boat. These feeds were transmitting to an on-board server. The performance chase boat contained a crew of four analytic experts, running real-time assessments of these multiple variables through time-based predictive algorithms to predict what the critical seconds ahead would look like and what might be done to take best advantage. On their wrist, each member of the sailing crew wore a Personal Digital Assistant that received real-time customised feeds from analytics designed to improve their performance.
It may not be sporting but it certainly worked. Data – big and small – has now become a major source of unsporting advantage. Companies are using it to predict consumer behaviour, reduce fraud, respond to weather patterns and solve major health problems (see sidebar). A major study of global firms showed that 80% claim Big Data has improved their decision-making.
“Statisticians are sexy” declares Hal Varian, Google’s lead statistician. But then he would say that. Big Data does run the risk of becoming just another cliché. How do we get past the usual suspects and the buzzwords?
First, we need to remove the hype. Data isn’t new. Business has always generated information, whether it’s cashflows or stock inventories. We “manage only what we can measure”, as the old saying goes.
That said, we need to acknowledge that something has changed. The volume of data collected is now increasing at an astounding rate. A staggering 90 per cent of the data in the world was created in the last two years, thanks to our online world. For example, your own personal outputs alone might consist of: your geo-locational information from your cell phone; the personal networks you reveal through calls you logged; your driver profile through satellite updates from your in-vehicle monitoring system; your consumer profile through financial information from bills you pay; supermarket data from your shopping; records from the petrol you bought or the purchases you made at cafés and bars through your day; your public heath profile from files relating to a doctor’s visit; your emails; your online statistics from websites visited, chats had, shows watched, tweets tweeted and music listened to; and on and on it goes.
In addition to the volume, the processing power of computers means data can be made meaningful. It’s now possible to do so many things that previously could not be done: spot business trends, know our customers better than they know themselves, prevent disease outbreaks, combat crime by targeting hotspots before an offence happens, map our ecological systems, develop bio-profiles and much more.
Managed well, data can be used to unlock new sources of economic and social value, to provide fresh insights for business and science, and to hold those in charge of resources to account. But this also comes with serious challenges and risks that must be understood and addressed.
As businesses, what do we do with all this information? How do we get smarter with data? A good place to start is with the three Vs: veracity (or accuracy), validity and value.
The first step in data analysis is ‘data conditioning’: getting records into a state where they are usable and can be trusted. In a business, without a quality-management strategy, data is usually messy, inconsistent and problematic. A clean-up can be major. There are tools to work with but human labour and decision-making is often required. Frequently there will be challenges: if data is missing, do you ignore the missing points? If data is incongruous, do you decide that the data is incorrect or that the incongruous data is telling its own story? The reality is that when dealing with historical information, you have to decide how to work with what you have, and develop a robust strategy for moving forward.
To ensure future improvement you need to establish consistent protocols for data management and set up the right systems for itscuration from now on. You can increase trust as the level of reliability and quality increases.
“All data – big or small – is subject to one overriding and inescapable source of bias: the beliefs and expectations of the analyst,” says data guru Nigel Hollis. We often fail to understand the patterns in information because we have not identified the problem we’re trying to solve. Data may be automated but the assumptions guiding the models used to interrogate the facts are human. Without proper theory to guide our analysis we can miss the point. For example, predictive algorithms are well-known tools of analysis and the changes they have already brought. Algorithms that predict stock prices have fundamentally changed the pace of Wall Street trading. Large organisations like Amazon, Wal-Mart and Linked In surprise us by ‘knowing’ what we are interested in or are connected to. But, as Nate Silver, author of The Signal and the Noise identifies, prediction in the Big Data era is not going so well. Despite having all the information to predict the event, America completely missed 9/11. Likewise, everyone failed to predict the global financial crisis.
The problem with many business data sets is that they appear as numbers on a table. Presenting information visually – in a dashboard or graph – is a key step on the analytic pathway where patterns become clearer and insights more available. For wider stakeholders to understand what the numbers mean they need to hear the stories the numbers are telling; good visual representation can greatly assist comprehension.
To achieve real value from data, insights must be actionable and embedded in the processes of the business. Best outcomes occur when the view is holistic and analytics resources are focused across areas that deliver the greatest value: marketing, finance, operations, supply chain management, risk, HR, and customer insights.
Does it work?
That all sounds like great management practice. But the question is, does make any difference? Yes. Two studies by the McKinsey Global Institute and Gartner have found that, first, big data can generate value independent of the sector you operate in and, second, that companies using unstructured data and external source data were predicting outsized return on investments (see the graphs).
The companies using big data successfully had a certain set of characteristics such as executive sponsorship and being “internet-centric”. Moreover these companies projected a return on investment on big data nearly three times that of the less internet-centric companies. Furthermore, it’s the depth of the behavioural data that internet-centric companies gather that seems to give them the proprietary insights that have led to effective development of new products and services, as evidenced in studies on Procter & Gamble and Netflix.
If fancy McKinsey studies don’t convince you then consider the success of local company WhereScape, a $20 million data warehouse and analysis company with more than 300 clients including Air New Zealand, Sky City, Nike, and Vodafone. The company’s so successful it’s considering a listing on the NZX.
Chief executive Michael Whitehead is a treasure of data business stories. One client in the United States used data to identify hospital purchasing inefficiencies, thereby giving the hospitals much greater bargaining power. An automotive company which checked on its overall international sales data found it was losing business on spare parts to the black market.
Whitehead says there is “low-hanging data fruit” that can improve a company’s bottom line. For example, a lot of companies can easily tell who their best customers are by revenue, but haven’t worked out their best customers by profit, which is a bit trickier to do. “I see a lot of companies who just don’t do that – they think revenue is a surrogate for value and the data often shows otherwise.”
Big data? Small data? There’s lots of data in your own company. The question is, are you getting smart with it? ×