In its global forecast for business intelligence industry up to 2021, MarketsandMarkets cited data-driven decision making as one of the four key drivers of the BI market growth. It’s interesting then that executives still get caught in the dilemma of using data analysis versus their gut feel to arrive at strategic decisions.
This article argues that it shouldn’t be a choice of one over the other. As our BI consulting experience shows, a combination of both gut feel and hard evidence is what can help managers make effective decisions, bring them strategic advantages, and position their companies at the forefront of competition. It also makes case for data democratization as an evolutional step of data science that has made data-driven decisions possible.
Data-driven decision making and gut feel often come up in a dichotomy as two extremes, with the pendulum of managers’ preferences swinging toward the later. The BI Survey research showed that 58% of the surveyed companies based half or more of their decisions on gut feel or experience. Most likely, this habit of using one’s intuition remains the key tool of decision-making simply because it had been there for centuries before computer-based data analysis came to challenge it.
Gut feel, or interoception as it’s known in academic circles, is the ability to sense bodily signals such as heart rate, pain, etc. Traditionally, this ability is connected to decision-making—the higher the interoceptive sensitivity, the greater the ability to arrive at better decisions.
In an interesting experiment, the team of scientists studied 18 traders’ ability to correctly count their heartbeat and correlated the results with their trading floor performance. The findings made it clear: those who were more in tune with their bodies and guessed the rate more accurately, were more successful at making risky decisions while also showing higher profitability than those with lower accuracy scores.
However, relying on one’s intuition also has downsides. Based on human perception, gut feel is prone to cognitive biases that may distort and undermine decision making.
Among them, there are the following common biases:
Confirmation bias—looking for information to confirm pre-existing beliefs and assumptions and discarding evidence that contradicts the decision maker’s opinion.
Anchoring—focusing intensely on one particular piece of information (usually the very first that comes across) and overlooking others.
Attentional bias—inability to take into account the entire range of impactful factors by randomly prioritizing some of them and ignoring the others.
Optimism bias—overestimating positive outcomes and underestimating the probability of negative ones.
In his Forbes op-ed, Bernard Marr also warns against the so-called HiPPO effect, which stands for over-relying on the highest paid person’s opinion in making decisions. This effect is mostly found in large companies with the typical top-down decision-making culture and, according to Marr, is among the key barriers to data-driven decision making that’s supported with evidence, not a single person’s beliefs.
Sometimes, HiPPO can wreck a business when not confronted with data analysis findings. In this infamous example, IBM went for selling 50% of their ROLM unit to Siemens despite the research, commissioned but abandoned by IBM’s executives, firmly showed that it would be a clear failure for the entire unit. Five years later, IBM was still dragging their irreversible financial losses resulted from that decision.
So should decision makers run with data or gut? The short answer is, with both.
At best, gut-based decisions should always be backed with data. In fact, both approaches share a similar mechanism by deriving an insight based on the past. While data-driven decision-making is impossible without a slew of past data at hand, our gut feel is associated with the outcomes of similar situations in the past, suggesting us the way toward a better result.
As put by Paolo Gaudiano, intuition becomes crucial when it comes to making sense of complex or conflicting outputs and navigating the sea of data insights. Experience should come into play as well. Overlooking managers’ expertise in favor of machine-generated recommendations would be a mishap since they can identify inconsistencies and unusual trends in data analysis outcomes.
Summing it up, effective decision-making is only possible with data in the picture, although this shouldn’t devalue managerial gut feel and experience. In this regard, data-driven decisions should rather be data-informed in case data is used as a starting point of decision-making, or data-verified when hard facts are used to reinforce or reject an assumption.
The numbers are telling: the BI Survey study cited above shows that 60% of top-performing companies rely on data when making the majority of their business decisions. This correlation is further reinforced in the 2018 Data & Analytics Global Executive Study and Research Report by MIT Sloan Management Review. Here, 59% of managers attributed their competitive advantage to the use of data analytics, up from 51% in 2015.
Data analysis and decision-making is a match made in heaven if you consider multiple strategic advantages that come from using them in tandem:
1. Growing business by detecting new opportunities and acting on them, for example, by entering new markets or launching new products or services based on identified niches.
2. Achieving cost-efficiency by decreasing expenses and optimizing business processes, like deciding on prolonging a contract with a vendor or cutting off poorly performing SKUs and replacing them with items that are likely to be in a higher demand.
3. Boosting customer engagement through a stronger focus on customer experience and journey. This is made possible with the use of multiple data sources to build customers’ detailed profiles and then acting on insights to decide on the ways and channels of personalizing service.
4. Improving a product or service based on customer feedback and sentiment analysis across channels to identify areas for improvements.
5. Refining a marketing strategy by identifying best-performing channels and cutting off/adding up advertising budgets, if necessary.
6. Taking HR management to a whole new level through automating employee surveys and candidate selection algorithms to create a healthier internal culture and make effective hiring decisions.
Until recently, pairing data analysis and decision making was a tough task that only those with advanced data science skills could accomplish. Now that two essential factors—proliferation of data sources and the adequate data processing technology—fall into place, we can speak of data democratization that puts data-driven decision making on every executive’s agenda.
Essentially, data democratization boils down to having the right tools to mine, visualize, and analyze data—all with minimum adoption barriers for non-tech users. Following the decades of data analysis confined to specialized IT departments, this is a breakthrough indeed that eliminates gatekeepers to valuable insights. According to Chad Bocklus, president and chief product officer for CarStory, analytics is what brings democratization to decision making by opening data to employees and making it transparent across an organization.
While opening up data to employees is largely a matter of corporate policy, minimizing barriers to embracing data analytics is more about technological innovations. From simple data visualization tools like Excel charts to self-service BI applications with machine learning at their core, data analysis software has come a long way and now can serve up information fully ready for digestion.
Advances in data virtualization, master data management, and cloud computing are all responsible for this paradigm shift. What was previously regarded as barriers to data-driven decision-making—data quality, fragmentation by departmental silos, integration of multiple sources, the complexity of dealing with unstructured and semi-structured data—now are giving way. Technopedia even metaphorically compares this evolution to the age of literacy when common people finally gained access to the Bible and more books, which eventually led to dramatic societal changes. Though data democratization is unlikely to bring such a tremendous impact, organizations will still feel incremental effects to their decision-making culture resulting from bringing data within everyone’s reach.
Decision-making is one of the most critical business functions delegated to managers. With the upsurge of data available to businesses today, the question of whether to use data for this purpose or resort to the old good intuition is becoming more pressing.
The answer is, one shouldn’t exclude the other. Although subject to cognitive biases, gut feel can still serve as a double-checker on analytical findings based on a manager’s experience and awareness of the context. At the same time, advanced data analysis, previously reserved to scientists and now available to the common public thanks to data democratization, allows decision-makers to mine petabytes of data for valuable insights and make sense of the outputs through handy visualizations.
When combined, both approaches are likely to bring in competitive advantages by helping managers to refine and validate their strategies—in product management, marketing, human resources, customer experience, and more domains.
Technology has arrived, now it’s up to business decision-makers to embrace this cultural shift for competitive gains.