The future of big data: 5 predictions from experts

6 min.

Can you believe that people have created 90% of today’s data in the last couple of years alone, at 2.5 quintillion bytes of data per day? This data comes mainly from the internet, including social media, web searches, text messages, and media files. IoT devices and sensors create a large volume of data as well, being among the key drivers for the growth of the global big data market, which already has reached 42 billion dollars in size.

The world now is powered by big data, but will it be the same in the future? In this article, you will find expert opinions and five predictions on the future of big data.

1. Data volumes will continue to increase

Most big data experts agree that the amount of generated data will continue to grow exponentially. IDC forecasts it will reach 44 zettabytes by 2020. To help you understand how much it is, let’s imagine the amount of digital data represented by the memory in a stack of 128GB Apple iPad devices. It would have stretched two-thirds of the distance from the Earth to the Moon in 2013, and grow to 6.6 stacks by 2020. Thus, the amount of data will double every two years.

What makes experts believe in such a rapid pace of growth? First, the increasing number of internet users doing everything online, from business communications to shopping and social networking. Second, billions of connected devices and embedded systems in the world that create, collect, and share a wealth of data every day.

Enterprises will create and manage 60% of this information in 2025 since they have the opportunity to store and analyze huge volumes of data. However, individual consumers have a significant role to play in data growth as well. IDC estimated that every person will be creating 1.7 MBs of data every minute by 2020.

2. The boom of machine learning will change the landscape

Machine learning, playing a huge role in big data, is another technology expected to boom in the not-too-distant future. Wei Lei, Vice President and General Manager at Intel, says, “Machine learning is becoming more sophisticated with every passing year. And, we are yet to see its full potential—beyond self-driving cars, fraud detection devices, or retail trends analyses.”

Machine learning is a rapidly developing technology. Its global market is growing at a CAGR of 44% from 2017 to 2020 and promises to reach 8.8 billion dollars, driven by the increasing availability of different types of data and technological advancements in the field. Experts believe that computers’ capability to learn from data will improve considerably due to more advanced unsupervised algorithms, deeper personalization, and cognitive services. As a result, there will be machines that are more intelligent and capable to read emotions, drive cars, explore the space, and heal the sick.

What fascinates me is combining big data with machine learning and especially natural language processing, where computers do the analysis by themselves to find things like new disease patterns, to find them in the data.

Bernard Marr

This is fascinating and scary at the same time. On the one hand, intelligent robots promise to make our lives easier. On the other hand, they may overtake humans in the near future, experts believe.

3. Data scientists and CDOs will be in high demand

The positions of data scientist and chief data officer (CDO) are relatively new, but the need for these specialists on the labor market is already high. As data volumes continue to grow, the gap between the need and the availability of data professionals will rise substantially. LinkedIn Workforce Report found a shortage of 150 thousand people with data science skills in the United States in 2018, which is six times more than in 2015.

The intensification of local shortages for data science skills

Metro area July 2015 July 2018 3Y Delta
New York City, NY
+4 132 +34 032 +29 900
San Francisco Bay Area, CA
+10 995 +31 798 +20 803
Los Angeles, CA
+425 +12 251 +11 826
Boston, MA +1 667 +11 276

+9 609

Seattle, WA +1 182 +9 688 +8 506
Chicago, IL -1 826 +5 925 +7 751
Washington, D.C. +735 +7 686 +6 951
Dallas–Fort Worth, TX -2 496 +3 641 +6 137
Atlanta, GA -2 301 +3 350 +5 651
Austin, TX +26 +4 949 +4 923



No wonder, data scientists rank among the top fastest-growing jobs today, along with machine learning engineers and big data engineers. Big data is useless without analysis, and data scientists are those professionals who collect and analyze data with the help of different analytics and reporting tools to turn it into actionable insights. To be a good data scientist, one should have a mix of soft and hard skills, as well as a deep knowledge of various data platforms and tools. Businesses striving to improve their operations and gain a competitive edge are willing to pay much money to such talents, so the future for data scientists looks bright.

The same is about a chief data officer, a C-level executive responsible for data availability, integrity, and security in a company. As more business owners realize the importance of this role, hiring a CDO will become the norm soon, with 90% of large global companies planning to fill this position by 2019.

4. Privacy will remain hot issue

Data security and privacy have always been the pressing issues, which will be snowballing, according to experts. The fast-growing volumes of data create additional challenges in protecting it from intrusions and cyberattacks since the levels of data protection don’t keep in pace with data growth rates.

Statistics will demonstrate the scale of the problem. The Raytheon 2018 Study found that 82% of organizations believe that unsecured IoT devices will cause a data exposure in the next three years, while 80% say this can be catastrophic for their businesses. The report also says that 67% of organizations worry that there will be more cyberattacks and data leakages within the specified period.

There are several reasons behind the data security problem:

  • Security skills gap, caused by a lack of education and training opportunities. This gap is constantly growing and will reach 3.5 million unfilled cybersecurity positions by 2021.
  • Evolution of cyberattacks. The threats used by hackers are evolving and become more complex by the day.
  • Lack of adherence to security standards and best practices. Although the governments are taking measures to standardize data protection regulations, GDPR being the example of that, most organizations still ignore data security standards.

Unless an effective method is found to address that situation, security and privacy will remain hot issues for the big data industry.

5. Fast data and actionable data will come to the forefront

Yet another prediction about the future of big data is related to the rise of what is called “fast data” and “actionable data.”

Some experts say that big data is dead and obsolete, and fast data will soon come to replace it. Unlike big data, typically relying on Hadoop and NoSQL databases to analyze information in the batch mode, fast data allows processing it in real-time streams. Because of stream processing, data is analyzed promptly—any event within just one millisecond. This brings more value to organizations by allowing them to make business decisions and take actions immediately when data arrives.

Actionable data is a missing link between big data and business value. As it was mentioned earlier, big data in itself is worthless without analysis, since it is too complex, multi-structured, and voluminous. Experts say that 99.5% of data is never analyzed, thus providing no valuable insights. Nevertheless, by analyzing data with the help of analytics platforms, organizations make information accurate and standardized—this is what characterizes actionable data. These insights help companies make more informed business decisions and improve their operations.

What the future holds for organizations

Being frightening and fascinating at the same time, the future of big data promises to change the ways businesses in finance, healthcare, manufacturing and other industries operate.

The overwhelming volumes of information may create additional challenges for the future, including data privacy and security risks, shortage of data professionals, and difficulties in data storage and processing

However, most experts agree that big data will bring big value. There will be new job categories and even whole departments responsible for data management in large organizations. New regulatory structures and standards of conduct will emerge as companies continue to use consumers’ personal data. Also, most companies will shift from being data-generating to data-powered organizations making use of actionable data and business insights. To do that, they will need experts in big data consulting capable to harness complex data processing.