Big data governance: roles, frameworks, and a case study

Big data governance Q&A

A big data governance plan should be tailored to business needs and industry laws while taking into account essential characteristics and requirements of big data processing.

Big data governance vs data governance: what’s the difference?

Actually, there shouldn’t be any difference—the same principles should apply to both. Big data governance is very similar to traditional data governance in terms of challenges and principles. However, the former has to take into account a number of peculiar big data characteristics:

• The volume of big data can reach petabytes and more
• Big data can be structured, semi-structured, or unstructured
• Big data repositories span files, NoSQL databases, data lakes, and streams 
• Data is extracted from internal and external sources, including connected devices
• Data is processed in real time

As traditional data governance tools on the market today can’t support big data processing requirements, it’s an imperative for businesses to rethink their data analytics strategy and for existing technologies to mature and evolve to meet new challenges.

Who is responsible for data governance?

Data governance crosses departmental borders and influences each department’s bottom line, either in good or bad ways. It requires collaboration across the entire enterprise and clearly defined roles and data ownership levels. When everybody knows who has data-related authority and responsibilities, it becomes possible to avoid chaos and mistakes and allows data governance socialization.

Depending on the organizational size and goals, the following roles may be needed for a powerful big data team:

• A data governance committee consists of top managers who are responsible for data strategy creation or approval, prioritization of projects, and authorization of data policies and standards.
• Chief data officers (CDOs) participate in data strategy development, oversee data framework implementation, and use data as a strategic asset. They create data standards, policies, and practices, and grow corporate data cultures.
• Big data architects are proficient in relevant technologies and understand relationships between them. They are responsible for designing big data processing solutions that are able to solve any data-related problem. They can be deeply engaged in data governance, automation and security.
• Data engineers set up systems for accumulating, cleaning, and organizing data from multiple sources, and transferring it to data warehouses.
• Data scientists/data analysts are responsible for analyzing large sets of structured and unstructured data, creating algorithms and predictive models, and extracting trends and insights relevant to the business.
• Data owners are team members who use data and are accountable for data assets in terms of quality and security within their teams. 
• Data stewards closely cooperate with data owners and oversee how the latter execute the data strategy and whether they follow data policies and standards. They also participate in training new data owners. As big data is often collected but not used due to the absence of professionals, currently there’s a significantly higher demand for data stewards and related expertise.
• Data users are team members who interact with data to perform their daily activities. They enter data, access different datasets, and generate reports.
• IT teams are responsible for technology implementation and customization, development of additional features for big data processing, audit, security, and maintenance.

It’s clear that big data technologies are able to replace some of the above-mentioned roles, like architects and data scientists, or some of the roles can merge and combine responsibilities. However, it’s important to involve the required professionals along the technology implementation.

What are data governance levels?

It’s important to understand that not all data can be governed in the same way, particularly when we deal with big data. There can be three levels of governance: 

• Strictly governed data is already vetted, rationalized, organized, and optimized for performance.
• Loosely governed data can be of two types. It can be data used by data scientists to run experiments, make approximations, and search for trends and patterns. Or it can be data that doesn’t require preparation or needs a minimal amount of rationalization, like key IDs. 
• Non-governed data is raw data in its purest form, with no additional keys. Such data can be used for ‘schema on read’ analysis—data is stored in an unorganized and unstructured format and gets organized for particular purposes. 

It’s important to monitor data across all governance levels—data is constantly changing and can require a shift to another governance level.

Itransition’s big data governance project

For one of our data analytics projects, Itransition partnered with a US-based multinational company that provided advanced pharmaceutical analytics and technologies. The customer accumulated over 500 million patient records of more than 50 thousand patents, not to mention petabytes of proprietary data. However, their legacy system limited their abilities to get more value out of this growing data, so they approached us to help them create a business intelligence project plan, migrate to the cloud, and improve data management capabilities. 

We redeveloped a BI platform

The customer’s data analytics platform comprised a toolset for generating reports based on multiple structured and unstructured data sources. The system couldn’t support the company’s needs and adapt to the changing market, so it required dramatic reengineering and optimization in terms of UI, data processing, and report generation. Itransition developed a new BI platform on the ASP.NET MVC framework, with Microsoft SQL Server as a database engine, where we delivered the redeveloped functionality, flexibility, and scalability. It resulted in 3-5x faster SQL querying and reduced RAM and CPU usage.

We delivered a data management and data visualization app

The legacy platform didn’t support multiple data source formats and had an outdated ETL configuration, which slowed down data processing. As a result, data processing could take days, with some sources excluded from processing. Additionally, non-tech users couldn’t participate in ETL processes and needed a user-friendly interface to interact with data.

We developed a data management app and integrated it with several database engines (Oracle, Microsoft SQL) and Apache Hadoop to enable distributed storage and processing of large datasets. It enabled 10x faster data processing and less memory and space usage. The app also became accessible for non-tech users, who could visualize data and get reports within minutes. The system was able to process various data sources, transform data, and prepare different output forms, be it databases or files. This way, users were able to deliver prepared data to other destinations, like cloud storages, FTP servers, or other teams.

We migrated to the cloud

To maintain a high system performance when the number of users was constantly growing, we initiated the system migration from the on-premises server to the cloud. Our DevOps specialists audited the existing infrastructure and prepared a migration roadmap. We designed a scalable and secure cloud infrastructure and deployed it to AWS. As a result, the customer got a virtual private cloud with private and public subnets, defined network gateways, and fine-tuned security settings.

To ensure security of large volumes of sensitive data, we used Amazon S3. Critical data was backed up via AWS tools. We utilized Amazon RDS to create and save automated backups of database instances. For enhanced security, we used AWS services to store passwords and license codes as encrypted parameters and enabled secure configuration of managed instances and password reset. 

We delivered excellent results for the long run

Our solution is now used by many leading pharmaceutical corporations, enabling them to handle data of multiple formats from various sources and manage their data assets efficiently and securely with big data governance tools.