Cloud business intelligence system for vehicle manufacturers

Customer

The customer supplies BI for the automotive industry all over the world. With decades of expertise in creating automotive BI solutions based on primary and secondary research datasets, the customer helps vehicle manufacturers plan and develop vehicles and related products, as well as launch and position them on local markets.

The company also offers research services for retailers and provides data feed tool kits for credit companies and banks, vehicle leasing organizations, and other data consumers. The customer’s solutions are also widely used in dealer and fleet management systems, dealer websites, showroom tools, and car locators.

Solution

First thing, Itransition recommended switching to a new microservices-based architecture to build the new BI portal, migrate all the data, and develop data validation and visualization tools. We also supported the customer’s idea to create an R&D center to develop cloud business intelligence tools that could fit end users’ ever-changing demands while bringing extra business value to the customer and their target audience.

The key goals of the project included:

• Automating data collection and cleaning
• Mapping metadata
• Ensuring the highest quality of data for accurate analytics and reports
• Integrating enterprise data with the customer’s data in one centralized system
• Providing the API for other companies to connect their own apps
• Extending and improving current services

Taking those steps would allow analysts to feed pre-processed data to analytics engines and machine learning algorithms, to utilize it for a variety of business needs.

Microservices-based architecture

Itransition analyzed the customer’s legacy system consisting of five geographically distributed databases. The system represented a desktop app with a data center deployed on a database server. The system got updates as soon as the enterprise data warehouse received new data. To move to a SaaS environment, we prepared the plan of system improvements and implemented the critical ones first before the migration. Then we developed a new architecture to meet the customer’s needs.

In the new architecture, Azure Search enables fast search in different languages, serving as a proxy for semantic data queries via Graph API. Cosmos DB hosts the core data. Semantic data sets are specific to each automotive client’s adopted vocabulary, with their own set of indexes and translations as illustrated below.

The new architecture ensures high search speed and control over user queries. It also supports automated translation of automotive terms to a variety of languages. Owing to the reworked architecture, the translation mechanism provides instant responses.

Data migration

Since data in the source database was poorly structured and stored in different formats, it was impossible to implement a transformation algorithm for all the data records. So, Itransition’s team drew up transformation rules to guide researchers in making new data entries.

We created several ETL pipelines for migrating data sets united into such categories as equipment, color, trim, dealer, price incentives, sales, etc.

Data cleaning, transformation and deduplication were parts of the migration. The customer’s researchers checked migrated data during each iteration and indicated revealed issues. We migrated only the verified data that was accurate and precise.

Itransition’s team was able to migrate 15% of historical data (about 5 TB) in 10 hours. As data inaccuracies and transformation issues were coming up after the migration, we changed our transformation rules and rewrote code for the next iteration, thus ensuring continuous improvement of the migrated data.

We also created a data validation and visualization tool using the Graph Database (GDB) as part of the quality assurance (QA) stage. By implementing validation rules and checkups, we ensured that the data coming to the system was homogeneous, accurate, reliable, and consistent.

R&D

To build a unified portal with cloud business intelligence, the project teams needed to evaluate the apps that would be highly demanded by data consumers, feasible to develop, and profitable from the business point of view.

With Itransition’s assistance, the customer set up an R&D center including researchers who analyzed various solutions with the focus on big data, machine learning, BI, and analytics.

The R&D specialists tested potential apps on real users to select only the most relevant and demanded cloud business intelligence solutions. The center operated in fast-paced cycles, where researchers demoed trial products to data consumers, got their feedback, and created MVPs.

The first product developed by the R&D center was a dealer value exchange app to analyze vehicle purchases. The app was integrated with Power BI to display statistical sales reports.

Process setup

SAFe and distributed collaboration

Apart from assisting the customer in their technological transformation, our team applied their expertise in business intelligence consulting to analyze the customer’s processes and set up the Scaled Agile Framework (SAFe). This allowed syncing all teams and training the customer’s in-house developers to adjust quickly to business changes.

The customer had already chosen SAFe in order to link their business and development and achieve visibility across the company in terms of goals, progress, issues, bottlenecks, and interdependencies. SAFe also helped the company align collaboration between distributed teams involved in the development of cloud business intelligence.

Since the projects involved three big teams located in different time zones, the customer requested Itransition to help them organize flawless software delivery processes and sync remote teams.

Itransition’s SAFe program consultant together with the customer's Agile delivery managers conducted SAFe injection workshops. It helped the whole team reorganize the working process into Agile Release Trains (ARTs), where stakeholders develop, test, and deliver solutions in short iterations.

PI planning and project structuring

Itransition also introduced co-located Product Increment (PI) planning events, held every ten weeks, to foster interaction between the teams. PI planning meetings allowed the teams to plan objectives, calculate risks, as well as highlight, discuss and resolve issues and dependencies during the next seven two-week sprints.

The development process was divided into epics, which the project management team would break down into features. Each feature consisted of a certain number of user stories. The stories included a list of tasks prioritized based on the Weighted Shortest Job First (WSJF) model.

We also worked on new and repeated issues, assigned SAFe action items, defined roles, and discussed business, technology and product vision. Visualizing all the teams and the new feature release processes allowed making the project predictable and transparent.

To plan each subsequent ten-week phases of the project, we analyzed the planned features in Portfolio Kanban. This way, we reviewed, implemented and released product increments, visualizing new features from their ideation to the launch. Such an approach was also useful for us in evaluating ideas against business opportunities, costs, market fluctuations, and risks.