Migrating to a microservices-based architecture with a 15-20x higher system throughput, building a scalable BI portal, and achieving a 100% predictability of processes across 3 remote teams.
Cloud business intelligence system for vehicle manufacturers
The customer supplies BI for the automotive industry all over the world. With decades of expertise in creating automotive BI solutions based on primary and secondary research datasets, the customer helps vehicle manufacturers plan and develop vehicles and related products, as well as launch and position them on local markets.
The company also offers research services for retailers and provides data feed tool kits for credit companies and banks, vehicle leasing organizations, and other data consumers. The customer’s solutions are also widely used in dealer and fleet management systems, dealer websites, showroom tools, and car locators.
The customer had been running a suite of BI products for globally distributed data consumers for more than 30 years.
Within the suite, the data from various sources passed through ETL pipelines in a semi-automatic mode, supervised by the customer’s researchers in different parts of the world. At the next stage, the data was packed into data marts and sent to consumers via FTP and database updates. End users could grab the processed data and integrate it with their own data to get accurate analytics and build comprehensive reports.
Over time, the system started to lag, which resulted in substantial technical debt, all the while the company was running high labor costs associated with the involved researchers who processed data manually. The data quality suffered due to different life cycle stages of processed data records. In some cases, data was inconsistent and contained errors. All considered, maintaining the legacy system was taking about 70% of the company’s annual budget.
What’s worse, complicated data recording, inaccurate incoming data, and related input errors caused substantial slowdowns in the company’s services. Therefore, the top priority was to switch to the software-as-a-service (SaaS) model and migrate their BI suite to a cloud-first, mobile-ready, microservices-based architecture.
The company also planned to launch a new BI portal packed with custom cloud business intelligence tools. To plan and develop the most suitable BI solutions, the company needed an R&D center to test-run potential BI software. Unfortunately, it was impossible to hit the new goals with the legacy BI suite due to recurring performance issues and its overall inability to scale with new features and products on demand.
To move from the legacy system to a new SaaS suite, the was looking for a technology partner experienced in automotive analytics and cloud migration. As Itransition had already completed projects involving business intelligence in the automotive industry and developed cloud business intelligence solutions at scale, the company chose us for the project.
First thing, Itransition recommended switching to a new microservices-based architecture to build the new BI portal, migrate all the data, and develop data validation and visualization tools. We also supported the customer’s idea to create an R&D center to develop cloud business intelligence tools that could fit end users’ ever-changing demands while bringing extra business value to the customer and their target audience.
The key goals of the project included:
Taking those steps would allow analysts to feed pre-processed data to analytics engines and machine learning algorithms, to utilize it for a variety of business needs.
Itransition analyzed the customer’s legacy system consisting of five geographically distributed databases. The system represented a desktop app with a data center deployed on a database server. The system got updates as soon as the enterprise data warehouse received new data. To move to a SaaS environment, we prepared the plan of system improvements and implemented the critical ones first before the migration. Then we developed a new architecture to meet the customer’s needs.
In the new architecture, Azure Search enables fast search in different languages, serving as a proxy for semantic data queries via Graph API. Cosmos DB hosts the core data. Semantic data sets are specific to each automotive client’s adopted vocabulary, with their own set of indexes and translations as illustrated below.
The new architecture ensures high search speed and control over user queries. It also supports automated translation of automotive terms to a variety of languages. Owing to the reworked architecture, the translation mechanism provides instant responses.
Since data in the source database was poorly structured and stored in different formats, it was impossible to implement a transformation algorithm for all the data records. So, Itransition’s team drew up transformation rules to guide researchers in making new data entries.
We created several ETL pipelines for migrating data sets united into such categories as equipment, color, trim, dealer, price incentives, sales, etc.
Data cleaning, transformation and deduplication were parts of the migration. The customer’s researchers checked migrated data during each iteration and indicated revealed issues. We migrated only the verified data that was accurate and precise.
Itransition’s team was able to migrate 15% of historical data (about 5 TB) in 10 hours. As data inaccuracies and transformation issues were coming up after the migration, we changed our transformation rules and rewrote code for the next iteration, thus ensuring continuous improvement of the migrated data.
We also created a data validation and visualization tool using the Graph Database (GDB) as part of the quality assurance (QA) stage. By implementing validation rules and checkups, we ensured that the data coming to the system was homogeneous, accurate, reliable, and consistent.
To build a unified portal with cloud business intelligence, the project teams needed to evaluate the apps that would be highly demanded by data consumers, feasible to develop, and profitable from the business point of view.
With Itransition’s assistance, the customer set up an R&D center including researchers who analyzed various solutions with the focus on big data, machine learning, BI, and analytics.
The R&D specialists tested potential apps on real users to select only the most relevant and demanded cloud business intelligence solutions. The center operated in fast-paced cycles, where researchers demoed trial products to data consumers, got their feedback, and created MVPs.
The first product developed by the R&D center was a dealer value exchange app to analyze vehicle purchases. The app was integrated with Power BI to display statistical sales reports.
Apart from assisting the customer in their technological transformation, our team applied their expertise in business intelligence consulting to analyze the customer’s processes and set up the Scaled Agile Framework (SAFe). This allowed syncing all teams and training the customer’s in-house developers to adjust quickly to business changes.
The customer had already chosen SAFe in order to link their business and development and achieve visibility across the company in terms of goals, progress, issues, bottlenecks, and interdependencies. SAFe also helped the company align collaboration between distributed teams involved in the development of cloud business intelligence.
Since the projects involved three big teams located in different time zones, the customer requested Itransition to help them organize flawless software delivery processes and sync remote teams.
Itransition’s SAFe program consultant together with the customer's Agile delivery managers conducted SAFe injection workshops. It helped the whole team reorganize the working process into Agile Release Trains (ARTs), where stakeholders develop, test, and deliver solutions in short iterations.
Itransition also introduced co-located Product Increment (PI) planning events, held every ten weeks, to foster interaction between the teams. PI planning meetings allowed the teams to plan objectives, calculate risks, as well as highlight, discuss and resolve issues and dependencies during the next seven two-week sprints.
The development process was divided into epics, which the project management team would break down into features. Each feature consisted of a certain number of user stories. The stories included a list of tasks prioritized based on the Weighted Shortest Job First (WSJF) model.
We also worked on new and repeated issues, assigned SAFe action items, defined roles, and discussed business, technology and product vision. Visualizing all the teams and the new feature release processes allowed making the project predictable and transparent.
To plan each subsequent ten-week phases of the project, we analyzed the planned features in Portfolio Kanban. This way, we reviewed, implemented and released product increments, visualizing new features from their ideation to the launch. Such an approach was also useful for us in evaluating ideas against business opportunities, costs, market fluctuations, and risks.
Itransition helped a global automotive BI vendor transform their legacy BI system into a robust suite in use by vehicle manufacturers and other commercial data consumers worldwide.
Throughout the project, Itransition’s team:
Find out about Itransition's high-profile BI consulting for a Canadian bank, including data architecture analysis and a BI strategy.
Itransition developed an ETL utility for automated data extracting and indexing, data mapping, transformation and saving as suitable formats.
Learn how Itransition delivered retail BI and deployed an ML-based customer analytics solution now processing 10TB of data.
InsightSoftware.com relies on Itransition’s expertise to evolve its market-leading reporting solution — InsightUnlimited™.
Itransition delivered a SaaS product that enable analytical processing of bulk data uploaded online.
Read how Itransition built a secure BI portal with online data gathering tools, third-party software modules, and a data processing center.
Learn how Itransition developed a high-load resistant solution able to handle tens of thousands of users simultaneously playing Facebook games.