Migrating big data and extending the new system with industry-specific features for 5x faster AIS message processing using 15x fewer resources.
The customer is a global leader in maritime data and shipping intelligence solutions. Using the expertise of 100+ analysts and 5K+ partners, the company provides maritime data analytics to clients in 200+ countries. The customer’s analytical system collects data from vessel movements, ships, companies, ports, credit reports, insurance companies, and the market.
Their ready-to-use maritime data is a combination of business intelligence (BI) with land and satellite automatic identification system (AIS) data. Key data consumers (insurers, manufacturers, and port agents) can build reports and make smart decisions on the basis of current and historical data as well as industry news. The collected insights help end users mitigate shipping risks and avoid legal issues.
The customer had a legacy system with three core components:
While processing hundreds of gigabytes of data, the system was delivering terabyte-large files with geodata to consumers. The legacy system was bulky and sluggish, especially when users filtered data by a large number of parameters. To overcome these issues and improve the quality of big data processing, the customer decided to create a new system based on Elasticsearch, which provides scalable search capabilities and supports multitenancy.
Together with a third-party software vendor, the customer partially migrated their maritime data to a new database and created a new website to display it. However, the largest part of business data was still being extracted from the legacy system. This caused the following problems:
Apart from these issues, the new system failed to go into production for several reasons:
The customer needed a technology partner to complete the transition to the new system, fix the issues with previously migrated data, and add the missing features. Our former client recommended Itransition to the company. We were chosen for the project thanks to our expertise in business intelligence implementation and a track record of successful data migration projects.
In order to move the new system to the production environment, we created a detailed migration roadmap for all types of maritime data featured on the customer’s website, including:
At the same time, the customer’s system processed three types of data:
Our team’s task was to make sure that all the data types were migrated completely and correctly.
Initially, the data obtained from the customer’s database was inconsistent and hard to update. To fix the issues with the existing maritime data analytics, we synchronized the data between the legacy system and the new one. We ensured that all data records were kept up-to-date regardless of their migration date.
Itransition’s team performed the following activities on the database:
Itransition’s team also redesigned operations around real-time processing of AIS data and its storage in Elasticsearch, as well as optimized the latter’s performance. Elasticsearch was lagging because of the 70% load on the Elasticsearch cluster. After our team redesigned and optimized the system’s architecture, the cluster load reduced to 5%. The team also facilitated data modification and data quality control, with 5x faster processing of AIS messages and 15x fewer resources involved.
Besides, our redesign and optimization services made AIS data processing predictable, trackable, and consistent. It also significantly improved the quality of the data generated from AIS messages. Based on the data processing results, the system can now generate and display ship positions on the map while showing the ship’s latest location and voyage history.
Itransition’s team also developed an admin panel for generating AIS messages, moving messages from ships, generating different data types, and configuring rules to match ships registered in the system with their AIS messages.
Previously, generating vessel data was expensive. The customer had to hire data experts to do research and obtain data from different sources. Within the redesigned system, the AIS data processing became faster and more cost-effective. Moreover, the whole system became much easier to customize and extend with new features.
Based on the modern requirements for real-time big data analytics in the maritime industry, our team extended the system with the following functionality:
Extended data query from AIS messages. Hundreds of vessels send dozens of messages per minute. The system could process such volumes of AIS messages, but only 5% of them got stored in the database, which was growing by 100 GB per month. Some clients needed to store all AIS messages, however, the legacy system couldn’t handle this task with a typical database.
To address this need, we turned to AWS Athena, which allowed data analysts to store, extract, process and compare big data according to preset rules. Using this unique feature, the customer’s clients get not only raw data (like vessel numbers and locations) but also a full data set on all tankers passing a particular area, coupled with detailed route information.
The feature allows the customer to automatically process hundreds of gigabytes of maritime data, match AIS messages with the corresponding vessels in the new database, add business data collected by in-house analysts, and send the compiled maritime data to a client.
To make the customer stand out on the market of maritime data systems, we extended the capabilities of the existing features:
Itransition’s team also added new pages with maritime analytics, collected instantly through Elasticsearch and supporting a variety of filters, tables, and graphs. We integrated these pages with Tableau to improve data visualization.
Itransition helped the customer to complete a large-scale migration of maritime data analytics while ensuring data consistency and timely updates. The project brought several impressive advantages to the customer, including:
Delegate data management to Itransition and turn your data into a unified, clean and secure source of value. Book your consultation now.
Find out 5 predictions of the future of big data up to 2025 and its influence on consumers and businesses worldwide according to experts.
Find out how Itransition migrated a BI suite to the cloud and delivered brand-new cloud business intelligence tools for the automotive industry.
Learn about the benefits of big data visualization and get a checklist of things to look for when picking the vendor.
Itransition delivered a SaaS product that enable analytical processing of bulk data uploaded online.
We tap into big data and its business impacts to show how to keep it under control and avoid common big data management pitfalls.
Find out about Itransition's high-profile BI consulting for a Canadian bank, including data architecture analysis and a BI strategy.