Maritime data analytics for a global shipping intelligence provider

Maritime data analytics for a global shipping intelligence provider

Migrating big data and extending the new system with industry-specific features for 5x faster AIS message processing using 15x fewer resources.

Case study

Maritime data analytics for a global shipping intelligence provider

Context

The customer is a global leader in maritime data and shipping intelligence solutions. Using the expertise of 100+ analysts and 5K+ partners, the company provides maritime data analytics to clients in 200+ countries. The customer’s analytical system collects data from vessel movements, ships, companies, ports, credit reports, insurance companies, and the market.

Their ready-to-use maritime data is a combination of business intelligence (BI) with land and satellite automatic identification system (AIS) data. Key data consumers (insurers, manufacturers, and port agents) can build reports and make smart decisions on the basis of current and historical data as well as industry news. The collected insights help end users mitigate shipping risks and avoid legal issues.

The customer had a legacy system with three core components:

  • A database with data collected from data providers’ websites
  • A website for data consumers to generate and download comprehensive transportation reports, as well as get shipment notifications
  • Admin areas for analysts to view and approve data

While processing hundreds of gigabytes of data, the system was delivering terabyte-large files with geodata to consumers. The legacy system was bulky and sluggish, especially when users filtered data by a large number of parameters. To overcome these issues and improve the quality of big data processing, the customer decided to create a new system based on Elasticsearch, which provides scalable search capabilities and supports multitenancy.

Together with a third-party software vendor, the customer partially migrated their maritime data to a new database and created a new website to display it. However, the largest part of business data was still being extracted from the legacy system. This caused the following problems:

  • External data inconsistency. The data on the new website didn’t match with the one from the legacy system. The data was either outdated or incomplete, or both.
  • Internal data inconsistency. Within the new website, the data differed on different pages, which could result in customer dissatisfaction.

Apart from these issues, the new system failed to go into production for several reasons:

  • Data obtained from the new database was unreliable and inconsistent.
  • The system processed AIS messages and generated alerts with bugs. There was no validation mechanism for vessel and area alerts, while AIS messages were duplicated or otherwise corrupted.
  • Many vital features were missing. It was impossible to generate data on class B vessels, port calls, area analytics, or extended data queries from AIS messages.
  • Important features from the old system were not implemented in the new one (for example, subscriptions to specified channels, such as tanker or gas channels for users interested in a specific data segment).

The customer needed a technology partner to complete the transition to the new system, fix the issues with previously migrated data, and add the missing features. Our former client recommended Itransition to the company. We were chosen for the project thanks to our expertise in business intelligence implementation and a track record of successful data migration projects.

Solution

In order to move the new system to the production environment, we created a detailed migration roadmap for all types of maritime data featured on the customer’s website, including:

  • Data on vessels and their characteristics (flags, names, ownership history, technical characteristics)
  • Vessel port calls
  • Vessel positions that could be tracked on the map (AIS messages, data on ship movements)
  • Vessel incidents, inspections, arrests, sanctions, hull risks
  • Companies (vessel owners, manufacturers, insurance companies)

At the same time, the customer’s system processed three types of data:

  • Data obtained from the customer’s database
  • AIS messages received from ships and containing data about their location, speed and course
  • User data (customized alerts, ship lists, user settings, saved searches)

Our team’s task was to make sure that all the data types were migrated completely and correctly.

Database data

Initially, the data obtained from the customer’s database was inconsistent and hard to update. To fix the issues with the existing maritime data analytics, we synchronized the data between the legacy system and the new one. We ensured that all data records were kept up-to-date regardless of their migration date.

Itransition’s team performed the following activities on the database:

  • Tracking operations by their ID to identify the scope of data sent to the system and reveal possible errors.
  • Developing an audit tool to reconcile data between the new and legacy systems. The tool allowed conducting automatic audits to reduce the number of data discrepancies while checking the data processing state through a number of tests.
  • Developing a retry mechanism that triggered repeated attempts to update data if the system failed to do so.
  • Creating a notification feature to alert users in case of unsuccessful data processing.
  • Carrying out manual checkups to ensure the data in the system corresponded to the data on the customer’s website.

AIS data

Itransition’s team also redesigned operations around real-time processing of AIS data and its storage in Elasticsearch, as well as optimized the latter’s performance. Elasticsearch was lagging because of the 70% load on the Elasticsearch cluster. After our team redesigned and optimized the system’s architecture, the cluster load reduced to 5%. The team also facilitated data modification and data quality control, with 5x faster processing of AIS messages and 15x fewer resources involved.

Besides, our redesign and optimization services made AIS data processing predictable, trackable, and consistent. It also significantly improved the quality of the data generated from AIS messages. Based on the data processing results, the system can now generate and display ship positions on the map while showing the ship’s latest location and voyage history.

Itransition’s team also developed an admin panel for generating AIS messages, moving messages from ships, generating different data types, and configuring rules to match ships registered in the system with their AIS messages.

Previously, generating vessel data was expensive. The customer had to hire data experts to do research and obtain data from different sources. Within the redesigned system, the AIS data processing became faster and more cost-effective. Moreover, the whole system became much easier to customize and extend with new features.

Based on the modern requirements for data analytics in the maritime industry, our team extended the system with the following functionality:

  • Data generation on class B vessels from AIS messages. Our team added a new feature to generate data on class B vessels from AIS messages, which became a new data source for the legacy database. In a week since the launch of the new system, we added 100K new vessels and reached the 300K+ count within the next 30 days.
  • Port call generation from AIS messages. The call generation feature developed by the previous vendor wasn’t operational due to a number of bugs. Itransition’s team cleaned up the code, redesigned and released the feature.
  • Area analytics. We developed a unique maritime analytics feature for users to specify areas on the map and find all the ships that passed through those areas in a defined period of time. Users can view the requested data in real time or save it as a detailed report to analyze the data later.
Area analytics

  • Extended data query from AIS messages. Hundreds of vessels send dozens of messages per minute. The system could process such volumes of AIS messages, but only 5% of them got stored in the database, which was growing by 100 GB per month. Some clients needed to store all AIS messages, however, the legacy system couldn’t handle this task with a typical database.

    To address this need, we turned to AWS Athena, which allowed data analysts to store, extract, process and compare big data according to preset rules. Using this unique feature, the customer’s clients get not only raw data (like vessel numbers and locations) but also a full data set on all tankers passing a particular area, coupled with detailed route information.

    The feature allows the customer to automatically process hundreds of gigabytes of maritime data, match AIS messages with the corresponding vessels in the new database, add business data collected by in-house analysts, and send the compiled maritime data to a client.

Extended features

To make the customer stand out on the market of maritime data systems, we extended the capabilities of the existing features:

  • Arrival alerts. Users receive email notifications 3-24 hours before planned arrival of ships at the port. We developed an algorithm to determine the ships’ upcoming stay on the basis of AIS messages and actual ship motion displayed on the map.
  • Sanction alerts. We expanded the number of sanction sources on the website and added alerts to notify users when the ship was sanctioned and the sanction duration.
  • Map access to users from China. Since the built-in maps were linked to Google Maps, they were unavailable to users from China. We made this feature independent from the map provider (Google or Bing), so that users could choose either service, or change the provider at any time.
  • Displaying all the ports in the world. We developed a feature to display 10,000 ports on any page without negatively affecting the solution’s performance. We also created features for regular map updates, ship tracking, ship clustering, displaying ship names on the map, etc.
Ship location on the map

  • Adding vessel images. We enabled image display, image uploads, integration with the customer’s several systems supplying users with images, and developed an admin panel for image editing and approval.
  • Custom API development. We developed a custom API from scratch to integrate the system with third-party services. We also expanded the number of services provided by the customer so that data consumers could integrate their own systems and build reports using combined data.
  • Ship location history. Users can track and analyze the ship's map positions and data on ship movements retrospectively. They can also select an area on the map and filter the ships that were in that area at a specific time.
  • The “Contact us” screen. The screen enables users to ask questions, send feedback, and sign up for training on every webpage.
  • Vessel hull risk. Users can calculate vessel hull risks, as well as customize the calculation depending on a ship’s characteristics.
  • Vessel lists. This feature helps users create, view and share a list of ships.
  • Tankers dashboard. The tankers' oil trade dashboard allows analysts to track tankers.
  • Saved searches. The feature allows users to apply filters, save searches for later to find data easily, track data changes, and share saved searches.
  • Calling statistics. Users can analyze the number of ship calls at a port on the map by region, country, or port.
Port calling statistics

  • Fleet statistics. Users can analyze different types of ships visualized as charts.
Fleet statistics

  • Container tracking. Users can track containers by a ship’s number. The website displays basic information on a particular container and on the ships that transported that container.

  • Law, regulation and insurance channel. The new channel allows users to view statistics on detentions, ship arrests, casualties, etc.
Casualties by type

New pages

Itransition’s team also added new pages with maritime analytics, collected instantly through Elasticsearch and supporting a variety of filters, tables, and graphs. We integrated these pages with Tableau to improve data visualization.

Results

Itransition helped the customer to complete a large-scale migration of maritime data analytics while ensuring data consistency and timely updates. The project brought several impressive advantages to the customer, including:

  • 5x faster AIS messages processing with 15x fewer resources
  • An extended vessel database enriched with 300K+ new vessels in just 30 days
  • Industry-unique features, such as area analytics and extended AIS data queries
  • Big data aggregated from disparate data sources without losses
  • An expanded set of available maritime data and easier integration of the system with third-party solutions