Maritime data analytics for a global shipping intelligence provider

Solution

In order to move the new system to the production environment, we created a detailed migration roadmap for all types of maritime data featured on the customer’s website, including:

• Data on vessels and their characteristics (flags, names, ownership history, technical characteristics)
• Vessel port calls
• Vessel positions that could be tracked on the map (AIS messages, data on ship movements)
• Vessel incidents, inspections, arrests, sanctions, hull risks
• Companies (vessel owners, manufacturers, insurance companies)

At the same time, the customer’s system processed three types of data:

• Data obtained from the customer’s database
• AIS messages received from ships and containing data about their location, speed and course
• User data (customized alerts, ship lists, user settings, saved searches)

Our team’s task was to make sure that all the data types were migrated completely and correctly.

Database data

Initially, the data obtained from the customer’s database was inconsistent and hard to update. To fix the issues with the existing maritime data analytics, we synchronized the data between the legacy system and the new one. We ensured that all data records were kept up-to-date regardless of their migration date.

Itransition’s team performed the following activities on the database:

• Tracking operations by their ID to identify the scope of data sent to the system and reveal possible errors.
• Developing an audit tool to reconcile data between the new and legacy systems. The tool allowed conducting automatic audits to reduce the number of data discrepancies while checking the data processing state through a number of tests.
• Developing a retry mechanism that triggered repeated attempts to update data if the system failed to do so.
• Creating a notification feature to alert users in case of unsuccessful data processing.
• Carrying out manual checkups to ensure the data in the system corresponded to the data on the customer’s website.

AIS data

Itransition’s team also redesigned operations around real-time processing of AIS data and its storage in Elasticsearch, as well as optimized the latter’s performance. Elasticsearch was lagging because of the 70% load on the Elasticsearch cluster. After our team redesigned and optimized the system’s architecture, the cluster load reduced to 5%. The team also facilitated data modification and data quality control, with 5x faster processing of AIS messages and 15x fewer resources involved.

Besides, our redesign and optimization services made AIS data processing predictable, trackable, and consistent. It also significantly improved the quality of the data generated from AIS messages. Based on the data processing results, the system can now generate and display ship positions on the map while showing the ship’s latest location and voyage history.

Itransition’s team also developed an admin panel for generating AIS messages, moving messages from ships, generating different data types, and configuring rules to match ships registered in the system with their AIS messages.

Previously, generating vessel data was expensive. The customer had to hire data experts to do research and obtain data from different sources. Within the redesigned system, the AIS data processing became faster and more cost-effective. Moreover, the whole system became much easier to customize and extend with new features.

Based on the modern requirements for real-time big data analytics in the maritime industry, our team extended the system with the following functionality:

• Data generation on class B vessels from AIS messages. Our team added a new feature to generate data on class B vessels from AIS messages, which became a new data source for the legacy database. In a week since the launch of the new system, we added 100K new vessels and reached the 300K+ count within the next 30 days.
• Port call generation from AIS messages. The call generation feature developed by the previous vendor wasn’t operational due to a number of bugs. Itransition’s team cleaned up the code, redesigned and released the feature.
• Area analytics. We developed a unique maritime analytics feature for users to specify areas on the map and find all the ships that passed through those areas in a defined period of time. Users can view the requested data in real time or save it as a detailed report to analyze the data later.

• Extended data query from AIS messages. Hundreds of vessels send dozens of messages per minute. The system could process such volumes of AIS messages, but only 5% of them got stored in the database, which was growing by 100 GB per month. Some clients needed to store all AIS messages, however, the legacy system couldn’t handle this task with a typical database.

  To address this need, we turned to AWS Athena, which allowed data analysts to store, extract, process and compare big data according to preset rules. Using this unique feature, the customer’s clients get not only raw data (like vessel numbers and locations) but also a full data set on all tankers passing a particular area, coupled with detailed route information.

  The feature allows the customer to automatically process hundreds of gigabytes of maritime data, match AIS messages with the corresponding vessels in the new database, add business data collected by in-house analysts, and send the compiled maritime data to a client.

Extended features

To make the customer stand out on the market of maritime data systems, we extended the capabilities of the existing features:

• Arrival alerts. Users receive email notifications 3-24 hours before planned arrival of ships at the port. We developed an algorithm to determine the ships’ upcoming stay on the basis of AIS messages and actual ship motion displayed on the map.
• Sanction alerts. We expanded the number of sanction sources on the website and added alerts to notify users when the ship was sanctioned and the sanction duration.
• Map access to users from China. Since the built-in maps were linked to Google Maps, they were unavailable to users from China. We made this feature independent from the map provider (Google or Bing), so that users could choose either service, or change the provider at any time.
• Displaying all the ports in the world. We developed a feature to display 10,000 ports on any page without negatively affecting the solution’s performance. We also created features for regular map updates, ship tracking, ship clustering, displaying ship names on the map, etc.

• Adding vessel images. We enabled image display, image uploads, integration with the customer’s several systems supplying users with images, and developed an admin panel for image editing and approval.
• Custom API development. We developed a custom API from scratch to integrate the system with third-party services. We also expanded the number of services provided by the customer so that data consumers could integrate their own systems and build reports using combined data.
• Ship location history. Users can track and analyze the ship's map positions and data on ship movements retrospectively. They can also select an area on the map and filter the ships that were in that area at a specific time.
• The “Contact us” screen. The screen enables users to ask questions, send feedback, and sign up for training on every webpage.
• Vessel hull risk. Users can calculate vessel hull risks, as well as customize the calculation depending on a ship’s characteristics.
• Vessel lists. This feature helps users create, view and share a list of ships.
• Tankers dashboard. The tankers' oil trade dashboard allows analysts to track tankers.
• Saved searches. The feature allows users to apply filters, save searches for later to find data easily, track data changes, and share saved searches.
• Calling statistics. Users can analyze the number of ship calls at a port on the map by region, country, or port.

• Fleet statistics. Users can analyze different types of ships visualized as charts.

• Container tracking. Users can track containers by a ship’s number. The website displays basic information on a particular container and on the ships that transported that container.

• Law, regulation and insurance channel. The new channel allows users to view statistics on detentions, ship arrests, casualties, etc.

New pages

Itransition’s team also added new pages with maritime analytics, collected instantly through Elasticsearch and supporting a variety of filters, tables, and graphs. We integrated these pages with Tableau to improve data visualization.