February 9, 2023
ML-enabled anomaly detection applications differ from traditional software in terms of detection technique:
Traditional anomaly detection solutions typically trigger an answer when one or more predefined conditions are violated.
Example: A credit card payment exceeding a certain threshold.
ML algorithms are trained to autonomously discover recurring patterns or clusters among key variables and data points by processing large datasets. Once an ML system runs into data that doesn't fit any existing pattern, it may have identified an anomaly.
Example: An unusual credit card payment deviating from its holder's typical purchasing patterns.
According to Technavio's 2022 Anomaly Detection Market Forecast and Analysis, machine learning consulting represents one of the main tech and business trends in the anomaly detection market, which is estimated to grow by $4.23bn from 2021 to 2026 at a maximum CAGR of 15.08%.
Data source: Technavio — Anomaly detection market by deployment and geography. Forecast and analysis 2022-2026
Market size growth (2021–2026)
Estimation of year-over-year growth rate of 2022
of the growth will originate from North America
CAGR with accelerating momentum
Anomalies can be classified into three main archetypes according to their relationship to the majority of the data under consideration.
An individual data point assumes an abnormal value compared to the common value range in the dataset.
Example: A suspiciously high-value card payment or bank deposit considering the account holder’s previous transactions.
This type of anomaly is context-specific as it entails a data point that is anomalous compared to most data points in the same scenario (typically from a temporal perspective).
Example: A spike in network traffic overnight or a skyrocketing sales growth outside the holiday season.
These are subsets of data points that might not seem anomalous data per se but raise suspicion when occurring together.
Example: Multiple login attempts from the same account or a sequence of unusually expensive purchases.
Let's take a look at some of the real-world data science and machine learning use cases for anomaly detection in key industry scenarios.
Machine learning-based anomaly detection systems, combined with financial predictive analytics tools, are commonly deployed by major financial players, such as Nasdaq. These embodiments of machine learning in the stock market can easily detect brokers' anomalous trading patterns to prevent fraud (including churning, spoofing, and wash trade) and ensure compliance with strict market regulations.
Machine learning-powered anomaly detection solutions can identify and report unusual transactions carried out by suspicious organizations, such as a small group of newly created companies located in tax havens and exchanging large sums of money despite their limited number of customers.
Machine learning-based systems can examine the companies’ general ledgers and recognize signs of tax fraud. Among suspicious anomalies, we can include inconsistent itemized deductions, multiple tax refunds filed from the same IP address, and significant changes in corporate sales.
faster radiotherapy planning
of money laundering cases detected
fraud investigation time
cosmetic defects detected
An ML algorithm can learn to identify patterns and anomalies via three different training techniques:
The anomaly detection algorithm is trained with already labeled data, namely the data already labeled as normal or anomalous.
Data engineers rely on several machine learning techniques and algorithms to build machine learning models for anomaly detection systems. Here's just a brief selection of the most common ones.
Data source: IEEE — Machine Learning for Anomaly Detection: A Systematic Review, 24 May 2021
Machine learning techniques
Kernel neural network (kNN)
Convolutional neural network (CNN)
Artificial neural network (ANN)
Restricted Boltzmann machine (RBM)
Self organizing map (SOM)
Recurrent neural network (RNN)
Hierarchical temporal memory (HTM)
Long short-term memory
AdaBoost
Fuzzy
A supervised learning algorithm that performs very well with large datasets but requires high computing power and is less reliable than other options when analyzing complex anomalies.
Another supervised learning algorithm following a tree-like decision-making model in which every branching represents the analysis of a specific variable to predict if a particular event is anomalous or not.
As well as the isolation forest, is a powerful algorithm combining multiple decision trees to analyze larger datasets and enhance its pattern recognition and anomaly detection capabilities.
A supervised learning algorithm designed to assess the probability of a certain outcome between two alternatives (normal event or anomaly) depending on a range of key variables.
A distance-based, supervised learning algorithm that predicts the nature of a potentially anomalous event by comparing it with similar events recorded in the past and defined as "neighbors".
Complex sets of deep learning algorithms comprising interconnected layers of artificial neurons that mimic the human brain's architecture, typically deployed to detect the most subtle patterns and anomalies via unsupervised learning. For example, Convolutional Neural Networks and Bayesian Neural Networks.
These are the main steps required to build and deploy an anomaly detection software solution using machine learning algorithms.
Data strategy
Business analysis and project framework
Tech stack identification
Data source selection
Corporate data assessment
Integration with external data assets
Data collection
ETL (extract, transform, load) pipeline setup
Data repository (data lake or warehouse) setup
Data preparation
Data cleaning
Data reengineering through feature selection
Data modeling
ML algorithm selection and design
Data processing for algorithm training
Software development
UI/UX design and software coding
Integration with your corporate ecosystem
Data analysis
Identification of patterns and anomalies
Data visualization (dashboards, visual reports)
Ongoing support
Post-release software adjustments
Continuous user support and training
ML-powered anomaly detection systems offer several advantages over traditional solutions.
Superior reactivity
Traditional methods rely on rules that should be manually updated to encompass unprecedented scenarios. ML algorithms, on the other hand, autonomously learn from the new datasets they process, ensuring higher reactivity to new anomalies.
Enhanced scalability
The same high-dimensional data that may be a burden for traditional anomaly detection methods represent a valuable training resource for machine learning algorithms, which enhance their capabilities as they process more data.
Wider data pool
While rule-based methods may have a hard time with unstructured data (such as medical images or handwritten reports), ML systems can process any dataset thanks to ML-fuelled technologies like computer vision and natural language processing.
Greater accuracy
ML-based anomaly detection systems consider a broader range of variables compared to rule-based solutions, resulting in superior accuracy, a lower rate of false positives that require manual examination, and improved cost-efficiency.
Potential challenge
Algorithm training for anomaly detection is a time-consuming and computationally demanding process, as the datasets should be large enough to provide sufficient examples of outliers.
Potential challenge
Recommendation
A common trick for training optimization is to select a smaller subset of essential features (such as IP address, transaction data, or payment method) and discard irrelevant attributes, depending on your scenario.
Potential challenge
The challenging trade-off between ML algorithms' data hunger and strict data management legislation can be a massive downside in highly regulated industries such as finance and medicine.
Potential challenge
Recommendation
Ensure that your ML-based anomaly detection solution complies with all major standards and regulations applicable to your industry, such as GDPR, HIPAA, and PCI DSS.
Potential challenge
Anomalies, by their very nature, are much less abundant than standard data points with normal behavior. This can make training datasets unbalanced and algorithms potentially biased.
Potential challenge
Recommendation
You can use synthetic minority oversampling or majority undersampling techniques to artificially reduce the number of outliers compared to normal data instances and therefore ensure a more balanced dataset.
ML-based anomaly detection systems have shown their potential in proactively addressing risks in different industries and applications, from fraud prevention and cybersecurity to advanced diagnostics and real-time asset monitoring. Furthermore, anomaly detection with machine learning has proved superior to its more traditional, rule-based counterparts, thanks to a successful mix of reactivity, scalability, and accuracy. Despite some algorithm training and compliance challenges, machine learning in anomaly detection can make the famous motto "prevention is better than cure" a reality. If you aim at enhancing your risk management capabilities, consider implementing a machine learning-based solution expertly crafted by Itransition.