Machine learning for anomaly detection: a technical overview
February 9, 2023
Anomaly detection market stats
Data source: Technavio — Anomaly detection market by deployment and geography. Forecast and analysis 2022-2026
Market size growth (2021–2026)
Estimation of year-over-year growth rate of 2022
of the growth will originate from North America
CAGR with accelerating momentum
- Key market trends
- AI & machine learning
- Internet of Things
- Data analytics
Types of anomalies
Anomalies can be classified into three main archetypes according to their relationship to the majority of the data under consideration.
Point anomalies
Point anomalies
An individual data point assumes an abnormal value compared to the common value range in the dataset.
Example: A suspiciously high-value card payment or bank deposit considering the account holder’s previous transactions.
Contextual anomalies
Contextual anomalies
This type of anomaly is context-specific as it entails a data point that is anomalous compared to most data points in the same scenario (typically from a temporal perspective).
Example: A spike in network traffic overnight or a skyrocketing sales growth outside the holiday season.
Collective anomalies
Collective anomalies
These are subsets of data points that might not seem anomalous data per se but raise suspicion when occurring together.
Example: Multiple login attempts from the same account or a sequence of unusually expensive purchases.
11 ML-based anomaly detection use cases by industry
Let's take a look at some of the real-world data science and machine learning use cases for anomaly detection in key industry scenarios.
Finance and banking
1 Stock market manipulation
Machine learning-based anomaly detection systems, combined with financial predictive analytics tools, are commonly deployed by major financial players, such as Nasdaq. These embodiments of machine learning in the stock market can easily detect brokers' anomalous trading patterns to prevent fraud (including churning, spoofing, and wash trade) and ensure compliance with strict market regulations.
2 Money laundering
Machine learning-powered anomaly detection solutions can identify and report unusual transactions carried out by suspicious organizations, such as a small group of newly created companies located in tax havens and exchanging large sums of money despite their limited number of customers.
3 Tax fraud
Machine learning-based systems can examine the companies’ general ledgers and recognize signs of tax fraud. Among suspicious anomalies, we can include inconsistent itemized deductions, multiple tax refunds filed from the same IP address, and significant changes in corporate sales.
Enhance your business with our machine learning solutions
Examples of ML-based anomaly detection
13x
faster radiotherapy planning
Microsoft's InnerEye team and the UK's NHS created an ML-based system to trace tumors and other anomalies in 3D radiological images and speed up radiotherapy and surgery planning.
+50%
of money laundering cases detected
Danske Bank deployed an ML-powered anomaly detection solution to identify money laundering cases, resulting in a massive reduction in false positives and an increase in true positives.
-70%
fraud investigation time
Capgemini developed an ML-based anomaly detection tool for credit card fraud prevention, which ensured a significant increase in detection rate and a reduction in investigation time.
+40%
cosmetic defects detected
Volvo Cars implemented Atlas, a vehicle body inspection system powered by machine learning and computer vision, to automate anomaly detection and enhance its QA operations.
Anomaly detection approaches
An ML algorithm can learn to identify patterns and anomalies via three different training techniques:
The anomaly detection algorithm is trained with already labeled data, namely the data already labeled as normal or anomalous.
Pros
This technique provides the ML system with clear examples of standard data points (for example, from lawful transactions) and enables outlier detection (such as fraudulent transactions).
Cons
It requires a time-consuming manual tagging process.
ML algorithms for anomaly detection
Data engineers rely on several machine learning techniques and algorithms to build machine learning models for anomaly detection systems. Here's just a brief selection of the most common ones.
Data source: IEEE — Machine Learning for Anomaly Detection: A Systematic Review, 24 May 2021
Machine learning techniques
Classification
SVM
- Random tree (RT)
- Random forest (RF)
- L48/C.45
- Entropy
Decision tree
- One-class SVM
- Two-class SVM
- Core vector machine (CVM)
- Kernel methods
Neural network
Kernel neural network (kNN)
Convolutional neural network (CNN)
Artificial neural network (ANN)
Restricted Boltzmann machine (RBM)
Self organizing map (SOM)
Recurrent neural network (RNN)
Hierarchical temporal memory (HTM)
Long short-term memory
Bayesian network
(BN)
K-nearest neighbors
(K-NN)
Conditional random field (CRF)
Local outlier factor
(LOF)
Optimization
- Genetic algorithm (GA)
- Linear embedding
Ensemble
AdaBoost
Rule system
Fuzzy
Clustering
- K-means
- Hierarchical clustering (HC)
- Fuzzy clustering
- Nearest clustering
Regression
- Logistic
- Linear
Support vector machine
A supervised learning algorithm that performs very well with large datasets but requires high computing power and is less reliable than other options when analyzing complex anomalies.
Decision tree
Another supervised learning algorithm following a tree-like decision-making model in which every branching represents the analysis of a specific variable to predict if a particular event is anomalous or not.
Random forest
As well as the isolation forest, is a powerful algorithm combining multiple decision trees to analyze larger datasets and enhance its pattern recognition and anomaly detection capabilities.
Logistic Regression
A supervised learning algorithm designed to assess the probability of a certain outcome between two alternatives (normal event or anomaly) depending on a range of key variables.
K-nearest neighbor
A distance-based, supervised learning algorithm that predicts the nature of a potentially anomalous event by comparing it with similar events recorded in the past and defined as "neighbors".
Neural networks
Complex sets of deep learning algorithms comprising interconnected layers of artificial neurons that mimic the human brain's architecture, typically deployed to detect the most subtle patterns and anomalies via unsupervised learning. For example, Convolutional Neural Networks and Bayesian Neural Networks.
The roadmap for adopting ML-based anomaly detection software
These are the main steps required to build and deploy an anomaly detection software solution using machine learning algorithms.
1
Data strategy
Business analysis and project framework
Tech stack identification
2
Data source selection
Corporate data assessment
Integration with external data assets
3
Data collection
ETL (extract, transform, load) pipeline setup
Data repository (data lake or warehouse) setup
4
Data preparation
Data cleaning
Data reengineering through feature selection
5
Data modeling
ML algorithm selection and design
Data processing for algorithm training
6
Software development
UI/UX design and software coding
Integration with your corporate ecosystem
7
Data analysis
Identification of patterns and anomalies
Data visualization (dashboards, visual reports)
8
Ongoing support
Post-release software adjustments
Continuous user support and training
Benefits of machine learning for anomaly detection
ML-powered anomaly detection systems offer several advantages over traditional solutions.
Superior reactivity
Traditional methods rely on rules that should be manually updated to encompass unprecedented scenarios. ML algorithms, on the other hand, autonomously learn from the new datasets they process, ensuring higher reactivity to new anomalies.
Enhanced scalability
The same high-dimensional data that may be a burden for traditional anomaly detection methods represent a valuable training resource for machine learning algorithms, which enhance their capabilities as they process more data.
Wider data pool
While rule-based methods may have a hard time with unstructured data (such as medical images or handwritten reports), ML systems can process any dataset thanks to ML-fuelled technologies like computer vision and natural language processing.
Greater accuracy
ML-based anomaly detection systems consider a broader range of variables compared to rule-based solutions, resulting in superior accuracy, a lower rate of false positives that require manual examination, and improved cost-efficiency.
Solving ML-based anomaly detection challenges
Potential challenge
Recommendation
Training times
Potential challenge
Algorithm training for anomaly detection is a time-consuming and computationally demanding process, as the datasets should be large enough to provide sufficient examples of outliers.
Potential challenge
Recommendation
A common trick for training optimization is to select a smaller subset of essential features (such as IP address, transaction data, or payment method) and discard irrelevant attributes, depending on your scenario.
Compliance
Potential challenge
The challenging trade-off between ML algorithms' data hunger and strict data management legislation can be a massive downside in highly regulated industries such as finance and medicine.
Potential challenge
Recommendation
Ensure that your ML-based anomaly detection solution complies with all major standards and regulations applicable to your industry, such as GDPR, HIPAA, and PCI DSS.
Unbalanced datasets
Potential challenge
Anomalies, by their very nature, are much less abundant than standard data points with normal behavior. This can make training datasets unbalanced and algorithms potentially biased.
Potential challenge
Recommendation
You can use synthetic minority oversampling or majority undersampling techniques to artificially reduce the number of outliers compared to normal data instances and therefore ensure a more balanced dataset.
Addressing risks with algorithms
ML-based anomaly detection systems have shown their potential in proactively addressing risks in different industries and applications, from fraud prevention and cybersecurity to advanced diagnostics and real-time asset monitoring. Furthermore, anomaly detection with machine learning has proved superior to its more traditional, rule-based counterparts, thanks to a successful mix of reactivity, scalability, and accuracy. Despite some algorithm training and compliance challenges, machine learning in anomaly detection can make the famous motto "prevention is better than cure" a reality. If you aim at enhancing your risk management capabilities, consider implementing a machine learning-based solution expertly crafted by Itransition.
