Anomaly detection with machine learning: spot risks before it's late

Anomaly detection with machine learning: spot risks before it's late

July 23, 2021

AI Researcher

"Prevention is the daughter of intelligence," said the famous English poet and navigator Walter Raleigh four centuries ago. Sadly, his wisdom and foresight didn't stop him from losing his head (not in the sense of losing his composure, he literally lost it).

However, this unfortunate epilogue does not belittle the meaning of this aphorism which, as certainly foreseen by Raleigh, is universally valid and holds even centuries later. What the explorer couldn't predict, on the other hand, is that the intelligence he was talking about would one day be artificial intelligence.

Indeed, AI has proven to be a reliable ally in preventing unwanted outcomes, thanks to the anomaly detection and forecasting capabilities of its sub-branch known as machine learning. But what are these powers based on and how can they be leveraged in various scenarios and AI use cases? Let's find out!

Cyber-detectives searching for clues

When it came to detecting anomalies, the typical way to go in many business contexts was traditionally based on predetermined sets of rules. For example, a fraud detection system could spot suspicious card payments which greatly exceeded a spending threshold.

The main problem with this approach is its lack of flexibility, given that the set of rules must be continuously updated to cope with ever-evolving scenarios, such as anomalous activity due to a new type of malware. Not to mention the fact that it would be unrealistic to draw up a list of rules sufficiently complete to cover any situation, including the most subtle nuances. And that’s where artificial intelligence solutions come into play.

Machine learning-based anomaly detection might seem like a rather complex technique. And don't get me wrong, it actually is. However, it is based on a very simple assumption: if a machine is able to learn what is ‘normal’, it will then understand when something deviates from the norm.

And here lies machine learning’s full potential. Any system fuelled with this technology is able to digest enormous datasets, autonomously identify recurring patterns and cause/effect relationships among the data analyzed, and create models portraying these connections. Once such models are properly trained, they will be capable of processing additional data to make predictions, further refining their skills through experience as they are fed with more and more information.

Norm vs anomalies

The recurring patterns that our model discovers by rummaging through data are the “norm” we were talking about. But what if the system ends up running into data that does not fit any existing pattern among those previously identified? Well, it’s likely to be an anomaly or an outlier.

Anomalies are typically classified into three archetypes:

  • Point anomalies: An individual data instance is anomalous with respect to the rest of the data, such as a suspiciously high-sum transaction.
  • Contextual anomalies: The anomaly is context-specific. For example, an increase in the network traffic during the night.
  • Collective anomalies: A set of data instances that may not be anomalous by themselves but look suspicious as they occur together, such as a sharp spike in login attempts or a series of unusually expensive purchases.

We can train a machine-learning system to identify the aforementioned anomalies as well as patterns and relations among data in different ways. The most common are:

  • Supervised learning: We provide the machine learning system with already labeled data, which is data that has been previously prepared and labeled as "nominal" or "anomaly".
  • Unsupervised learning: This is the approach to take when we don't know exactly what we are looking for because we are faced with an unknown scenario. Therefore, we do not provide labeled data and let the system define eventual categories and relationships.
Unsupervised machine learning for anomaly detection

Whether trained via supervised or unsupervised learning, the advantage of deploying these solutions for anomaly detection is that they don’t require pre-compiled sets of rules and are very adaptive, as machine learning systems can learn over time and fine-tune their models with new data. Let's have a look at some real scenarios in which we can harness these capabilities.

1. Fighting fraudsters and cybercriminals

The "dark side" of the massive shift towards a fully digitalized economy, further catalyzed by the recent COVID-19 pandemic, has been an exacerbation of pre-existing fraud trends combined with a steady rise in new forms of fraud and cybercrime.

According to PwC's 2020 Global Economic Crime and Fraud Survey, 56% of the surveyd US companies experienced some kind of fraud or economic crime in the previous 24 months, compared to only 38% in 2016, and this negative trend seems to be shared with the rest of the world. Specifically, the top 3 list of the most common types of fraud included customer fraud, cybercrime, and accounting fraud.

Fraud and financial crime rates, 2009-2020

Considering these trends, as well as the growing importance for companies and public institutions to protect their sensitive data, networks, and financial assets, it should come as no surprise that machine learning's anomaly detection capabilities have been widely leveraged to prevent and counter the aforementioned threats.

The idea behind implementing machine learning in fraud detection is to scan data regarding money transactions or interactions with applications, platforms, and corporate networks to probe behaviors that look out of the ordinary. These anomalous actions may be signs of potential criminal attempts and can be flagged by the system to require an additional human inspection.

Many threats, one solution

Fraud can take a myriad of different forms, as the inventiveness and creativity of fraudsters and cybercriminals could rival that of Leonardo Da Vinci, albeit for far less noble purposes. Fortunately, machine learning can fairly cope with most of them:

  • Cyber-intrusion: Hackers may try to violate corporate systems or networks in order to compromise them or steal assets and data. A machine learning-powered intrusion detection system (IDS) using network behavior anomaly detection (NBAD) can deal with similar attempts by tracing any atypical event, such as coordinated access via multiple accounts provoking a spike in traffic volume and bandwidth, and flag it as a potential cyberattack.
  • Electronic payment fraud: Fueled by the growth of ecommerce and other online transactions, electronic payment fraud has become a major threat for banks, shopping platforms, and, of course, their customers. Machine learning-driven systems can de-escalate this trend by spotting anomalous account behaviors (such as a rising transaction frequency and a change in IP addresses or login times), flagging suspicious users, and even blocking them.
  • Stock market manipulation: Machine learning-based anomaly detection techniques have been widely adopted by major financial actors, including Nasdaq, to prevent potential threats and comply with growingly stringent market regulations. Specifically, machine learning in the stock market is commonly leveraged to scan brokers' activity and track inconsistent trading patterns, which may be a sign of fraudulent equity orders such as churning, spoofing, and wash trade.
  • Money laundering: "Following the money" becomes a lot easier when a cyber eye looks for it. Machine learning-powered anomaly detection systems represent a valuable tool in the fight against money laundering, as they are capable of detecting atypical transactions carried out by suspicious organizations: for example, a small group of newly created companies located in tax havens and exchanging large sums of money despite having a limited number of customers.
  • Tax fraud: Machine learning-based systems are much faster and more accurate than any human auditor in scrutinizing the general ledger and recognizing signs of tax fraud. Among the anomalies that may arouse suspicion, we can include inconsistent itemized deductions, multiple tax refunds filed from the same IP address, and significant changes in the corporate gross sales.

2. Improving medical diagnostics

Deploying machine learning solutions is not just a matter of business but, literally, of life and death. In this regard, we cannot exempt ourselves from stressing the advantages ensured by machine learning-powered anomaly detection and predictive modeling in healthcare.

Medicine is undoubtedly a sector in which the predictive capabilities of machine learning can shine. In fact, these forecasting powers allow physicians to quickly identify patient condition anomalies that could be clues to upcoming health complications (including brain aneurysms and tumors) and set up preventive measures before it's too late.

This can be achieved by training machine learning systems with physiological data from previous clinical cases. The algorithms will process such information, spot recurring patterns related to the standard or non-standard health conditions, and build a model capable of recognizing any sign of deviation from the norm.

Example of unsupervised medical anomaly detection

Regarding the training process, it's interesting to notice that labeled datasets are typically not so well-balanced and therefore may be biased because most of the samples, for a pure matter of statistics, come from healthy patients rather than actually sick people. That's why semi-supervised or unsupervised approaches are generally more common in the medical field, and in many other contexts too, as they're not based on labeled data processing.

Another trick to avoid potential bias is to perform the so-called synthetic minority oversampling or undersampling techniques, which "artificially" reduce the number of records from the majority class (healthy persons, in this case) to equalize it to the minority class (patients suffering from a specific disease or health issues).

Accuracy, speed… and feelings

Once our machine learning-based system has been trained with clinical data, it will be able to detect any medical anomalies and integrate physicians’ expertise and "gut" by providing them with valuable insights into patients' health conditions. This combination can greatly streamline the medical workflow, improving both the accuracy and speed of the diagnostic process.

Regarding the first factor, namely accuracy, a 2018 report showed how a deep learning-based system powered by convolutional neural networks (CNN) detected 95% of skin cancers compared to 86.6% identified by human dermatologists.

As for the speed of diagnoses, it's worth mentioning a project carried out by the InnerEye team at Microsoft in collaboration with the UK's NHS and focusing on anomaly detection for diagnostic radiology. The project developers created a machine learning-based system to automate the tracing of tumors and other anomalies in 3D radiological images, in order to speed up radiotherapy and surgery planning.

In simple words, a machine replaced clinicians in performing a time-consuming procedure that typically involves the visual analysis of numerous top-to-bottom scans of a body organ to spot tumor outlines. This solution accelerated the overall radiotherapy planning process by 13 times.

The technological progress driven by the implementation of AI in radiology and other medical disciplines not only implies increased performance but also indirectly ensures a better patient experience and a more humane approach to the medical profession. Indeed, the speeding up of clinical procedures allows doctors to spend more time with their patients, taking care of their psychological well-being in a time of suffering and building a relationship based on trust, respect, and empathy: something that machines still cannot offer.

3. Enhancing maintenance operations

Mentioning faulty machines after discussing "faulty" organs may seem trivial or even inappropriate. But considering the damage that a serious breakdown of an industrial plant, especially in such high-risk sectors as energy, can cause both in economic and human terms, it is certainly worth talking about.

The most efficient way to contain these costs is certainly by predicting failures before they occur rather than trying to fix what's already broken. In this regard, the forecasting powers of machine learning based on anomaly detection have given a significant boost to condition monitoring and predictive maintenance.

These techniques involve collecting via sensors and processing through machine learning systems a massive amount of data regarding the typical functioning of machinery, power grids, and other industrial components. Once the systems are trained, they can easily spot any shift from the ideal performance of this equipment and send an alert, as such anomalies may involve an impending failure.

An efficient and rapidly spreading approach

The success of this approach is proved by statistics. According to IoT Analytics' 2019 Predictive Maintenance Report, the global predictive maintenance market was valued at $3.3 billion in 2018 and is anticipated to reach $23.5 billion by 2024 at a CAGR of 39%. Similar expectations are shared by Grand View Research, which also pointed out how North America is probably set to hold its dominant position thanks to higher investments in AI, IoT, and machine learning.

The global predictive maintenance market, 2018-2024

Speaking of North America, a successful example of machine learning-based anomaly detection for predictive maintenance comes from San Diego Gas & Electric, a major public utility that was facing a widespread energy leakage problem. The utility's system failures, which were becoming increasingly common due to the age of the transmission and distribution lines, were difficult to fix as the assets had been buried in high-density urban areas.

Such issues have been largely solved by implementing a machine learning solution to preventively detect high-risk T-splices, which were a major cause of the increase in asset malfunctions.

A proactive approach to risk management

An indisputable strength of machine learning is that it enables humans to predict and proactively address potential dangers instead of dealing with them when the damage is done. As we've seen, machine learning-based anomaly detection has proven to be a valuable tool for protecting both physical and digital assets and, most importantly, helping save lives. This technology can be leveraged in many scenarios, including:

  • Anti-fraud measures
  • Medical diagnostics
  • Predictive maintenance

Of course, we should keep in mind that training machine learning systems can be demanding as it requires huge data sets to correctly represent phenomena such as anomalies which are by definition statistically unlikely.

Despite these challenges, machine learning in anomaly detection may be the key to making the famous motto "prevention is better than cure" a reality.