July 23, 2021
Anomaly detection with machine learning: spot risks before it's late
"Prevention is the daughter of intelligence," said the famous English poet and navigator Walter Raleigh four centuries ago. Sadly, his wisdom and foresight didn't stop him from losing his head (not in the sense of losing his composure, he literally lost it).
However, this unfortunate epilogue does not belittle the meaning of this aphorism which, as certainly foreseen by Raleigh, is universally valid and holds even centuries later. What the explorer couldn't predict, on the other hand, is that the intelligence he was talking about would one day be artificial intelligence.
Indeed, AI has proven to be a reliable ally in preventing unwanted outcomes, thanks to the anomaly detection and forecasting capabilities of its sub-branch known as machine learning. But what are these powers based on and how can they be leveraged in various scenarios and AI use cases? Let's find out!
When it came to detecting anomalies, the typical way to go in many business contexts was traditionally based on predetermined sets of rules. For example, a fraud detection system could spot suspicious card payments which greatly exceeded a spending threshold.
The main problem with this approach is its lack of flexibility, given that the set of rules must be continuously updated to cope with ever-evolving scenarios, such as anomalous activity due to a new type of malware. Not to mention the fact that it would be unrealistic to draw up a list of rules sufficiently complete to cover any situation, including the most subtle nuances. And that’s where artificial intelligence solutions come into play.
Machine learning-based anomaly detection might seem like a rather complex technique. And don't get me wrong, it actually is. However, it is based on a very simple assumption: if a machine is able to learn what is ‘normal’, it will then understand when something deviates from the norm.
And here lies machine learning’s full potential. Any system fuelled with this technology is able to digest enormous datasets, autonomously identify recurring patterns and cause/effect relationships among the data analyzed, and create models portraying these connections. Once such models are properly trained, they will be capable of processing additional data to make predictions, further refining their skills through experience as they are fed with more and more information.
The recurring patterns that our model discovers by rummaging through data are the “norm” we were talking about. But what if the system ends up running into data that does not fit any existing pattern among those previously identified? Well, it’s likely to be an anomaly or an outlier.
Anomalies are typically classified into three archetypes:
We can train a machine-learning system to identify the aforementioned anomalies as well as patterns and relations among data in different ways. The most common are:
Whether trained via supervised or unsupervised learning, the advantage of deploying these solutions for anomaly detection is that they don’t require pre-compiled sets of rules and are very adaptive, as machine learning systems can learn over time and fine-tune their models with new data. Let's have a look at some real scenarios in which we can harness these capabilities.
The "dark side" of the massive shift towards a fully digitalized economy, further catalyzed by the recent COVID-19 pandemic, has been an exacerbation of pre-existing fraud trends combined with a steady rise in new forms of fraud and cybercrime.
According to PwC's 2020 Global Economic Crime and Fraud Survey, 56% of the surveyd US companies experienced some kind of fraud or economic crime in the previous 24 months, compared to only 38% in 2016, and this negative trend seems to be shared with the rest of the world. Specifically, the top 3 list of the most common types of fraud included customer fraud, cybercrime, and accounting fraud.
Considering these trends, as well as the growing importance for companies and public institutions to protect their sensitive data, networks, and financial assets, it should come as no surprise that machine learning's anomaly detection capabilities have been widely leveraged to prevent and counter the aforementioned threats.
The idea behind implementing machine learning in fraud detection is to scan data regarding money transactions or interactions with applications, platforms, and corporate networks to probe behaviors that look out of the ordinary. These anomalous actions may be signs of potential criminal attempts and can be flagged by the system to require an additional human inspection.
Fraud can take a myriad of different forms, as the inventiveness and creativity of fraudsters and cybercriminals could rival that of Leonardo Da Vinci, albeit for far less noble purposes. Fortunately, machine learning can fairly cope with most of them:
Deploying machine learning solutions is not just a matter of business but, literally, of life and death. In this regard, we cannot exempt ourselves from stressing the advantages ensured by machine learning-powered anomaly detection and predictive modeling in healthcare.
Medicine is undoubtedly a sector in which the predictive capabilities of machine learning can shine. In fact, these forecasting powers allow physicians to quickly identify patient condition anomalies that could be clues to upcoming health complications (including brain aneurysms and tumors) and set up preventive measures before it's too late.
This can be achieved by training machine learning systems with physiological data from previous clinical cases. The algorithms will process such information, spot recurring patterns related to the standard or non-standard health conditions, and build a model capable of recognizing any sign of deviation from the norm.
Regarding the training process, it's interesting to notice that labeled datasets are typically not so well-balanced and therefore may be biased because most of the samples, for a pure matter of statistics, come from healthy patients rather than actually sick people. That's why semi-supervised or unsupervised approaches are generally more common in the medical field, and in many other contexts too, as they're not based on labeled data processing.
Another trick to avoid potential bias is to perform the so-called synthetic minority oversampling or undersampling techniques, which "artificially" reduce the number of records from the majority class (healthy persons, in this case) to equalize it to the minority class (patients suffering from a specific disease or health issues).
Once our machine learning-based system has been trained with clinical data, it will be able to detect any medical anomalies and integrate physicians’ expertise and "gut" by providing them with valuable insights into patients' health conditions. This combination can greatly streamline the medical workflow, improving both the accuracy and speed of the diagnostic process.
Regarding the first factor, namely accuracy, a 2018 report showed how a deep learning-based system powered by convolutional neural networks (CNN) detected 95% of skin cancers compared to 86.6% identified by human dermatologists.
As for the speed of diagnoses, it's worth mentioning a project carried out by the InnerEye team at Microsoft in collaboration with the UK's NHS and focusing on anomaly detection for diagnostic radiology. The project developers created a machine learning-based system to automate the tracing of tumors and other anomalies in 3D radiological images, in order to speed up radiotherapy and surgery planning.
In simple words, a machine replaced clinicians in performing a time-consuming procedure that typically involves the visual analysis of numerous top-to-bottom scans of a body organ to spot tumor outlines. This solution accelerated the overall radiotherapy planning process by 13 times.
The technological progress driven by the implementation of AI in radiology and other medical disciplines not only implies increased performance but also indirectly ensures a better patient experience and a more humane approach to the medical profession. Indeed, the speeding up of clinical procedures allows doctors to spend more time with their patients, taking care of their psychological well-being in a time of suffering and building a relationship based on trust, respect, and empathy: something that machines still cannot offer.
Mentioning faulty machines after discussing "faulty" organs may seem trivial or even inappropriate. But considering the damage that a serious breakdown of an industrial plant, especially in such high-risk sectors as energy, can cause both in economic and human terms, it is certainly worth talking about.
The most efficient way to contain these costs is certainly by predicting failures before they occur rather than trying to fix what's already broken. In this regard, the forecasting powers of machine learning based on anomaly detection have given a significant boost to condition monitoring and predictive maintenance.
These techniques involve collecting via sensors and processing through machine learning systems a massive amount of data regarding the typical functioning of machinery, power grids, and other industrial components. Once the systems are trained, they can easily spot any shift from the ideal performance of this equipment and send an alert, as such anomalies may involve an impending failure.
The success of this approach is proved by statistics. According to IoT Analytics' 2019 Predictive Maintenance Report, the global predictive maintenance market was valued at $3.3 billion in 2018 and is anticipated to reach $23.5 billion by 2024 at a CAGR of 39%. Similar expectations are shared by Grand View Research, which also pointed out how North America is probably set to hold its dominant position thanks to higher investments in AI, IoT, and machine learning.
Speaking of North America, a successful example of machine learning-based anomaly detection for predictive maintenance comes from San Diego Gas & Electric, a major public utility that was facing a widespread energy leakage problem. The utility's system failures, which were becoming increasingly common due to the age of the transmission and distribution lines, were difficult to fix as the assets had been buried in high-density urban areas.
Such issues have been largely solved by implementing a machine learning solution to preventively detect high-risk T-splices, which were a major cause of the increase in asset malfunctions.
An indisputable strength of machine learning is that it enables humans to predict and proactively address potential dangers instead of dealing with them when the damage is done. As we've seen, machine learning-based anomaly detection has proven to be a valuable tool for protecting both physical and digital assets and, most importantly, helping save lives. This technology can be leveraged in many scenarios, including:
Of course, we should keep in mind that training machine learning systems can be demanding as it requires huge data sets to correctly represent phenomena such as anomalies which are by definition statistically unlikely.
Despite these challenges, machine learning in anomaly detection may be the key to making the famous motto "prevention is better than cure" a reality.
Find out how machine learning in ecommerce is reshaping the way we buy and sell on the web.
In a world moving towards full digitalization, find out how machine learning can help organizations manage the growing spread of fraud and cybercrime.
Find out how machine learning in the stock market is reshaping the way of trading and investing.
NLP in healthcare provides a wide array of tools to assist medical professionals and improve patient experience. How? Let's find out.
Learn about the effects of AI in the workplace, how it's reshaping our jobs and careers, and what challenges may arise from its implementation.
One of the main barriers to AI adoption is identifying the right AI use cases. Find out the best practices to make it less challenging.