June 3, 2021
The stock market and machine learning: how to invest with algorithms
In ancient Rome, well-respected priests skilled in the art of divination and known as haruspices examined the entrails of sacrificed animals to predict the future. Nowadays, after realizing that these old-fashioned methods are rather ineffective (as well as quite creepy), investors prefer to rely on modern, AI-powered oracles, capable of rummaging through huge datasets to forecast the stock market's upcoming trends.
Let's discover the secrets of these virtual seers and try to understand if traders' trust is well-placed.
In recent years, the notoriously volatile nature of the stock market, further destabilized by global-impact events such as the COVID-19 pandemic, has prompted a growing number of financial institutions to explore new ways of enhancing their decision-making, such as through predictive analytics in finance. In such a chaotic environment, indeed, predicting how the market will evolve to maximize trading profits can be particularly challenging, especially when taking into account all the economic, social, and political factors which may influence these trends.
Not to mention the fact that humans, including the most experienced financial professionals, are prone to making irrational decisions. As pointed out by cognitive psychologists, humans typically suffer from loss aversion (we prefer to avoid losses than to achieve equivalent gains) and confirmation bias (we tend to confirm pre-existing beliefs with new evidence).
That’s the reason why this sector is hardly new to combine brokers’ gut feeling with the massive use of computers and statistics when it comes to investigating and predicting market trends. However, alongside traditional mathematical models and information technologies (including stock trading software), major financial players have increasingly implemented AI-based solutions to shed light on these mysterious dynamics.
Indeed, the role played by artificial intelligence is constantly growing so that, according to Gartner’s estimates, three-quarters of venture capitalists globally will take advantage of AI-based tools to make their decisions by 2025.
Despite the importance of artificial intelligence as a whole, the absolute protagonist of this new approach that could reshape the way of trading and invest is undoubtedly one of its newest branches, namely machine learning (ML).
Machine learning focuses on creating computer algorithms that can automatically improve their performance through experience. Specifically, ML algorithms can recognize patterns and relations among the data they are trained with, build mathematical models concerning such patterns, and use the same models to make predictions or decisions even without being explicitly programmed to do so.
The "learning" part in this technology's name is not there by chance. The more information ML-based systems can process, the more patterns will be detected and the aforementioned models will be polished. This operation allows algorithms to upgrade their analytical and forecasting performance with time.
Such capabilities, as you might imagine, are invaluable to financial firms, which are actually investing more and more in AI implementation and machine learning consulting. By delving into the depths of big data (including stock market trends, GDP growth, unemployment rate, corporate performance, news, consumer behavior, and social media information), ML-powered systems will pinpoint the cause-effect relationships between all these variables. Based on these findings, they may provide market players with useful insights and recommendations on future economic tendencies.
A particularly fascinating aspect of big data analytics, that is recently catching the attention of many shareholders, is the study of online financial news and social media sentiment. The assumption of this increasingly widespread approach is that taking into consideration purely economic variables is not enough to predict stock market trends, even by deploying cutting-edge technologies such as machine learning.
Instead, financial experts should leverage text analysis and natural language processing to identify the sentiment from sources like social media posts or financial news articles, i.e. to understand if such texts reflect a positive or negative opinion on specific financial matters.
These techniques have already been adopted by various financial giants. J.P. Morgan Research has created an ML solution tested on 100,000 news articles covering global equity markets, in order to assist experts in future equity investment decisions. Blackrock, instead, has leveraged text analysis techniques to predict future changes to company earnings guidance.
As you may have gathered, the most crucial application of machine learning in the stock market involves big data analysis to provide forecasts on financial trends and insights into trading and investing possibilities.
An excellent indicator of ML’s forecasting potential lies in the interest aroused among scientists and top financial players, which are investing large sums to develop these technologies and, for some years now, have achieved very promising results.
In its Innovations in Finance with Machine Learning report, J.P. Morgan described an initiative focused on interest rate markets. An ML-powered system based on the random forest algorithm was fed with a wide variety of data collected from 2000 to 2016, including international interest rates and the Federal Reserve meetings' calendar. The aim was to suggest the timing and sizing of the 2017 trades.
The difference in terms of returns between buying bonds through conventional methods and following an ML-based approach was pretty impressive, as you may see from the graph below. The third and sixth bars show the performance of standard operations without ML, which served as a control. The first and fourth bars, on the other hand, indicate returns from short selling, and the second and fifth bars from both buying and selling with ML guidance.
Other encouraging data comes from research on bond default forecasts and AI-led hedge funds. Regarding the first, a team of Brazilian academics from the Universities of Brasília and Uberlândia discovered that ML-based predictive techniques are approximately 10% more accurate than traditional methods at estimating the likelihood of bond defaults.
As for the second point, a study found in the August 2020 Cerulli Edge Global edition showed that the cumulative return of AI-driven hedge fund trading from 2016 to 2019 was almost three times higher than that achieved by traditional hedge fund investments in the same period (33.9% vs 12.1%).
Not all prophecies come true. Otherwise, we would all have died several times, according to the numerous predictions about the world’s end. The same statement is valid for machine learning in the stock market.
Despite the massive potential of ML-based forecasting, this technology is far from being perfect. Once implemented in a real-world scenario, it may give rise to some unexpected distortions, as brilliantly pointed out by Harvard Business Review.
Machine learning in the stock market is not just about predicting upcoming economic trends. We can also take advantage of algorithms to fight fraud and improve back-office activities.
ML-powered tools can take on the role of detectives with the same zeal with which they play that of clairvoyants. Specifically, they may take advantage of their data processing capabilities to prevent or detect fraud, money laundering, and other financial crimes.
All they have to do is to cross-check transactions and traders' personal data to spot inconsistencies in the information provided or anomalies in their typical activity, which could be clues of potential misconduct. ML solutions can also adapt to new fraudulent behaviors breaking standard patterns, and consequently fasten reaction times of competent authorities when facing unprecedented threats.
Nowadays, this aspect of AI application in finance is more important than ever, given the increasingly strict trading regulations and the need for greater compliance in asset management. In this regard, machines can be a valuable ally, ensuring greater accuracy and, at the same time, lower operating costs.
Indeed, many major financial players have understood the potential of these technologies in fighting crime, including Nasdaq that is testing an ML-based surveillance system to deal with any attempts at trade manipulation. Its algorithms are leveraged to identify irregular patterns and send an alert to exchange officials.
“It [points] out what we call an ‘interesting event’. It’s not necessarily a prohibited activity, but it’s what the model has deemed to be interesting because it’s not normal market behavior.”
Senior Vice President, Head of Artificial Intelligence and Investment Intelligence Technology, Nasdaq
For every hyperactive trader who screams in the halls of the stock exchange to liquidate their positions, there is one (or more) white-collar who takes care of the paperwork. Indeed, the "hidden" side of trading is a massive amount of back-office procedures that represent the gears of the financial bureaucracy.
Think of all those activities related to administration and support services, but most of all to accounting, including financial and tax reporting. This last point is essential to ensure maximum transparency to customers and achieve proper audit and compliance with current regulations.
ML-powered systems can work side by side with other technologies (for instance, RPA in finance) to automate some of these tasks, such as recording transactions and generating the related documentation quickly, with almost absolute accuracy and in total autonomy.
The main difference between machine learning and other less advanced solutions is that the former doesn't just automate business processes but also makes them smarter. For example, an ML-based financial and accounting platform could autonomously understand how human operators work and replicate their actions, even adapting to different operational scenarios. Or it could record voice-based transactions and interact with employees to assist, thanks to ML-related technologies such as natural language processing.
Despite the generally growing adoption of machine learning in the stock market, by investigating this trend in detail we might unveil conflicting situations. On the one hand, major asset managers are leading the way with significant investments in ML implementation and financial software development, but also by securing top talent. On the other hand, small businesses may have a hard time keeping up.
Based on CFA Institute, indeed, only 10% of portfolio managers surveyed said they had adopted ML techniques in the previous 12 months.
Similar results were described by Statista in their report on the use of artificial intelligence for portfolio management in 2020, according to which only 6% of asset owners and investment managers surveyed declared applied the technology. Nevertheless, 76% of the respondents showed interest in the future adoption of AI for portfolio management.
No matter how things go, the race is on and the most forward-looking financial players may derive massive benefits from ML implementation. In this regard, machine learning has already proven to have several tricks up its sleeve, ensuring:
As we have already clarified, all of these advantages come along with some challenges. After all, ML is an amazing technology but it's not magic, despite previous simplifications involving clairvoyants and crystal balls. And even if we really wanted to see it as a crystal ball, we'd still need a wizard to look into it, just as we'll still need human financial experts to properly leverage ML solutions in the years to come.
Learn the most potent use cases for RPA in banking and figure out the best implementation strategies with Itransition.
Discover the contribution of RPA to finance and accounting, its top use cases, and the good practices to implement it properly.
One of the main barriers to AI adoption is identifying the right AI use cases. Find out the best practices to make it less challenging.
The role of AI in sports is growing rapidly. Find out how this technology can be leveraged and what challenges its implementation will entail.
Which industries are benefitting from predictive analytics tools right now? Learn more about their business value from our analysis.
In the face of the growing uncertainty surrounding machine learning adoption in the banking industry, Itransition responds with an actionable framework.