The stock market and machine learning: how to invest with algorithms
June 3, 2021
In ancient Rome, well-respected priests skilled in the art of divination and known as haruspices examined the entrails of sacrificed animals to predict the future. Nowadays, after realizing that these old-fashioned methods are rather ineffective (as well as quite creepy), investors prefer to rely on modern, AI-powered oracles, capable of rummaging through huge datasets to forecast the stock market's upcoming trends.
Let's discover the secrets of these virtual seers and try to understand if traders' trust is well-placed.
A new fortune-teller in town
In recent years, the notoriously volatile nature of the stock market, further destabilized by global-impact events such as the COVID-19 pandemic, has prompted a growing number of financial institutions to explore new ways of enhancing their decision-making, such as through predictive analytics in finance. In such a chaotic environment, indeed, predicting how the market will evolve to maximize trading profits can be particularly challenging, especially when taking into account all the economic, social, and political factors which may influence these trends.
Not to mention the fact that humans, including the most experienced financial professionals, are prone to making irrational decisions. As pointed out by cognitive psychologists, humans typically suffer from loss aversion (we prefer to avoid losses than to achieve equivalent gains) and confirmation bias (we tend to confirm pre-existing beliefs with new evidence).
That’s the reason why this sector is hardly new to combine brokers’ gut feeling with the massive use of computers and statistics when it comes to investigating and predicting market trends. However, alongside traditional mathematical models and information technologies (including stock trading software), major financial players have increasingly implemented AI-based solutions to shed light on these mysterious dynamics.
Indeed, the role played by artificial intelligence is constantly growing so that, according to Gartner’s estimates, three-quarters of venture capitalists globally will take advantage of AI-based tools to make their decisions by 2025.
An algorithm-powered crystal ball
Despite the importance of artificial intelligence as a whole, the absolute protagonist of this new approach that could reshape the way of trading and invest is undoubtedly one of its newest branches, namely machine learning (ML).
Machine learning focuses on creating computer algorithms that can automatically improve their performance through experience. Specifically, ML algorithms can recognize patterns and relations among the data they are trained with, build mathematical models concerning such patterns, and use the same models to make predictions or decisions even without being explicitly programmed to do so.
The "learning" part in this technology's name is not there by chance. The more information ML-based systems can process, the more patterns will be detected and the aforementioned models will be polished. This operation allows algorithms to upgrade their analytical and forecasting performance with time.
Such capabilities, as you might imagine, are invaluable to financial firms, which are actually investing more and more in AI implementation and machine learning consulting. By delving into the depths of big data (including stock market trends, GDP growth, unemployment rate, corporate performance, news, consumer behavior, and social media information), ML-powered systems will pinpoint the cause-effect relationships between all these variables. Based on these findings, they may provide market players with useful insights and recommendations on future economic tendencies.
Feelings matter!
A particularly fascinating aspect of big data analytics, that is recently catching the attention of many shareholders, is the study of online financial news and social media sentiment. The assumption of this increasingly widespread approach is that taking into consideration purely economic variables is not enough to predict stock market trends, even by deploying cutting-edge technologies such as machine learning.
Instead, financial experts should leverage text analysis and natural language processing to identify the sentiment from sources like social media posts or financial news articles, i.e. to understand if such texts reflect a positive or negative opinion on specific financial matters.
These techniques have already been adopted by various financial giants. J.P. Morgan Research has created an ML solution tested on 100,000 news articles covering global equity markets, in order to assist experts in future equity investment decisions. Blackrock, instead, has leveraged text analysis techniques to predict future changes to company earnings guidance.
Supporting traders with forecasting
As you may have gathered, the most crucial application of machine learning in the stock market involves big data analysis to provide forecasts on financial trends and insights into trading and investing possibilities.
- Trading strategies: Betting on the right bonds, stocks and commodities is always a hard choice. Even more so when half the world is in lockdown, the Suez Canal is blocked by a ship larger than the town you live in, and people dressed as shamans storm Capitol Hill.
Combining data mining and ML algorithms, it's possible to create solutions that forecast stock fluctuations, volatility, and risks to recommend the most promising trading strategies. Such suggestions come from the analysis of numerous factors, including global financial trends, corporate earnings, market prices, and investors' sentiment. Of course, stock rankings aren't necessarily binding, as humans still have a final say on what action to take. This is a fundamental point, as we will explain later.
- Portfolio management: the same approach based on algorithms can be a game-changer when it comes to choosing the best investment options. ML-powered platforms and digital advisors can process gargantuan amounts of information, evaluate potential asset allocations, and help investors build a well-balanced portfolio.
It is worth noting that such tools are becoming more and more accessible, opening the doors of the stock market and offering financial advice at reasonable costs even to private investors. Such tips are highly customizable, as they are modeled not only on general economic trends but also on customers' specific characteristics and needs, such as their income and risk tolerance.
Does the seer deserve your coin?
An excellent indicator of ML’s forecasting potential lies in the interest aroused among scientists and top financial players, which are investing large sums to develop these technologies and, for some years now, have achieved very promising results.
In its Innovations in Finance with Machine Learning report, J.P. Morgan described an initiative focused on interest rate markets. An ML-powered system based on the random forest algorithm was fed with a wide variety of data collected from 2000 to 2016, including international interest rates and the Federal Reserve meetings' calendar. The aim was to suggest the timing and sizing of the 2017 trades.
The difference in terms of returns between buying bonds through conventional methods and following an ML-based approach was pretty impressive, as you may see from the graph below. The third and sixth bars show the performance of standard operations without ML, which served as a control. The first and fourth bars, on the other hand, indicate returns from short selling, and the second and fifth bars from both buying and selling with ML guidance.
Other encouraging data comes from research on bond default forecasts and AI-led hedge funds. Regarding the first, a team of Brazilian academics from the Universities of Brasília and Uberlândia discovered that ML-based predictive techniques are approximately 10% more accurate than traditional methods at estimating the likelihood of bond defaults.
As for the second point, a study found in the August 2020 Cerulli Edge Global edition showed that the cumulative return of AI-driven hedge fund trading from 2016 to 2019 was almost three times higher than that achieved by traditional hedge fund investments in the same period (33.9% vs 12.1%).
What if the prophet fails?
Not all prophecies come true. Otherwise, we would all have died several times, according to the numerous predictions about the world’s end. The same statement is valid for machine learning in the stock market.
Despite the massive potential of ML-based forecasting, this technology is far from being perfect. Once implemented in a real-world scenario, it may give rise to some unexpected distortions, as brilliantly pointed out by Harvard Business Review.
- Machines suffer from bias too: Although ML-based systems are unaffected by human emotions, the effectiveness of their discernment depends on the quality of the information they are trained with. Therefore, insufficiently representative datasets could lead to bias. For example, a machine could place greater emphasis on certain patterns and correlations while underestimating others that would be far more critical in shedding light on important financial trends.
That is why it is still essential to rely on data scientists and other qualified professionals to select the right data sources and integrate the judgment of machines with that of humans.
- The stock market could be too chaotic: Finance is like a rough sea, and even the most skilled captain, trained to interpret the movement of the stars and the breath of the winds, could end up lost once an unexpected storm hits. This holds true for algorithms, which build their models on historical data to predict financial trends but may struggle to face unprecedented scenarios, such as a global pandemic and its tremendous impact on the markets.
Moreover, stock exchange data is rather limited, as the financial market how we know it is relatively young and a good portion of human history (including some of the most cataclysmic events) is unknown to algorithms.
- A winning strategy cannot last if everyone follows it: "When you play the game of thrones, you win or you die." What about the game of trading? Well, it's not such a dramatic scenario, but still, someone wins and someone loses money. This rule could not be broken even if all traders relied on machine learning. Rather, we would be faced with a paradoxical situation in which every investment firm follows identical ML-prompted investment strategies, buys or sells the same equity securities, and ends up vaporizing any potential gain.
A similar episode, known as the Quant Meltdown, happened in 2007 when a bunch of relevant hedge funds guided by similar quantitative models sold their stock simultaneously and suffered massive losses.
Not only predictions: other uses of ML
Machine learning in the stock market is not just about predicting upcoming economic trends. We can also take advantage of algorithms to fight fraud and improve back-office activities.
Eradicating stock-market abuses
ML-powered tools can take on the role of detectives with the same zeal with which they play that of clairvoyants. Specifically, they may take advantage of their data processing capabilities to prevent or detect fraud, money laundering, and other financial crimes.
All they have to do is to cross-check transactions and traders' personal data to spot inconsistencies in the information provided or anomalies in their typical activity, which could be clues of potential misconduct. ML solutions can also adapt to new fraudulent behaviors breaking standard patterns, and consequently fasten reaction times of competent authorities when facing unprecedented threats.
Nowadays, this aspect of AI application in finance is more important than ever, given the increasingly strict trading regulations and the need for greater compliance in asset management. In this regard, machines can be a valuable ally, ensuring greater accuracy and, at the same time, lower operating costs.
Indeed, many major financial players have understood the potential of these technologies in fighting crime, including Nasdaq that is testing an ML-based surveillance system to deal with any attempts at trade manipulation. Its algorithms are leveraged to identify irregular patterns and send an alert to exchange officials.
“It [points] out what we call an ‘interesting event’. It’s not necessarily a prohibited activity, but it’s what the model has deemed to be interesting because it’s not normal market behavior.”
Enhancing back-office workflows
For every hyperactive trader who screams in the halls of the stock exchange to liquidate their positions, there is one (or more) white-collar who takes care of the paperwork. Indeed, the "hidden" side of trading is a massive amount of back-office procedures that represent the gears of the financial bureaucracy.
Think of all those activities related to administration and support services, but most of all to accounting, including financial and tax reporting. This last point is essential to ensure maximum transparency to customers and achieve proper audit and compliance with current regulations.
ML-powered systems can work side by side with other technologies (for instance, RPA in finance) to automate some of these tasks, such as recording transactions and generating the related documentation quickly, with almost absolute accuracy and in total autonomy.
The main difference between machine learning and other less advanced solutions is that the former doesn't just automate business processes but also makes them smarter. For example, an ML-based financial and accounting platform could autonomously understand how human operators work and replicate their actions, even adapting to different operational scenarios. Or it could record voice-based transactions and interact with employees to assist, thanks to ML-related technologies such as natural language processing.
Who will win the tech race?
Despite the generally growing adoption of machine learning in the stock market, by investigating this trend in detail we might unveil conflicting situations. On the one hand, major asset managers are leading the way with significant investments in ML implementation and financial software development, but also by securing top talent. On the other hand, small businesses may have a hard time keeping up.
Based on CFA Institute, indeed, only 10% of portfolio managers surveyed said they had adopted ML techniques in the previous 12 months.
Similar results were described by Statista in their report on the use of artificial intelligence for portfolio management in 2020, according to which only 6% of asset owners and investment managers surveyed declared applied the technology. Nevertheless, 76% of the respondents showed interest in the future adoption of AI for portfolio management.
No matter how things go, the race is on and the most forward-looking financial players may derive massive benefits from ML implementation. In this regard, machine learning has already proven to have several tricks up its sleeve, ensuring:
- Superior data analytics and trend predictions
- Portfolio and investment optimization
- Targeted actions against financial fraud
- Back-office workflow streamlining
- Enhanced access to wealth management via digital advisors
As we have already clarified, all of these advantages come along with some challenges. After all, ML is an amazing technology but it's not magic, despite previous simplifications involving clairvoyants and crystal balls. And even if we really wanted to see it as a crystal ball, we'd still need a wizard to look into it, just as we'll still need human financial experts to properly leverage ML solutions in the years to come.
Contact us
United States
720 207 2820Sales inquiries