Recommendation systems and machine learning: driving personalization

Recommendation systems and machine learning: driving personalization

August 3, 2022

Andrea Di Stefano

Technology Research Analyst

Imagine having a really fashionable friend, we'll call her Nancy, working in a large clothing store and clearly hiding some sort of portal to Milan and Paris instead of Narnia in her wardrobe.

This lady literally watched you grow up and got an idea of your tastes, their evolution over the years and how your passions, friendships, and a thousand other factors have influenced them. Therefore, you decide to pay her a visit during her shift and rely on her wise advice and in-depth knowledge of the shop’s assortment to make over your wardrobe.

But after imagining all this, you remember it's 2022 and the pandemic radically changed our shopping habits. So, you opt for perusing the options online…like many other people nowadays, according to Statista.

Global ecommerce sales trend and forecast, 2014-2025

However, having a companion to help you out wouldn't be too bad. Well, machine learning-based recommendation systems may be a less chatty but very efficient alternative. Let's explore the nature and different design approaches of these tools and find out how, assisted by machine learning experts, retailers are leveraging them to redefine the shopping experience and drive sales.

What is a machine learning-based recommendation system?

Machine learning-based recommendation systems are powerful engines using machine learning algorithms to segment customers based on their user data and behavioral patterns (such as purchase and browsing history, likes, or reviews) and target them with personalized product and content suggestions.

The role of machine learning

To be fair, basically everyone, including non-tech-savvy users, has heard of recommender systems and has a very rough idea of their basic functioning, even for a simple matter of personal experience. What's not so well known, however, is the role of machine learning in their underlying mechanisms.

Machine learning (ML) is a sub-branch of artificial intelligence focusing on the creation of computer algorithms that can process huge datasets, identify recurring patterns and correlations among multiple variables, and build mathematical models portraying them.

That "learning" in its name is not there by chance, since machine learning systems as well as reinforcement learning applications leveraging these algorithms can actually enhance their capabilities over time through experience. The more data they process, the more relationships among data points they'll spot, and the better they will fine-tune their models.

Such models, typically used for predictive analytics in marketing as well, represent a window into what customers think, as they allow us to frame the logic behind specific purchase patterns and shed light on current sales trends or even predict future developments. For example, a machine learning-powered solution might notice a recurring connection between the age of customers and their preference for one brand over another.

Machine learning modeling

How ML-powered recommendation systems work

To clarify how ML-based recommender systems actually perform their duties, let's go back to the fashion connoisseur example. Your friend Nancy can boast solid expertise in both the latest fashion trends and the shop's product range. Furthermore, she's already framed your tastes as she perfectly knows what you usually buy and wear. Not to mention a long list of personal aspects that may have impacted your style (cultural interests, social environment, profession, and more).

But not all shopkeepers have this privilege. Many sellers see you for the first time as you enter their store. Therefore, they'll need to interact with you, gather information, understand what type of customer you are, and recommend merchandise matching your preferences. In marketing terms, they have to segment you, namely categorize you into a certain customer archetype or buyer persona according to your characteristics (purchase patterns, interests, gender, etc.) and target you with a suitable product suggestion reflecting your archetype.

Market segmentation variables

In virtual marketplaces, recommendation systems play a similar role, replacing sales assistants in segmenting and recommending products to customers. The major difference is that human sellers are driven by their intuition and experience to investigate a small fraction of the aforementioned variables during a short chat with purchasers.

Recommendation engines, instead, rely on machine learning to process huge customer datasets and consider a broader range of parameters to perform this classification and targeting process. These include browsing behavior, purchase history, content usage, personal information from user profiles, product reviews, and access devices.

Multiply this for each user on a given platform and you'll understand how a recommendation system can get a pretty clear idea of both individual purchasers and the audience as a whole, as well as figuring out underlying sales dynamics that a human observer would struggle to grasp.

Furthermore, machine learning algorithms can take into account a massive range of purely contextual parameters not strictly related to customers. For example, as December approaches, the ML-based recommendation engines of a major web store would start recommending typical Christmas products. On the other hand, a streaming platform may adapt its recommendations from the day of the week, offering family-friendly films and documentaries over the weekend.

Consult our experts to implement machine learning the right way

Turn to Itransition

Recommendation system market trends

The range of capabilities of machine learning-based recommendation systems certainly represents one of the key catalysts driving their adoption across industries. According to Mordor Intelligence's 2021 Recommendation Engine Market report, these benefits and opportunities will also lead to a sharp growth experienced by the global recommendation system market in the years to come (from $2.12 billion in 2020 to $15.13 billion by 2026).

Global recommendation engine market forecast

In their 2021 Recommendation Engine Market Size, Share & Trends Analysis Report, Grand View Research's analysts confirmed these optimistic predictions, pointing out that the retail industry accounted for the largest revenue share in 2020. The study also provided further insights regarding recommendation systems' market share by type, reporting that collaborative filtering-based engines still ranked first while the hybrid system segment seems set to expand at the highest CAGR. However, we'll detail such categories, along with their pros and cons, a bit later.

Top recommendation systems on the internet

Nowadays, all major digital service providers and ecommerce enterprises rely on recommendation systems to deliver a customized user experience and enhance sales performance or advertising revenues:

  • Amazon leverages a recommendation algorithm to recommend products or search results, combining in-site suggestions based on several strategies (recommended for you, bought together, recently viewed, etc.) with off-site recommendations via email. The ecommerce leader deployed its collaborative filtering-based recommender engine between 2011 and 2012, recording an outstanding 29% sales increase in the second fiscal quarter of 2012.
  • YouTube implemented a recommendation system to prioritize certain videos, suggest channel subscriptions, and provide relevant news. The engine takes into account a variety of parameters, defined as "signals" to better frame user interests, including clicks, likes and dislikes, watch time, and shares.
  • Facebook uses a recommendation engine based on deep learning and neural networks (known as DLRM or Deep-learning Recommendation Model) for friend suggestions and News Feed sorting, but also to recommend groups, pages you may be interested in, or products on its Marketplace.
  • Netflix relies on a recommendation system to provide movie recommendations. Its algorithms consider variables such as user features (including browsing history and ratings issued), movie type and popularity, seasonal trends, and item-item similarity with previous content to sort the groups of movies displayed in horizontal rows on its home page. 
  • LinkedIn deployed a recommendation system to suggest job ads, connections, and courses. One of its core applications is LinkedIn Recruiter, a powerful HR tool capable of compiling lists of suitable candidates for an open position and ranking them depending on their skills, experience, and likelihood of a response.

ML-based recommendation system benefits

Following the transition from brick-and-mortar sales to ecommerce, coupled with the spread of online platforms ensuring remote access to digital content, retail corporations and service providers have relied on machine learning-powered recommendation systems and other big data ecommerce solutions to achieve five essential goals:

  • Better user experience: Recommendation systems help replicate the in-store customer care and personalized shopping experience, offered by a real salesperson who provides an undecided purchaser with expert guidance, in a virtual environment.
  • Focus on the right product: Recommender systems mitigate the so-called information overload as they direct customers towards the product (be it physical or digital) they really want, hidden amid an overwhelming offer of merchandise and content.
  • Sales drive: Personalizing the shopping experience and highlighting relevant products result in a higher number of items per order, superior average order value, and enhanced customer lifetime value.
  • Data-driven decision-making: recommendation systems gather customer and sales data and compile detailed reports, providing managers with valuable insights to enhance their decision-making in terms of marketing, logistics, and pricing strategies.
  • Revenue growth: As a consequence of the previous points, recommendation systems can act as powerful revenue boosters. In this regard, McKinsey's 2019 The Future of Personalization article highlighted that product recommendation solutions may help improve marketing-spend efficiency by 10-30% and increase revenues by 5-15%.

Approaches to recommendation system design

Most recommendation engines fall into three major sub-categories, depending on the approach embraced to select and recommend products or services meeting each customer's needs:

  • Recommendation systems adopting collaborative filtering
  • Recommendation systems leveraging content-based filtering
  • Hybrid recommendation systems
Recommendation system approaches

We’ll start with the most popular type on the market, which is collaborative filtering.

Collaborative filtering: a focus on user similarity

How it works: "Show me what people similar to you buy, and I will tell you what you may like". That's a good approximation of collaborative filtering, a recommendation approach grouping users with shared characteristics and purchase patterns into clusters and providing them with product suggestions based on their similarity.

The focus, in this case, is on customers, their opinions on products, and their interactions with the online platform, rather than on the items' features. This implies that recommender systems in this category will rely on machine learning algorithms (such as clustering models, K-nearest neighbors, matrix factorization, and Bayesian networks) to survey customers’ perception of products via user rating, understand who likes what, and offer items already bought by other users with comparable tastes.

Collaborative filtering

As we've previously said, such analyses require huge datasets to properly train machine learning algorithms. There are two possible ways of gathering this data:

  • Explicit data collection: asking users to compile a list of favorite items and rate previously purchased products on a scale or from the most favorite to the least favorite.
  • Implicit data collection: scanning user activity on AI-driven social media, such as likes and dislikes, or monitoring customer purchases, views, and viewing times on ecommerce websites.

The pros: Collaborative filtering systems can be extremely accurate. In fact, to further expand the range of parameters considered and better measure user similarity, many engines are now designed to monitor other context-related variables, as we've previously mentioned in our overview of machine learning applications. This approach, known as context-aware collaborative filtering, complements user data with contextual information (region, time, device, etc.) to accurately define the scenario in which a customer operates and provide more effective suggestions.

Excellent accuracy, however, is not the only strength of collaborative filtering. Another relevant benefit is that these systems can literally predict users' interest in a product they didn't know existed by observing that the same item has caught the attention of other customers with similar interests.

Furthermore, by taking into account the relationship between products and the audience more than the product itself, recommendation engines based on this approach can work pretty well even without understanding the nature of each item. This also implies that platforms implementing such systems won't need to accurately describe each product in order to make the system operate properly and boost sales. For platforms offering a massive range of products such as Amazon, this represents a significant plus.

The cons: Among the potential drawbacks of collaborative filtering, we might mention:

  • Cold start: providing valuable suggestions to new customers with no purchase history can be challenging even considering several other parameters.
  • Scalability: using machine learning algorithms to search for purchase patterns among a constantly growing number of customers and products requires huge computational power.
  • Rich-get-richer effect: algorithms generally recommend products with many excellent reviews, further increasing their popularity at the expense of new items on the market.
  • Data sparsity: the tremendous offer of content and merchandise on all major platforms risks to reduce recommendations' accuracy, since each product may receive an insignificant number of user reviews.
  • Shilling attacks: because of the previous point, new products are easily vulnerable to rating manipulations (such as negative reviews from competitors).

Content-based filtering: a focus on product similarity

How it works: While being less popular than collaborative filtering, content-based filtering still has a couple of tricks up its sleeve. Here, we have a quite different approach to recommendation system design, since the focus partially shifts from customer similarity to product similarity. In fact, this model mainly considers the item's characteristics, such as price, category, and other features defined by assigning specific keywords and tags, along with user preferences interpreted from their purchases and related feedback.

Based on these metrics, a machine learning algorithm (be it Bayesian classifiers, decision trees, clustering, etc.) will investigate customers' purchase patterns and recommend other products sharing similar features with those previously bought and positively reviewed.

The most advanced content-based recommendation systems can also enrich this set of variables by probing text reviews through AI-based natural language processing. This type of textual feedback represents a valuable source of implicit data both in terms of item features (which may be mentioned by reviewers) and user rating (extracted via sentiment analysis from the same reviews).

Content-based filtering

The pros: The key plus of this approach over collaborative filtering is that it mitigates the aforementioned problems with new products on sale, as the system can already count on a good deal of information regarding each item's features thanks to the assigned keywords.

The cons: the tagging procedure implies a massive workload, especially on the largest platforms. Second, the cold start issue is still there, as the historical data associated with new customers is very limited. Furthermore, algorithms may show a rather conservative, risk-free behavior, recommending categories of products and content already purchased by a certain user while avoiding new, potentially interesting items.

Regarding the picture above, for example, our Leopold may also be interested in hair gel or mustache wax like many other bearded hipsters, even if they do not strictly fall into the category of products ordered before. To spot such nuanced, more subtle relationships, the collaborative approach has proven far superior.

Hybrid systems: the best of both worlds

How it works: To find a reasonable compromise between collaborative and content-based filtering aimed at maximizing the respective pros and minimizing their cons, many recommendation systems have embraced a hybrid approach.

There are several ways to hybridize these two recommendation system types. The mixed hybridization technique involves providing users with both collaborative and content-based suggestions at the same time. The weighted technique, on the other hand, merges the score calculated via two different approaches. Another combination trick, namely meta-level, implies using the output of the first approach (basically the machine learning model built by algorithms) as an input source for the second one.

Meta-level technique

The pros: According to research, hybrid models significantly enhance recommendation systems' performance. This may explain why hybrid systems are the fastest growing segment in the market, as indicated by Grand View Research's aforementioned study.

Platforms like Netflix or Spotify, for example, have already adopted a combination of both models to spot similarities among several users through collaborative filtering while identifying movies and songs with the same characteristics via content-based filtering.

The cons: It goes without saying that merging the mechanisms of both approaches into a single system requires more complex architectures and superior computing power.

Rely on Itransition’s machine learning solutions to reimage your shopping experiences

Machine leaning consulting

Machine leaning consulting

Our consultants will identify retail processes and tasks that would benefit from ML-based automation and aid with planning and building a suitable solution.

ML-based recommender engines implementation guidelines

ML-based recommender engines are complex tools with intricate architectures, and reaping the benefits of their adoption without knowing how to properly implement them may prove challenging. Here are some factors to keep in mind:

  • Approaches: The aforementioned design approaches determine recommendation systems' underlying mechanics and the variables to be prioritized. The choice depends on your business scenario, including your target audience and product range. 
  • User-driven strategies: The most effective recommender systems can switch between different recommendation strategies depending on the customer journey stage. For example, first-time users may be directed towards the "most popular" products, as we still can't frame their interests, while visitors with a long purchase history can be targeted with tailored recommendations based on user affinity with certain categories of merchandise.
  • Page context-driven strategies: The above holds true also for different page contexts, which may require specific recommendation strategies. A standard choice for homepages is the “most popular” strategy, while recommendations based on “similar products” can work great for product detail pages. Regarding cart pages, instead, it's worth embracing a “bought together” strategy.
  • Ready-made solutions: Nothing prevents you from creating your own, fully tailored recommendation system if you're ready to face higher upfront costs for its development. However, you can also opt for off-the-shelf solutions such as Evergage, Adobe Target, Optimizely, IBM Watson Real-Time Personalization, and many more.

Sales booster or threat to privacy?

Recommendation systems offer a fairly balanced synthesis between customers' wishes for a fully customized shopping experience and the need for retail companies and digital service providers to improve their sales performance in an increasingly competitive market. As highlighted by McKinsey, AI and machine learning in ecommerce and retail as a whole have further fostered this synergy, allowing enterprises to fully personalize customer journeys end-to-end.

On the flip side, these technologies may easily create tension between sheer performance and the growing interest of public opinion and legislators for privacy and data protection issues, since machine learning-powered systems require huge amounts of customer data to work properly.

Not to mention that these systems are generally considered as black boxes, which means they're great at interpreting data to identify relationships and patterns, but no one really knows how they end up providing a certain recommendation. In short, if we asked our  usually fashionable friend Nancy to explain her recommendations, her answer would probably reflect our behavioral patterns, style, and other aspects of our life that she knows very well. A machine learning system, instead, would randomly bip.

This isn't necessarily a big deal, as we can still enjoy life without talking to a machine. But the other issues will surely need to be addressed with proper data management policies to ensure that recommendation systems deliver their consistent benefits without endangering user privacy.