Methodologies for Improving Modern Industrial Recommender Systems

Recommender system (RS) is an established technology with successful applications in social media, e-commerce, entertainment, and more. RSs are indeed key to the success of many popular APPs, such as YouTube, Tik Tok, Xiaohongshu, Bilibili, and others. This paper explores the methodology for improving modern industrial RSs. It is written for experienced RS engineers who are diligently working to improve their key performance indicators, such as retention and duration. The experiences shared in this paper have been tested in some real industrial RSs and are likely to be generalized to other RSs as well. Most contents in this paper are industry experience without publicly available references.

Introduction. This paper elaborates on the methodologies for improving industrial recommendation systems (RS). How does the industry measure whether an experiment makes the RS better or worse?

Discussion / Conclusion. This paper summarizes my understanding of how to improve an industrial Recommendation System (RS). At the early stage of developing an RS, efforts should be focused on improving models. Without reliable predictions of clicks and engagements, many of the methods introduced in this paper will not function effectively. Once the two-tower and I2I retrieval models, the three-tower pre-ranking model, and the wide&deep multi-task ranking model are established, the RS can be considered modern. After numerous updates, improving the pre-ranking and ranking models through feature engineering and enhanced neural network architectures becomes increasingly difficult. At this point, it is time to consider online learning and long-sequence modeling (e.g., SIM); these approaches will significantly increase the accuracy of the models’ predictions. Whether they will be implemented depends on the strength of your infrastructure and how you balance costs and returns. Eventually, particularly after the launch of online learning, it becomes incredibly challenging to further enhance the ranking and pre-ranking models.