Yandex presents a method for training large transformer recommenders with up to 1B parameters

In an era where personalization drives user engagement and customer satisfaction, traditional recommender systems are increasingly falling short. Many platforms still rely on shortened user interaction histories, which miss long-term preferences, and struggle to scale as interactions reach the billions.

Developed by Yandex, ARGUS (AutoRegressive Generative User Sequential modeling) is a novel, AI-powered architecture that enhances recommender systems by leveraging transformers — similar to those powering large language models (LLMs). The research paper describing this approach was recently published on arXiv, making Yandex one of the few companies — alongside Google, Netflix, and Meta — to successfully overcome significant technical challenges and achieve a breakthrough in recommender systems.

Addressing industry challenges

Recommender systems face persistent challenges, including limited ability to process long-term user histories, scalability issues, and struggles with evolving user preferences. Traditional systems often rely on truncated interaction data, missing nuanced behavioral patterns, and lack the flexibility to adapt to dynamic user needs or seasonal trends. These limitations result in less relevant recommendations, reduced user engagement, and the lack of opportunities for discovery.

The recommender transformers trained with ARGUS analyze a significantly longer history of user interactions — several times more extensive than before — and identify connections, including non-obvious ones, between them. These algorithms better detect how user needs evolve over time and account for seasonal patterns. For instance, if a user consistently buys tennis balls from the same brand every summer, the system will proactively remind them when the season approaches.

Key technical innovations behind ARGUS

ARGUS delivers a leap forward in recommendation quality by leveraging principles from large language models (LLMs), reinforcement learning (RL), and state-of-the-art transformers. The method employs several key innovations:

• Generative sequential modeling: ARGUS employs an autoregressive approach, analyzing entire anonymized user histories — including context and full-spectrum feedback (not just positive interactions) — to capture nuanced user intent and behavior.
• Multi-Headed learning objectives: The architecture is trained simultaneously on next-item prediction (capturing both logged policy imitation and user intent) and multi-type feedback prediction, enabling preference modeling and effective knowledge transfer.
• Scalability and efficiency: ARGUS was tested using transformer models of up to a billion parameters. Each increase in model size enhances showed consistent improvements in model performance, as measured by global temporal split and pairwise ranking accuracy (PairAccuracy).
• Unified embeddings and context awareness: Advanced embedding techniques handle item IDs and categorical features across billions of records, with periodically recalculated user and item representations supporting offline two-tower ranking and scalable deployment.
• Accelerated training and inference: ARGUS introduces innovations that enable processing year-long user histories in a single transformer pass, improving retraining and inference efficiency compared to impression-level models.

Nikolai Savushkin,
Head of Recommender Systems at Yandex

In recent years, the quality of recommender algorithms has plateaued. To push them to the next level, we needed to adopt generative models, which demand significant computational resources. We developed a neural architecture that is more efficient in training and requires fewer resources, proving that the leap in quality seen in language models is achievable in recommendations. Only a handful of companies worldwide — such as Yandex, Google, Netflix, and Meta — have solved this challenge. These algorithms, built on our new architecture, will be rolled out across most of the Yandex services.

Real-world impact and future applications

Early implementations of ARGUS across several Yandex services have demonstrated substantial improvements in total listening time (TLT), click likelihood, and “novelty/discovery” scenarios, further amplified by its integration as a feature in downstream ranking models.

ARGUS demonstrates that transformer architectures, traditionally used in natural language processing, are highly effective for recommender systems where long-term context is critical. Yandex is advancing large neural architectures, exploring runtime adoption of ARGUS, feature engineering for pairwise ranking, and extending the approach to high-cardinality, real-time recommender domains.

Kirill Khrylchenko,
RecSys R&D Team Lead at Yandex

Although developed by our Research Team, ARGUS isn't just a research experiment — it's a live technology already improving recommendations across Yandex services today. We are excited to share this knowledge and advance the field of recommender systems globally. Ultimately, it'll enhance personalization across numerous services we use every day, helping us find what we need faster and making our digital experiences more satisfying.

About Yandex

Yandex is a global technology company that builds intelligent products and services powered by machine learning. Its goal is to help consumers and businesses better navigate the online and offline world. Since 1997, Yandex has delivered world-class, locally relevant search and information services and developed market-leading on-demand transportation services, navigation products, and other mobile applications for millions of consumers worldwide.