Yandex and School of Data Analysis Release Open-Source Benchmark for AI Reasoning

Yandex engineers and students at the Yandex School of Data Analysis have developed and released MTGBench, an open-source benchmark that evaluates how well large language models reason through complex, multi-step tasks. It can help companies and developers compare reasoning models and choose those best suited to their needs. A research paper describing MTGBench has been published in Findings of EMNLP 2026.

People often refine their requests to AI assistants as a conversation unfolds. Someone might ask for a travel itinerary, then add dates and a budget in a follow-up. The model has to incorporate this new information while keeping track of what came before. Multi-turn games offer a way to test this ability: each move builds on previous ones. This is the approach behind MTGBench, which evaluates not only whether a model completes a task successfully but also, where applicable, how many steps it takes to reach a solution.

MTGBench includes seven text-based games played over multiple turns: Wordle, Word Chain, Situation Puzzles, 20 Questions, Wheel of Fortune, 20 Features, and Alias. They test whether a model can reason logically, use new information, plan its next moves, and remain consistent with its previous answers. Four games use algorithms to check the model's responses, while the other three rely on an AI judge whose accuracy was validated by human annotators. All seven games are available in both English and Russian, with tasks generated separately for each language.

The researchers tested MTGBench on 16 language models, ranging from smaller Qwen variants to larger Gemini and GPT models. The results showed that the benefits of reasoning depend heavily on the task. Step-by-step reasoning proved especially useful in games involving letters and symbols, interpreting clues, and eliminating incorrect options. Its benefits were less consistent in tasks requiring knowledge retrieval or semantic understanding, with results varying between models. Performance also differed considerably across games: a high score on one task did not necessarily translate into success on another. Models generally performed worse in Russian than in English.

The researchers also compared model performance with that of human participants playing the Russian-language games. People performed well on letter-based puzzles, but the strongest models with reasoning enabled matched or exceeded the collected human baseline across all seven games.

Researchers and developers can access MTGBench on GitHub to test and compare language models.