Brace for a defeat in chess matches against Transformers on Lichess platform
In a groundbreaking development, researchers have used transformer models to play chess, marking a significant shift away from traditional search-based algorithms. The models, which can have up to 270 million parameters, were trained on the extensive ChessBench dataset, consisting of 10 million annotated games by Stockfish, a leading chess engine.
During training, the transformer models were labeled with "bins" representing different levels of confidence or likelihood for moves. Each game position was also labeled with action-values, recommending moves based on the game's current position, and state-values, likely representing the outcomes of the game from that position. The models were trained using supervised learning.
The transformer models demonstrated a high level of chess play, achieving an Elo rating close to that of grandmasters (Lichess blitz Elo of 2895). However, they fell short of engines like Stockfish when making quick, tactical moves and struggled when playing non-standard chess games, like Fischer Random Chess.
Although the transformer models performed well, they only represent human play, possibly hindering further advancement without engine-derived datasets. The models were tested against well-known chess engines like AlphaZero and Leela Chess Zero, and although they almost matched these engines without using search during play, they still could not surpass them.
The transformers' performance drops significantly when playing non-standard chess games, indicating generalization to similar but non-identical scenarios remains a challenge. This study suggests that large transformers can handle planning problems without search algorithms, but they still lack the deep strategic "understanding" found in traditional AI strategies that employ adversarial search techniques like alpha-beta pruning for top-level planning.
The Difference between Large Transformers and Traditional AI
The planning approach in large transformers without search and memorization is based on pattern recognition and learned statistical associations over games. In contrast, traditional search-based AI, such as Alpha-Beta pruning engines, use explicit adversarial search with pruning, lookahead, and evaluation functions to deeply analyze tactical sequences and strategize several moves ahead.
The internal representation in large transformers is based on statistical embeddings and associations from training data, while traditional AI systems have explicit game states and computations of move consequences. Performance limitations in large transformers include plateaus below expert level, while traditional AI consistently performs near or surpasses human expert level with combinatorial search.
Moreover, transformers have shown unique mathematical shortcuts for other dynamic prediction tasks by aggregating information hierarchically rather than step-by-step simulation, reflecting a fundamentally different reasoning style than search-based AI. This suggests transformer chess models attempt to predict good moves via association rather than explicit planning, which can limit their strategic foresight compared to search-based algorithms.
Conclusion
While large-scale transformers exhibit some strategic reasoning in chess, they currently cannot match the strength or reliability of traditional AI systems based on adversarial search and pruning algorithms for top-level chess planning. This gap underscores the importance of explicit lookahead, memory of past states, and structured search in chess AI—capabilities that pure transformer-based approaches have yet to fully replicate. The study demonstrates that large-scale transformers can nearly match top chess engines without using search-based algorithms, but further advancements may require a combination of both approaches.
Artificial Intelligence, in the form of large transformer models, has shown significant progress in playing chess while ditching traditional search-based algorithms, but it fails to surpass the tactical capabilities of engines like Stockfish or perform well in non-standard games like Fischer Random Chess. Unlike traditional AI strategies that employ adversarial search techniques and deep analysis, large transformers base their planning approach on pattern recognition and statistical associations over games.
The internal representation in transformer-based AI varies from traditional AI systems, using statistical embeddings and associations from training data, while traditional AI systems have explicit game states and computations of move consequences. This difference in design and approach results in performance limitations in transformers, as they struggle to match the strength and reliability of traditional AI systems in top-level chess planning.
