Evaluating Chess Moves by Analysing Sentiments in Teaching Textbooks
No Thumbnail Available
Date
2025
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
the University of Manchester
Abstract
The rules of playing chess are simple to comprehend, and yet it is challenging to make accurate
decisions in the game. Hence, chess lends itself well to the development of an artificial
intelligence (AI) system that simulates real-life problems, such as in decision-making processes.
Learning chess strategies has been widely investigated, with most studies focused on
learning from previous games using search algorithms. Chess textbooks encapsulate grandmaster
knowledge, which explains playing strategies. This thesis investigates three research
questions on the possibility of unlocking hidden knowledge in chess teaching textbooks.
Firstly, we contribute to the chess domain with a new heterogeneous chess dataset “LEAP”,
consists of structured data that represents the environment “board state”, and unstructured
data that represent explanation of strategic moves. Additionally, we build a larger unstructured
synthetic chess dataset to improve large language models familiarity with the chess teaching
context. With the LEAP dataset, we examined the characteristics of chess teaching textbooks
and the challenges of using such a data source for training Natural Language (NL)-based
chess agent. We show by empirical experiments that following the common approach of
sentence-level evaluation of moves are not insightful.
Secondly, we observed that chess teaching textbooks are focused on explanation of the
move’s outcome for both players alongside discussing multiple moves in one sentence, which
confused the models in move evaluation. To address this, we introduce an auxiliary task by
using verb phrase-level to evaluate the individual moves. Furthermore, we show by empirical
experiments the usefulness of adopting the Aspect-based Sentiment Analysis (ABSA)
approach as an evaluation method of chess moves expressed in free-text. With this, we have
developed a fine-grained annotation and a small-scale dataset for the chess-ABSA domain
“ASSESS”. Finally we examined the performance of a fine-tuned LLM encoder model for
chess-ABSA and showed that the performance of the model for evaluating chess moves is
comparable to scores obtained from a chess engine, Stockfish.
Thirdly, we developed an instruction-based explanation framework, using prompt engineering
with zero-shot learning to generate an explanation text of the move outcome. The
framework also used a chess ABSA decoder model that uses an instructions format and evaluated
its performance on the ASSESS dataset, which shows an overall improvement performance.
Finally, we evaluate the performance of the framework and discuss the possibilities
and current challenges of generating large-scale unstructured data for the chess, and the effect
on the chess-ABSA decoder model.
Description
Keywords
Chess, ABSA, Sentiment Analysis, Large Language Models, LLMs, BERT, GPT, Text Generation, Move Evaluation