craiget.com

Extracting Chess Positions from Screenshots

Sun, 26 Jan 2025 00:00:00 +0000

Introduction

I play a lot of online chess.

Quite badly, according to my ELO.

Recently, I became interested in writing a computer program to automatically extract positions from screenshots of chess boards.

Why bother? I dunno, it seems interesting and it's outside my normal area of expertise, which is web programming.

In this post, I will describe how to do that.

This is a pet project, but feedback is welcome!

Ground Rules

I started with a fairly specific idea in mind:

The input should be a screenshot of a top-down, 2D board
The output should be a FEN string (the position as text)
It should work with different icon sets (pieces and boards)
It should be stateless (no move history)
It should use a neural net for piece classification

Basically, using python, we want this:

def extract_fen(screenshot: Image) -> str:
    # do image stuff

The "stateless" criteria means we won't track castling, en-passant, or the turn-number. Those are key concerns for generating useful game analysis! But we'll ignore them for now.

The program will run a single step, capturing one board from one image. But of course, if you have lots of images, you could wrap the logic in a while-loop and capture the entire game.

The Approach

In pseudo-code, we want something like this:

board = array(64)
image = open("screenshot.png")
for i in range(1, 64):
    square = crop(image, i)
    board[i] = classify(square)
result = generate_fen(board)

That is to say:

open the image file of the board
for each of the 64 squares..
    crop, to slice out an individual square
    classify its contents (either a specific piece, or "empty")
    accumulate the results into an array
convert the array into a FEN string

We'll develop each part in detail, but basically:

cropping can be done using any image library
classification will use a neural net, and we'll need to train it
FEN strings can be generated from the python-chess library

Setting up the environment

I am currently using uv to manage python environments. But the setup will be similar with most package managers. We will need:

tensorflow
tensorflow-macos
numpy
pillow
albumentations
scikit-learn
chess
jupyter (optional)
cairosvg (optional)

I had to fiddle with the installation a bit to satisfy my older Macbook M1. It wanted the ARM version of python for tensorflow, and numpy 1.x, instead of 2.x (YMMV)

uv init -p cpython-3.10.16-macos-aarch64-none
uv add chess pillow scikit-learn jupyter cairosvg albumentations
uv pip install tensorflow tensorflow-macos
uv pip install 'numpy<2'

Once complete, you should be able to import tensorflow

import tensorflow as tf

Cool.

Getting the images

The next thing we're gonna need is some images of chess boards.

Specifically, we need an image that looks like this:

The graphical style of the board and icons aren't important. But the position of the pieces is critical. We'll use this image for our training data. We need:

a normal starting position, but..
2 extra queens (d2 and d7)
2 extra kings (e2 and e7)

Pawns on b2, c2, f2, g2, b7, c7, f7, g7 are optional. You can keep or remove them. The code below will work the same.

Why do we need the extra kings and queens? The neural net will learn better if we show it an example of every possible piece and square combination. The normal starting position lacks 4 possibilities:

white king on a light square
white queen on a dark square
black king on a dark square
black queen on a light square

By creating an image that includes those, we can supply complete training data.

There are lots of ways to create the image above. You can..

download it
use python to create a PNG file
use a website and capture a screenshot

To generate the image using the python-chess package, you need to have the optional cairosvg package installed.

Then run:

import os
import chess
import chess.svg
import cairosvg

fen = "rnbqkbnr/p2qk2p/8/8/8/8/P2QK2P/RNBQKBNR w KQ - 0 1"
board = chess.Board(fen)
svg_data = chess.svg.board(board, coordinates=False, size=256)
with open("board.svg", "w") as f:
    f.write(svg_data)
cairosvg.svg2png(url="board.svg", write_to="board.png")

Alternatively, you can generate the image by creating an "analysis board" on any of the popular chess websites like chess.com or lichess.org.

Then take a screenshot.

It needs to be a GOOD screenshot!

256x256 or larger
square (or close, +/- 5px)
well-clipped (just the board, no borders)

If the screenshot is really bad, these image processing steps won't work well. Use an image editor if necessary to get a clean image.

Extracting individual squares

The next step is to start processing the image. We need the ability to extract each individual square from an 8x8 chess board.

There are many image manipulation libraries, but Pillow (PIL) is widely used and I like it.

Let's import libraries and declare the size of our board:

import math
import numpy as np
from typing import List
import albumentations as A
import tensorflow as tf
from PIL import Image, ImageDraw, ImageFont
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.preprocessing.image import ImageDataGenerator, img_to_array
from tensorflow.keras.optimizers import Adam
from sklearn.model_selection import train_test_split

BOARD_SIZE = 256
SQUARE_SIZE = 32

Next, load the screenshot and resize it to the target size.

board = PIL.open('path/to/my_screenshot.png')
board = board.resize((BOARD_SIZE, BOARD_SIZE))

Then, define a function to extract the contents of an individual square.

def get_square(board: Image, pos: str) -> Image:
    file, rank = pos[0].lower(), pos[1]
    file_num = ord(file) - ord('a')
    rank_num = 8 - int(rank)
    left = file_num * SQUARE_SIZE
    upper = rank_num * SQUARE_SIZE
    box = (left, upper, left + SQUARE_SIZE, upper + SQUARE_SIZE)
    return board.crop(box)

Our function will use chess rank and file notation to refer to the squares. For example, to see the contents of square "a1", the bottom-left corner, we would use:

a1_img = get_square(board, 'a1')
a1_img.show()

And we should see this:

Now we can loop through the squares of the board, and process each square separately. By placing the ranks in an outer loop, we can enumerate the positions in a natural order, starting from the top-left corner.

def extract_board(board: Image) -> List[List[str]]
    result = []
    for rank in '87654321':
        row = []
        for file in 'abcdefgh':
            pos = file + rank
            img = get_square(board, pos)
            piece = predict_piece(img)
            row.append(piece)
        result.append(row)
    return result

All that's left is to implement the predict_piece() function, which so far remains undefined.

Preparing the data

To write that function, we will train a neutral net to look at a square and predict the most likely piece.

Let's start by defining our labels. We'll use the notation of the python-chess library, with uppercase letters for white, lowercase for black, and "." for an empty square.

LABELS = ['.', 'P', 'N', 'B', 'R', 'Q', 'K', 'p', 'n', 'b', 'r', 'q', 'k']

The task for our network will be to process an image, and decide which of those 13 labels the image represents. The neural network doesn't actually learn the labels themselves (P, N, B, etc). Rather, it learns the integer offsets of the labels. If you ask the neural net what piece is on e1, will respond "6" instead "K", because "K" is the 7th element of the zero-indexed array.

We need one example for each type of piece. We can define that using a lookup table.

# (piece, light_sq, dark_sq)
LOOKUP = (
    ('.', 'b3', 'a3'),
    ('p', 'h7', 'a7'),
    ('n', 'g8', 'b8'),
    ('b', 'c8', 'f8'),
    ('r', 'a8', 'h8'),
    ('q', 'd7', 'd8'),
    ('k', 'e8', 'e7'),
    ('P', 'a2', 'h2'),
    ('N', 'b1', 'g1'),
    ('B', 'f1', 'c1'),
    ('R', 'h1', 'a1'),
    ('Q', 'd1', 'd2'),
    ('K', 'e2', 'e1'),
)

Each entry in this table states where the corresponding piece may be found in our screenshot.

For example, this line ('b', 'c8', 'f8') means:

one black bishop is on c8 (a light square)
one black bishop is on f8 (a dark square)

The dot "." indicates an empty square. We need to have both a light and dark square example for each piece, including the kings and queens.

So far we have 26 labeled images. But to train a network, ideally we need hundreds or thousands of examples. Capturing that many screenshots would be impractical. Instead, we can generate synthetic data using a library.

One such library is Albumentations, which can be used to make small adjustments to each image, for example by adjusting the contrast, shifting the pixels up or down, or rotating the whole image. Training the neutral net with a diverse set of images prevents over-fitting (memorizing the data) and makes it more robust in real usage.

transform = A.Compose([
    A.Rotate(limit=10, p=0.8),
    A.RandomSizedCrop(min_max_height=(28, 34), size=(32, 32), p=0.8),
    A.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.8),
    A.ShiftScaleRotate(shift_limit=0.2, scale_limit=0.2, rotate_limit=0.0, p=0.8),
    A.HorizontalFlip(p=0.5),
])

After defining our transformation parameters, we convert each image to grayscale, and then repeatedly call the transform() function to apply some random adjustments to the image. This gives us 5000 samples per piece.

num_images = 5000
label_list = []
image_list = []
for piece, light_sq, dark_sq in LOOKUP:
    class_num = LABELS.index(piece)
    for pos in (light_sq, dark_sq):
        img = get_square(board, pos)
        img = img.convert('L')
        img_array = img_to_array(img) / 255.0

        k = 0
        while k < num_images:
            augmented = transform(image=img_array)
            new_img = augmented['image']
            image_list.append(new_img)
            label_list.append(class_num)
            k = k + 1

Training the model

The next step is to convert the data into the required format, and then split into testing and training sets. Tensorflow wants to receive numpy arrays for its inputs.

image_arr = np.array(image_list)

label_arr = np.array(label_list)
label_arr = to_categorical(label_list, num_classes=len(LABELS))

X_train, X_test, y_train, y_test = train_test_split(image_arr, label_arr, test_size=0.2)

For the network itself, we will use a Convolutional Neural Network (CNN) model.

inputs = Input(shape=(32, 32, 1))
x = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
x = MaxPooling2D((2, 2))(x)
x = Conv2D(64, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2))(x)
x = Flatten()(x)
x = Dense(32, activation='relu')(x)
x = Dropout(0.2)(x)
outputs = Dense(len(LABELS), activation='softmax')(x)
model = Model(inputs=inputs, outputs=outputs)

model.compile(optimizer=Adam(learning_rate=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])

The most important parts of this are the input and output

The input shape (32, 32, 1) describes our 32x32 pixel, 1-channel grayscale images
The output shape, Dense(len(LABELS)) will give us one output per label

Then we will fit the model to the data. This is the famous "training" step.

model.fit(X_train, y_train, epochs=10, shuffle=True)

You'll see a bunch of output. On my computer it takes 5-10 minutes to run.

So uh.. did it actually work?

Let's find out by checking the accuracy of the model on our "test data" which was intentionally omitted from the training.

test_loss, test_accuracy = model.evaluate(X_test, y_test)
print(f"Test Loss: {test_loss}")
print(f"Test Accuracy: {test_accuracy}")

# Test Loss: 0.05167254060506821
# Test Accuracy: 0.9879999756813049

98.8% accuracy is pretty good (maybe even "suspiciously good" after just a few epochs)

However, we are classifying 64 squares each move, so this error rate means we'll still make a mistake every 2-3 turns.

In practice, this won't be a big problem. As we'll see in a moment, the transformations that we performed on the synthetic data are aggressive. A real game screenshot won't have the rooks rotated 20 degrees and the knights facing backwards. The end notes also suggest a few alternative ways to improve the accuracy beyond "more data".

Let's spot-check a few squares.

We want a function that accepts an image and returns the piece.

The predict_piece() function first converts an input image to the numpy format required by Tensorflow. Then it calls model.predict() and the result is a list of class probabilities. The function then selects the index of the class with the highest probability, and finally it returns the associated label (the name of the piece)

def predict_piece(model: Model, labels: List[str], image_sq: Image.Image) -> str:
    image_sq = image_sq.convert('L')
    image_arr = img_to_array(image_sq) / 255.0
    image_arr = np.expand_dims(image_arr, axis=0)
    predictions = model.predict(image_arr, verbose=False)
    offset = np.argmax(predictions)
    return labels[offset]

We can now check that our original screenshot has a rook on "a1"

image_sq = get_square(board, 'a1')
piece = predict_piece(model, LABELS, image_sq)
print(piece)

# "R"

And it does! Nice!

Putting it all together

Okay, prediction appears to work.

Performance can be tuned by adjusting model parameters such as the layer sizes, the quantity and variety of training data, and the number of training epochs.

Earlier, we wrote an extract_board() function. Now that we also have implemented predict_piece(), we can put this to use for the entire board.

def extract_board(model: Model, labels: List[str], board: Image.Image) -> List[List[str]]:
    result = []
    for rank in '87654321':
        row = []
        for file in 'abcdefgh':
            pos = file + rank
            img = get_square(board, pos)
            piece = predict_piece(model, labels, img)
            row.append(piece)
        result.append(row)
    return result

Cool. Are we done?

Let's try it on a brand new screenshot.

This was taken from the Opera Game after move 11.. bd7

To make it more fun, I edited that image in Photoshop to blur and shift some of the pieces, so that it wouldn't be too easy.

opera_board = Image.open('path/to/opera-game.png')
extract_board(model, LABELS, opera_board)

The output is a double-nested list. We can see that each piece matches the screenshot. So it works!

[['r', '.', '.', '.', 'k', 'b', '.', 'r'],
 ['p', '.', '.', 'n', 'q', 'p', 'p', 'p'],
 ['.', '.', '.', '.', '.', 'n', '.', '.'],
 ['.', 'B', '.', '.', 'p', '.', 'B', '.'],
 ['.', '.', '.', '.', 'P', '.', '.', '.'],
 ['.', 'Q', '.', '.', '.', '.', '.', '.'],
 ['P', 'P', 'P', '.', '.', 'P', 'P', 'P'],
 ['R', '.', '.', '.', 'K', '.', '.', 'R']]

Finally, we need a way to convert a board into a FEN string.

Below is a code snippet to build a representation of the board using the popular python-chess module and then calling board.fen()

def to_fen(board_array):
    board = chess.Board(None)
    for row in range(8):
        for col in range(8):
            piece = board_array[row][col]
            if piece != '.':
                color = chess.WHITE if piece.isupper() else chess.BLACK
                piece_type = {
                    'p': chess.PAWN,
                    'r': chess.ROOK,
                    'n': chess.KNIGHT,
                    'b': chess.BISHOP,
                    'q': chess.QUEEN,
                    'k': chess.KING
                }[piece.lower()]
                board.set_piece_at(chess.square(col, 7 - row), chess.Piece(piece_type, color))
    return board.fen()

This FEN string is something that we'll need in later stages of the pipeline. It can also be copy/pasted into chess engines and analysis programs, to help double-check the accuracy of the piece classification.

Let's wrap this up into the single function promised at the beginning:

def extract_fen(screenshot: Image.Image) -> str:
    board = screenshot.resize((256, 256))
    board_arr = extract_board(model, LABELS, board)
    return to_fen(board_arr)

That's it! We can extract the position from any screenshot (well, any screenshot that uses same set of icons as our original)

Debugging

It was very useful to visualize the pieces, labels and classification errors while writing this post.

The function below draws a grid of pieces. It can also optionally draw a red overlay to highlight classification errors.

def visualize(x, y, width=16, height=None, predict=False):
    height = height if height else math.ceil(len(x) / width)
    size = width * height
    y_orig = np.argmax(y, axis=1)
    labels = [LABELS[i] for i in y_orig]

    sheet = Image.new('RGBA', (SQUARE_SIZE * width, SQUARE_SIZE * height))
    mask = Image.new('RGBA', (SQUARE_SIZE * width, SQUARE_SIZE * height))
    text = Image.new('RGBA', (SQUARE_SIZE * width, SQUARE_SIZE * height))
    draw = ImageDraw.Draw(text)
    font = ImageFont.load_default()

    def draw_piece(x, i):
        img = np.squeeze(x[i])
        img = Image.fromarray(img * 255.0)
        pos = (SQUARE_SIZE * (i % width), SQUARE_SIZE * (i // width))
        sheet.paste(img, pos)

    if predict:
        y_pred = model.predict(x)
        y_pred_labels = np.argmax(y_pred, axis=1)
        pred_labels = [LABELS[i] for i in y_pred_labels]

        for i in range(size):
            draw_piece(x, i)
            pos = (SQUARE_SIZE * (i % width), SQUARE_SIZE * (i // width))
            pred_label = pred_labels[i]
            orig_label = labels[i]
            draw.text(pos, pred_label, font=font, fill=(0, 0, 0, 200))
            color = (255, 0, 0, 128) if orig_label != pred_label else (0, 0, 0, 0)
            mask.paste(Image.new('RGBA', (SQUARE_SIZE, SQUARE_SIZE), color), pos)
    else:
        for i in range(size):
            draw_piece(x, i)
            pos = (SQUARE_SIZE * (i % width), SQUARE_SIZE * (i // width))
            label = labels[i]
            draw.text(pos, label, font=font, fill=(0, 0, 0, 200))

    sheet = Image.alpha_composite(sheet, mask)
    sheet = Image.alpha_composite(sheet, text)
    return sheet

I used this for several things while debugging the code.

It was worth it to spot-check the training images and labels. It's super-easy to make array indexing errors with numpy. On several occasions, I had messed up and the inputs were mislabeled (which makes training impossible).

It was also nice to see the classification errors visually.

Below is an example of some of the "synthetic" images from the testing data.

visualize(X_test, y_test, width=16, height=16, predict=True).show()

Pretty cool looking..! Right?

Also, you can see that two rooks are misclassified.

A bit later, I realized that I could also visualize "ordered" results for specific transformations, such as translation or rotation. The next image shows each piece rotated from 0-50 degrees in both directions.

(These particular images were generated from a separate process and don't appear in the training or testing data)

We can see that the unrotated images are all classified correctly. But as the angle of rotation increases, we get more mistakes. A rotated pawn starts to look like a bishop, and rook classification drops of pretty quickly.

The overall "shape" of these errors has some explanatory power.

If there were zero errors, that might hint at memorization or pollution of training data into the "test" split. If we saw totally random errors, that could suggest insufficient data or model size. Mistakes like one specific piece being totally incorrect, could mean we have a class imbalance or other issues with the data processing pipeline. I know because all of those problems happened while I was debugging!

Next Steps and What Abouts..

So this basically works and I'm pretty happy with it.

There's lots that could be done to make it more robust. Also, my own code is organized a bit differently than what I've shown here. I wanted to present the snippets in a logical order, but my actual code is more modularized and organized into class files, to make it easier to experiment with multiple boards.

My eventual goal is to develop a program that can automatically capture, analyze, and provide commentary for live, in-progress games. We'll see how far I get with it! Hopefully there will be more posts on that topic soon.

Below are some FAQ style "what about.." questions covering issues I ran into, and options that I didn't pursue yet. After playing around with this a bit, I've got a million new questions and things to try.

Some of the questions below mention improving the accuracy. At 99%, why bother? Well, the testing we did here was totally synthetic and omitted a lot of problems that can happen with live screen capture - we made it pretty easy for the classifier! I am considering some future ideas that are less easy, so just thinking ahead a bit..

What about cheating?

We extracted positions from screenshots and output FEN, which happens to be the language of Stockfish.

Could you extend this program to cheat with Stockfish? Yes, you could, but uh.. please don't do that..

Cheating is a real problem for online chess.

That said, there are MUCH easier ways to cheat, like.. using any engine.. opening an analysis board in a private window.. or just asking a stronger player to stand over your shoulder and tell you the best moves.

I don't think I'm enabling or encouraging cheating here.

I also don't think a lot of cheaters are gonna bother with building an image recognition model.

But if they do, then at least it's educational!

I never cheat. But I have played over 5000 games and I still suck. Sometimes I wish that Chess.com would ban me! It would be a tremendous time-saver if they just didn't let me play anymore.

What problems did you run into?

More than I would like to admit for a couple hundred lines of code..

It took me a bit to get comfortable with the inputs and outputs of different model layers. ChatGPT was super helpful here.

I repeatedly forgot to re-scale the pixel values to [0.0 - 1.0] for the model, and then back to [0 - 255] to preview the images. Every time, I was left my scratching my head for 15-30 minutes before figuring it out.

I ended up fiddling with the model architecture a lot. I wanted to get the smallest model that still performed well, but without memorizing the data. I also wanted to try different options and start building some intuition about what works, and why.

At one point, I seemed to be getting instant "perfect" classification with a single tiny layer. I knew that was impossible. Which led to discovering the next problem..

Before I started using Albumentations to generate the synthetic images, I was using a different library. The idea with data augmentation is to increase the quantity of training data by starting with a small number of labeled source images, and then introducing random minor transformations to get more images with the same labels.

Well, for some reason, that randomization was not working how I expected. I kept getting the same "random" transformations for each piece, over-and-over again. This was hard to spot visually, but I eventually got suspicious enough that I went back and used a hash table count the instances and verify that every synthetic image was unique. Well, they weren't! My 5000 "unique" images per class were actually closer to 80. Switching libraries fixed this problem immediately, and other things snapped into place.

Why did you use synthetic data in the test split?

True, you're not supposed to do that.

Typically you would only use synthetic data for training.

But.. I wasn't gonna take hundreds of screenshots to get "real" test data. This is just a tutorial, so I used synthetic data and just verified that it didn't overlap with the training data. Really though, you should be validating model performance against real, unseen data.

What about different icon sets?

Here, we trained using a single set of icons. Any other "normal" set of icons should work just fine too.

But what about multiple sets of icons? Can the model generalize if we train it on lots of different icons? Can it learn the concept of "this is what a knight looks like"?

My guess is "yes, up to a point" (even in the best case, you could imagine an adversarial set of icons)

I want to investigate this a lot more.

What about image pre-processing?

I think this might improve the results.

I wasted quite a few hours experimenting with blurring, sharpening, masking, normalizing and a variety of FFT-based image processing techniques. The idea was to modify the screenshots in some way that makes them easier to learn.

None of that produced reliable improvements across many different sets of icons and boards. My understanding is that pre-processing is less common these days, because the models just end up learning the same thing, with or without it.

That said, I still suspect that background removal could be useful. Some boards are styled to look like "wood" or "marble", which creates significant artifacts that may confuse the classifier. I experimented with U-Net architectures to automatically generate image masks for background separation as a pre-processing step. The masks I got were "fine", but not amazing.

Maybe user error! I might return to this idea in the future.

What about ChatGPT and LLMs?

I did a very quick check on whether ChatGPT would ace this.

Not yet! But also not bad!

For my quick test, I uploaded a screenshot of a chess puzzle with 18 pieces to ChatGPT (January 2025).

It placed 13 of the 18 pieces correctly. You could say that's 72% correct. You could also say it's 92% correct, since it got 59 of the 64 squares right. Either way, it seems possible that commercial AI models will perform near-perfectly on this task in the future.

Note that I was asking ChatGPT to extract the entire board at once. In the code above, we were splitting the board and performing 64 separate classifications. It's a different problem. I will test ChatGPT on the single-square version sometime soon.

What about visual artifacts, cursors, icons, etc?

Due to being screenshots, some input images contain visual artifacts. This can be tough for the classifier, which prefers "clean" images. Artifacts that I encountered were:

mouse cursors
highlighted squares
arrows and non-piece icons
rank/file labels on the squares

These artifacts don't appear in the training data, so the model doesn't know what to do with them. An arrow or large icon that obscures 25% of a piece will likely mess up the classification.

The easiest solution here is to include examples of those situations in the training data. Perhaps the data augmentation step could randomly add cursors or small icons to the training examples.

What about state-specific rules?

Chess is stateful. You can't determine the entire game status from one screenshot with just the squares. There are special rules for:

These rules require knowing aspects of the game history. Here we ignored that because we were only concerned with the image recognition task.

But technically, that means our final output, a FEN string, is partly wrong. We fail to set the special bits for castling, en passant, and the half-move number.

What about live-streaming, video, or 3d boards?

I have ideas to handle live-streaming/video for 2D boards in the near future.

I barely know where to start with 3D. You've got to worry about lighting, shadows, bad camera angles, and piece orientation. Even worse, people are constantly putting their hands into the scene!

Seems hard!

I did zero research because I wanted to do this project for fun, without looking at other people's work. I might do some research in the future to learn what the state of the art is. We have driverless cars (sort of?), so it must be possible to recognize pieces on a real-life, 3d chess board (right?)

What about "sliding windows" for classification?

Another way to detect/classify pieces without a neural net is to "slide" an image or mask across every square of the board. The best "match" will have the most overlapping pixels.

I didn't try that because I wanted to use neural nets.

My gut feeling is that sliding windows should work pretty well. But it might fail in the same difficult cases that confound the model described here.

Maybe this is better or faster! I dunno! Something to investigate.

What about diff-ing the boards between moves?

Perhaps instead of continously detecting the entire board state, we can subtract the previous board from the current board to find out "which two squares changed" and use that to build up the game state over time.

from_sq, to_sq = current_board - last_board

We don't need to do any processing on the CONTENTS of the squares. Just knowing WHICH two squares changed on a given turn is (almost) sufficient to fully determine the last move.

Is this better?

Speed and accuracy are definitely advantages. It requires minimal image processing to calculate the "from" and "to" squares, and the results are deterministic. This method also completely avoids problems with image artifacts and non-standard graphics for the pieces and board.

Pawn promotion is a major problem though. Without external information, there is no way to immediately determine the promotion type. Either image recognition on the promotion square, or user input, would still be required to fully handle promotions.

Another minor drawback is that this method is stateful. A neural net can detect the whole board at any moment of the game. This method requires that we see every move, in order, from the very beginning. Does that matter? Maybe or maybe not, depending on the specific application.

What about using legal moves to improve correctness?

An interesting possibility is to use the current legal moves to re-weigh the neural net's class probability predictions for each square. For example, suppose the class probabilities for the square e5 are:

{R: 0.50, Q: 0.45, P: 0.05}

In this situation, our network predicts a white rook (R), because 50% is the highest probability. But suppose we possess external knowledge that a rook on e5 is impossible (maybe all 4 rooks have been captured).

If we discard the rook on e5 as impossible, then we can recalculate the class probabilitites for that square and predict a queen as the most likely piece instead.

{Q: 0.90, P: 0.10}

Using the last board position and legal moves to constrain the class probabilities seems powerful, especially in cases where the neural net is uncertain. Maybe worth exploring!

Solving XKCD 3045

Tue, 09 Feb 2021 00:00:00 +0000

Introduction

The chess bot AlphaMove was proposed in xkcd #3045.

The text of the comic describes the logic as simply:

sort the list of legal moves alphabetically and pick the middle one

From the image, we know:

sorting is case-insensitive
comparisons use SAN rather than UCI

And from the alt-text, we also know:

round down, for an even number of moves
it's possible to get 6 knights on the board

I wanted to know:

Can an AlphaMove bot actually reach the position shown in the comic?

The answer is: yes!

1. e4 Nc6 2. f3 e5 3. d4 Nxd4 4. c4 Bc5 5. h3 Nc6 6.
Kd2 Nd4 7. Kd3 Qg5 8. h4 Qxg2 9. Kc3 Ne7 10. Kd3 a5 
11. Kc3 Qg3 12. Kd2 d5 13. Kc3 Qf2 14. f4 dxe4 15. Ne2

The AlphaMove Algorithm

The snippet below implements the algorithm as described, using the Python chess package, with AlphaMove playing against itself:

import chess

def alpha_move(board):
    san_moves = [board.san(move) for move in board.legal_moves]
    sorted_moves = sorted(san_moves, key=lambda x: x.lower())
    if len(san_moves) % 2 == 1:
        middle_move = len(san_moves) // 2 
    else:
        middle_move = len(san_moves) // 2 - 1
    return sorted_moves[middle_move]

def play_game():
    board = chess.Board()
    while not board.is_game_over():
        move = alpha_move(board)
        board.push_san(move)
    return board

play_game()

Initially my code was rounding up, playing 1. f3.

While poking around online, I ran into Explain XKCD and they give moves for the "6 knights game" mentioned in the alt-text. They also started with 1. e4.

I hadn't been certain the "6 knights" was a real thing, or that it was the expected behavior of AlphaMove vs. AlphaMove. But after some more fiddling, I was able to get my code rounding down and was able to reproduce that game too.

It looks like this after 19. gxf8=N Nge4

And here is the full game (by default, the python chess module halts on 5-fold repetition)

1. e4 e6 2. f3 f5 3. e5 g5 4. d4 d5 5. exd6 g4 6. d7+
Kf7 7. dxc8=N Ke8 8. fxg4 h6 9. gxf5 Kd7 10. g4 h5 11.
fxe6+ Ke8 12. g5 Na6 13. h3 Nc5 14. h4 Ne7 15. Kd2 Ne4
+ 16. Ke1 Nf5 17. g6 Nf6 18. g7 Ng3 19. gxf8=N Nge4 
20. Ke2 Ng4 21. Kf3 Ngf2 22. Ke2 Nh3 23. Ke3 Nhf2 24. 
Nb6 Nh3 25. Na4 Nhf2 26. Nac3 Nxc3 27. Kxf2 Nxd1+ 28. 
Kf3 Qc8 29. c4 Ne3 30. Ke4 Nf5 31. Kd3 Ng3 32. e7 Nxh1 
33. Kc2 Qb8 34. d5 Kxe7 35. d6+ Kf6 36. dxc7 Nf2 37. 
c8=R Ng4 38. Kd2 Nh2 39. Ke3 Ng4+ 40. Kd4 Nh2 41. Kd5 
Nxf1 42. Nc3 Nh2 43. Nce2 Ng4 44. Nd4 Nh6 45. Nd7+ Kf7 
46. Ndf3 Qd6+ 47. Ke4 Qd2 48. Nf8 Qd5+ 49. Ke3 Qd2+ 
50. Ke4 Qd5+ 51. Ke3 Qd2+ 52. Ke4 Qd5+ 53. Ke3 Qd2+ 
54. Ke4 Qd5+ 55. Ke3 Qd2+ 56. Ke4

Given that we (probably) now have the correct implementation of AlphaMove, can we try to figure out what sequence of moves led to the board shown in the comic?

Is Game 3045 Findable?

This is the target board:

Which is represented by this partial FEN string

r1b1k2r/1pp1nppp/8/p1b1p3/2PnpP1P/2K5/PP3q2/RNBQ1BNR

Since this is xkcd, we'll assume this is actually a reachable board state using AlphaMove. That's PROBABLY the case, but we don't know 100% for certain beforehand. It could just be a random board.

The bigger concern is whether this state is FINDABLE. After just 4 "half-moves" (2 moves each), there are already 197,281 possible positions. Then after just 15 half-moves, there are 2,015,099,950,053,364,471,960 positions. Simply because a position exists, that doesn't mean that we can find it in a reasonable amount of time.

As a casual player, it's easy to invent a series of moves that would result in the position above (not "good moves" necessarily.. just legal moves). But we don't want just "legal moves", we need to reach this position after white plays the "alphabetical middle" move each time. That makes the task much harder.

We do have a few things going in our favor:

First, since we know that white is using AlphaMove, we only need to figure out black's moves. White's moves are totally pre-determined by what black does.

Second, just from looking at it, we can see this state should be reachable in roughly 10-20 moves. Most of the pieces are still on the board and many remain on their home squares. The game isn't so far along that a "directed search" is hopeless.

Third, you can sort of "see" how some pieces must have moved. For example, that knight on d4 probably moved Nb8 -> Nc6 -> Nd4. It COULD have moved Nb8 -> Na6 -> Nb4 -> Nc2 -> Nd4, but it probably didn't because that would be weird.

Together these criteria significantly restrict the search space.

Is that enough? (Yes)

The Code

The usual candidates for searching are:

Depth first
Iterative deepening, like DFS, but fancy
Breadth first
Best first, like BFS, but fancy

After some experimenting, I decided to implement a "best first search", using the hints above as heuristics to restrict the possibilities.

With "best first search", you need to define what "best" means. The algorithm needs to know whether it is getting closer to the goal.

After a bit of experimentation, I landed on two useful metrics:

number of mismatched squares
number of total moves

Basically, we ask the algorithm to favor "short solutions that already have lots of squares correct" and that should guide it to quickly finding a solution with "all the squares correct"

This is the code. Kinda long for a blog post.

import heapq
from chess import Board
import chess.pgn

# 1D string representation of the target board state
target_a64 = 'r.b.k..r.pp.nppp........p.b.p.....PnpP.P..K.....PP..Nq..RNBQ.B.R'

# Human-generated list of moves that might be made by black
likely_black_moves = [
    'a6', 'a5',
    'd6', 'd5', 'dxe4', 'dxe5', 'dxe6',
    'e6', 'e5', 'e4',
    'Nc6', 'Nd4', 'Nxd4',
    'Qe7', 'Qf6', 'Qg5', 'Qh4',
    'Qh2', 'Qg2', 'Qf2', 'Qd2', 'Qc2',
    'Qh3', 'Qg3', 'Qf3', 'Qd3', 'Qc3',
    'Qg3', 'Qxg3', 'Qf2', 'Qxf2', 'Qxg2', 'Qg3', 'Qf3',
    'Bc5', 'Bd6', 'Be7',
    'Ne7',
]

# Human-generated list of moves that might be made by white
likely_white_moves = [
    'c3', 'c4',
    'd3', 'd4', 'd5',
    'e3', 'e4', 'e5',
    'f3', 'f4',
    'g3', 'g4', 'g5',
    'h3', 'h4',
    'Ne2',
    'Ke1',
    'Kc2', 'Kd2', 'Ke2', 'Kf2', 'Kg2', 'Kh2',
    'Kc3', 'Kd3', 'Ke3', 'Kf3', 'Kg3', 'Kh3',
    'Kc4', 'Kd4', 'Ke4', 'Kf4', 'Kg4', 'Kh4',
]

def alpha_move(board):
    """
    Selects the move that is alphabetically in the middle of the list of legal moves.
    """
    san_moves = [board.san(move) for move in board.legal_moves]
    sorted_moves = sorted(san_moves, key=lambda x: x.lower())
    if len(san_moves) % 2 == 1:
        middle_move = len(san_moves) // 2 
    else:
        middle_move = len(san_moves) // 2 - 1
    return sorted_moves[middle_move]

class QueueItem:
    """
    A node in our search tree.
    This holds the current board state and the fitness of that state.
    """
    def __init__(self, board):
        self.board = board
        # 1D string representation of the board state
        self.a64 = str(board).replace(' ', '').replace('\n', '')
        # The number of squares that are different from the target state
        wrong_squares = sum(1 for a, b in zip(self.a64, target_a64) if a != b)
        # The number of moves made so far
        move_num = len(board.move_stack)
        # Fitness is the product of the two values above
        self.fitness = wrong_squares * move_num

    def __lt__(self, other):
        # Compare based on fitness
        return self.fitness < other.fitness

def best_first_search(start_board):
    # the priority queue
    priority_queue = []
    # avoid repeating the board state
    seen_positions = set()

    # push the first item onto the queue
    start_move = QueueItem(start_board)
    heapq.heappush(priority_queue, start_move)

    while priority_queue:
        # pop the item with the best fitness
        current_stack = heapq.heappop(priority_queue)
        current_board = current_stack.board

        # check every possible continuation for black
        for black_move in current_board.legal_moves:
            black_move_san = current_board.san(black_move)

            # skip if that move was unlikely to be made
            if black_move_san not in likely_black_moves:
                continue

            # make the move
            next_board = current_board.copy()
            next_board.push_san(black_move_san)

            # the white move is the middle of the alphabetically sorted list
            white_move_san = alpha_move(next_board)

            # skip if that move was unlikely to be made
            if white_move_san not in likely_white_moves:
                continue
            next_board.push_san(white_move_san)

            # we made 2 moves that are likely
            next_stack = QueueItem(next_board)

            # skip if we have already seen this board state
            if next_stack.a64 in seen_positions:
                continue
            seen_positions.add(next_stack.a64)

            # check if we are we done
            if next_stack.a64 == target_a64:
                return next_stack.board

            # we like the progress, but we aren't done yet
            # so push this new state into the queue
            heapq.heappush(priority_queue, next_stack)
    return None

def make_pgn(board):
    """
    Helper function to print the moves in PGN format
    """
    game = chess.pgn.Game()
    game.setup(chess.Board())
    node = game
    for move in board.move_stack:
        node = node.add_variation(move)
    return game.variations[0]

board = Board()
board.push_san(alpha_move(board))
found = best_first_search(board)
print(make_pgn(found))

And here is the solution, which it finds after a reasonable ~3200 steps.

1. e4 Nc6 2. f3 e5 3. d4 Nxd4 4. c4 Bc5 5. h3 Nc6 6.
Kd2 Nd4 7. Kd3 Qg5 8. h4 Qxg2 9. Kc3 Ne7 10. Kd3 a5 
11. Kc3 Qg3 12. Kd2 d5 13. Kc3 Qf2 14. f4 dxe4 15. Ne2

A few aspects of the search are worth noting:

Pruning the unlikely and impossible moves is crucial. This prevents the algorithm from needing to examine A TON of board states that definitely never happened in the target game. For example, we know there is a pawn on c4. We don't know for certain whether that was due to c2c4 versus c2c3 .. c3c4, but we know for sure that c4c5 was never played. We speed things up a lot by limiting the search to moves that could have actually happened in this particular game. These lists were guesses based on intuition after manually inspecting the board. The move Bb4 was possible, but I thought it was unlikely that the bishop had retreated, so I didn't include it. The move Qh4 seemed likely, so I included it, but it wasn't actually played and the queen stopped at Qg5 instead.

Tracking the list of "seen positions" is necessary. This prevents "loops" where the pieces just shuffle around and repeat the same moves. The existence of repeats could cause the search to run forever, or to try the same few positions over and over again.

The choice of fitness metric matters. I checked what happens if either the "number of incorrect squares" or "path length" metric is used by itself. Both succeed, but much more slowly.

The "path length" metric finds the solution after ~85,500 steps. I believe that using "path length" alone is equivalent to performing breadth-first search.

The "incorrect squares" metric finds the solution after ~103,200 steps. It's not immediately obvious that "number of incorrect squares" is even a good fitness metric, since it gives the same penalty to a piece that is off by 1 square and a piece that is on the wrong side of the board. My guess is that a smarter "edit distance" metric for chess would be expensive to compute, so the trade-off might not be worth it.

What's interesting is that combining the two metrics is so effective (~3200 steps), since neither is amazing by itself.
Conclusion

This was amusing. I made a few mistakes along the way:

My initial code was "rounding-up". I didn't realize that until learning that AlphaMove vs AlphaMove should result in the "6 knights" game, which my code didn't do.

I kept going back and forth between trying to find the position with a tree search and trying to find it by hand with computer assistance. I felt like I was getting somewhere on intuition. But looking at the actual solution, I now suspect that finding it by hand is borderline impossible.

Finally, I was missing a few items from the "likely moves" lists at first. The "likely moves" heuristic is great for pruning the search space, but if the list is missing any of the "actual moves", then the algorithm will fail to find a solution. That's a problem when you don't know for sure whether a solution exists. I made several edits to the "likely moves" lists before getting it right.

That's it. I was initially interested in how AlphaMove plays, which pieces it favors, how oftens it draws against other bots, etc. But then I got intrigued by trying to generate the board from the comic because it seemed feasible. I might follow up on those other questions soon, TBD.