RoboChess

On May 11, 1997, IBM’s Deep Blue defeated Garry Kasparov in the final game of their rematch. It was the first time a computer had beaten the reigning world chess champion under standard tournament conditions. Kasparov, widely considered the greatest chess player in history, had held the world title for over a decade. He lost to a machine running on 480 specialized chips, capable of evaluating 200 million positions per second.

Many expected chess to die as a game after that. It didn’t. If anything, the opposite happened. Chess became the benchmark, the go-to proving ground for artificial intelligence. Can your system plan ahead? Does it reason under uncertainty? Can it beat the best human? If you wanted to demonstrate that a new form of artificial intelligence had arrived, you played chess.

Now, artificial intelligence is moving beyond the screen and into the physical world. It is picking up objects, navigating spaces, operating tools. It is doing the things that used to require years of robotics engineering to hard-code, and it is starting to do them from raw sensory input. We are talking about robotics, and specifically about a new generation of robotic systems that are trained more than they are programmed.

So we asked ourselves: Can chess serve as a meaningful benchmark for embodied artificial intelligence? And what does it take to turn it into one? Not to play the game; computers solved that in 1997. No, we want the AI to physically execute chess moves, guided by nothing more than a camera and a natural-language instruction. That is what this project set out to explore.

From Hand-Built Pipelines to Foundation Models

To be clear: robots playing chess is not new. Several systems already do it, and they do it well. The standard approach is a carefully engineered pipeline. A computer vision module identifies the pieces and their positions. A chess engine decides the move. A motion planner computes a collision-free path. A grasping routine picks up the piece and places it. Each component is specialized, hand-designed, and reliable within its scope. Hand-built, specialized pipelines work, but they are rigid. Change the chess set, swap the robot, alter the lighting, or step outside the designed scenario and the pipeline will most likely fail. To make it work again, you are back to engineering. For every. single. change.

But things are changing. Robotics is currently undergoing a paradigm shift that closely mirrors what happened to natural language processing (how machines understand natural language) a few years ago. Back then, you also had meticulously engineered pipelines for different tasks, e.g., sentiment analysis or recommendation systems. ChatGPT wasn’t just a better engineered pipeline; it was a paradigm shift: from hand-built pipelines to foundation models. Large neural networks, pre-trained on vast and diverse data, adaptable to a wide range of downstream tasks with relatively little additional training. We are experiencing this ChatGPT-moment in robotics right now. Robot Foundation Models (RFMs) are large models pre-trained on diverse robot interaction data spanning many different robots, tasks, and environments. The promise is familiar: one generalist model, capable of adapting to new tasks, robots, and environments without rebuilding the perception and control stack from scratch. Fine-tuned, rather than programmed.

The most prominent architecture for such Robot Foundation Models is called a Vision-Language-Action model (VLA). It is essentially a Vision Language Model, i.e., a model that can understand both text prompts and image and generates natural language tokens, adapted to be able to control robots. In addition to the prompt and camera images, it also understands the robot state (joint rotations, sensor readings, etc.) and uses that information to generate some form of movements and actions for the robot. This means you have an end-to-end architecture that can control a robot based on text prompts.

Teaching a Robot Arm to Play Chess

Our hardware setup looks as follows: We have a UR5 robot at the Innovation Hacking office in Unterföhring: a six-joint industrial robot arm. Along with that, we have a Schunk EGH 80 gripper, a two-finger gripper. In this setup, we are using two cameras: a front camera that gives us an overview of the robot and workspace from the front, and a camera mounted on the gripper (in the following called wrist camera), which captures a detailed view of grasping and placing objects.

We set out to explore the current capabilities of Robot Foundation Models by teaching this robot arm to play chess. For the RFM this means performing chess moves based on user prompts. We do not want the model to be a chess engine, i.e., to come up with chess moves itself by looking at the board or checking the legality of moves it receives as instructions. This would be a much more complex task to teach the model. The RFM receives a text prompt, e.g., “Pick up the white pawn at e2 and place it on e4”, as well as the current frames from the two cameras in a resolution of 224x224, and should output actions for the arm and the gripper to successfully execute this move. For the first iteration, we exclude moves like castling, promoting, or capturing, which would require moving multiple pieces.

The model at the center of this project is an RFM from NVIDIA: GR00T N1.5, a 3-billion-parameter vision-language-action model. It takes camera images and a natural-language instruction as input and outputs joint commands for the robot arm. By default, GR00T has no concept of time, choosing each action from the current camera frames with no memory of the past. No dedicated object detector, no motion planner, no hand-crafted grasping routine. End to end: camera and prompt in, motor commands out. For more in-depth information on GR00T, check out our post on teaching GR00T how to play video games on teaching GR00T how to play video games.

The training paradigm most often applied to fine-tune RFMs for your use case is called imitation learning, i.e., learning from some kind of demonstration how the task should be solved. So how do we generate a dataset of chess demonstration?

How to: Adapting RFMs for Specific Tasks

There are different ways of generating demonstration data for fine-tuning RFMs.

The easiest way is teleoperation, where a human performs the task, by remote controlling a real robot. By recording all camera inputs, robot states, and actions, we generate a complete dataset for fine-tuning RFMs, requiring only the addition of a prompt. Teleoperation can come in many different shapes and sizes. For our dice sorting showcase, we used two pairs of LeRobot SO-101 robot arms. One pair are the leaders, i.e., a human can grab them and move them around. The second pair are the followers; they are synced with the leaders and “follow” according to how the teleoperator moves the leaders. With that, you can manually record demonstrations the model can learn from. For the Unitree G1 we use a VR headset, that tracks the head and hands of the operators, and moves the G1 in the same way.

While teleoperation is a convenient solution for some use-cases, it may not always be a viable option.

A significant potential blocker is the absence of a teleoperation setup for the specific robot. The SO-101 and the G1 come with very straight-forward teleoperation setups. The UR5 doesn’t. Even though we are able to control the UR5 manually with a control panel, we can only adjust one joint at a time with it. Recording demonstrations manually for this task would therefore take ages, as would building a functional teleoperation setup.

Another consideration is the amount of needed training data and the labour cost associated with it. For simple tasks a 50-100 demonstrations can be enough data to fine-tune the model. For the Dice-sorting task, we needed less than 200 demonstrations with a fixed prompt: “sort the dice”. Performing chess moves is more complex, both for the data collection and as a task for the RFM. The prompt isn’t constant, it changes for every move and the RFM has to take this into consideration. Instead of tens to hundreds, we now need thousands of demonstrations. The more data, the better, up to a certain level. And as one demonstration for us means executing one chess move on the board, a clip of roughly 15 seconds. Doing this 1600 times, even with a good teleoperation setup, would still take ages. And apart from recording an episode, you have to manually set up the board, document the current prompt, and so on. Ain’t nobody got time for that!

The alternative to teleoperation is recording expert demonstrations in simulation. In a digital twin of our setup, we know where each object is and can write a simple bot to perform the demonstrations for us. No human labour required. But first, we have to rebuild the real scene in simulation, then record the expert demonstrations there, fine-tune the model on that simulated data via imitation learning, and finally transfer the resulting model onto the real robot.

Rebuilding the Real World in Simulation

We used NVIDIA Isaac Sim, a physics-based robotics simulation platform, to reconstruct the setup of our real robot. Same arm, same gripper, same table, same chessboard dimensions, same camera positions. The goal was a virtual copy, a digital twin, accurate enough that a model trained entirely inside it would still work when deployed in the real world.

Getting the scene into simulation was a project in itself. We rebuilt it from a mix of sources: public 3D models where they existed, such as the official UR5 arm; hand-modeled assets for the simpler custom parts, such as the base the UR5 is mounted on and an image-to-3D pipeline that reconstructs an object from photographs, which we used for the chess pieces.

To build and script the scene programmatically rather than by hand, we worked in NVIDIA Isaac Lab, the framework that sits on top of Isaac Sim and lets you define everything in code. This allowed us to configure the board state freely, automate the placement of pieces and the other assets, and generate the corresponding natural-language prompt for each demonstration.

The huge benefit: Unlike the in real wold, finding the chess board and pieces is easy in simulation. We can read their position directly from the simulation instead of having to rely on computer vision. With the hardest part - scene understanding - out of the way, we can program a rather simple scripted state machine that moves the arm in simulation to pick and place the piece that should be moved. Now we only need to run the state machine while recording the camera frames, robot states and action and store that together with the text prompt to generate one demonstration without any human input.

Another big benefit of collecting data in simulation is scale and parallelism. We are not limited by the physical availability of robots and tables; we can replicate our simulation setup as many times as our GPU memory can handle. Running four simulation environments simultaneously, we generated hundreds of demonstrations in the time it would take to record a handful on the real UR5.

Improving the Model

With a well defined setup and the imitation learning locked in, we started an iterative process of improving our model from something completely unusable to something showcase-worthy. First, we wanted to push model performance as far as possible within simulation by adjusting the scene, the training data, and the information available to the model. The goal was not just better numbers but genuine insight into what the model actually needs in order to learn the task correctly. The second phase was to take those insights and apply them to the real world, where the constraints are different and some of the solutions that work in simulation simply are not available.

Insights Gained in Simulation

The model receives many signals at once: camera images from two viewpoints, the current joint positions of the arm, and the language prompt describing the move. In theory, it should learn to combine all of these. In practice, it will use whatever signal is cheapest to exploit.

The simplest example would be if we trained on a single move only. The model would only be able to solve that one, and performance would collapse when the model is instructed to execute another move. This is the easiest example of “giving the model a clear signal that loses generalization”. But it goes beyond this. Even with different moves in the demonstrations, if the board is always placed the exact same way, the cheapest signal is to remember what move description correlates with which movement. If the board is slightly shifted, it will move to the wrong place, because the demonstrations didn’t teach the model to use the camera to adjust positioning based on what it sees. Most of these biases can be removed by increasing variance in the demonstrations by applying domain randomization. This means that in every episode recorded in simulation, we use a different move, a different chess position on the board, slightly nudged pieces, a slightly nudged board on the table, slightly tilted and offset cameras, and so on. When the board is shifted in every episode, the model can’t just learn to remember “I should pick up from e4, so I move my arm exactly there”. It has to learn to use the camera to see how the board is positioned and adjust its position based on that.

We found that the model performs best if we reduce the correlation between a specific square and its position in front of the robot as much as possible. In short: Increasing variance makes the model robust. Shifting the board pushes the model not to remember a correlation between a certain square and a fixed position in front of it, but it requires the model to also consider where the board is placed. However, the correlation that was still left is that, for example, the square “a8” is on the far left side of the board relative to the robot. Just increasing the randomization of the board position in x and y doesn’t change that. To further break the correlation of “a8 is always on the far left corner of the board in front of me”, we additionally rotated the board by up to 360 degrees. This means there is no spatial correlation left between where certain squares are relative to the robot. The model must now look at the board and really understand the chessboard logic to navigate on the board. This is the most robust signal the model gets to generalize to any positioning of the board in front of the robot and to any move to be executed.

However, the core requirement is that there is sufficient visual information for the model to navigate above the board. A standard chessboard has letters and numbers along its edges. In the 224x224 pixel images the model receives, those edge labels are simply too small to read. They are also not visible in the wrist camera when the arm is near the center of the board. The solution was to add an explicit alphanumeric label to every individual square.

So, to summarize:

• Unwanted biases in the demonstrations can be removed by increasing the variance applied to the scene in simulation.
• The most reliable signal we found for the model to solve this task was purely visual-and-prompt based, with no spatial prior, i.e., no remaining memorization of how squares should be positioned in front of the robot.
• To push a model to follow your intended signal, you have to ensure that all the information needed is given in the demonstrations. In this case, this wasn’t the case until we added per-square labels.

Transferring Learnings into the Real World

Transferring the model onto the real robot was hard. Some things are simply given and cannot be changed, and some methods found in simulation to improve performance won’t work in the real world. The following documents the fixed constraints we decided on in the real world, and what changes we made to keep the model’s performance high.

First, though, let’s highlight why transferring the model into the real world is such a hassle. Since the simulated environment was designed to closely resemble its real-world counterpart, one might reasonably expect the model to transfer with little difficulty. Well… see for yourself.

Even though we can get the same assets with the same dimensions and a similar look into simulation, there are still clear differences: apart from visual differences in lighting, background, or general colors, there are also differences in physics or the movement/control of the robot. This difference, or gap, we are trying to bridge is called the sim-to-real gap. And no matter the efforts to keep it small, it is still significant, so transferring a model trained solely in simulation onto a real robot isn’t trivial. However, closing the gap isn’t the only way to bridge it. Building a robust model is the other one.

The Constraints

In simulation we found that rotating the board to remove a fixed spatial signal improves robustness. However, this required providing sufficient visual information in the form of labels on the board. We decided against it. Adding labels to each board square is messy, and any other kind of visual cue would not look much better on the board. Instead, we decided to keep the original texture. This also means we cannot give the model additional information on it, and therefore cannot include arbitrary rotation of the board. This means the orientation of the board is fixed, just slightly altered by a few degrees. So we are giving the model a bit of a spatial signal to navigate, which is actually closer to how a human would navigate on the board. A chess player would know that the board doesn’t rotate arbitrarily in front of them during the game, and knows which region of the board the squares are in.

The Learnings

The following are the core learnings from transferring a model trained in simulation onto a real robot:

Hardware behavior and control differ. The simulation uses the exact same robot, but how you control it, and how the parts behave, is different in the real world. In simulation, control is simple: you send the arm a waypoint and it moves there. The real UR5 is more involved: its driver takes waypoints, plans trajectories, and eases in and out of them, so the motion comes out much more stepwise. The gripper was another difference. In simulation we could just tell the model to close the fingers as far as they go. On the real robot that backfired: it closed too hard, the force got too high when pressing the fingers against a chess piece when grasping, and the gripper’s driver software gave up. It would put itself in an protective error state that could only be reset by restarting the driver. Since different pieces are thicker in different spots, “just close as far as it goes” was by far the simplest thing to teach, and all of a sudden didn’t work on our gripper. Our fix was to 3D-print our own gripper fingers out of a flexible material. Now the extra force bends the fingers instead of loading the gripper and triggering a protective stop.

Nailing the visuals. The visuals are by far the biggest gap between simulation and reality, and we tackled it from two sides: making the simulated images look as close to real as we could, and adding enough visual variance that the model didn’t overfit to the exact look of the simulation. But visuals are not only colors and appearance. They also include the camera’s intrinsic parameters. For a long time our simulated cameras had different intrinsics than the real ones. The color mismatch didn’t hurt much, since we heavily randomized colors during training anyway, but the field of view was another story: it directly changes how the model reads depth and perspective from an image. Only after we had precisely replicated the intrinsic parameters of the real cameras in the simulation was the model able to transfer its sense of depth to the real world.

It’s a data problem. Matching the simulation to reality matters, and so does tuning the model, but in the end this is a data problem: a model can only learn what its demonstrations show it. Even with a perfect sim-to-real match and perfect settings, anything missing from the data stays missing. Because the real arm control dynamics are different, it drifted off the trajectory it had learned much sooner than it ever did in simulation. Our demonstrations always followed the same path: move above the target piece, come straight down, grasp, lift, and carry it to the destination. In practice the descent wasn’t always perfectly straight, so the arm would end up slightly beside the piece instead of above it, close to the board but not lined up with anything. The model had never seen that. It was out of distribution, and it had no idea how to recover. The fix was to widen the data: we started the arm from many more positions, including ones low over the board and far from the piece, so it had to find its way back. Once the demonstrations showed that the approach often needs correcting on the way down, the model learned to correct too, and its approach became far more precise.

What We Found

The result is a showcase: you enter a move as a prompt, and the model executes it end to end on the real robot, with no hand-crafted pipeline in between.

The demo aside, we came away with a handful of concrete lessons about fine-tuning a Robot Foundation Model on simulation data alone.

Sim-to-real transfer works well. A model trained only on simulated images ran on the real robot and coped with the visual differences between the two. That is worth pausing on. A classic computer-vision pipeline would not survive it: change the lighting or the color of a piece and a hand-tuned detector falls over. Our model held up because it does not learn to see from scratch. It starts from the broad visual knowledge already inside the foundation model, and fine-tuning just points that knowledge to our task. Even so, the smaller the gap the better it works, and the camera settings mattered most. The model reads distance and depth straight from the image, so when the simulated camera’s field of view did not match the real one, its sense of depth was off in a consistent way. Getting those camera parameters exactly right was not optional.

Performance scales with data, up to a point. Foundation models are supposed to pick up a new task without enormous amounts of data, and that held: we did not need hundreds of hours of footage. We grew the dataset in stages, 400 episodes, then 800, then 1600, where one episode is a single chess move of about 15 seconds. The model got better at every stage. Going from 400 to 800 helped the most; 800 to 1600 helped noticeably less. We did not run a clean controlled comparison and we do not have a scaling curve, so we will not put a hard number on it: the gains were clearly shrinking, which hints at a plateau ahead, but that is a guess rather than a measurement. It also depends on the task. Ours leans heavily on the language prompt, with many different instructions, so it is hungrier for data than a single-instruction task like dice sorting, which managed on fewer than 200 episodes.

Good demonstrations are hard to produce. The simple scripted state machine we started with soon became the bottleneck. It is not just how much data you have, but how good and how varied it is. One thing shaped everything: GR00T N1.5 has no memory of the past. It chooses each action from the current camera frames alone, with nothing about the moments before. So the data must never contain a situation the model cannot distinguish from a single frame. If the arm hovering over a piece looks the same whether it is on its way down to grab it or on its way back up after placing one, the model is handed two different “correct” actions for what looks like the same moment, and the data contradicts itself. That is why the demonstrations had to be built so carefully: every path has to be readable from a single frame, with no moment that could be read two ways.

The model is only as good as the demonstrations. GR00T is far from perfect. Sometimes it approaches the wrong piece, and past a certain point it commits: it has settled on the piece it is grabbing and will close on whatever is there, right or wrong. This is not a quirk you can tune away, it is the basic limit of imitation learning. The model can only reproduce behaviors it has actually seen. To fix a failure you have to show the fix in the data, which means anticipating and recording every way things go wrong: every mis-approach, every recovery, every awkward grasp. That adds up quickly. A method that can discover its own corrections, like reinforcement learning, would not be stuck this way, but in pure imitation learning, if it is not in the data, the model cannot do it.

Push the model past what it saw in training, and the same limit shows up in different ways:

Conclusion

We began by asking whether chess can serve as a meaningful benchmark for embodied artificial intelligence, and what it would take to turn it into one. Our results suggest that it can: not as a test of strategic chess playing, but as a test of perception, spatial grounding, language understanding, and precise physical execution. However, current Robot Foundation Models are still far from mastering this task, and turning chess into a robust benchmark requires large amounts of task-specific demonstration data. This project explored one scalable path toward that goal: using simulation to generate expert demonstrations for fine-tuning.

What’s Next?

Reintroduce board rotation, cheaply. The spatial-grounding problem is not solved, and there is a clear next step. In simulation, rotating the board forced the model toward genuine visual attention, but it required per-square labels, which we did not want on the real board. A lighter-weight idea: instead of labelling every square, place a single visual marker (for example, a red dot) at one corner of the board. That one reference point could be enough for the model to infer the board’s orientation and derive every square’s position from it, a minimal visual signal rather than labelling all 64 squares.

Use this setup as a benchmark for comparing models. So far we used this project as a benchmark solely for GR00T N1.5. The model family has since moved to newer versions, and other VLA models exist, including those from Physical Intelligence. Benchmarks only get meaningful when used to compare different models, so the natural next step is to use the showcase as a probe for vision-language-action models. Chess-move execution is unlikely to be well represented in any of these models’ pretraining data, which is exactly what makes it useful: not a benchmark for general manipulation, but specifically for how well a model can be adapted to a task requiring strict spatial grounding and instruction following under scene variation. The engineering infrastructure for running and evaluating such comparisons is already in place.