A truco agent that learns to bluff

The problem

Truco is interesting for reinforcement learning for one specific reason: much of the information is hidden, and lying about it is part of the rules. An agent that only computes hand strength plays badly. It has to learn that raising with a bad hand works sometimes, and that it works because the opponent cannot see the cards.

That changes the problem. It is not optimising against a visible board; it is learning a policy in a game where the same observable state could have been produced by completely different hands.

Technical decisions

Illegal actions are masked, not punished

Against what: letting the agent pick any action and handing back a negative reward when the action is invalid.

Training uses PPO with action masking: at every state the engine reports which moves are legal, and the agent only chooses among those.

What it cost: the rules engine stopped being merely executable and had to become queryable: for every state it must be able to answer “which actions are valid right now” before any move happens. The truco, envido and flor state machines had to be written with that question in mind from the start, which is more work than validating at the moment of the play.

Two implementations of the same rules, on purpose

Against what: a single engine, with training talking to the C# side across the Unity boundary.

The game has a pure C# engine (tested with xUnit); training has a second engine in Python (tested with pytest). Training across a process boundary would make each episode far too expensive, and there are millions of episodes.

What it cost: two implementations of the same rules can drift apart, and silent drift is the worst kind. That is why both sides carry their own test suite over the same card-hierarchy and hand-resolution rules. It is a permanent maintenance cost accepted in exchange for training that is actually feasible.

The model is served over HTTP, not embedded in the game

Against what: exporting the model and running it inside Unity.

The trained agent sits behind a FastAPI service, and the Unity client asks it for the move.

What it cost: the game stopped being self-contained: with the service down, there is no opponent. In exchange, the model can be retrained and swapped without touching the game build, and you can play against different versions of the agent to compare them.

What was left out

Trained models are not versioned in the repository. And the agent today plays 1v1; the 2v2 and 3v3 rules exist in the C# engine, but the training environment does not yet cover partner play, where the implicit communication between teammates changes the whole game.