October 02, 2026
The best truco agent I trained was not the last one
truco-rl is a 1v1 Truco Gaúcho rule engine in Python and an agent that learns to play against copies of itself, with no human games and no hand-written strategy. I trained it for 5 million steps, an hour of CPU on a laptop, and the model that came out at the end is worse than one it had been halfway through.
This post is about how I noticed, because the metric the training prints was never going to tell me.
The training metric is zero by construction
In self-play, the opponent is a copy of the agent itself, refreshed every 20k steps. The last line of the training log says:
ep_rew_mean | 0.0045Mean reward close to zero. That makes sense: a player against itself wins half its games. And that is true at step 250k and at step 5 million. The number the training shows does not go up when the agent gets better, because the opponent gets better with it.
So to know whether the agent was learning anything, I needed an opponent that does not change.
What to measure against
The obvious first candidate was a random player, which picks any legal action with equal probability. The agent beats it around 80 to 90% of the time. That looks good, until you remember that the random player answers bets by coin flip and calls truco without looking at its hand. Beating it says little.
So I wrote a second opponent, a rule-based bot that plays like a cautious beginner: it only calls envido with a good score, only calls truco with a strong card or a round already won, and plays the cheapest card that wins each round. It beats the random player 89% of the time. Neither of them appeared during training.
For a sense of what a draw looks like: random against random comes out at 51%, and the bot against itself at 50.5%. Every number below is over 2,000 games, with a margin of about 2 points either way.
The rule-based bot found a bug before it measured anything
The first time I put the bot in a game, the game did not end.
Nothing in the engine limited envido raises. One player called Real Envido, the other answered with Real Envido, and so on forever. The bot always raised when it had a good score, and both sides had a good score. The environment has no step limit per episode, so self-play training could fall into the same loop and I would not see it, I would just have a stuck training run.
The fix was to make the chain only go up: Envido at most twice, Real Envido once, and Falta Envido ends it. The rule lives in one place, and the agent’s action mask only offers the raises that are still legal.
What bothers me is that the random player never found this bug. It raises sometimes and accepts sometimes, so the chain always ended by chance. It took an opponent with a strategy, even a dumb one, to follow the same path all the way.
The curve
I saved a checkpoint every 250k steps and evaluated all of them against the two fixed opponents:
| Checkpoint | vs random | vs rule-based bot |
|---|---|---|
| 250k steps | 89.6% | 38.0% |
| 750k | 88.4% | 54.1% |
| 1.25M | 83.5% | 63.9% |
| 2.25M | 83.9% | 73.1% |
| 3.0M | 81.5% | 60.1% |
| 3.5M | 79.0% | 52.6% |
| 5.0M (final) | 80.2% | 62.8% |
The agent passes the rule-based bot around 750k steps and peaks at 73% at 2.25M. After that it gets worse. It drops to 53%, nearly a draw against a beginner bot, and recovers to about 63% at the end.
Had I taken only the final model, which is what the training script saves under the name “final”, I would have reported 63% and assumed that was the ceiling. The best model was 10 points higher, two thirds of the way back.
Why it gets worse
This is the classic problem of naive self-play. The opponent is always the latest version of the agent, so the agent learns to beat that opponent. When it changes strategy, the new version no longer needs to remember how to beat the old ones, and it forgets. Strategies can go in circles: A beats B, B beats C, C beats A, and training keeps running with the reward at zero the whole time.
There is a sign of this in the other column of the table. Against the random player, the agent starts at 90% and ends at 80%, worse than the rule-based bot against the same random player. My reading is that it specialized in playing against an opponent that plays well, and an opponent that answers bets with no reasoning stopped being something it remembers how to face. I have not tested this properly; it is a hypothesis.
The known fix is a league of opponents: instead of always playing the newest version, the agent plays against a draw from older versions and fixed bots, and cannot forget any of them. I did not implement it. It is the obvious next step, and the README lists it as not done.
What I would take from this
Two things, and neither is specific to truco.
The first is that the metric training optimizes and the metric that says whether the model is any good can be different things, and in self-play they are different by definition. Without a fixed opponent from outside, I would have had a nice-looking training log and no idea what was happening.
The second is that the evaluation opponent has to be good enough to actually lose. The random player did not find the envido bug and does not show the drop: against it, the checkpoint that looks best is the one at 250k steps, which is exactly the one that loses to the rule-based bot 62% of the time. It was a beginner rule-based bot that showed both.
The code, the results and how to reproduce them are in the repository.