Reverse Engineering a Game's Screen Capture Protection

It started with a simple frustration: my friends and I play an online game together, and we can’t share our screens on Discord. The game blocks it. Black screen, every time.

I could have accepted that. But something about knowing that a piece of software on my own computer was deliberately stopping me from doing something harmless made me curious. I spent the whole day down that rabbit hole.

The first clue

Before touching any tooling, I noticed something odd: weeks earlier, for a few minutes, everyone in the group could share their screen at the same time. Then it stopped again. For everyone, simultaneously.

If the protection were purely client-side, each computer would decide this independently. The fact that it broke and came back in sync pointed at something server-side.

A second detail pushed the same way: streamers broadcast this game without any trouble. So the block is not unconditional: there is a per-account whitelist somewhere, and the client asks the server whether to apply it. When that service blinked, every client fell back to not blocking, which is exactly the synchronized window we had seen.

Finding the mechanism

A quick search confirmed the API in use: SetWindowDisplayAffinity, from user32.dll. With the WDA_EXCLUDEFROMCAPTURE flag, any screen capture (Discord, OBS, a browser) sees the window as black.

The obvious tools, locked out

The natural first move is to watch the program run: attach something that logs the call the moment it happens. I tried API Monitor, filtered for SetWindowDisplayAffinity, and nothing showed up at all. I tried attaching a debugger, x32dbg, even as administrator, and the process refused to even appear in the attach list. The game’s anti-cheat was actively blocking anything that tried to observe it from the outside.

That is what pushed me to static analysis. If I could not watch the process while it ran, I would read the binary while it sat still.

Opening the black box with Ghidra

I had never done reverse engineering before. I installed Ghidra, loaded the game’s executable, and searched the import table for SetWindowDisplayAffinity. A single result: a single place in the entire codebase where that function was called.

void FUN_00be4080(int param_1, char param_2) {
    DWORD dwAffinity = 0x11;
    if (param_2 == '\0') 
    {
        dwAffinity = 0;
    }
    SetWindowDisplayAffinity(*(HWND *)(param_1 + 4), dwAffinity);
}

The decompiled code around it was more convoluted than a function this small has any right to be, and the file paths baked into the binary explained why: the client is written in Rust (the tao windowing crate turned up in the symbols). Unusual for a game, and the reason one function further up the chain was simply too large for Ghidra to decompile. This wrapper only executes the decision, too; the real choice is a flag sitting in a per-account structure, set from whatever the server sends back.

I patched the executable directly, swapping the instruction that passed the blocking flag for one that always passed zero, and watching the decompiler update in real time to reflect the change was the moment I viscerally understood what reverse engineering is. It never reached a running game, though: the launcher hashes the binary on startup, saw that my modified file was not the original, and quietly re-downloaded and ran the real one. In the decompiler the patch looked right; whether it actually worked I never found out, because the integrity check undid it before a single frame rendered.

The network layer

Capturing traffic with Wireshark, I found that the decision didn’t come from the game’s own servers, but from a third-party SDK (network optimization) embedded in the client. That explained the synchronization: when their server went down, the blocking signal disappeared for everyone at the same time.

What stuck

I didn’t solve the original problem: my friends still can’t watch me stream the game. But I came out understanding, hands-on, how Windows controls screen capture at the API level, how an anti-cheat shuts the door on anyone trying to watch or modify the process from outside, and how a client defends its own binary against exactly the kind of patch I made.

The best part wasn’t any specific discovery. It was treating something opaque and seemingly fixed as a system that could be understood.