Reverse Engineering a Game's Screen Capture Protection

It started with a simple frustration: my friends and I play an online game together, and we can’t share our screens on Discord. The game blocks it. Black screen, every time.

I could have accepted that. But something about knowing that a piece of software on my own computer was deliberately stopping me from doing something harmless made me curious. I spent the whole day down that rabbit hole.

The first clue

Before touching any tooling, I noticed something odd: weeks earlier, for a few minutes, everyone in the group could share their screen at the same time. Then it stopped again. For everyone, simultaneously.

If the protection were purely client-side, each computer would decide this independently. The fact that it broke and came back in sync pointed at something server-side.

Finding the mechanism

A quick search confirmed the API in use: SetWindowDisplayAffinity, from user32.dll. With the WDA_EXCLUDEFROMCAPTURE flag, any screen capture (Discord, OBS, a browser) sees the window as black.

Opening the black box with Ghidra

I had never done reverse engineering before. I installed Ghidra, loaded the game’s executable, and searched the import table for SetWindowDisplayAffinity. A single result: a single place in the entire codebase where that function was called.

void FUN_00be4080(int param_1, char param_2) {
    DWORD dwAffinity = 0x11;
    if (param_2 == '\0') 
    {
        dwAffinity = 0;
    }
    SetWindowDisplayAffinity(*(HWND *)(param_1 + 4), dwAffinity);
}

I tried patching the binary directly, swapping the instruction that passed the blocking flag for one that always passed zero. It worked. And watching the decompiler update in real time, reflecting the change, was the moment I viscerally understood what reverse engineering is.

The network layer

Capturing traffic with Wireshark, I found that the decision didn’t come from the game’s own servers, but from a third-party SDK (network optimization) embedded in the client. That explained the synchronization: when their server went down, the blocking signal disappeared for everyone at the same time.

What stuck

I didn’t solve the original problem: my friends still can’t watch me stream the game. But I came out understanding, hands-on, how Windows controls screen capture at the API level, how anti-cheats protect process memory, and how hooks work at the byte level.

The best part wasn’t any specific discovery. It was treating something opaque and seemingly fixed as a system that could be understood.