The Gamers Do Not Understand Anti-Cheat
push to talk #8 // feat. people who've led anti-cheat for Riot Games, Roblox, and Fortnite
In recent years a number of game devs—most recently the Helldivers 2 team—have faced widespread outcry from players after implementing kernel-level anti-cheat drivers in their games.
To learn more about this phenomenon, I decided to interview people who've led anti-cheat teams on games like Fortnite, Roblox, and VALORANT.
Find that story below. First up, this week's spiciest news links.
Scuttlebutt and Slackery
The week’s most-shared, oft-Slacked, and spiciest games industry news links.
Final Fantasy VII Rebirth is Universally Acclaimed - Apparently the latest game in the ongoing FFVII remake series absolutely rules, and I’m pissed about it. My third child is going to be born sometime in the next couple of months, which means I probably won’t have time to play it until sometime in 2025. In the meantime, I hope the rest of you enjoy it. Jerks. (Metacritic)
Sony is Lowkey Roasting Bungie’s Leadership in Public Now - In a Q&A after a financial results briefing, Sony president and PlayStation chairman Hiroki Totoki said that he feels Bungie has “room for improvement from a business perspective with regard to areas such as the use of business expenses and assuming accountability for development timelines. I hope to continue the dialogue and come up with some good solutions.” (Video Game Chronicle)
What Went Wrong with Immortals of Aveum - Stephen Totilo has a fantastic interview with Ascendant Studios lead Bret Robbins about the commercial failure of his team’s recent FPS release. “You don't hear a lot of stories about what happens when not everything goes right,” Robbins says. “I think it's useful for people to hear about it.” (Game File)
Bring Your Gun to a Swordfight - The greatest game trailer of 2024 has already been released. Don’t believe me? Just watch through at least the 23-second mark of the official announce trailer for Kingmakers. (YouTube)
An Analysis of Xbox’s Game Pass Business - Simon Carless from GameDiscoverCo has been consistently on-the-mark in his predictions about Xbox’s strategy for Game Pass, so his recent deep dive into the biz is welcome. Key quote: “Unless you can prove prestige first-party titles are retaining subscribers, there’s going to be more focus on GaaS-ish Grounded & Sea Of Thieves-like games that can also monetize on other platforms.” (GameDiscoverCo)
Why Can't Gamers and Devs Agree on Anti-Cheat?
The big story this past week has been the rapid rise of Helldivers 2. The game's a monster word-of-mouth hit, and continues to break its own concurrent player count records (and its servers) day after day.
Basically everyone agrees that Helldivers 2 is an incredible game, but—alas—it only has a 71% positive review rating on Steam. The problem according to players (aside from server hiccups) seems to be its anti-cheat software, nProtect GameGuard, which uses a kernel-level anti-cheat driver.
Helldivers 2 Technical Director Peter Lindgren (who politely declined to comment for this piece) recently put out a detailed statement aimed at reassuring players about the studio’s approach to anti-cheat.
Lindgren’s post is detailed and reasonable. And yet, the gamers remain suspicious, despite Lindgren stating outright that claims like those from the above reviewer—for example, that GameGuard “still lingers after uninstallation"—are simply not true.
So who's right: the players or the devs?
I decided to ask one of the smartest guys I know: Paul Chamberlain.
On Security and Paranoia
Chamberlain is currently Principal Software Engineer at Odyssey Interactive (disclaimer: I lead marketing for Odyssey) but he previously led the anti-cheat team on Fortnite. Before that, he worked at Riot Games, where he led the anti-cheat team on League of Legends. It doesn’t stop there, folks—Chamberlain’s résumé is insane. He has worked in cybersecurity for Google, the Australian Federal Police, and the Australian Signals Directorate, which is basically Australia's version of the NSA.
My man understands cybersecurity. So it may interest you to hear his reaction to the Steam review I shared above.
"Anti-cheat is such a cursed field to work in," Chamberlain says. "Developers have no incentive to steal your data or hurt your computer, and if an evil developer did want to harm their players then they don't need anti-cheat software to do it. Installing their game would be enough."
This caught my attention. Chamberlain isn't saying that players should worry less—he's saying players are worried about the wrong thing.
"Stealing your nudes, getting your passwords, stealing your bank info... none of these things require a kernel-level driver," Chamberlain says. "All of that is possible with a regular application that you install on your computer. I don't need a kernel driver to stealthily record your webcam. I don't need a kernel driver to get your credit card info."
What is a kernel-level anti-cheat driver?
Essentially, kernel-level drivers run at a very low level of your computer's operating system, and (usually) boot up before other software. Anti-cheat devs like this because it gives them a more comprehensive view of what software is running on your machine, which lets them detect cheat software more easily.
Chamberlain gives the following illustrative example of the sorts of challenges game developers face when they don't use a kernel-level anti-cheat driver:
"Whoever's driver loads first gets to observe or influence everything that happens on your computer after that point. It's kind of like a race.
The first famous cheat to do this was WoWGlider—the botting service for World of Warcraft. They were hiding from Warden, which was Blizzard's anti-cheat.
When Warden is scanning it would hide. When Warden wasn't scanning it would put it back."
Ultimately, Blizzard struggled for years to stop WoWGlider's influence, and they only managed to shut the service down in 2011 via a lawsuit against WoWGlider's developers.
Clint Sereday, the Head of Anti-Cheat at Roblox, agrees with Chamberlain. "Any software you put on your machine can be used to take it over," he tells me.
If any game developer—even those who don't use kernel driver anti-cheats—could in theory steal your information, how can players trust anyone?
When asked this, Chamberlain throws his arms up in the air in a what can I say? gesture. "You already have to trust the people who provide the software you run. When you run software on your computer, it's acting on your behalf. It can do anything you can do. If you can use your webcam, it can use your webcam. Kernel or not kernel, it does not make a difference to the level of danger posed to you by unknown software. The whole argument is kind of a distraction."
Ultimately, Chamberlain says, it comes down to one question: "Do you trust the people making the software to do right by you?"
And with this question, we get closer to the real root of the problem.
The Trust Problem, and Risks of Kernel Drivers
If you've ever worked on video games, you know how important player trust can be, and how hard it is to keep. When I worked as a spokesperson for games like League of Legends, PUBG, and Apex Legends, I've often been frustrated by the cynicism of players online, particularly the most vocal ones who tend to dominate discussion in online communities.
The only reliable cure for cynicism is good, honest, transparent communication, as well as a track record of following through on your promises. I often tell devs that if you don't explain why you're doing what you're doing, somebody will make up an uncharitable explanation and people will believe that instead.
I believe this dynamic is at play with the discussions around anti-cheat in games. Developers are often hesitant to discuss the nuanced details of the anti-cheat solutions they're using because they don't want to give information (or "signal," in industry parlance) to players. As a result, players assume the worst: this is all some ploy to steal my data.
And, in truth, there are additional risks to users when you deploy software that operates at the kernel level. One of the most infamous examples of this is Sony BMG Music Entertainment's Extended Copy Protection, or XCP, software originally released in 2005. The software was essentially secretly installed, without warning to users, for anyone trying to listen to one of ~52 albums like "B in the Mix: The Remixes" by Britney Spears or "Rebirth" by Jennifer Lopez using their home computer's CD drive.
The XCP software was invasive and, crucially, it was also poorly-designed. It had vulnerabilities that allowed anyone making malware or viruses to make their software run invisibly on XCP-infected machines simply by putting the text string “$sys$” at the start of their process's name.
This, as you can imagine, was an absolute boondoggle. The resulting controversy went international. For a laugh, listen to this 2005 radio segment where NPR tries to explain to its audience what a "rootkit” is.
These sorts of mistakes have happened with games, as well. Roblox head of anti-cheat Clint Sereday recalls the time that a badly-designed anti-cheat solution added via an update to Street Fighter V introduced a back-door vulnerability.
"They slapped together some kernel-based driver," says Sereday, "and for years that became the main attack vector that cheat-makers would use to load their code and run cheats even for games that weren't Street Fighter V.”
You'd see people at Street Fighter V tournaments holding up signs that just said “ROOTKIT,” says Sereday with a laugh. Still, after the vulnerability was patched, Sereday still installed Street Fighter V and played it.
The lesson? "Any time you work in the kernel you have to be very careful."
As for his own company's approach to anti-cheat, Roblox doesn't use a kernel-level driver. Sereday says that although Roblox does have some powerful custom-built anti-cheat solutions, it all runs at the user-level.
The reasoning? Partly, it's because of Roblox's multiplatform nature. "It's hard to have good security when you're on 10 different devices," Sereday says, "and Roblox is on so many devices." But there's also a difference in the types of attacks that Roblox faces compared to other games.
Cheating in Roblox, Sereday says, “is a different beast." Sereday's team has made an effort to reach out to Roblox cheat developers that has resulted in the game's cheat problems being relatively limited. "We've seen a dramatic reduction in exploits from what were staggering numbers," he says.
However, Sereday says, some teams in the industry are up against much more aggressive cheat-makers and cheaters. He points at the anti-cheat team at Riot Games as the leaders in the space: "Riot is far and away the best," Sereday says, calling the League of Legends and VALORANT developer "the industry leader from a technical standpoint."
Says Sereday: "The VALORANT competitive ladder is one of the safest competitive ladders in gaming."
To find out why Riot is considered best-in-class at anti-cheat, I decided to reach out to Phillip Koskinas, the Head of Anti-Cheat at Riot Games.
Riot Games and Its Approach to Anti-Cheat
There is a caveat that you always have to share upfront when talking about anti-cheat techniques in public, which is that the game devs can't tell you anything that might help the cheat-makers make better cheats.
This, says Koskinas, is why you don't see anti-cheat devs talking very often. "We're always trying to detect people who are willing to do just about anything to be undetected, and any time we open our mouths or do a banwave, we give them more signal into what we do or don't do."
Lots of players don't understand this basic dynamic, and it's not uncommon to see players demanding more transparency than would actually make sense for devs to offer.
"People might like to say 'just open source the anti-cheat,' but if we did that, we couldn’t exactly call it an anti-cheat," says Koskinas. Players say "'Just show me all the checks you're doing and I will feel comfortable.' And so would every cheater! Hopefully one day this kind of cat and mouse thing won't be required, but until Windows tightens up, we have to do a lot of the lifting ourselves."
That being said, Koskinas is willing to talk about Riot's widely discussed kernel-level anti-cheat driver for VALORANT, known as Vanguard.
Ever since Vanguard was first announced, players and cheat-makers have cried foul. But the Riot Games team has held firm in their defense of the kernel-level tech, and regularly put out comms discussing updates to their approach.
“It’s mostly about ensuring the host’s security and having a stronger perimeter around the game client,” says Koskinas. Simply put, Vanguard's kernel driver reduces what anti-cheat professionals call attack surface. "With the driver, we cut our attack surface in half," he says.
By catching most cheats early and labeling cheaters and their machines, Riot can effectively make it so that cheaters are forced to buy new hardware if they want to keep playing VALORANT. "Continually buying new hardware to avoid identity bans slowly drives up the cost of cheating," says Koskinas, "and we’re really into increasing the likelihood that Mom notices charges on the AmEx.”
That said, even when using Vanguard, there are still other ways that cheat-makers can use to try to slip around Riot's defenses.
These “goofy” attack vectors, Koskinas says, are things “like DMA (reading system memory with external hardware), input injection (like Arduinos spoofing mouse input into the game), and image classification (screen reading for heads to click)."
You might have noticed that acronym that Koskinas used above: DMA. That stands for "direct memory access," and it refers to an increasingly rampant type of cheating method that involves side-loading cheats onto a machine via an actual physical hardware device. The idea is to run the nasty cheat-related code on a separate computer—sometimes literally something as simple as a Raspberry Pi—and then disguise that as a harmless peripheral like a mouse or keyboard.
Although Riot is widely known for Vanguard, anti-cheat experts throughout the games industry tell me that they're also world-class in anti-DMA techniques. When asked about this, Koskinas grins and admits that, basically, he's not able to say much about it publicly. The technology is that secretive and proprietary.
"Almost six years ago now, Everdox cooked up a theory to detect and prevent DMA cheats through a scholarly level of hardware research. It’s been a slow rollout since then, but we’ve managed to stay ahead of the worst of it, for the most part due to his expertise.”
The 10,000 Pound Panda in the Room
One of the biggest challenges for the Riot Games anti-cheat team is public perception of the studio's relationship with its parent company, China-based Tencent. A common concern from players is that, via Vanguard, their data might be being scraped and sent overseas.
Koskinas—with the backing of Riot's corporate comms team, who signed off on Koskinas's statements for this piece—flatly denies this, with two main supporting arguments.
First, Koskinas challenges the idea that Vanguard is inappropriately gathering player data:
“The driver itself has no connectivity to a server," says Koskinas. “At boot, it’s just a bunch of preventative checks on whether Windows is currently in a trusted state. Once you launch a game, Vanguard confirms the driver thinks everything’s good to go, and if it is, you can play the game. Even once we’re running, we’re not like sending files back to our servers.”
According to Koskinas, it’s just bad risk management to gather unnecessary data.
“Every piece of data we hold at rest is [a] risk for Riot to deal with, so when we do have to collect something, the retention rates [author’s note: the amount of time the studio holds onto data] are absurdly low," says Koskinas. "A good percentage of the magic is just shipped as queries to the client with replies that are only binary (true or false)."
Secondly, Koskinas says, there are strict firewalls in place between Riot and Tencent when it comes to Vanguard in particular.
"Tencent doesn't have any access to Vanguard," Koskinas says. "I've met them like twice in the last ten years and the only thing we exchanged was high-fives."
In fact, Koskinas says, Tencent doesn't use Vanguard in China, in part because they have totally different problems to solve. "Their attack surface is also huge. They still have to support Windows 7, and luckily, we don’t have to play on inferno difficulty."
But What About GameGuard?
Before we close, let’s bring it back to Helldivers 2 and its kernel-level anti-cheat solution, nProtect GameGuard.
I wondered: what do anti-cheat leads in the games industry think about the software? Is it as bad as players think?
“I'd call it mid,” says Paul Chamberlain. “It’s ok tech, but not a leader.”
Riot’s Phillip Koskinas has memories of GameGuard stretching back over a decade. “A version of it has been around since even I was a young lad cooking cheats on MapleStory and Gunz,” he says. (Almost all anti-cheat developers were once cheaters themselves, as they’ll gladly tell you.)
“Back then,” says Koskinas, “GameGuard was actually one of the ‘stronger’ anti-cheats, largely due to their willingness to do crazier stuff. A lot of these techniques are actually common sense or useless now, but back then, it was sort of atypical to be that wide of a watchdog. These days, it has held onto a reputation for being invasive, especially because it takes so much of its anti-cheat actions locally and instantly (closing windows, blocking processes, etc.)."
Roblox anti-cheat lead Clint Sereday was similarly unperturbed. He says that the negative player reviews Helldivers 2 is getting are off the mark.
“This is exactly the same concern as people have with any anti-cheat,” Sereday says. “The concern is a little more exaggerated here because Helldivers 2 is a PVE game. nProtect GameGuard could also be adding to the concern since it is a less-used 3rd party.”
But that’s not stopping Sereday from playing Helldivers 2. “Great game by the way,” he says.
To all my dearly beloved gamers reading this post, I know what you’re thinking: Yeah right, bro! Of COURSE these corporate goons say their rootkits are safe. That’s what THE MAN wants you to think!
And look, I get it. There’s a lot of Reddit karma to be farmed by posting something cynical about these guys who have spent their entire careers leading anti-cheat on the world’s biggest games. What do they know, right?
But hear me out.
Maybe, just maybe, the average Steam reviewer’s understanding of the risks of various anti-cheat solutions isn’t always fully informed. Maybe (hear me out, hear me out!) the risks of installing games on your machine are actually more scary than you previously expected, but for different reasons than those you’ve been told.
Is it possible that the world is messier than we’ve previously believed? Do we live in a world with loads of bad actors who exist simultaneously alongside the majority of game devs who are just regular people doing their best to make entertaining experiences that aren’t ruined by cheaters?
Is acquiring more knowledge about anti-cheat a form of curse in itself?
“For in much wisdom is much vexation,” wrote the ancient philosopher king who authored Ecclesiastes, “and he who increases knowledge increases sorrow.”
I’m just tossing out ideas, here, man.
That’s all for today’s post. I’m gonna stick a Ricky Martin CD from 2005 in my laptop’s disk drive and try not to get my identity stolen.
See you next Friday.
I got to really question the knowledge of those interviewed people. Someone who is saying that a software running in the application layer can do the same things as a kernel driver has either not knowledge and is ignorant about the topic OR is just LYING. Why do we even need all of those layers if they all can do the same thing anyway?
Something so very aggravating about clicking through to that Steam post and seeing that the top reply basically just says "you're lying." And it even uses the word "verifiably"--naturally, it doesn't actually link or even reference any evidence that Lindgren is wrong. Maybe this is just all Discourse online now, but it's depressing how predictably and quickly gamers get whipped up into a righteous fervor. It's been like 15 years of gaming discourse feeling like a never-ending loop of this. And when you get through all the froth, there doesn't seem to be any substance. Like, I thought the complaints about Steam Deck compatibility were legit, and then I realized the post was weeks old and you can play the game just fine on Steam Deck (a tad bit finicky but that's very normal with the Steam Deck)… the complaints were pre-release and purely hypothetical. And, as has often been in the case in the past, I'm guessing there won't really be any evidence for the performance issues either.
Anyways, I really appreciated your piece and have subscribed. Good to hear from people in the industry and in particular I'm glad Riot has been willing to lift the veil a tiny bit. I will say that as someone who is especially paranoid about my security and privacy, I do dislike the idea of kernel anti-cheats, but I recognize I'm talking about a really marginal risk here and one that even paranoid people like me are already accepting on a daily basis. I don't play any games that use 'em, but that's not a principled boycott or anything. I just haven't fallen in love with one of those games yet. Though Riot's about to expand Vanguard to LoL, so that might change soon.
I will say that even though I know it's a great slam dunk for anti-cheat, I hate that Riot's Vanguard implementation requires it to be continuously on from device startup. That's a lot of trust I'm handing over in the sense that you really gotta believe they've done their best to avoid performance impacts, compatibility issues, and security risks. But in the age of SSDs and 10 second restarts, even my paranoid self can't really complain too much.