18 Comments
Mar 18·edited Mar 18

I got to really question the knowledge of those interviewed people. Someone who is saying that a software running in the application layer can do the same things as a kernel driver has either not knowledge and is ignorant about the topic OR is just LYING. Why do we even need all of those layers if they all can do the same thing anyway?

Expand full comment
Feb 24Liked by Ryan K. Rigney

Something so very aggravating about clicking through to that Steam post and seeing that the top reply basically just says "you're lying." And it even uses the word "verifiably"--naturally, it doesn't actually link or even reference any evidence that Lindgren is wrong. Maybe this is just all Discourse online now, but it's depressing how predictably and quickly gamers get whipped up into a righteous fervor. It's been like 15 years of gaming discourse feeling like a never-ending loop of this. And when you get through all the froth, there doesn't seem to be any substance. Like, I thought the complaints about Steam Deck compatibility were legit, and then I realized the post was weeks old and you can play the game just fine on Steam Deck (a tad bit finicky but that's very normal with the Steam Deck)… the complaints were pre-release and purely hypothetical. And, as has often been in the case in the past, I'm guessing there won't really be any evidence for the performance issues either.

Anyways, I really appreciated your piece and have subscribed. Good to hear from people in the industry and in particular I'm glad Riot has been willing to lift the veil a tiny bit. I will say that as someone who is especially paranoid about my security and privacy, I do dislike the idea of kernel anti-cheats, but I recognize I'm talking about a really marginal risk here and one that even paranoid people like me are already accepting on a daily basis. I don't play any games that use 'em, but that's not a principled boycott or anything. I just haven't fallen in love with one of those games yet. Though Riot's about to expand Vanguard to LoL, so that might change soon.

I will say that even though I know it's a great slam dunk for anti-cheat, I hate that Riot's Vanguard implementation requires it to be continuously on from device startup. That's a lot of trust I'm handing over in the sense that you really gotta believe they've done their best to avoid performance impacts, compatibility issues, and security risks. But in the age of SSDs and 10 second restarts, even my paranoid self can't really complain too much.

Expand full comment
Feb 24Liked by Ryan K. Rigney

I love your writing style Ryan. I miss it on apex with every update.

Expand full comment
Feb 23·edited Feb 23Liked by Ryan K. Rigney

I wonder when Koskinas made Maplestory cheats. @Ryan if you get a chance, could you ask him if he ever worked on Gamekillers Terminal or Gamersouls Blight, or if it was way earlier?

Expand full comment
author

I just asked Koskinas this for you and he said "oh no, GK/GS were way after my time"

Expand full comment

As a game developer, I would say that most peoples knows that game developers don't want to steal your data. The issue with kernel-level anti-cheat is that you end up having a lot of them installed on your computer. If only one is compromised with a supply chain attack, the attacker can access all the players computers.

A significant example of a supply chain attack was the Solarwinds cyber attack, in which their build server got compromised. Luckily for their client, Solarwinds products don't run in ring 0, so the attack impact was limited. Still, if they did, the attacker would have completely controlled all their customer's servers, including Microsoft, and all the data of Microsoft clients. If SolarWinds and Microsoft can get compromised, then a game studio or an anti-cheat developer can also have their build server compromised.

Nobody should have full access to your computer besides you because you can't audit their security and be sure they have good security practices or won't be breached.

Expand full comment

Meanwhile.. the chinese classified information dump was almost 100% all related to online hacking and stealing of US information from various platforms... including gaming..

but yeah.. no big deal.. Lets have Tencent sponsor every US release, no harm no foul right?

/eyeroll give me a break. Anyone who ISNT concerned about PC vulnerability is a complete toolbag. Full of Dollar General products no less.

Jazz hands and well wishing isnt going to distract people from the simple fact that between Russia and China, US based systems are under constant attack 24/7 365.

How long till they find the way in? .. more importantly, why are people giving them a head start by installing unreliable and dangerous rootkit software that has proven time and time again to be dangerous to the end user?

Post all the articles you want, but the fact you completely "left out" everything we just learned about China's info leak... is a HUGE red flag. Not to be trusted.

Expand full comment
Mar 18·edited Mar 19

Hi, I am a software engineer who used to work on video game (engine and graphics) so I think I have a decent understanding of how game dev works. I always roll my eyes at articles like this because when people get harmed by others, there are usually two reasons: 1) maliciousness, 2) incompetence. Game developers love to latch on to (1) and claim they have no incentive to harm their players etc, which may be true, but keep in mind we don't know what deployment or code review policies each team uses and from having worked in games before a lot of teams have a pretty lax code policy compared to other areas I have worked in (aerospace). It's not crazy to assume that backdoors could still be injected by malicious employees even if the company didn't want to.

The more important issue is (2) (incompetence). Programmers like to think of themselves as rockstars, but *everyone* makes mistakes. Having a kernel level anti-cheat significantly increases the attack vector compared to a user-level app, and if you have an unfortunate bug or supply chain attack (e.g. via a third-party package) the end result is more catastrophic. In security, we believe in defense in depth, and using a kernel level driver directly goes against that. When you talked about trust, note that you are asking for people to not just trust that you are not malicious, but that trusting you are not incompetent.

This also leads to the comment that a user-level program can access the webcam. I don't think this is necessarily true. It depends on the OS / hardware, but there are permissions that you have to seek before you can access the webcam. A kernel level driver can potentially do other things such as being able to record silently without turning on the light, etc. You essentially get to bypass all of the safeguards modern OSes have put on to prevent such malicious usages. Riot's Vanguard system also runs at boot, meaning that you are vulnerable all the time, rather than only when playing the game. Even if Vanguard doesn't talk to the internet, it talks to other programs on your PC, and those programs (including the software updater that updates Vanguard) talk to the internet. It's a layer of indirection, but if there are flaws in the way the IPC (interprocess communication) then Vanguard could be exploited all the same. The fact that it's an always-on kernel driver gives a lot of incentives for attackers to target it.

Ultimately, I get it. It's hard to do anti-cheat on PCs because of the power users have (although other OSes like macOS actually have stuff like app attestation that help in providing user-level apps some safeguards against cheats), but for me personally I value my security more than being able to headshot someone.

In order for this article to really be fair, you really should have interviewed third-party security professionals as well, aka people who work in security, but *not* in video game anti-cheat software. Obviously people who work on anti-cheat themselves are going to say their stuff works, no shit.

Expand full comment

Great article, but the one thing you don't seem to touch on is the fact that anti-cheat just plain doesn't work. From Roblox to Tarkov, Call of Duty to Helldivers 2, GTA V to Diablo 4, there are always going to be cheaters. Most anti-cheat solutions don't even reduce the frequency of the cheaters that much, because it only takes one smart hacker to break the anti-cheat and then distribute their knowledge or software to thousands of users, who can all exploit the hole.

Folks like Koskinas might compare anti-cheat to whack-a-mole, but hey, isn't that exactly what they'd be doing if they *didn't* have an invasive kernel module loaded onto my system? And by not shipping said kernel module, they would be allowing their game to run on Mac and Linux through Wine, and eliminating any possibility that their anti-cheat kernel module might be crashing peoples' systems, which is a widely reported issue; or that the anti-cheat kernel module exposes vulnerabilities of its own and gives programs (or even servers online) low-level access to your system through an unintended backdoor.

If kernel anti-cheat actually worked, I might even accept it as a useful tool in the toolbox. But other than Valorant, which just appears to be significantly ahead of the cheaters' technology right now (but I guarantee that will change in a few months) -- most kernel anti-cheats are useless.

They're even more useless when they don't evolve and constantly improve, stepping up their anti-cheat game in the never-ending cat and mouse. nProtect is especially poor in that area, as we're about 2 weeks into Helldivers 2, and nProtect is already losing the battle against the cheaters. Cheating by memory editing is becoming very pervasive already.

Kernel anti-cheat is a solution looking for a problem, but in the end, it just ends up creating more problems. User trust problems, technical glitches, unintended vulnerabilities, Mac and Linux compatibility showstoppers, and in the end, for what? Most people with 40+ hours in Helldivers 2 at this point have seen at least one cheater; if they haven't, they're not paying attention. No, you and your team didn't manage to loot 99 rare samples on that mission.

Expand full comment

Good article in terms of giving insight of the developers view. But the ending is a bit condescending towards end-users, while there certainly is a lot of misinformation, there is also a lot valid complaints from them. Many which developers want to sweep under the rug, but in the end, many potential users dont consider kernel-level anti-cheats to be worth the cost in privacy, PC security, closing off the system, risks of it messing with OS and hardware etc., for the moderate increase in anti-cheat capabilities.

And it makes sense for devs in kernel anti-cheat to see Vanguard as the industry leader, but the future most likely lies in server-side anti-cheats and ML.

Expand full comment

Anti-cheat software should be server side. So the software can monitor all interactions from the server side that comes from the client. Now that we have player data that can be logged and analyzed. Now we use just the tiniest amount of science. We know that within the confines of the game there are limits and that humans have limits. Of the two the human limits are easier as the former limitation data is dependent on the game. That being said we have plenty of accurate data on how humans aim in games and on human reaction times and behaviors from the bottom to the top. Using that data and the science of reaction speed you could make an extremely hard to trick anti-cheat. Combined with in game stat tracking and game specific mechanics {I.E. Ghost players to spoof the aim-bot} you could make a nearly perfect anticheat. Basically if you focus on the code your going to get nowhere. You need to focus on the player and their capabilities vs the capabilities of a computer.

Expand full comment

From a scientific perspective an anti-cheat should at the verry least be able to accurately detect 90% of cheats. Player reaction aiming type and button presses are all unique enough that it can easily be differentiated by a machine. We know scientifically the limits of human reactions. We also have plenty enough data on stream for how people, even good people aim. We have the data its just not being used. If someone who plays your game can recognize cheats a computer should be able to do it faster and better.

Expand full comment

Hey, it's me - "this guy" from the steam review. Still crazy to me how it blew up like that.

Yes, I continued playing Helldivers 2 for quite a while longer after giving it a negative review. I enjoy the game, but I don't enjoy what's come with it - the out-of-touch balancing decisions, the whole PSN and region-locking debacle, in addition to bundling gameguard with it. Reviews are one of very few effective ways for me as a consumer to communicate with the developer/publisher about such issues. When it comes to kernel anti-cheat, or specifically gameguard - as I understand it, the damage was already done if I've installed the game and not practically factory-reset my system.

At the time of posting that review, my understanding of kernel-level anti-cheat was pretty surface level - still is, on the scale of things - but the more I've learned about it since, the more vindicated I feel in my stance on the matter.

I've also played a ton of other games that use kernel-level anti-cheat. In each of those cases, you aren't allowed to play the game without accepting the installation and operation of such software - and I often want to play the game its self enough to reluctantly, begrudgingly compromise on that. Doesn't stop me from being very uncomfortable with anyone who isn't me - and especially for-profit third-parties - having such unfettered access to my machine. Especially when it's less than a comprehensive solution to the problem it's meant to solve, and not significantly better than alternatives for the trouble.

If I want to play the game it's attached to, I have no choice but to hope that they won't mess something up, have their own security compromised, or otherwise mishandle or take advantage of that level of access, in a way that negatively effects me. Many people have reported issues with the game, if not the PC its self, as a direct result of gameguard. Sure, the veracity of each of these reports is dubious, but the state of the industry being what it is, it wouldn't surprise me to learn if any were true.

To make matters worse - I imagine a majority of customers do not understand this covert contract they're signing up to. If you were to ask every HD2 player for example, "Are you okay with our software running on your PC at a higher level of authority than your very operating system, in order to [maybe, hopefully] catch/prevent cheaters?" how many do you reckon would say "yes"? How many do you reckon would still say "yes", if their ability to play the game they'd paid for, weren't contingent on their answer being "yes"? If that proportion isn't a large one - is that solely due to a lack of understanding?

I don't believe anti-cheat developers need kernel access to client PCs, in order to create effective anti-cheat. Just look at VAC (and CS:GO's Overwatch system). In my experience, across all the games that've used kernel anticheat, they've not been significantly better when it comes to mitigating the number of cheaters I encouter, compared to VAC or anything like it. Granted, that's a hard thing for me to quantify as a mere player without access to hard data or the education to interpret it.

And this is all in relation to a co-op game, where the extent to which a cheater could negatively affect my enjoyment is relatively minimal. The worst I've heard of HD2 cheaters accomplishing is dumping a bunch of in-game currency on the 3 other players, or "spoiling" upcoming content that has been implemented, but is yet to be officially released. I can just kick them from the game if I'm lobby host, or leave and join/start another if I'm not. The currency thing, I imagine would be relatively easy to detect server-side, and it's kinda crazy to me that that's even a thing in spite of the almighty kernel anti-cheat. It's especially a non-issue, if I'm only playing with people I know.

In any case, it's been infrequent enough for me to have never encountered it myself. Is that down to the anti-cheat operating on the kernel level? If so, why is it stopping some but not all? Does gameguard effectively stop people from using even more egregious cheats - to an extent that it otherwise couldn't, if it were designed to run at a lower level of privilege? Or, is it just that the "incentives" aren't really there to cheat in a co-op game?

Of course, there will always be cheaters, and the arms-race between them and anti-cheats is constant. I come from the era of multiplayer games where the majority, if not all online servers were privately hosted - which often had regular, recurring communities build around them, and live moderation through server admins, who were given adequate tools to effectively supplement anti-cheat software, and supported by said communities in doing so.

With the mainstream trending toward "live-service" models, matchmaking and more atomised player-bases - where moderation and control is more consolidated with the developer - I realise that that's less practical. One small part of why I hate said trends.

I reckon the best anti-cheat - if not, at least the kind I'd be most comfortable with, and is the best compromise between security against cheaters, and privacy and system security - will always be human oversight and moderation, with the tools to facilitate it. Replays and demos, actively moderated community servers, and/or official game moderators; reporting systems and moderation outsourcing like CS' Overwatch. Between that, and user-level or server-side anti-cheat software - whatever kernel access brings to the table alongside its risks, seems to outweighed from what I've learned, and according to my anecdotal experiences.

Expand full comment

disappointing missed interview opportunity most the article didn't talk about nGuard and when it did it didn't address much i.e. comments I seen such as: "Most ACs if they think is a cheat, close the game.

This AC, if detecting something it thinks is a cheat, shuts the 'cheat' down. If it erroneously thinks a windows process is a cheat, whoops, crash. If it thinks your CPU cooling fan is a cheat, PC may burn. It's known to cause issues where it interferes with programs it has no business interfering with, the damage it causes to hardware, the lies about how the program will uninstall with the game and that it only runs with the game, even after you manage to get rid of it, it will still leave backdoors into your system completely bypassing firewall and AV, its also the fact that this malware blocks several firewalls and anti-virus softwares"

I get that if an AC was such a widespread hardware killer it may be more known though it don't help posts have been getting deleted by Helldivers2 mods about AC complaints

Expand full comment

Paranoia aside, the funny part of using an outdated, often-proved ineffective tool such as nProtect GameGuard is that it took cheaters less than a week to discover flaws in the game protection enough to ruin some players' progression. This is a really weird call considering there are better players in the same industry, such as Easy Anti-Cheat.

Expand full comment

Easy isn’t doing the job, just ask Apex players.

Expand full comment

Apex is competitive vs Helldivers 2 being a non-competitive PvE game. Easy Anti-Cheat would be more beneficial for Helldivers 2 than nProtect GameGuard, its flaws, and the current cheaters in the game.

Expand full comment

Yeah, maybe. I don’t pretend to know how they work, I just notice when they don’t.

Expand full comment