Blog

How to solve the cheaters problem in Counter Strike, with or without machine learning

Cheating, or hacking, as it often called in this game, takes two basic forms. One of them is knowing where the opponents are, most often seeing them through the walls (hence wallhacking). The other one is aim assistance. If you’ve played any single-player shooting games, you know that sometimes, on lower difficulty settings, the game helps you with aiming. You don’t need to be precise, just aim at the general direction of the enemy. In multiplayer games, with humans on both sides, this wouldn’t work. But cheaters use software to achieve the same effect. Both these forms introduce a profound unfair advantage and ruin the experience for other players.

Countermeasures

How does Valve detect cheaters? There are two ways. Unfortunately, in practice both of them mostly don’t work at all.

The first line of defense is VAC, which stands for Valve Anti-Cheat. It’s a software that runs on a player’s computer and detects some cheats (like an antivirus detects viruses). More on this below.

The second way uses community’s help. This system goes by the name Overwatch (there is an unrelated game of the same title). People report suspected cheaters and then good players review recorded gameplay demos of suspects and issue a verdict. Then it’s back to Valve’s black box and some people get banned in the end. You can see how it works from the judging player’s perspective in this WarOwl’s video.

Valve uses Overwatch to get labelled data for their machine learning system, called VACnet. The idea is to learn to detect cheaters, and based on that, automatically select players for review in Overwatch. The diagrams come from a presentation on Using Deep Learning to Combat Cheating in CSGO, in which John McDonald talks at length about the topics that we skim here.

The presenter would have you believe that VacNet has been a great success. In a sense, it has. If you watch the video, you can see that the conviction rate in Overwatch went waaaay up after introduction of VACnet in early 2017. This probably nudged Valve to make the game free-to-play in late 2018. They must have known that it would encourage cheating, but wanted to believe that VACnet would cope.

It wouldn’t. The cheaters are still there. A LOT OF THEM.

Better labels

In Overwatch, the verdicts are formulated on a “beyond reasonable doubt” basis. This means that there should be few false positives, but unfortunately many false negatives. In other words, the precision is high, but the recall is low.

This is a problem, because a system constructed like that can only detect blatant cheaters. It is certainly good to ban them, but it doesn’t address the real issue.

Moreover, the underlying reality is that more often than not, it’s difficult to be 100% sure that someone is cheating just by viewing his gameplay. For a smart cheater (an oxymoron, we know) it is actually quite easy to stay unbanned.

For example, cheaters can toggle their hacks on and off. When someone reviews the demo, one shot may look suspicious. Then another one. Then the cheater toggles off, and is back to his mediocre self. A couple of poor plays and the reviewer comes to a conclusion that the previous good shots were just lucky. Everybody gets lucky from time to time, you know.

Still, a person with some experience develops a nose for cheaters. While you cannot be 100% sure in some cases, you can give an estimate. For example “probably not cheating”, or “might be cheating”, or “pretty sure he’s cheating”. These words express probability, and in our opinion it would be very beneficial if Valve introduced probabilistic, percentage based verdicts into Overwatch. Just add a slider, or a list with 0%, 10%, …, 90%, 100% to select from.

Note that these will work just fine for machine learning models. Log loss, or cross-entropy, the basic loss function (and evaluation metric) used in classification, should work with probabilistic labels. As far as the math goes, percentages are as good as zeros and ones. One could also try training a regression model.

The reason for the current all-or-nothing setup is pretty obvious: they don’t want to ban innocent people. It’s similar to the situation with spam filtering - better to let a few spam messages slip into the inbox than to put a legitimate mail into the spam folder.