Anatomy of a DDoS: what happened on 20 October, and what we changed overnight

Pročitaj ovaj tekst na srpskom

If you were in the city around 00:37 UK time on Tuesday 20 October, you saw what everyone else saw: the world froze, a timer started counting, and then "connection timed out". Balkan went first, English a minute later.

We were hit by a DDoS attack. This post explains what that means in plain English, what we changed the same night, and what we learnt. (We have left out anything that would help the people doing it.)

The short version

• Someone flooded both game servers with junk traffic, so much that real players' traffic couldn't get through.
• The servers never crashed. Your character, cars, money and items were safe the whole time. You were knocked off because your game couldn't hear the server for a few seconds, not because anything broke.
• That same night we put new protection in place. When someone tried again two hours later, nobody even noticed.

What is a DDoS?

Imagine the only road into the city. Normally it carries a steady stream of cars: that's you and everyone else playing. A DDoS is someone sending thousands of empty lorries down that road at once. Nothing is wrong with the city itself, but nobody real can get in.

In computer terms, everything you do in game (walking, driving, talking) is sent to the server as tiny messages called packets. The server reads them one by one. Flood it with millions of fake packets and it spends all its time reading junk. Your real packets wait in the queue, and after a few seconds your game gives up.

"Distributed" just means the junk comes from lots of places at once, which makes it much harder to block than one person.

How big was it?

On a normal night, about 500 packets a second arrive at our servers. Here is what that night looked like, minute by minute:

• 00:37: traffic jumps to 475,000 packets a second, almost a thousand times normal. Balkan is hit first, then English.
• 00:42: the first wave stops.
• 00:45 to 00:54: a second, smaller wave hits English.

The normal traffic is there on that chart. It is just so small next to the attack that you can barely see it.

Inside the attack

Here's a closer look at how it actually worked.

Tiny packets, on purpose

The junk packets were tiny: about 70 bytes each, roughly the size of a short text message. That sounds harmless, but it was deliberate.

There are two ways to overwhelm a server. One is to send huge amounts of data and fill up its internet connection, like blocking the road with enormous lorries. The other is to send huge numbers of small packets, like sending a million motorbikes. Each one is light, but the server still has to stop, look at every single one and decide what to do with it. That's what this attack did. It wasn't trying to fill the pipe; it was trying to keep the server too busy checking junk to read anything real.

Why it went for the network thread

A FiveM server has a part whose only job is reading incoming traffic. Think of it as the post room. Your movements, your car, your chat all arrive there first and get passed on to the game. During the attack, Balkan's post room went from almost idle to around 80% busy in a minute. It was opening millions of junk letters, and so many were arriving at once that the pile overflowed and some of them, including real ones, were thrown away before anyone read them.

The game itself, the part that runs the city, was never overloaded. It just stopped receiving your messages.

Two kinds of junk

The two waves hid in different ways, and that mattered for how we fought back.

• Wave one came from a botnet. These are ordinary home computers, routers and phones around the world that have been infected with malware and are being remote-controlled, almost certainly without their owners knowing. Each one sent a few thousand packets a second. On its own that's not much, but together they were a wall. We checked every one we could identify against our records: none of them had ever played on our servers. This wasn't a player using their own connection; it was rented or hijacked equipment.
• Wave two used fake addresses. Every packet on the internet carries a "from" address, like the return address on an envelope. Nothing checks it's true. In wave two the attackers wrote made-up return addresses. In just 20 seconds we saw more than 135,000 different "senders", which were really a much smaller number of machines pretending to be lots of people. Blocking addresses doesn't help when the addresses are made up, and that is exactly why the attackers did it.

It moved around

The attack didn't just hit one server. It hit Balkan for about two minutes, then switched to English, then came back for English again a few minutes later with the second wave. Switching targets like that is a common trick: it checks whether the defences are the same everywhere, and it makes the attack harder to pin down. Because we were recording every minute, we could see each switch as it happened.

What we changed overnight

1. A bouncer on the door

This is the big one.

When your game joins the server, it always does a quick "knock" first, a normal web connection, before it starts sending game traffic. You never see it happen. The clever part: a fake address can't knock. The server answers the knock, and that answer goes to the real owner of the address, not to the attacker. So the attacker never gets to finish it.

Now the server keeps a guest list of everyone who knocked properly. Game traffic from someone on the list goes straight in. Everything else is thrown away at the door, before the game server even looks at it.

For you, joining works exactly as before. For a flood of fake packets, it is a brick wall.

2. Speed limits and a block list

Every sender now has a sensible speed limit, far above what any real player needs. We also block the kinds of traffic attackers like to bounce off other people's servers to make their floods bigger. The botnet machines from that night are now on a permanent block list.

3. Alarms

The servers now record the traffic every minute, and our network security team gets an alert the moment things look wrong. The alert says which server is being targeted, whether the senders look fake, and whether players are being affected. When it's over, it posts a summary.

4. Locking the other doors

When someone tries one door, you check all of them. We went through everything on our servers that could be reached from the internet and shut anything that didn't need to be open.

Did it work?

Two hours after the main attack, someone tried again. A single machine started firing about a thousand packets a second at the English server. It had never played and never knocked.

The bouncer turned away about 128,000 of its packets in two minutes. The players online saw nothing: no lag, no timeouts. The alarm went off in our security channel while it was happening, and the sender went straight onto the block list.

That kind of small "test" is usually someone checking whether the door is still open. It wasn't.

Behind the scenes

None of this happens by itself. Behind Roleplay Project is the Phytoventures team, and a big part of what we do never shows up in the city: watching, learning and building against exactly this kind of attack.

• We monitor our network 24 hours a day, 7 days a week. Traffic, server health and player connections are measured every minute, and our network security team is alerted the moment something looks wrong, day or night.
• That's why outages are measured in minutes, not hours. On the night, both waves were spotted as they started, and the new defences were in place before the end of the night. The faster we see an attack, the faster we can shut it down.
• We work with our network provider. Their systems spot large attacks heading for us and flag them, and they follow up with the networks the attack traffic came from. They are part of the team, not just the people who plug the cables in.
• Every attack makes the next one harder. Each incident gets studied afterwards: what it looked like, what got through, and what we can build so it doesn't get through next time. Everything in this post came out of that process.

Why we don't see this as a bad thing

It might sound odd, but we don't see an attack as a defeat. Every attack teaches us something. Each one shows us what the attackers are trying, how their traffic looks, where it comes from and how it changes when we push back. That builds up our knowledge, and our defences, for next time.

The bouncer, the alarms and the block list in this post all exist because of that one night. The attackers who tried again two hours later ran straight into what their own attack had taught us. That's how it works from here on: every attempt makes the next one harder, and the community better protected.

What we learnt

• Measure everything. Recording the traffic every minute turned "the server went down" into a clear timeline. It also let us prove the new protection works, instead of just hoping it does.
• Check who is real, don't chase who is fake. Blocking bad addresses works against real infected machines, but not against made-up ones. Checking that a player is genuine works against both.
• Our own protection has a limit. Our server can throw junk away very quickly, but only once it has arrived. A big enough flood can clog the road before it ever reaches the door. The answer to that is filtering further out, on a much bigger network than ours.
• Small tests come before big attacks. So we keep the alarms on.

What's next

• Protection further up the road. We are trialling a protection service built for FiveM servers. Players connect through its huge network, and the junk is filtered out long before it reaches us. We will try it on Balkan first.
• An even faster bouncer. We'll move the filter to the very first point where traffic enters the server, so junk costs us almost nothing to throw away.
• Separate roads for the websites and the game servers, so an attack on one can't slow down the other.

A word about DDoS attacks: they are a crime

Let's be clear about this, because some people think of DDoS attacks as a prank or part of gaming culture. They are a serious crime.

• In the UK, attacking a computer system so it stops working properly is an offence under the Computer Misuse Act 1990, with a maximum sentence of 10 years in prison.
• Across the EU, including the countries our Balkan community lives in, deliberately disrupting a computer system is a criminal offence under national laws based on the EU directive on attacks against information systems. The US and most other countries have similar laws.
• Paying for one is a crime too. Websites that sell DDoS attacks often call themselves "stressers" or "booters" and claim to be for "testing your own server". Using one against someone else's server is still an attack, and buying it makes you part of it.

It's also never as anonymous as people think. Police forces around the world, including the UK's National Crime Agency, regularly take down DDoS-for-hire websites and seize their customer lists. Under an international operation called PowerOFF, police have shut down dozens of these services, gone through their records, and contacted, warned, arrested and charged the people who used them. Some of those "booter" websites were actually run by the police, just to find out who would try to buy an attack.

And what does it actually get you? A few minutes of people being disconnected from a game. In exchange you risk a criminal record that shows up on job applications, travel restrictions, losing your computer to the police, and being permanently banned from the communities you were part of. It is never worth it.

We keep records of attacks, and they are reported to the networks they come from. If you are ever tempted, or you know someone who is: please don't. If you have a problem with us, open a ticket. We'd much rather talk.

How you can help

If you ever get knocked off along with everyone else, open a ticket and tell us what time it happened (and your time zone). Times are the most useful thing you can give us. We can line them up with the traffic records and tell you exactly what was going on.

And to whoever spent their night on this: it didn't work, and it's not worth it.