• Please make sure you are familiar with the forum rules. You can find them here: https://forums.tripwireinteractive.com/index.php?threads/forum-rules.2334636/

Server KF2 (or any Unreal Engine 3) server on RedHat/CentOS/Rocky/Alma Linux DDoS defense with the help of firewalld

A quick calculation of the amplification with a Wireshark test is approximately 1:700 over 200 seconds as per below. According my steam servers list there are 2536 killing floor 2 servers online. If the attacker sends 15000 packets to each server we get 2536x15000=38 040 000. Assuming 90% loss for things like servers not responding, network loss,etc,etc we get 3 804 000 packets per second and this will go on for 200 seconds minimum. This number of packets will ddos most firewalls and most servers.

1627839256943.png
 
Upvote 0
I appreciate the efforts that's been put into fixing this issue! I was wondering how I can know if the script runs correctly, it seems like the attacks get cut off pretty quick. I can't tell if it's because of the script or if they're just being more sporadic because of my manual blocking of IPs. When I run the "firewall-cmd --list-all" command I don't see any IPs added, but when I run the script manually it echos "success."
 
Upvote 0
Upvote 0
--list-all is a wrong command you should run the command that displays the content of the network block, e.g.:

firewall-cmd --ipset=networkblock --get-entries

Also, at this point as an end customer who may not be into all the nuts and bolts I'd rely on Zmey's solution:

Now I'm getting an "Error: INVALID_IPSET: networkblock" error. Both with the script and with --get-entries.

I can try the other solution, but I've never used an executable in linux before and this one doesn't seem to run like normal ones do?
 
Upvote 0
Glad to hear that you got it working. I guess I'm going to supply commands on how to create a network block and how to associate it with a block zone so others can pick it up easily. As well as crontab instructions to ensure this is called automatically once per minute or so.

As for GenZmey solution it's written in Golang which implies that would need the interpreter installed on your Linux machine.

PS updated the OP with more detailed instructions on how to operate firewalld and put the command into a crontab.
 
Last edited:
Upvote 0
Always fun to learn new stuff on Linux. I went ahead and used the sudo crontab because I assume it's needed to edit the block. Kinda surprised it took a few years before I ran into this attack because they seem quite persistent.

I'd assume that having the traffic dropped wouldn't be noticeable by the attacker, especially with any replies that do happen going to the spoofed IPs. So hopefully there's someone out there wasting their time and resources with these attacks.
 
Upvote 0
It's actually possible for the attacker to learn if the server being used for amplification is no good anymore, I imagine they include their own IPs among the ones attacked via spoofing, if they receive any amplified traffic back then the amplification server is still good, if none or not for long then it's being protected.

Plus, even without it, they have a general feel of the attack effectiveness by observing the availability of the service that is being attacked. A successful attack disrupts the service, if not -- well, something is wrong.
 
Upvote 0
I added more diagnostic commands to the original post.

Try this one before the one-liner that analyzes the log and after and see if any IPs get added.

Code:
firewall-cmd --permanent --info-ipset=networkblock

Ensure that the ipset is associated with the drop zone:

Code:
firewall-cmd --permanent --zone=drop --add-source=ipset:networkblock

Check if the script is able to identify the offending IPs at all by running half a command:

Code:
tail -10000 /home/kf2-admin/kf2-server/KFGame/Logs/Launch.log|awk '/NetComeGo: Open/ {a[$5]++} END { for (b in a) { if (a[b]>50) {print b} } }'

Ensure that you have the path to the directory where the logs are right and you don't modify the file name in KF2 config but if you do -- modify the command as well to reflect it.

Ensure that there are indeed malicious connections in the log file in the vicinity of 10000 or whatever number you use lines from the bottom of the file.
 
Upvote 0
I played around with it a bit and I ended up having to increase the number of lines being read by quite a lot. Once I counted the lines myself the issue became pretty obvious. I was running the script every 6 or so minutes and in that time around 80k lines were getting added to the log file. I'm gonna try lowering the number of lines and running it every minute or so instead and see how that goes.

Thanks a lot for putting up with my dumb butt :)
 
Upvote 0
80K sounds like a whole lot, I suggest you run it every minute and decrease the number of lines to say 20K.

If you keep analyzing 80K batches imagine what's going to happen when the DDoS ends. Regular legit games don't fill the log as much, therefore, your script is going to end up analyzing long periods of time spanning probably even days. During a few days it becomes more and more probable that regular sane players connect/disconnect to your servers 50 times per 80K lines. So the script will start banning IPs of legit players.

To avoid that, either use additional scan suggested in one of the comments on first page for player genuinity (but it means more resources spent on this job), or decrease the number of lines to analyze, but also ensure the script is invoked frequently enough, if once per minute isn't enough you can use splits of a minute by modifying the script such that it runs, sleeps for needed number of seconds, runs needed number of times to fill up the time <1 minute, then exits until it gets invoked by a cron daemon again the next minute.

In short, first tune the frequency down to every 1 minute (that's the lowest number possible with cron), but if found to be inadequate then tune the number of lines.
 
Upvote 0
So, I was just getting hit by 6-7 entire /24 subnets which was generating about 200k lines a minute. It seems to have defeated the script though? Even when I set it to review the 1.2mil lines created from the start of that attack, it still didn't come up with any IPs to block. I know it works under normal circumstances because it's been successfully working for a decent many attacks so far.

EDIT: Actually, it appears that it got two of the subnets
 
Upvote 0
That's unusual, everyone who used it so far reported great 100% success and even overreach for legit players. I myself have been running half a dozen servers DDoS free for almost a year since then -- I get attacked occasionally but the script does its job and stops the attack at most 1 minute after it begins.

I fully expect the script to be capable of extracting ALL the spoofed source IPs from the Launch.log. But I'm curious what doesn't work as expected in your case -- do you mind sharing one of your Launch.log files for which you believe the script isn't effective, so I can work on refining it?
 
Upvote 0
Rating limiting the number of packets per ip gives the script time to block the ip. This prevents the killing floor from crashing or maxing a CPU core before the script does it's job. This wont help if the attacker is using lots of different ip addresses to attack your server. With the rate of 1000 packets per second i don't have any drops from normal game play.


Code:
iptables -A INPUT -p udp -m conntrack --ctstate RELATED,ESTABLISHED -m udp --dport 7770:7780 -j GAME-RATE-LIMIT

iptables -A GAME-RATE-LIMIT -m hashlimit --hashlimit-upto 1000/sec --hashlimit-burst 1000 --hashlimit-mode srcip --hashlimit-name conn_rate_limit -j ACCEPT

iptables -A GAME-RATE-LIMIT -m limit --limit 1/sec --limit-burst 1 -j LOG --log-prefix "GAME-RATE-LIMIT-Dropped: "

iptables -A GAME-RATE-LIMIT -j DROP
 
Upvote 0