• Please make sure you are familiar with the forum rules. You can find them here: https://forums.tripwireinteractive.com/index.php?threads/forum-rules.2334636/

Server KF2 (or any Unreal Engine 3) server on RedHat/CentOS/Rocky/Alma Linux DDoS defense with the help of firewalld

So i have switch again to slightly better solution. In the log files you can find lines like this which can be phrased out to get the client ip address. This allows us to create a whitelist of known clients using a ipset.


My ip tables setup now looks like:
Allow existing(RELATED,ESTABLISHED) game sessions -> rate limit total packets to 600 / second per ip
Allow known client new session(ipset) -> rate limit to 20 NEW connections / second per ip address
Allow unknown client new session -> rate limit to 20 connections / second for the subnet /8. This subnet can probably be made larger if you have been adding client ip addresses to the ipset whitelist for a while.

Code:
iptables -I FORWARD 1  -p udp -m conntrack --ctstate RELATED,ESTABLISHED -m udp --dport 7778:7779 -j GAME-RATE-LIMIT
iptables -I FORWARD 6  -p udp -m conntrack --ctstate NEW -m udp --dport 7778:7779   -m set --match-set KNOWN-CLIENT-RATE-LIMIT -j NEW-GAME-RATE-LIMIT
iptables -I FORWARD 7 -d <KF2 server ip> -p udp -m udp --dport 7778:7779 -m connlimit --connlimit-above 20 --connlimit-mask 8 --connlimit-saddr -m limit --limit 1/sec --limit-burst 1 -j LOG --log-prefix "Drop-Connection-Limit: "
iptables -I FORWARD 8 -d <KF2 server ip> -p udp -m udp --dport 7778:7779 -m connlimit --connlimit-above 20 --connlimit-mask 8 --connlimit-saddr -j DROP
iptables -I FORWARD 9 -p udp -m conntrack --ctstate NEW -m udp --dport 7778:7779 -j ACCEPT

ipset
Code:
ipset create KNOWN-CLIENT-RATE-LIMIT hash:net family inet hashsize 32768 maxelem 655360
ipset add KNOWN-CLIENT-RATE-LIMIT 10.0.0.1
ipset add KNOWN-CLIENT-RATE-LIMIT 10.0.0.2
ipset add KNOWN-CLIENT-RATE-LIMIT 10.0.0.3


This works well. It stops any false positives from known clients.

Note: To optimize your firewall rules and reduce CPU load / latency all frequently used rules like RELATED,ESTABLISHED statements should be at the beginning of your firewall rules. All firewall rules using ipset should be as close to the bottom of the firewall rules as possible to reduce the number of packets that they have to do a lookup on. I did a small test on a VM with 2 cores and a ipset rule with 655360 lines. I was able to put a gigabit of traffic through the server with 60-75% CPU usage.

Edit diagram didn't upload first time:

View attachment 2337216

Hey everybody,

It happened last week for the first time when my server started acting weird (teleporting around the map, player pings jumping like crazy etc).
So I reinstalled the server, moved it to windows for testing purpose etc
Of course nothing helped and I'm very glad that I found this thread.
For now I'm using GenZmeY's anti ddos solution which does it's job so far.
Thanks for that tool GenZmeY, we can play again :)
However he highly recommends using bazztheallmighty's solution but that's a bit too much for me.
So I asked a few friends who are "quite good" with linux and their reaction was "oh no, this is complicated af and doesnt seem to be a tutorial"
So maybe someone can help setting this up.
Saw you can buy whole books about that iptable stuff...oh boy :|
I also tried to contact you baz, but even when I just want to check your profile I receive "permission denied..."
Im on the latest Ubuntu Server

Thank you

 
  • Like
Reactions: GenZmeY
Upvote 0
Hey everybody,

It happened last week for the first time when my server started acting weird (teleporting around the map, player pings jumping like crazy etc).
So I reinstalled the server, moved it to windows for testing purpose etc
Of course nothing helped and I'm very glad that I found this thread.
For now I'm using GenZmeY's anti ddos solution which does it's job so far.
Thanks for that tool GenZmeY, we can play again :)
However he highly recommends using bazztheallmighty's solution but that's a bit too much for me.
So I asked a few friends who are "quite good" with linux and their reaction was "oh no, this is complicated af and doesnt seem to be a tutorial"
So maybe someone can help setting this up.
Saw you can buy whole books about that iptable stuff...oh boy :|
I also tried to contact you baz, but even when I just want to check your profile I receive "permission denied..."
Im on the latest Ubuntu Server

Thank you


I'm just out of the previous ban and will likely get banned soon permanently. But you can take advantage of it while I'm still here, I'm willing to help.

On the Ubuntu check if you use firewall-cmd for managing your firewall:

Code:
firewall-cmd --state

If you get "running" just issue the following commands under root. It will persist reboots and so far has proven to not interfere with KF2 in the last 2 years we've been using it. If you host something else on your server, the result is hard to predict.

Code:
firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p udp --dport 7777:7797 -m connlimit --connlimit-above 5 --connlimit-mask 20 -j DROP
then this command so the previous one takes effect immediately:
Code:
firewall-cmd --reload

This assumes that your KF2 server game port is in the 7777:7797 range. If not, you can learn it by checking the Port= line in your LinuxServer-KFEngine.ini file of your KF2 server, which is located KF2-directory (can be different for different installs)/KFGame/Config. So if what you've learned is different, repalce 7777:7797 with your range. The first number is the first port of the range, the seond is the last, everything in between is included.
 
  • Like
Reactions: n1ouk and GenZmeY
Upvote 0
Hi Sir,

thanks for your detailed instructions.
Indeed I already tried your code few days ago, which did not work.
So I was testing some stuff yesterday and RELOADED the firewall.
That did the trick!
Now I only get 5 notifications from 1 IP until it stops (I think you could even set the connlimit to 3 or 4?)
Anyway this seems to be all that is needed in my case.

But that firewall rule was just 1 step from your tutorial.
You also did some log reading stuff etc I think.
What is the benefit from that?
When I tried baz's solution I got too many "command not found" etc.
Anyway, thanks again :)

btw why previous ban? why probably banned permanently soon?
 
Upvote 0
Now I only get 5 notifications from 1 IP until it stops (I think you could even set the connlimit to 3 or 4?)

Yes, you are free to explore the boundaries of this solution. Theoretically, 1 KF2 game needs only 1 connection during the game, but there are marginal scenarios when it may need at least two. First, password protected servers under the hood require KF2 games to reestablish the connection after the password is entered. It's not clear if the first finishes gracefully, if not, after the password enter there will be 2 from the same IP. Another is MOTD images, if you serve them from the same server, this also may need at least 1 separate connection.

The number 5 in this rule is completely arbitrary, what felt OK for everyone who researched this issue. There's no solid research behind it.

But that firewall rule was just 1 step from your tutorial.
You also did some log reading stuff etc I think.
What is the benefit from that?

You've implemented the rate limiting solution. Now your server can never get stressed by too many requests, because the rate limiting rule will not allow ridiculous things like 5000 requests per second, which result in the server lagging tremendously, leaking memory, hanging up and crashing.

Still, the bad guys will keep sending your way malicious traffic to which your server is going to respond. The log reading stuff helps you with completely banning all the IPs from which the traffic comes on an ongoing basis, so even if they change, they soon get picked up and added to the ban list. This ensures there's no malicious traffic whatsoever exists, unless it comes from new IPs that weren't included into the ban list just yet.

It's a bit prone to false positives when KF2 game crashes. If you have players who crash often, multiple times a day, the script may ban their IPs.

btw why previous ban? why probably banned permanently soon?

Some obscure KF2-related drama, I'm not allowed to talk about it here. I'm being watched and not ready for the next ban just yet.
 
Upvote 0
Attacks are on going again.
Difference to past is that now all servers on "default ports" (in line after first) are getting attacked. Actually only instance that is let running alone is server that has been attacked alone before.
I´m looking now is it possible to make some firewall rules to Ubiquiti EX4 to prevent this. If not, I guess I need to start making pfsense firewalls.
 
Upvote 0
Attacks are on going again.

Yes, they are.

With most equipment that targets home users you can't do much. I just searched online and it doesn't seem like filtering traffic by TTL, which is the best working currently method against the DDoS, is available on Ubiquiti.

Advanced (and yet cheap) stuff like pfsense should do it though. There's a proven working configuration exists for Mikrotik as well, both for TTL and rate limiting.

Also, this thread will not be supported anymore very soon as I will not be able to answer here.
 
Upvote 0
It is getting out of hands, can't have a HoE 10 waves game rn. The attackers wait for us to be on wave 10 most of the time to crash the server. I even think attackers are tracking people. Any news from TWI? I don't even see their servers to play HoE 10 waves, do they even have 1? If I use the ig option to make a game and auto find a server I am never on officials, but end up with 100+ ping on US servers. I'm in France btw.
 
Upvote 0
Does anyone know if you can get logs for successful(verified) queries to the query port?

Context: The query port 27015(by default) does verification of the player before responding. It sends each player a random string that the player has to reply with. If the player replies with the same string they are then given the server status(Number of players, wave,etc,etc) This verifies that the player is real. Amplification attacks can't do this verification as the victim is the one receiving the random string.

There is normally a couple of seconds between the query and user connection. This will allow would the creation of a whitelist of play ip addresses.

The process would look something like this:
1) User queries server status
2) Verification happens
3) Log file is written
4) Script reads log files and updates a whitelist.
5) Players is allowed to connect to server.
 
Upvote 0
I'm working on a feature in killinuxfloor that should alleviate this, hopefully it'll be ready by the end of the week.

BTW does anyone know if it's legally OK to publish a docker image that includes all the KF2 binaries preinstalled?
Alright, this didn't work out due to tech limitations.

In any case, I'm trying to figure out where to host my server, but there's no visitors, only maybe 1 every day or so.

Could someone please try and play a match on one of my servers?

- kf.noobient.com
- kf-do.noobient.com

You can just press F3 in the main menu, then "open kf.noobient.com" and it'll join to this server. They're both set to 10 waves HoE ATM.

I'm curious if they can withstand a full length match without being brought down. I've already played one endless on them, I even got a "visitor" by around wave 3, got the dreaded "connection lost", but it actually recovered after like 5 seconds, and was okay after that until wave 13, when for whatever reason I was given a "beefcake" wave, except it wasn't beefcake but 37 fleshpounds, and I wiped rofl.
 
Upvote 0
So I did not know KF2 still had issues with DDOS attacks etc.

Going to post some points from RS2:V in hopes it may help, as it has been massively successful for us in RS2:V.

For RS2:V we have been using pfsense very successfully, we have done the following

Added perm blocks for known "source port" attacks towards game ports, so for example if an IP connects and the source port it is connecting from is a bad one its silently dropped and ignored.

I have a long list of these source ports too, some of the devices being abused were unknown to me, but using wireshark we were able to see where the attacks were mainly coming from, 99% of the time they are STATIC source ports used over many IPs, so you can block them fairly easily if you know the source port to block.

Good advice is to block incoming based on source port matching 0:150 (first 150 ports) and then the other bad ones:
Code:
1434 - SQL Browser Service Attack Vector
389 - LDAP Attack Vector
3702 - Web Services Discovery Attack Vector
394 - embl-ndt Attack Vector
2268 - AMT  Attack Vector
3283 - Nmap Attack Vector
3785 - BFD Echo Attack Vector
2222 - Unknown Attack Vector
3306 - MySQL Server Attack Vector
1900 - SSDP, UPnP Attack Vector
13000 - UT Attack Vector
11211 - Memcache  Attack Vector
1230 - Unknown Attack Vector
We also had out provider datacenter block these in their upstream carries towards us based on the source port towards our game ports, so massively reduced the load the pfsense has to deal with.

BOGON network IPs also need to be considered, as these are massively spoofed in attacking RS2:V servers, so i imagine its the same for KF2.

Finally something pfsense does is "UDP states" so we can count UDP connections.
So normal operation a single IP that's attacking might attack 100s of times per second, what the UDP states do is allow you to limit that IP as much as you want.

So UDP can be counted the same as TCP connections.
You can then using UDP states limit an IP to say x states over 5 minutes (or more) depending on the attack, this is also great for controlling server queues I have found, as they will get the "unable to connect to server" message after just a minute or two in queue and be blocked from seeing the server until the states time out!

Lastly make sure you are limiting ICMP to just request echo and request reply if ping matching is used. Using pfsense we limit the number of IPs that can send us pings over 10 minutes to certain number or so and those that do can only send enough to match make etc to the server.

With RS2:V I have found that ICMP ping attacks hit the game process but using the windows firewall I have restricted pings to the host o/s as that's all that is needed for match making on RS2:V Epic clients.

Pfsense also has a plugin called Snort, using this we also capture bad packets and fragmented packet type attacks, so also worth looking into.
 
Last edited:
Upvote 0
Looks like the ones responsible for the amplifier attack, found a way around the ttl limit since, my 30 OVH hosted servers got taken out due to this exploit, while i’ve been using the ttl iptable block which worked untill Wednesday, trying to solve this with the OVH support.. also not entirely sure if the iptables on my specific servers randomly resetted, but be cautious i’d say.
 
Upvote 0
I have a solution that works to block all of these amplification attacks now. However it is not trivial to understand or implement.

NOTE:I have glossed over some of the very complicated technical details regarding NFQUEUE like how to install,config,etc.

What are the most common ways players connect to a KF2 server:
1) They open KF2 and click browse and find a server
2) They open steam and go to the game servers option
3) They recieve an invite from a friend in steam.
4) They open KF2 and go to the console and type open IPADDRESS : port

With options 1 & 2 the player will send out a query to most, if not all KF2 servers to see who is on the server, what map is being played,etc. This is normally on the default port 27015 or whatever the server has it configured to. It looks like this:
1707750850719.png
Breaking down this communication:
1) Player sends a hello packet to the server "????TSource Engine Query\000" In this case some unprintable characters represented by ?. No. 61
2) Server response with a random string "????A?E?\v" No. 62
3) Player response with "????TSource Engine Query\000????A?E?\v" which includes the random string . This effectively authenticates them as real.
4) Server then sends the name of the server, who is playing,etc.

Amplification attacks can't replicate this challenge response process as the target of the amplification attack will never respond with the challenge as above.

So the solution is just to log all of the ip addresses who successfully do the challenge response and create a firewall rule to accept connections from those ip addresses. Easy......... If you could add one line to the source code of the game to print it to the logs. Hence this request: https://forums.tripwireinteractive....henticated-queries-to-the-query-port.2340028/


Not so easy if you have to do a network capture phrase the packets to find the right information and then log it. The way i have set this up is using iptables and NFQUEUE. In iptables you have 3 options as an action Accept, Drop or NFQUEUE. NFQUEUE allow you to write an application to receive the packets, then do whatever you want with them. In this case i used python check if the packet contained my server name "BAz", than log the ip destination (players ip) to disk and forward the packet on.

This iptables command passes the packet to 1 of three NFQUEUE's.
Code:
iptables -I INPUT -s <SERVER IP ADDRESS> -i enp5s0 -p udp -m udp --sport 27015:27056 -j NFQUEUE --queue-balance 1:3

This code logs the ip address to disk and forwards the packet onwards.
Code:
#! /usr/bin/env python3
from netfilterqueue import NetfilterQueue
from scapy.all import *
import os


def modify(packet):
    pkt = IP(packet.get_payload())
    #skip small packets
    if len(pkt) > 100:
        f.write("{}\n".format(str(pkt.dst)))
        f.flush()
    packet.accept() #accept the packet


nfqueue = NetfilterQueue()
#1 is the iptabels rule queue number, modify is the callback function
nfqueue.bind(1, modify)
f = open("/var/log/kf2client", "a")
try:
    print("[*] waiting for data")
    nfqueue.run()
except KeyboardInterrupt:
    pass



This list of ip addresses can then be turned into a ipset once sorted and unquie'd. E.g

Code:
create ActiveUsers1 hash:net family inet hashsize 32768 maxelem 1000000 bucketsize 12 initval 0xee41b63f
add ActiveUsers1 192.168.0.1
add ActiveUsers1 192.168.0.2
etc

Which results in a very large ipset. 197684 enties logged over two months or so. Which is probably also the approximate active player based assuming 1 ip per player.

Code:
Name: ActivePlayers
Type: hash:net
Revision: 7
Header: family inet hashsize 131072 maxelem 1000000 bucketsize 12 initval 0xee41b63f
Size in memory: 6246576
References: 3
Number of entries: 197684

This ipset can then be used to match players connecting to the server.

Code:
iptables -A INPUT-p udp -m conntrack --ctstate NEW -m udp --dport 7779:7779 -m set --match-set ActivePlayers src -j ACCEPT


Once this has been left running for a week or so you will have a nice list of ip addresses that covers 99% of KF2 players. Any other iptables rules that accept packets can then be removed. This doesn't cover the other ways of connecting to a KF2 server but chances are the player has already queried the server just by opening the server browser. I haven't had any false positives that i know about yet.
 
  • Like
Reactions: GenZmeY and Ruuvi
Upvote 0