• Please make sure you are familiar with the forum rules. You can find them here: https://forums.tripwireinteractive.com/index.php?threads/forum-rules.2334636/

Server KF2 (or any Unreal Engine 3) server on RedHat/CentOS/Rocky/Alma Linux DDoS defense with the help of firewalld

Same problem. Yeah. But what shall i do, if we have casual router and no Linux?

If we enable AntiDDOS feature on router, it does not block source of the attack from outside internet (hundreds of IP addresses, like botnet or so), but router blocks local network IP address of KF2 server, where is traffic including DDOS attact redirected using port forwarding (KF2 server ports 7777, 8080, 20560, 27015). Is there a possibility to use some feature on Windows 10? Does Windows 10 firewall have such feature to prevent this?

Only solution we have found was to reduce ConnectionTimout to 10s and even with this setting, connections hit 2k in some moments.

[IpDrv.TcpNetDriver]
...
ConnectionTimeout=10.0

Before this fix, we could only watch like connections hit 10k and server or router went down. Please help or add feature to KF2 server.
 
Last edited:
Upvote 0
Is there a possibility to use some feature on Windows 10? Does Windows 10 firewall such a feature to prevent this?
To the best of my knowledge, and I looked for a while into this, one cannot rate limit the traffic per source IP on Windows firewall. You need either a Linux machine or some commercial firewall, most of which are linux machines with netfilter under the hood anyway. You can check your router documentation to see if there's rate limiting per source IP option or if you are allowed to access the shell directly and issue iptables commands. If so, the commands are supplied in the OP.
Only solution we have found was to reduce ConnectionTimout to 10s and even with that setting, connections hit 2k in some moments.
This works poorly in the long run. You may consider to keep switching the game port number once in a few days (until the attack resumes).
Please help or add feature to KF2 server.
TWI isn't in position to do that, they just rely on a 3rd party product they themselves don't know on a good enough level to do anything which is Unreal Engine 3. Complaints should be routed to Epic Games. TWI can help though with opening support tickets with Epic and talking them from the position of a customer, so it may worth creating a support ticket with TWI as well, go ahead and do that.

 
Upvote 0
Same problem. Yeah. But what shall i do, if we have casual router and no Linux?

If we enable AntiDDOS feature on router, it does not block source of the attack from outside internet (hundreds of IP addresses, like botnet or so), but router blocks local network IP address of KF2 server, where is traffic including DDOS attact redirected using port forwarding (KF2 server ports 7777, 8080, 20560, 27015). Is there a possibility to use some feature on Windows 10? Does Windows 10 firewall have such feature to prevent this?

Only solution we have found was to reduce ConnectionTimout to 10s and even with this setting, connections hit 2k in some moments.

[IpDrv.TcpNetDriver]
...
ConnectionTimeout=10.0

Before this fix, we could only watch like connections hit 10k and server or router went down. Please help or add feature to KF2 server.

I see a couple of solutions:
1) buy a linux box
2) Run a linux vm on windows and install kf2 in there.

3) A udp proxy
e.g. https://github.com/fragtion/python-udp-relay

However unless you have programming skills and some experience with programming UDP you are in for a very steep learning curve. BTW i have done a little testing with the UDP proxy before I put a linux box in place. It needs to be fast. In my testing Python adds to much latency to the connection. 50-90ms on a lan. The proxy probably has to be written in something like C or C++.
 
Upvote 0
I implemented rate limiting (i.e. DDoS protection) in killinuxfloor using firewalld's builtin function for this. No log parsing required. It simply drops packets if they exceed the set limit for new connections. The installation is automatic and enabled by default.

Supports Ubuntu 20.04 / 22.04, AlmaLinux 8 / 9, and Fedora 37. You just "install.sh", then maybe "klf apply", and it's pretty much done.

The curious can also enable some statistics about such attacks via the "klf ddos" command, but it may get really resource intensive, at some point it exceeded 1 million dropped packets within a day...

I just released killinuxfloor 3.0 with this feature (and many others, like "klf workshop"). For the interested:


Cheers.
 
Last edited:
Upvote 0
I implemented rate limiting (i.e. DDoS protection) in killinuxfloor using firewalld's builtin function for this. No log parsing required. It simply drops packets if they exceed the set limit for new connections. The installation is automatic and enabled by default.

Supports Ubuntu 20.04 / 22.04, AlmaLinux 8 / 9, and Fedora 37. You just "install.sh", then maybe "klf apply", and it's pretty much done.
Howdy Noobient! Didn't know who you are here! Great work. Thank you. With your help at making it user-friendly we may very well be in position to retain a good share of community KF2 servers. The overall number of KF2 servers shrunk to below 1800 in just less than 2 months, losing >200 servers, most likely because of these attacks, such a pace of the server loss is unprecedented.

I've used killinuxfloor back in the day before I've developed my own scripts for that so it suits my needs better.

Have you figured out BTW how to make the workshop content working again? If not I can help with that so the hosters struggle less.
 
  • Like
Reactions: bviktor
Upvote 0
Howdy Noobient! Didn't know who you are here! Great work. Thank you. With your help at making it user-friendly we may very well be in position to retain a good share of community KF2 servers. The overall number of KF2 servers shrunk to below 1800 in just less than 2 months, losing >200 servers, most likely because of these attacks, such a pace of the server loss is unprecedented.

I've used killinuxfloor back in the day before I've developed my own scripts for that so it suits my needs better.

Have you figured out BTW how to make the workshop content working again? If not I can help with that so the hosters struggle less.
Thanks :) Sure, the workshop downloads have been "fixed" a while back :)


I just wish TWI would stop linking to random versions of this library...
 
Upvote 0
As for server numbers, are you sure it's caused by these DDoSes?

I skimmed through your article about Tamari et al, do you think they're behind this? I was wondering for quite a while now, who and why would DDoS KF2 servers? What's the rationale? Now, this Tamari stuff might be just that.

Btw. to me it seems after a while they give up. At least on my server the number of attacks seem to be decreasing all the time.

Which is a bit worrysome as it seems they adapt, which in turn makes me wonder what kind of bullcr@p they come up with next...
 
Upvote 0
I have no idea for sure, but why else the server numbers went down so quickly and so much? About 1 year ago we've had roughly 2,300 KF2 servers. 2 years ago 2,500. Now it's a bit below 1,800. From 2,000 at the beginning of 2023.

Yes, we've mostly got to the bottom of this.

Tamari has nothing to do with it, unless by pure coincidence, but even that is not likely, he's a game developer and gamer, the people behind these DDoS attacks are pros who most likely sell their as they say "booter" or "stresser" services to people who may want to take revenge on someone or get competitive business advantage by taking their competitors offline.

The way it works is through amplification. Gaming, for latency reasons, relies on UDP protocol, which, unlike TCP, doesn't require 2-way handshake protocol negotiation. This means that one can send UDP datagrams with spoofed source IPs, meaning that they craft UDP datagrams and, unlike the network driver who would typically put your own source IP into the datagram, their software uses some other source IP address. Because still there exist last mile ISPs who do not filter ingress traffic from their customers by source IP and do not drop all IP ranges on their routers that don't belong to this ISPs, according to the BCP 38, customers of such ISPs can send UDP datagrams with spoofed source IP.

The destination IP of such datagrams would be any UE3 servers. The issue with them is their responses are 700 times more voluminous in size packet wise and 900 times more voluminous byte wise. You can check it yourself, but using a simple command like netcat:

nc 5.161.149.88 -u 7779

replace the IP with your KF2 IP and 7779 with the port on which your KF2 server listens on. The command will listen for your input, press any key and then enter and then watch in awe how much garbage a KF2 server spits out in return to just 1 byte.

But because the source IP was spoofed, all this garbage, unlike in the "netcat" experiment, is going to be sent to the spoofed IP address. Which can be abused to focus the responses from thousands of UE3 servers on one single online business and take it offline by just fully saturating their internet connection.

Quick back of the napkin calcualtions. Suppose that your business has 1 Gbit/sec connection to the internet. Depending on how much useful traffic is going through this pipe, you need to generate something like 0.4Gbit/sec-0.99Gbit/sec of traffic from UE3 servers. Say, that we want to limit ourselves with sending only 50 spoofed IP datagrams to a single UE3 server so it doesn't crash too soon, which is totally realistic. In the worst attack on my servers I've seen 1700 requests per second, 50 is rather mild and wouldn't crash the server too fast. Assume 8 bits per byte. 1 response is amplified 700 times. In response to 50 requests we get 50 * 700 = 35000 bytes. Let's take 1KB=1000 bytes for simplicity and so on. That's 280 Kbit/sec. Thus, in order to saturate 1,000,000,000 (=1 Gbit/sec) pipe we would need 1,000,000,000 / 280,000 = 3571 server. It's easy to send 500 datagrams instead of 50, this way we would need only 357 servers to fully saturate the pipe.

Of course, if we go overboard, it's likely that the KF2 servers start locking up, crashing, lagging and such and their owners would either take them offline, or put behind firewalls, etc -- thus reducing your amplification factor. But if you keep it mild you can keep the intensity of the attack for a long time.

There are tens of thousands of UE3 servers out there.

We know for a fact, by just analyzing the spoofed source IP datagrams to which our KF2 servers send the traffic to, that mostly these are various cloud hosters, typically 2nd or 3rd grade, such as digital ocean, hetzner and such, sometimes even smaller ones. We know that they have some beef with Roblox, which is online gaming and they have been attacking their servers for years now.

As I mentioned, there are 2 ways to monetize these services, 1 is to charge other cybercriminals for the "stresser" service time and another to directly request ransom to be paid from their attacked victims, which happens in about 7% of cases, according to one of the CloudFlare DDoS reports.

This attack is called "amplification volumetric L3/L4 distributed denial of service" attack.

An infamous story about a very similar botnet from Brian Krebs:

 
Upvote 0
the people behind these DDoS attacks are pros who most likely sell their as they say "booter" or "stresser" services to people who may want to take revenge on someone or get competitive business advantage by taking their competitors offline.
But that's precisely the puzzling part - what kind of advantage or gain would anyone get by taking offline a KF2 server? This is a coop game, there's literally nothing to gain from taking offline a server - unless you want people to use your own servers, that is. And that's where Tamarin might come into picture.
 
Upvote 0
But that's precisely the puzzling part - what kind of advantage or gain would anyone get by taking offline a KF2 server?
Out of envy or because of the competitors interfering with the expected profits, but here it's irrelevant. This type of attacks we are talking about in this topic are targeting some online business (e.g. "Roblox") internet connections, KF2 servers are a collateral here and are exploited to amplify the amount of garbage traffic sent to those internet connections.
 
Upvote 0
So i've been running a block script for exactly a week now, and we've blocked 640 ips thus far, still wondering why only one port is getting hit, 7797-7798
not that i can complain this way i don't have to merge LOG files, and filter every server, anyone else that has this happening on these set of ports or just random?

I have it like that, I host 3-6 servers per VM, sometimes all of them suffer from it, but sometimes just a share or even 1. We've already established that the attacks are rolling and for a certain period of time only some ports may be attacked, but then others get picked up while the previous get dropped from the attack.

In order not to bother with identifying if/when and what servers are under attack I run both rate limiting and IP banning scripts on standby at all times since about 2nd half of February 2023, before that I've been running just the IP banning script since 2021.

At some points my servers weren't attacked at all.
 
Upvote 0
So i have switch again to slightly better solution. In the log files you can find lines like this which can be phrased out to get the client ip address. This allows us to create a whitelist of known clients using a ipset.
[9363.55] DevOnline: EndRemoteClientAuthSession: ClientAddr: xx.xx.xx.xx:13542, ClientUID: 76xxxxxxxxxxxxxxxxxx

My ip tables setup now looks like:
Allow existing(RELATED,ESTABLISHED) game sessions -> rate limit total packets to 600 / second per ip
Allow known client new session(ipset) -> rate limit to 20 NEW connections / second per ip address
Allow unknown client new session -> rate limit to 20 connections / second for the subnet /8. This subnet can probably be made larger if you have been adding client ip addresses to the ipset whitelist for a while.

Code:
iptables -I FORWARD 1  -p udp -m conntrack --ctstate RELATED,ESTABLISHED -m udp --dport 7778:7779 -j GAME-RATE-LIMIT
iptables -I FORWARD 6  -p udp -m conntrack --ctstate NEW -m udp --dport 7778:7779   -m set --match-set KNOWN-CLIENT-RATE-LIMIT -j NEW-GAME-RATE-LIMIT
iptables -I FORWARD 7 -d <KF2 server ip> -p udp -m udp --dport 7778:7779 -m connlimit --connlimit-above 20 --connlimit-mask 8 --connlimit-saddr -m limit --limit 1/sec --limit-burst 1 -j LOG --log-prefix "Drop-Connection-Limit: "
iptables -I FORWARD 8 -d <KF2 server ip> -p udp -m udp --dport 7778:7779 -m connlimit --connlimit-above 20 --connlimit-mask 8 --connlimit-saddr -j DROP
iptables -I FORWARD 9 -p udp -m conntrack --ctstate NEW -m udp --dport 7778:7779 -j ACCEPT

ipset
Code:
ipset create KNOWN-CLIENT-RATE-LIMIT hash:net family inet hashsize 32768 maxelem 655360
ipset add KNOWN-CLIENT-RATE-LIMIT 10.0.0.1
ipset add KNOWN-CLIENT-RATE-LIMIT 10.0.0.2
ipset add KNOWN-CLIENT-RATE-LIMIT 10.0.0.3


This works well. It stops any false positives from known clients.

Note: To optimize your firewall rules and reduce CPU load / latency all frequently used rules like RELATED,ESTABLISHED statements should be at the beginning of your firewall rules. All firewall rules using ipset should be as close to the bottom of the firewall rules as possible to reduce the number of packets that they have to do a lookup on. I did a small test on a VM with 2 cores and a ipset rule with 655360 lines. I was able to put a gigabit of traffic through the server with 60-75% CPU usage.

Edit diagram didn't upload first time:

kf2Flow.png
 
Last edited:
Upvote 0
Great results, but I don't think it's warranted to do all that ultimately, as it may be an improvement at finesse at the cost of complexity and the impact on performance.

A small chance of a false positive is fine with me so far, given that it happens super rarely, mainly because KF2 crashes (which can be fixed and the players whose KF2 crashes should really work on optimizing it), and also because I flush the bans once in 3 days because it doesn't make much sense to keep accumulating the IPs as the attacks keep rolling and shifting to other IP ranges.

I also think that --connlimit-above 5 for /20 networks should work better, although I have no numbers to back it up, just a qualitative assessment that everything seems to work just fine and no legit players seem to be dropped.
 
Upvote 0
Will TripWire fix this properly so that their server binaries do the rate limiting internally? I'm running a KF2 server on a Windows 2008 server, and it's recently been overloaded and brought to its knees by this attack. There's nothing I can do either since it's Windows 2008, and `new-NetFirewallRule` cmdlets don't exist in powershell for that OS. I'm also running ESET Internet Security, and their firewall doesn't appear to have a way to rate limit connections. Anyone have any ideas? For now, I've turned my KF2 server off as it was crashing my entire Windows server due to this attack. :(
 
  • Like
Reactions: ®omano
Upvote 0
Anyone have any ideas?

The best would probably be to put a Linux VM/OpenWRT/DD-WRT router with a capability to run firewalld/iptables commands. Keep in mind that you want to alter the FORWARD chain in this case instead of the INPUT, although it may work with the INPUT just as well if no other stuff is running on this box that this thing may interfere with. If not and you think you can do the scripting -- adapt the baz's script above on the 1st page to use netsh advfirewall firewall or netsh firewall commands instead. Parallel to that, Windows QoS still can limit traffic by bandwirdth (not the same as connection rate limiting) for outbound traffic, which may also be helpful at mitigating the DDoS, it may be available for Windows 2008. And lastly, uninstall Windows, install Linux. I get that this is a lot of unpaid frustrating work, but that's how our life is, I can't think of anything else. With Windows you are on your own and will have to spend countless hours in your own research. For Linux we've already done it (I personally put hundreds of hours into the research and trial/error experiments) and you can use our ready to use tried and verified solution in the OP.
 
Upvote 0
Will TripWire fix this properly so that their server binaries do the rate limiting internally? I'm running a KF2 server on a Windows 2008 server, and it's recently been overloaded and brought to its knees by this attack. There's nothing I can do either since it's Windows 2008, and `new-NetFirewallRule` cmdlets don't exist in powershell for that OS. I'm also running ESET Internet Security, and their firewall doesn't appear to have a way to rate limit connections. Anyone have any ideas? For now, I've turned my KF2 server off as it was crashing my entire Windows server due to this attack. :(
Run a linux vm on the windows box?
1Gb of linux vm with 2vCPU should work totally fine for filtering. Might even be able to do less memory.
 
Upvote 0
From testing, it looks like the correct iptables rule is the following:

Code:
iptables -A INPUT -p udp -m udp --dport 7777:7779 -m connlimit --connlimit-above 3 --connlimit-mask 20 --connlimit-saddr -j DROP

The ones posted previously in this thread didn't actually work or mitigate the attack. The firewall-cmd generated iptables rule (what I grabbed above), did work, so there's your correct iptables rule.

Also, here's a perl script (I modified to do multi-threaded connections to test rate limiting) to test whether or not your server is vulnerable. You may need to install some perl packages using apt to test in linux.

Code:
use strict;
use warnings;
use threads;
use IO::Socket::INET;
use IO::Socket::Timeout;
use diagnostics;

sub threaded_task {
    threads->create(sub { 
        my $thr_id = threads->self->tid;
        my $Line = $ARGV[0];
        print "Starting thread $thr_id\n";
        my @ipinfo = split /:/, $Line;
        my $ip = $ipinfo[0];
        my $port = $ipinfo[1];
        my $connAttempts = $ipinfo[2];

        print "Checking IP: $ip Port:$port\n";
        my $sock = new IO::Socket::INET(PeerAddr => $ip,
                                PeerPort => $port,
                                Proto => 'udp', Timeout => 1) or die('Error opening socket.');
        IO::Socket::Timeout->enable_timeouts_on($sock);

        my $data = "\xff\xff\xff\xff\x00";


        $sock->send($data) or die "Error on send Payload...\n";

        $sock->read_timeout(2);

        my $output = <$sock>;

        close($sock);

        if(not defined $output or length($output) == 0){

            print "[!] $ip:$port Not Vulnerable\n\n";

        }else{
            my $amp_factor = length($output);
            print "[+] $ip:$port Vulnerable [Amplification Factor $amp_factor Bytes]\n\n";
        }
        
    });
}

my $Line = $ARGV[0];

if (@ARGV != 1){
    print "\n";
    print "XLabs Information Security www.xlabs.com.br\n";
    print "America's Army Proving Grounds DDoS Amplification Checker\n";
    print "Developed by Mauricio Correa\n";
    print "Contact: mauricio\@xlabs.com.br\n";
    print "Usage: perl $0 192.168.1.1:7778:6\n";
    print "Third parameter after server port is number of connection attempts to make simultaneously\n";
    print "\n";
}else{

    print "\n";
    print "XLabs Information Security www.xlabs.com.br\n";
    print "America's Army Proving Grounds DDoS Amplification Checker\n";
    print "Developed by Mauricio Correa\n";
    print "Contact: mauricio\@xlabs.com.br\n";
    print "Usage: perl $0 192.168.1.1:7778:6\n";
    print "Third parameter after server port is number of connection attempts to make simultaneously\n";
    print "\n";

    my @ipinfo = split /:/, $Line;
    my $ip = $ipinfo[0];
    my $port = $ipinfo[1];
    my $connAttempts = $ipinfo[2];
    my $i = 0;
    while ($i < $connAttempts)
    {
        threaded_task();
        $i++;
    }
    $_->join() for threads->list();

}

You run it like this:

Code:
perl scriptname.pl kfserverIP:kfserverPort:NumSimultaneousConnections

So, for example:

Code:
perl scriptname.pl 1.1.1.1:7777:10

If your iptables or firewall-cmd rule is working, you'll see:

Code:
Starting thread 1
Checking IP: 1.1.1.1 Port:7777
Starting thread 2
Checking IP: 1.1.1.1 Port:7777
Starting thread 3
Checking IP: 1.1.1.1 Port:7777
Starting thread 4
Checking IP: 1.1.1.1 Port:7777
Starting thread 5
Checking IP: 1.1.1.1 Port:7777
Starting thread 6
Checking IP: 1.1.1.1 Port:7777
Starting thread 7
Checking IP: 1.1.1.1 Port:7777
Starting thread 8
Checking IP: 1.1.1.1 Port:7777
Starting thread 9
Checking IP: 1.1.1.1 Port:7777
Starting thread 10
Checking IP: 1.1.1.1 Port:7777
[!] 1.1.1.1:7777 Not Vulnerable

[!] 1.1.1.1:7777 Not Vulnerable

[!] 1.1.1.1:7777 Not Vulnerable

[!] 1.1.1.1:7777 Not Vulnerable

[!] 1.1.1.1:7777 Not Vulnerable

[!] 1.1.1.1:7777 Not Vulnerable

[!] 1.1.1.1:7777 Not Vulnerable

[+] 1.1.1.1:7777 Vulnerable [Amplification Factor 21 Bytes]

[+] 1.1.1.1:7777 Vulnerable [Amplification Factor 21 Bytes]

[+] 1.1.1.1:7777 Vulnerable [Amplification Factor 21 Bytes]

3 connection attempts should work (where it says vulnerable, while the rest shouldn't).
 
Upvote 0
What i ve heard is .. that purpose of such a attack is to forward attack to Google DNS. I dont know if thats true or not, but somewhere i read they abuse KF2 servers to attack Google DNS servers. If i use my casual router DDOS blocking / preventing feature, it blacklists not the source of attack or multiple sources of attacks, but mine server with KF2 server software. And that is pain.

Having no other option. Cant use whole hardware like PC or VM machine with linux as router or firewall. Having 3 routers, one old, one new.. both with not many options and vulnerable and very same old router with OpenWRT. But i didnt achieved to block or filter attacks directed to port 7777 yet. These are thousands of connections per minute .. we are speaking about 150mbit /s attack. Router can handle, but cant handle big number of connections / packets.

Now trying to use that above described method with iptables inside OpenWRT, but have not been sucessfull yet.

Code:
iptables -A INPUT -p udp -m udp --dport 7777:7779 -m connlimit --connlimit-above 3 --connlimit-mask 20 --connlimit-saddr -j DROP

This does not work for me.
 
Upvote 0