Server KF2 (or any Unreal Engine 3) server on RedHat/CentOS/Rocky/Alma Linux DDoS defense with the help of firewalld

[o2xVc3UuXp0NyBihrUnu]

I can no longer support this topic here.

Please read the info that was previously posted here on the URL below which is now expanded to include new developments. I'm removing it here, because the situation is developing, the information becomes no longer valid and I will no longer be available to change it here because of the upcoming ban. I'm currently back from a temporary ban and the next one will likely be permanent.

Link removed by moderation team

I'm available over there for support and will no longer be able to respond in this thread.

 

[Pricetx]

EDIT: From further reading, it's sounding like I may have been receiving traffic from these IPs in an attempt to max out connection limits on the server, and the response traffic is much greater than the inbound traffic - a sort of amplification attack. I sadly don't have any network dumps from the attack period to confirm that myself.

Hey,

It's good to see I'm not alone in this issue.

Back on Tuesday, my server hosted with Kimsufi (OVH) was shut down by an automatic process that considered my server to be hacked. The logs they provided showed a very large number of UDP packets originating from my KF2 server to a singular IP address. Their statistics showed 140,000 packets per second, and about 68 Mbps of bandwidth.

I've included a screenshot from their ticket here:



It's taken me until last night to finally get access to my data (long, irrelevant story), and I've been having a look through my Launch.log. I'm seeing basically the same as what's already been posted in terms of log entries. Many thousands of "NotifyAcception Connection", followed by timeouts.

However, based on my own stats, it doesn't look like my server was the victim of a DDoS, but rather all of the traffic was OUTBOUND, so it looks like my server had actually become the attack vector. I've included a screenshot below of the network monitoring showing the traffic all being outbound. This matches the information OVH provided:



At this time, i'm not sure how somebody is managing to use my KF2 server daemon to generate all of this traffic, as I'm not seeing a high volume of inbound traffic to generate it. I do use FirewallD, so I can potentially try to do something that quickly kills excessive connections to an IP, but it's very much a band-aid fix. I suspect there may be an exploitable flaw in the server software itself.

I did some quick analysis on the destinations for this traffic which i've included below:

grep "NetComeGo: Open TheWorld" Launch.log | sed 's/.* //' | sort | uniq -c | sort -nr 56719 104.153.108.19 51109 199.231.233.15 46622 73.247.224.154 33927 178.237.56.153 30389 12.1.21.1 20091 24.167.129.214 19530 74.74.74.74 15279 73.73.73.73 14309 178.237.56.155 14087 104.153.108.192 9365 80.80.80.80 7627 192.172.226.159 7565 70.177.113.16 5316 50.7.8.131 4591 1.32.42.3 4443 171.33.133.153 3015 68.142.122.37 2748 8.41.7.62 2600 68.185.71.105 1533 45.48.19.195 1510 8.41.7.41 1383 8.44.63.63 871 192.172.226.78 667 8.41.7.56 616 144.172.73.69 564 185.53.90.81 558 172.104.213.248 550 70.70.70.79 500 69.141.13.26

I would be very interested to learn more about what other people have seen though. If somebody could DM me a link to any relevant Discords that would be greatly appreciated. In the meantime, once I've finished recovering my server I think i'll have to unfortunately temporarily disable my KF2 server until I understand more.

 

[Pricetx]

There's a couple of threads in the steam Discussions section which i'm linking below for extra info:

Malicious activity on my kf2 server [HELP] :: Killing Floor 2 General Discussions

I believe the culprit could be this name which joined my server repeatedly: MR Bullet-Proof Attacking from an IP in Manchester, UK Often when my server is live, I get repeated messages in-game "MR Bullet-Proof joined, MR Bullet-Proof left" basically over and over. He never stayed long enough to...

steamcommunity.com

Is this an intentional DoS attack on my server or a bug? :: Killing Floor 2 General Discussions

I run a small vanilla KF2 server, and over the past two days I noticed something strange happening. The server will suddenly start using way more CPU than usual, the console shows a single IP address will start opening thousands of connections, and it maxes out the 20mbps of upstream bandwidth I...

steamcommunity.com

 

[o2xVc3UuXp0NyBihrUnu]

...

You are correct that the replies are much more sizeable than the initial requests to connect. Here's how it looked like on my servers. A downwards (means outbound) spike starts when a whole bunch of new IPs are added to the attack and then ends when my banning bot picks it up and blocks the traffic:



You are also correct that it might be a part of a broader DDoS that uses KF2 servers as amplification beacons. But for our purposes this is irrelevant as whatever the small volume of traffic comes to our servers it is still enough to bring our servers down and as I noted previously I witnessed my servers hanging up completely after receiving (and responding to) merely ~5K malicious requests. Therefore it still should be considered a DDoS attack against KF2 servers as it still reaches the goal of bringing them down, even if it's a collateral damage situation.

Also, by now in my research I established that, from all the attacked hosts I analyzed, all of them listened on default port UDP 7777. This is also true for your server per the hoster notification you provided, thank you. This fully explains why TWI doesn't care about it and was reluctant to react in any way, as almost all if not all of their KF2 servers use weird ports, typically it's some random number from the range 27000-30000 UDP. Thus, by the nature of this attack, only lone (because if a person hosts >1 on a single IP they have to change the default ports on all but 1st server) community servers are the victims.

This suggests a stupid and simple solution to the DDoS problem that still amazingly works, which is to just change the listening port from the default one to any other random value. This has proven to prevent the attack, even if one unbans all the IPs one has banned previously to provoke it. With great success this technique has been implemented on a few hosts I've dealt with, so if you don't like the IP banning solution outlined in the OP (which still works flawlessly and prevents hangups and complaints from the hoster), you can just change the listening port to some random value. The only downside to this I know is it effectively removes the server from frequenters favorite list, as it's populated by IPort and they will have to readd it.

 

[baztheallmighty]

The windows powershell version:

#Run once to create the firewall rule 80.80.80.80 is one of the known bad addresses.
new-NetFirewallRule -DisplayName "Block DDOS" -Direction inbound –LocalPort Any -Protocol any -Action Block -RemoteAddress "80.80.80.80/32"


#Log location
cd "C:\KF2Server\KFGame\Logs"
while(1){

    $Files=Get-ChildItem Launch_*.log
    #Get existing rules
    $IPString = (Get-NetFirewallRule -DisplayName "Block DDOS" | Get-NetFirewallAddressFilter ).RemoteAddress
    #Setup the firewall rules
    foreach ($file in $files){
        #Get the ip addresses from the file, only look at the last 3000 lines to speed it up
        $IPs=Get-Content $file.fullname -Tail 3000 | where { $_ -like "*NetComeGo: Open TheWorld*" } | % { $_ -replace "\[\d+.\d+\] NetComeGo: Open TheWorld \d+\/\d+\/\d+ \d+\:\d+\:\d+ ",'' } | Group-Object
        #Must be more than 50 connections
        $NewIPS=$IPS | where { $_.count -gt 50 } | % { "$($_.name.trim())/32" }
        if($NewIPS){
            write-host $file.fullname
            write-host $NewIPS
            $IPString+=$NewIPS
            $IPString=$IPString | sort -Unique
            #Ban the ips
            set-NetFirewallRule -DisplayName "Block DDOS" -Direction inbound –LocalPort Any -Protocol any -Action Block -RemoteAddress $IPString
            write-host "--------"
        }
    }
    write-host -NoNewline "."
    sleep 120
}

 

[baztheallmighty]

Good news: I have been able to successfully replicate attack being used and have sent the details to the tripwire team.

I also have updated my script to output all ip's I block to a file. This file can be found here: https://stats.mtreadwell.com/knownddosip.txt

 

[o2xVc3UuXp0NyBihrUnu]

Good news: I have been able to successfully replicate attack being used and have sent the details to the tripwire team.

I also have updated my script to output all ip's I block to a file. This file can be found here: https://stats.mtreadwell.com/knownddosip.txt

Any clues who might be behind this attack? Also, my set of IPs:

50.7.8.131/32
80.80.80.80/32
8.44.63.63/32
58.69.140.17/32
158.69.140.17/32
135.125.87.35/32
68.142.122.37/32
47.185.107.147/32
192.119.8.93/32
178.237.56.153/32
73.247.224.154/32
73.247.224.15/32
67.193.92.90/32
70.177.113.166/32
70.51.0.226/32
37.9.175.20/32
98.217.116.40/32
69.28.48.201/32
66.254.114.41/32
91.220.101.13/32
34.224.160.149/32
3.143.34.186/32
172.86.114.77/32
47.153.186.164/32
194.146.239.129/32
66.151.138.194/32
192.200.158.178/32
192.119.8.82/32
112.132.221.121/32
74.91.119.228/32

The order is chronological. The IPs you provided seem to be ordered to a degree, are they chronological?

 

[baztheallmighty]

Unfortunately my ip's are not in a chronological order they are in sorted order from PowerShell.

As for who is behind it, no idea. If you put them through something like a "bulk ip lookup tool" you get all kinds of ISP's and hosting platforms(aws,OVH,etc) around the world. This would suggest some kind of bot net.

 

[o2xVc3UuXp0NyBihrUnu]

Yeah. I also sent a couple of complaints about the IPs to a few hosters, but we all know that most of it ends up in garbage and they certainly won't disclose any info about who rented the server/IP from them, unless there's a subpoena or something.

 

[DoM!niC]

Some could help with my Debian Server I use IPTables and currently my Servers are maintance because UDP Flood Stuff....

 

[o2xVc3UuXp0NyBihrUnu]

Some could help with my Debian Server I use IPTables and currently my Servers are maintance because UDP Flood Stuff....

1. Shut down the KF2 server.
2. Change the "Port=7777" in LinuxServer-KFEngine.ini file to "Port=7878"
3. Ensure that this port is accessible from the Internet (e.g. firewall openings, etc)
4. Start the KF2 server.

 

[DoM!niC]

1. Shut down the KF2 server.
2. Change the "Port=7777" in LinuxServer-KFEngine.ini file to "Port=7878"
3. Ensure that this port is accessible from the Internet (e.g. firewall openings, etc)
4. Start the KF2 server.

ok but other Ports got attacks too, someone know the StartParameter to change the Launch.log for each Server to assign a own log to use Fail2Ban to prevent the Attacks ?

ok got it -Log=NAME

 

[o2xVc3UuXp0NyBihrUnu]

ok but other Ports got attacks too

Surprised to hear that, do you have any evidence to support that? I oversee multiple servers and for all of them the attacks have stopped once the port was changed from the default one.

 

[DoM!niC]

Surprised to hear that, do you have any evidence to support that? I oversee multiple servers and for all of them the attacks have stopped once the port was changed from the default one.

I create a simple filter for the Log I hope this will prevent my problems.... but very strange why only KF2 Servers are attacked, I host a DNS and so on and all is fine without any attacks on this Services.

 

[GenZmeY]

o2xVc3UuXp0NyBihrUnu,​

thanks for the info, your script helped a lot
However, it is worth adding a check for a player to it, because in my case there were players among them. Something like this:

tail -50000 ./Launch.log | \
awk '/NetComeGo: Open/ {a[$5]++} END { for (b in a) { if (a[b]>50) {print b} } }' | \
while read IP
do
    if grep -qF "VerifyClientAuthSession: ClientIP: $IP, ClientUID" ./Launch.log ; then
        echo "player: $IP"
    else
        echo "botnet: $IP"
    fi
done

I am very surprised that there are players on this list, but this is a fact. In the attachment to the message, an example - at first the player spams a lot, but eventually connects. I don't know what it is - maybe their computers are infected or someone near them is infected (NAT) or even the kf2 client can also cause this? But these are definitely real players. they have profiles in steam and kf2 in which many hours are played.

I should also point out that changing the default port is not a complete solution. I have servers in the 7778-7796 port range and 7781 suffers the most from the attack. Obviously, the attacker can choose any port, not only the default.

and here is my list of ip from botnet (cleared of players):
46.80.57.82
46.80.57.93
73.73.73.73
69.162.108.171
98.26.1.73
128.116.28.8
73.9.3.211
52.148.210.120
13.91.167.147
37.0.11.253
51.105.225.244
104.26.8.109
35.190.45.146
172.67.73.245
52.210.95.185
54.208.10.149
71.199.120.136
185.173.21.73
172.67.73.20
83.229.42.195
18.235.229.245
8.9.230.245
68.142.122.37
51.89.240.144
213.87.146.252
178.237.56.153
178.237.56.155
116.16.128.108
54.39.67.22
80.80.80.80
144.44.189.92
8.45.43.20
100.15.34.146
98.26.1.73
128.116.74.211
51.89.66.42
142.44.189.92
123.30.215.48
54.208.10.149
45.130.53.68
99.83.188.88
71.199.120.136
188.214.128.77
178.237.56.153
178.237.56.155
213.192.54.136
18.235.229.245
24.15.187.173
108.55.19.51
8.9.230.245
68.142.122.37
75.2.84.139
85.249.37.170
178.44.224.229

 

[baztheallmighty]

Surprised to hear that, do you have any evidence to support that? I oversee multiple servers and for all of them the attacks have stopped once the port was changed from the default one.

This no longer works for me either. Against a persistent attacker it was never going to work long term.

The current solution is to use a "fail2ban" script to block the attacks based on the logs which show a flood of connections. The frequency of the attacks has also increased for me as well.

 

[deckardblade]

1. Shut down the KF2 server.
2. Change the "Port=7777" in LinuxServer-KFEngine.ini file to "Port=7878"
3. Ensure that this port is accessible from the Internet (e.g. firewall openings, etc)
4. Start the KF2 server.

One of my servers is on 7780, gets hit. May decrease the frequency. I've done some digging and would not be surprised if this is being used or has been modified Fake Players by Luigi Aurriemma.

 

[o2xVc3UuXp0NyBihrUnu]

One of my servers is on 7780, gets hit. May decrease the frequency. I've done some digging and would not be surprised if this is being used or has been modified Fake Players by Luigi Aurriemma.

Again, can you supply some evidence of that from your log files? Thanks for the tool tip, I'll see how it works and if it resembles what I've seen on the network.

If you are still attacked though on the non-standard port and don't want to change it the fail2ban type of log analyzer from the OP works wonders.

 

[deckardblade]

Again, can you supply some evidence of that from your log files? Thanks for the tool tip, I'll see how it works and if it resembles what I've seen on the network.

If you are still attacked though on the non-standard port and don't want to change it the fail2ban type of log analyzer from the OP works wonders.

Port 7780, 41,000 login attempts. I just null route the IPs and have the bandwidth where it doesn't matter.

 

[deckardblade]

Surprised to hear that, do you have any evidence to support that? I oversee multiple servers and for all of them the attacks have stopped once the port was changed from the default one.

Zero surprise found here