• Please make sure you are familiar with the forum rules. You can find them here: https://forums.tripwireinteractive.com/index.php?threads/forum-rules.2334636/

Server KF2 (or any Unreal Engine 3) server on RedHat/CentOS/Rocky/Alma Linux DDoS defense with the help of firewalld

Port 7780, 41,000 login attempts. I just null route the IPs and have the bandwidth where it doesn't matter.
Thanks, yeah, this proves it. Now I wonder if one uses a more random port which is far from the 7777 value, something like TWI does, will it help? Their servers aren't attacked according to the last info I have, but it's from the last week, who knows by now.
 
Upvote 0

Thanks for your input! No idea why would regular players connect and disconnect for 50 times in a row. I mean, there are weidos out there and sometimes yes, someone would join, stay a bit and then leave and so on for a dozen of times, but 50?

But okay, based on your report I'll review the addition and will update the OP script today. It's gonna be a little slower, but we'd better be safe in order not to touch the real players...
 
Upvote 0
Thanks for your input! No idea why would regular players connect and disconnect for 50 times in a row. I mean, there are weidos out there and sometimes yes, someone would join, stay a bit and then leave and so on for a dozen of times, but 50?

I'm as surprised at this as you are. I can share the logs and ip list of such players, if it can help. Perhaps you will find some patterns that I did not notice
 
Upvote 0
Hi guys,

Mine servers were affected too (started 04.07)... Port 27015 were used to ddos other 3rd party server... It looks like someone have found vulnerability in KF2 server files and uses it for attacks.
Hi!

omg, you say your server was used to attack other servers? Can you tell us more? Why did you come to this conclusion? This is interesting and might explain the spam from real players that I found in my logs
 
Upvote 0
Well, yeah. So it increasingly looks like changing the default port doesn't help long term.

Theoretically, the port your server listens on is discoverable via Steam Client or maybe also via Steam API, so it shouldn't be a big deal to run a discovery and then attack what's discovered. The reason why changing the port at least temporarily stops the attack is probably because the discovery is run once in a while, so if you changed the port, the attack stops until the next discovery event happens and your server ends up in the list of the ones the dude wants to attack at the moment. Then, again, theoretically, immediately changing the port after the attack is detected should again stop it until the next discovery event, so, theoretically one can develop a script that detects the attack, stops the server (if noone's playing), changes the port to some random value, starts the server. The downside is the server gets removed from the favorites tab and also it's temporary and one will need to keep changing it making the favorites functionality useless for his users.

It is unclear why TWI said that they aren't under attack. Aren't they really? Or their pipes are so big that they just don't care about traffic, they don't have NG firewalls that would start dropping it and their servers don't hang up? It's unlikely IMO given what we know that it's TWI themselves behind the attack (although so far it seems that only the community servers are affected) as it's a criminal offense and one would think the company wouldn't risk doing that.

I'm actually lost at guessing the motives of someone behind the attacks. I've heard that it's very typical, say, in Minecraft scene to DDoS the servers of one's competitors. It kind of makes some sense at least. Could it be the motive here? Say, some rather big community servers owner tries to knock down his competitors. Kind of ridiculous to me, but that's what happens in the Minecraft scene. E.g.:


Could very well be a typical journo BS, hard to tell. But hey, maybe I should start thinking about providing a DDoS KF2 hosting services or DDoS KF2 protection consulting services.
 
Upvote 0
Hi!

omg, you say your server was used to attack other servers? Can you tell us more? Why did you come to this conclusion? This is interesting and might explain the spam from real players that I found in my logs
Yes, exactly. For now i have closed my KF2 servers due to that issue...
I have received info from 3rd server that was "ddosed" by my server, IP/MAC anonymized ofc

Dear Sir/Madam,

We have detected abuse from the IP address XXX, which according to a abusix.com is on your network. We would appreciate if you would investigate and take action as appropriate.

Log lines are given below, but please ask if you require any further information.

If you are not the correct person to contact about this please accept our apologies - your e-mail address was extracted from the whois record by an automated process.
[SPOILER="Nothing interesting"]
This mail was generated by Fail2Ban.
The recipient address of this report was provided by the Abuse Contact DB by abusix.com. abusix.com does not maintain the content of the database. All information which we pass out, derives from the RIR databases and is processed for ease of use. If you want to change or report non working abuse contacts please contact the appropriate RIR. If you have any further question, contact abusix.com directly via email ([email protected]). Information about the Abuse Contact Database can be found here: https://abusix.com/global-reporting/abuse-contact-db
abusix.com is neither responsible nor liable for the content or accuracy of this message.[/SPOILER]

Note: Local timezone is +0200 (CEST)

Jul 4 18:00:15 www3 kernel: [581803.956597] INPUT LOG: IN=eth0 OUT= MAC=XYZ SRC=XXX DST=ZZZ LEN=197 TOS=0x00 PREC=0x00 TTL=57 ID=13061 PROTO=UDP SPT=27015 DPT=57163 LEN=177 <-- this repeated hundread of times...
EDIT:

Btw my log file which was created 18.07 and stopped today (28.07) have 12 075 638 lines and is full of two states:
[670423.20] NetComeGo: Close TcpNetDriver_32 TcpipConnection_2385975 IP HERE
[670423.20] DevNet: TcpNetDriver_32 Connection timed out after 200.000000 seconds (279.929688)
[670423.20] DevNet: TcpNetDriver_32 Connection state = 3
[670742.23] DevNet: NotifyAcceptingConnection: Server TheWorld accept
[670742.23] NetComeGo: Open TheWorld IP HERE
EDIT 2:

Only two of my three kf2 servers were affected (all same IP) - affected ports (7781 + 27019 and 7777 + 27015; 7784 + 27022 unaffected). Fun fact - first server is password protected and have been compromised...
 
Last edited:
Upvote 0
Check out from what address I'm being hammered now!

I do not believe that anyone has hacked google DNS to attack kf2 servers.
looks like spoofing the sender's ip address. This explains a lot, but it is bad because the simple ip ban is not enough now.
 

Attachments

  • Capture8888.PNG
    Capture8888.PNG
    259.5 KB · Views: 22
Last edited:
Upvote 0
in addition to the previous post:

if we want to defend without DPI, then I see only this option: analyze logs in real time, ban ip temporarily.
for requests come from fake 8.8.8.8 or something similar that is needed for work maybe use some more complex rule that will restrict it only for the kf2 server, e.g.:
firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address=<ip> port protocol=udp port=<kf2_ports_range> reject"
 
Last edited:
Upvote 0
I have been running wireshark on my system and have discovered the below when i was checking my log. Most people will recognise this ip as googles dns servers, the ip in red is mine. Obviously these packets are not actually from google they are from the attacker. The attacker must be connected to a ISP that has poor security and doesn't filter packets based on source ip. This means my replies to there connection requests will go to googles dns. My server is sending UDP packets to googles dns server(or any other server) effectively making it part of a udp reflection attack( https://docs.aws.amazon.com/whitepa...s-ddos-resiliency/udp-reflection-attacks.html ). This also explains why people that are using hosted services are getting banned because there are technically part of a udp reflection attack.

As such i have updated my powershell code to prevent my server from sending packets to the victim once discovered and a few other tweaks to reduce the likelyhood of blocking legitimate services my server might need to use.



Capture.PNGCapture2.PNG





Code:
#Run once to create the firewall rule
new-NetFirewallRule -DisplayName "Block DDOS" -Direction inbound –LocalPort Any -Protocol any -Action Block -RemoteAddress "80.80.80.80/32" -Program "C:\KF2Server\Binaries\Win64\KFServer.exe"
new-NetFirewallRule -DisplayName "Block DDOSout" -Direction outbound –LocalPort Any -Protocol any -Action Block -RemoteAddress "80.80.80.80/32" -Program "C:\KF2Server\Binaries\Win64\KFServer.exe"


#Log location
cd "C:\KF2Server\KFGame\Logs"
while(1){

    $Files=Get-ChildItem Launch_*.log
    #Setup the firewall rules
    foreach ($file in $files){
        #Get the ip addresses from the file, only look at the last 3000 lines to speed it up
        $IPs=Get-Content $file.fullname -Tail 1000 | where { $_ -like "*NetComeGo: Open TheWorld*" } | % { $_ -replace "\[\d+.\d+\] NetComeGo: Open TheWorld \d+\/\d+\/\d+ \d+\:\d+\:\d+ ",'' } | Group-Object
        #Must be more than 50 connections
        $NewIPS=$IPS | where { $_.count -gt 50 } | % { "$($_.name.trim())" }
        if($NewIPS){
            #Get existing rules
            $IPString = (Get-NetFirewallRule -DisplayName "Block DDOS" | Get-NetFirewallAddressFilter ).RemoteAddress
            write-host $file.fullname
            write-host $NewIPS
            $IPString+=$NewIPS
            $IPString=$IPString | sort -Unique
            #Ban the ips
            set-NetFirewallRule -DisplayName "Block DDOS" -Direction inbound –LocalPort Any -Protocol any -Action Block -RemoteAddress $IPString -Enabled True
            set-NetFirewallRule -DisplayName "Block DDOSout" -Direction outbound -LocalPort Any -Protocol any -Action Block -RemoteAddress $IPString -Enabled True
            write-host "$(get-date)--------"
        }
    }
    write-host -NoNewline "."
    sleep 15
}
 
Upvote 0
You are correct that the replies are much more sizeable than the initial requests to connect. Here's how it looked like on my servers. A downwards (means outbound) spike starts when a whole bunch of new IPs are added to the attack and then ends when my banning bot picks it up and blocks the traffic:

View attachment 2336424

You are also correct that it might be a part of a broader DDoS that uses KF2 servers as amplification beacons. But for our purposes this is irrelevant as whatever the small volume of traffic comes to our servers it is still enough to bring our servers down and as I noted previously I witnessed my servers hanging up completely after receiving (and responding to) merely ~5K malicious requests. Therefore it still should be considered a DDoS attack against KF2 servers as it still reaches the goal of bringing them down, even if it's a collateral damage situation.

Also, by now in my research I established that, from all the attacked hosts I analyzed, all of them listened on default port UDP 7777. This is also true for your server per the hoster notification you provided, thank you. This fully explains why TWI doesn't care about it and was reluctant to react in any way, as almost all if not all of their KF2 servers use weird ports, typically it's some random number from the range 27000-30000 UDP. Thus, by the nature of this attack, only lone (because if a person hosts >1 on a single IP they have to change the default ports on all but 1st server) community servers are the victims.

This suggests a stupid and simple solution to the DDoS problem that still amazingly works, which is to just change the listening port from the default one to any other random value. This has proven to prevent the attack, even if one unbans all the IPs one has banned previously to provoke it. With great success this technique has been implemented on a few hosts I've dealt with, so if you don't like the IP banning solution outlined in the OP (which still works flawlessly and prevents hangups and complaints from the hoster), you can just change the listening port to some random value. The only downside to this I know is it effectively removes the server from frequenters favorite list, as it's populated by IP:port and they will have to readd it.

View attachment 2336425
There's nothing distributed about this, it's a literal DoS. I haven't ever been hit by more than 2 IPs at once concurrently.
 
  • Like
Reactions: GenZmeY
Upvote 0
Hilarious stuff! You are right, at this point I'm convinced that it's an amplification DDoS with spoofed source IPs that are necessarily coming from a source (which we won't be able to establish) that doesn't do egress filtering on their routers. Otherwise we'll have to assume that it's Google DNS that is either compromised or participates in a DDoS, both of which are very low likelihood scenarios.

Also, great idea on altering the script so it doesn't do full IP bans, but bans only for traffic that's coming to/from KF2 server ports. Since I run multiple servers on the same IP I will have to incorporate the knowledge of which KF2 server's log is being analyzed to set proper ports for firewall rules. I also wonder what firewalld type of rule would suit this best from the performance standpoint, as one could potentially run out of rules or something. The ipsets from my experience can pretty much be filled with a real ton of records, but rules -- I dunno, needs to be investigated.
 
Upvote 0
There's nothing distributed about this, it's a literal DoS. I haven't ever been hit by more than 2 IPs at once concurrently.
Yeah, but as it turns out these IPs are targets (source IP is likely spoofed), not true sources. The attacker most likely uses many KF2 servers at once to amplify the attack against Google DNS and other targets. Your particular server is just one pawn out of the many, so it's still a DDoS. Another consideration that the true sources may be many, you just can't know it as all of them use the same source IP which is spoofed, so your server responds to them.
 
Upvote 0
1. Shut down the KF2 server.
2. Change the "Port=7777" in LinuxServer-KFEngine.ini file to "Port=7878"
3. Ensure that this port is accessible from the Internet (e.g. firewall openings, etc)
4. Start the KF2
Unfortunately my ip's are not in a chronological order they are in sorted order from PowerShell.

As for who is behind it, no idea. If you put them through something like a "bulk ip lookup tool" you get all kinds of ISP's and hosting platforms(aws,OVH,etc) around the world. This would suggest some kind of bot net.
I have received very few proper hosting platform IPs. I've had one total at OVH, nothing else. T-Mobile hotspots and then private IPs
 
Upvote 0
Well, yeah. So it increasingly looks like changing the default port doesn't help long term.

Theoretically, the port your server listens on is discoverable via Steam Client or maybe also via Steam API, so it shouldn't be a big deal to run a discovery and then attack what's discovered. The reason why changing the port at least temporarily stops the attack is probably because the discovery is run once in a while, so if you changed the port, the attack stops until the next discovery event happens and your server ends up in the list of the ones the dude wants to attack at the moment. Then, again, theoretically, immediately changing the port after the attack is detected should again stop it until the next discovery event, so, theoretically one can develop a script that detects the attack, stops the server (if noone's playing), changes the port to some random value, starts the server. The downside is the server gets removed from the favorites tab and also it's temporary and one will need to keep changing it making the favorites functionality useless for his users.

It is unclear why TWI said that they aren't under attack. Aren't they really? Or their pipes are so big that they just don't care about traffic, they don't have NG firewalls that would start dropping it and their servers don't hang up? It's unlikely IMO given what we know that it's TWI themselves behind the attack (although so far it seems that only the community servers are affected) as it's a criminal offense and one would think the company wouldn't risk doing that.

I'm actually lost at guessing the motives of someone behind the attacks. I've heard that it's very typical, say, in Minecraft scene to DDoS the servers of one's competitors. It kind of makes some sense at least. Could it be the motive here? Say, some rather big community servers owner tries to knock down his competitors. Kind of ridiculous to me, but that's what happens in the Minecraft scene. E.g.:


Could very well be a typical journo BS, hard to tell. But hey, maybe I should start thinking about providing a DDoS KF2 hosting services or DDoS KF2 protection consulting services.
Why they're not attacked? Not running high population servers. They only run 6 man servers. Attacking a server hosting 2 people doesn't cause much grief. When you poll the player count and target based on that - TADA. This isn't an automated attack and it is not distributed. It is being manually kicked off. If it were distributed I'd be getting hit with hundreds/thousands of IPs from a botnet. Instead it has been no more than 3 at any one time. Usually just 1, sometimes 2.

I'm pretty sure I have the guy in my discord unless his specific IP got spoofed during gameplay.
 
Upvote 0
Yeah, but as it turns out these IPs are targets (source IP is likely spoofed), not true sources. The attacker most likely uses many KF2 servers at once to amplify the attack against Google DNS and other targets. Your particular server is just one pawn out of the many, so it's still a DDoS. Another consideration that the true sources may be many, you just can't know it as all of them use the same source IP which is spoofed, so your server responds to them.
Not a DDoS. If it were the service I pay for would have kicked off, and it does when it's a legit one. Which happens every 3-8 months. I have 20gpbs mitigation enabled, this attack does not touch that threshold at all and it does not come from a distributed source.

1627619409856.png

E.G. from one provider. Also TWI will not ever be able to do anything about this, they're trying to figure out how to build a streaming community on a dead game.
 
Upvote 0
Not a DDoS. If it were the service I pay for would have kicked off, and it does when it's a legit one. Which happens every 3-8 months. I have 20gpbs mitigation enabled, this attack does not touch that threshold at all and it does not come from a distributed source.

E.G. from one provider. Also TWI will not ever be able to do anything about this, they're trying to figure out how to build a streaming community on a dead game.
Your server(s) is not the target of the attack. Your server is participating in the DDOS of the target by amplifying the amount of data sent to the target.

From: https://blog.cloudflare.com/reflections-on-reflections/
To bake a reflection attack, the villain needs four ingredients:
  • A server capable of performing IP address spoofing.
  • A protocol vulnerable to reflection/amplification. Any badly designed UDP-based request-response protocol will do.
  • A list of "reflectors": servers that support the vulnerable protocol.
  • A victim IP address.
The general idea:
  • The villain sends fake UDP requests.
  • The source IP address in these packets is spoofed: the attacker sticks the victim's IP address in the source IP address field, not their own IP address as they normally would.
  • Each packet is destined to a random reflector server.
  • The spoofed packets traverse the Internet and eventually are delivered to the reflector server.
  • The reflector server receives the fake packet. It looks at it carefully and thinks: "Oh, what a nice request from the victim! I must be polite and respond!". It sends the response in good faith. (
  • The response, though, is directed to the victim.


As we can see here from my wireshark capture we have a request from the attacking server which is 67 bytes in size. Then we can see my servers response which is 191 bytes in size. This means the attacker is getting a amplification of ~3 minimum. If the server re-sends the data because it doesn't get a response from the client after a timeout period this could be an amplification of ~6. So if they send 10Gb/s of traffic to Killing floor 2 servers all over the world the victim of the attack will receive 60Gb/s of traffic. Now that we have seen the range of IP addresses being targeted and the persistence of the attacks this is unlikely to be someone with an Axe to grind with Tripwire or the KF2 community.
Capture.PNG


The real problem is the fix. Scanning log files and putting ip bans in place will work short term. However the real fix will be to put some kind of rate limit in the network code.
 
Upvote 0
Not a DDoS. If it were the service I pay for would have kicked off, and it does when it's a legit one. Which happens every 3-8 months. I have 20gpbs mitigation enabled, this attack does not touch that threshold at all and it does not come from a distributed source.
I'm going to repeat it for the second and last time and ignore subsequent iterations unless new information is provided.

You CANNOT know if it "does not come from a distributed source" because the source IP addresses are spoofed. It could very well be that most of the datagrams that reach your server are indeed from many different sources, but because the source IP is spoofed you don't know it.

Your server isn't alone, it's used as one of the intermediaries all of which are tricked to send traffic to the same set of IP addresses (e.g. 8.8.8.8) in hopes to saturate their bandwidth and make the service unavailable. This makes it both DoS and distributed which is DDoS, in particular, with amplification.

It's on TWI to provide a long term solution and they should implement some additional verification code that would avoid sending responses of bigger size than the requests or space them out if the requests are found to be sketchy (e.g. >50 attempts in a short period of time with no subsequent authentication with steam id as we use now).
 
Upvote 0
Try this. I've tested it on my old log files and it works, but unfortunately I can't say anything about its real effectiveness...
A couple of days ago, I turned off the server that was attacked. Today I turned it on again, but it looks like they stopped attacking me because of this.

There is also a windows version, but I haven't tried using it. If someone shares their experience it will be great.
 
Last edited:
Upvote 0
Looks like I was wrong, I'm still under attack
However, this thing does a good job, now the attacker manages to send no more than 500 requests from one ip (before, it could have been up to 15000), and the server copes with this number. I have not noticed any increase ping during the game or something similar

iJBZ0tb.png


of course it would be more efficient to do something similar in the kf2 server itself, but this is only available to TWI
 
Last edited:
Upvote 0