• Please make sure you are familiar with the forum rules. You can find them here: https://forums.tripwireinteractive.com/index.php?threads/forum-rules.2334636/

Server KF2 (or any Unreal Engine 3) server on RedHat/CentOS/Rocky/Alma Linux DDoS defense with the help of firewalld

Jump to latest Follow Reply
Sort by date Sort by votes
#81
Emenblade said:
I wonder if other games are affected with the same thing?
Click to expand...
Good question, I don't host other UE3 servers and don't know anyone who would. We would need input from major multi-game hosters to learn that or from cloud hosters who bothered to investigate this issue for their customers. So, no idea, but to the best of my understanding any UE3 game server can be abused this way.
 
Upvote 0
#83
What I've found is with severe rate limiting firewall rule in action the logs don't get populated with enough offending IPs frequently enough for the banning script to pick them up at all times. The worst offenders do get picked up, but minor IPs that don't hit your server as often slip through the cracks. I'm currently testing a more robust approach that involves two things:

1) changing the
Code: 
[IpDrv.TcpNetDriver]
...
ConnectionTimeout=20.0

from the default 180. That's how much time KF2 server waits before hanging up on a client. I don't feel like it's worth it to wait that long.

2) relying on the results of this one-liner to extract the offending IPs from the logs:

Code: 
tail -5000 ServerVanillaSeventh.log|grep -F -A2 'Connection timed out after'|awk -F" |:" '/Close/ {print $7}'|uniq

The previous IP extracting one-liner had a requirement to extract them ASAP, so it relied on the "Open TheWorld" keyword, which is logged first when we have a connection. The upside of it is you get this ASAP, but the downside is a good share of them are legit connections which need to be filtered out by relying on the fact that legit clients rarely connect >50 times in a row.

What I've found is legit clients, even if they connect >50 times in a row, close their connections themselves, they never just "expire" by timeout. While all the DDoS connections do expire, always.

This way even very mild DDoS IPs, that can hit your server just 2 times, will still get banned. With the initial one-liner they get banned only if they hit >50 times.

The full crontab line looks like this:

Code: 
* * * * * tail -5000 /home/kf2-admin/kf2-server/KFGame/Logs/ServerVanillaSeventh.log|grep -F -A2 'Connection timed out after'|awk -F" |:" '/Close/ {print $7}'|uniq|while read ip; do firewall-cmd --permanent --ipset=networkblock --add-entry=$ip/20;done && firewall-cmd --reload >/dev/null 2>&1

So far so good.
 
Upvote 0
#84
Another observation, I logged the dropped UDP packets wtih connlimit 5 for /20 networks for whole day today.

Then, I've collected a bunch of legit authenticated players IPs from the logs and correlated them with what I extracted from 'messages' with the firewall logs. There were 0 hits. Which means that, despite >100 players joining total, the rate limiting script with connlimit 5 and mask /20 did not drop a single UDP datagram from a single player who authenticated.

So far connlimit 5 and /20 networks for rate limiting look very robust. Will keep watching them more.

PS Of course, the logging rule doesn't log EVERY packet that gets dropped because of the logging limit of 1 line per second, so there could be some intersection. Watching it.
 
Last edited:
Upvote 0
#85
I have my KF2 server sitting behind my Fortinet UTM Firewall, and my logs are showing UDP flood attacks much like everyone else has been seeing.

I'm seeing several IP addresses trying to connect every 5-20 seconds, with each IP address establishing over 10000+ sessions per second.

I set my session UTM threshold to 90 concurrent sessions and it automatically blocks the IP if it exceeds it.

It definitely helps having my server behind my own UTM appliance because it is constantly being flooded with this UDP attack, and 99% of the blocked IP's are trying to access port 7777 (or any other Game port you designated for your KF2 server).
 
  • Like
Reactions: Emenblade and o2xVc3UuXp0NyBihrUnu
Upvote 0
#86
baztheallmighty said:
I have been running wireshark on my system and have discovered the below when i was checking my log. Most people will recognise this ip as googles dns servers, the ip in red is mine. Obviously these packets are not actually from google they are from the attacker. The attacker must be connected to a ISP that has poor security and doesn't filter packets based on source ip. This means my replies to there connection requests will go to googles dns. My server is sending UDP packets to googles dns server(or any other server) effectively making it part of a udp reflection attack( https://docs.aws.amazon.com/whitepa...s-ddos-resiliency/udp-reflection-attacks.html ). This also explains why people that are using hosted services are getting banned because there are technically part of a udp reflection attack.

As such i have updated my powershell code to prevent my server from sending packets to the victim once discovered and a few other tweaks to reduce the likelyhood of blocking legitimate services my server might need to use.



View attachment 2336441View attachment 2336442





Code: 
#Run once to create the firewall rule
new-NetFirewallRule -DisplayName "Block DDOS" -Direction inbound –LocalPort Any -Protocol any -Action Block -RemoteAddress "80.80.80.80/32" -Program "C:\KF2Server\Binaries\Win64\KFServer.exe"
new-NetFirewallRule -DisplayName "Block DDOSout" -Direction outbound –LocalPort Any -Protocol any -Action Block -RemoteAddress "80.80.80.80/32" -Program "C:\KF2Server\Binaries\Win64\KFServer.exe"


#Log location
cd "C:\KF2Server\KFGame\Logs"
while(1){

    $Files=Get-ChildItem Launch_*.log
    #Setup the firewall rules
    foreach ($file in $files){
        #Get the ip addresses from the file, only look at the last 3000 lines to speed it up
        $IPs=Get-Content $file.fullname -Tail 1000 | where { $_ -like "*NetComeGo: Open TheWorld*" } | % { $_ -replace "\[\d+.\d+\] NetComeGo: Open TheWorld \d+\/\d+\/\d+ \d+\:\d+\:\d+ ",'' } | Group-Object
        #Must be more than 50 connections
        $NewIPS=$IPS | where { $_.count -gt 50 } | % { "$($_.name.trim())" }
        if($NewIPS){
            #Get existing rules
            $IPString = (Get-NetFirewallRule -DisplayName "Block DDOS" | Get-NetFirewallAddressFilter ).RemoteAddress
            write-host $file.fullname
            write-host $NewIPS
            $IPString+=$NewIPS
            $IPString=$IPString | sort -Unique
            #Ban the ips
            set-NetFirewallRule -DisplayName "Block DDOS" -Direction inbound –LocalPort Any -Protocol any -Action Block -RemoteAddress $IPString -Enabled True
            set-NetFirewallRule -DisplayName "Block DDOSout" -Direction outbound -LocalPort Any -Protocol any -Action Block -RemoteAddress $IPString -Enabled True
            write-host "$(get-date)--------"
        }
    }
    write-host -NoNewline "."
    sleep 15
}
Click to expand...

On my windows sometimes it won't add the ddos address to my firewall filter, I add manually on the firewall UI and it still doesn't work, needs to restart PC then I can manual add it but PS still doesn't add new address.

The current filter has 18 ip addresses, and powershell will still print the addresses that the filter already has.
 
Upvote 0
#87
I checked my logs again and it seem a little over 30% of my attacks are coming from the 128.116.0.0/16 block of addresses. That's just my particular server, but I'm curious if it's a block of IP's from a botnet thats targeting KF2 server ports specifically. You can set your firewall to block that IP address range as I did. But there's still scans from IP's looking for port 7777 and 7778 specifically in my logs. I'm lucky enough to be behind my Fortinet UTM, but if possible, I'd suggest getting a cheaply built pfsense firewall box setup in front of your KF2 server to stop the UDP floods.
 
Upvote 0
#88
baztheallmighty said:
This explains why i have some many extra players in my servers. Probably hit 10k this month.
Click to expand...
Great stats! How do you extract them from the game? I know there's a stats directory with .stats files generated after each match, I've always wondered if there's a way to parse it as it seems binary.
 
Upvote 0
#90
o2xVc3UuXp0NyBihrUnu said:
Great stats! How do you extract them from the game? I know there's a stats directory with .stats files generated after each match, I've always wondered if there's a way to parse it as it seems binary.
Click to expand...
I have two PowerShell scripts. Both are terrible abominations, both in how they are written and designed. I would not repeat this error again, however at this point it's to much work to redo. The first script is a scraps the web admin interface for all the useful data every 5 seconds and puts the data into a database. It also does a bunch of other stuff like tell players there stats when they join the server. The second one reads and computes the data, then generates 1000's of html files, which make up the web site.


If you would like a copy PM me, but as mentioned it's a real mess.
 
Upvote 0
#91
Racpy said:
On my windows sometimes it won't add the ddos address to my firewall filter, I add manually on the firewall UI and it still doesn't work, needs to restart PC then I can manual add it but PS still doesn't add new address.

The current filter has 18 ip addresses, and powershell will still print the addresses that the filter already has.
Click to expand...
At this point, Fail2ban style protective measures like my script don't work. It's too slow and the attacker changes target ip addresses frequently. An endless game of whack a mole. I suggest either a commercial firewall solution or a linux iptables rate limiting solution at this point.


The current Linux solution as per previous posts rate limits new connections to the server from within a /20 ( 4096 address block). Here is some example code:
Code: 
iptables -I FORWARD 1 -d 10.0.0.1/32 -p udp -m udp --dport 7777:7779 -m connlimit --connlimit-above 20 --connlimit-mask 20 --connlimit-saddr -m limit --limit 1/sec --limit-burst 1 -j LOG --log-prefix "Drop-Connection-Limit 23: "
iptables -I FORWARD 2 -d 10.0.0.1/32 -p udp -m udp --dport 7777:7779 -m connlimit --connlimit-above 20 --connlimit-mask 20 --connlimit-saddr -j DROP
 
Upvote 0
#92
charlierunkle said:
I checked my logs again and it seem a little over 30% of my attacks are coming from the 128.116.0.0/16 block of addresses.
Click to expand...
That's actually 128.116.0.0/17, which is "Roblox". It's some kind of online game framework for simplistic games or something like that. Seems to be pretty popular. They've been under this attack for years, so I've included the blanket ban of their whole range into the method suggested in the OP.

Code: 
0 6 * * * firewall-cmd --ipset=networkblock --get-entries|while read ip; do firewall-cmd --permanent --ipset=networkblock --remove-entry=$ip;done;firewall-cmd --permanent --ipset=networkblock --add-entry=128.116.0.0/17 >/dev/null 2>&1
 
  • Like
Reactions: charlierunkle
Upvote 0
#93
New developments regarding the current WIP banning script.

After a few days of running 2 issues were identified:
1) 20s server timeout isn't enough, as people with HDD get disconnected sometimes during the map change when they have to download the map from the workshop. I now changed it to 60s.

2) Players whose KF2 crashes will timeout and thus will be counted as bots by the script and their IP range will get banned. To address that I've introduced the counter of 5, which should trigger fine on bots, but players will have to crash at least 6 times in a relatively short period of time (e.g. 1 hour or so) to get their IP banned, which should be rare enough. Now the banning script looks like this:

Code: 
*/15 * * * * tail -5000 /home/kf2-admin/kf2-server/KFGame/Logs/ServerVanillaSeventh.log|grep -F -A2 'Connection timed out after'|awk -F" |:" '/Close/ {a[$7]++} END {for (b in a) {if (a[b]>5) {print b}}}'|uniq|while read ip; do firewall-cmd --permanent --ipset=networkblock --add-entry=$ip/20;done && firewall-cmd --reload >/dev/null 2>&1

Since the rate limiting rule in the firewall does a decent job at mitigating the intensity of the attack it's fine IMO to launch the banning script only once in 15 mins.

Currently the banning script picks up and bans about 250 /20 IP ranges per full 24 hours cycle, including the default "Roblox" /17 range. All ranges get unbanned once a day because 1) I haven't fine tuned the solution good enough to fully exclude marginal scenarios with legit players 2) the attack is rolling anyways and in my tests roughly ~30% of IP ranges from the previous day roll over to the next one, while ~70% are new ones, so it's not a big deal to flush them once a day. But maybe it's too frequent and flushing it once in 3 days would be better.

I'm not updating the OP message just yet as it doesn't feel "production ready" enough yet.
 
Upvote 0
#94
Also, here's another idea.

It's well known that TWI's official KF2 servers use very non-standard game ports from the 27000-30000 range of UDP. There are also no reports that their servers are under attack. Maybe they use some kind of next gen firewall that does the filtering for them or anti-DDoS CDN like CloudFlare to scrub the bad traffic. But maybe they aren't attacked at all.

I'm going to verify this hypothesis partially soon. The idea is as follows. If I were a sane attacker I would do my best to ensure the future of my operation by not going overboard and not going too noisy, because, let's say, I followed the Mirai botnet story and learnt my lessons. Therefore, I would exclude all major server hosters from my list of the servers used for amplification, such as TWI servers. How would I distinguish TWI from community hosted? There are two ways that come to my mind which are 1) filter out servers that have "Tripwire" in their title or 2) filter out all the TWI's server ranges.

The second maybe also helpful for another reason. Suppose you are a backbone ISP and have to deal with this BS. These attacks can focus on bringing > 1 Tbps of garbage traffic, capable of saturating the pipes of even major providers. Of course, if you are a backbone ISP you want none of that as you don't want to spend resources on providing routing for garbage traffic. They may very well engage in throttling of the garbage traffic. What criteria they would use for that? One good suggestion is by just relying on port ranges, knowing that the default and most popular UE3 game port is 7777 and >1 UE3 server on the same IP gets next port in line, e.g. 7778 and so on by default. Thus, they may engage in throttling of 7777 and plus say hundred UDP ports. Of course, if they did this, there would be no incentive to go public about it because it's all bad publicity and they would treat potential complaints with obscure explanations regarding performance issues and overall useless helpdesk stalling etc. Of course, the fact that TWI (and maybe other major hosters) don't use the default ports (BTW, does anyone know the reason why?) ensures that they don't complain.

Coincidentally, in the past 2 or so weeks the players on my servers were complaining about lags and jitter more often than usual and I know for sure it's not because of my servers. Is it a coincidence? Or is this how the aforementioned throttling may manifest itself?

If that was the case, one would do themselves good service by switching their servers from the default ports range up to something that TWI themselves uses. So I'm going to do this right now and report back the results.

Expected results: 1) attack stops at least for a while, until new addresses are rediscovered via Steam API 2) attack may stop altogether if the filtering by port numbers is in use 3) there may be less lags and jitter complaints because of less/no throttling.
 
Upvote 0
#96
  • Like
Reactions: o2xVc3UuXp0NyBihrUnu
Upvote 0
#97
I think i have found a way to fix this completely. When kf2 client is opened it downloads a list of servers(well i assume it downloads a list i haven't verified this part, it's not overly import). The client then sends out a ping to all servers on the query port. Then the client displays the list of servers to the player. The player selects a server and joins it. Wireshark confirms this.

So effectively we have a means of authentication a little like port knocking. If the ip connects to the query port, then they are probably a real player and should be allowed to connect to server. This works the same for the kf2 native client and the steam client. Is there a EPIC client that shows a server list???? This only breaks down if the player connects directly to the server with the open command as far as I can tell. Interestingly this allows us to collect the vast majority of kf2 players ip addresses.

A possible untested solution:
We could then use something like a script or fail2ban with a custom command to scan the log file created by a iptables log rule and put the ip address into an ipset which is part of a iptables accept rule for connecting to the kf2 server port.




o2xVc3UuXp0NyBihrUnu said:
Also, here's another idea.

It's well known that TWI's official KF2 servers use very non-standard game ports from the 27000-30000 range of UDP. There are also no reports that their servers are under attack. Maybe they use some kind of next gen firewall that does the filtering for them or anti-DDoS CDN like CloudFlare to scrub the bad traffic. But maybe they aren't attacked at all.

I'm going to verify this hypothesis partially soon. The idea is as follows. If I were a sane attacker I would do my best to ensure the future of my operation by not going overboard and not going too noisy, because, let's say, I followed the Mirai botnet story and learnt my lessons. Therefore, I would exclude all major server hosters from my list of the servers used for amplification, such as TWI servers. How would I distinguish TWI from community hosted? There are two ways that come to my mind which are 1) filter out servers that have "Tripwire" in their title or 2) filter out all the TWI's server ranges.

The second maybe also helpful for another reason. Suppose you are a backbone ISP and have to deal with this BS. These attacks can focus on bringing > 1 Tbps of garbage traffic, capable of saturating the pipes of even major providers. Of course, if you are a backbone ISP you want none of that as you don't want to spend resources on providing routing for garbage traffic. They may very well engage in throttling of the garbage traffic. What criteria they would use for that? One good suggestion is by just relying on port ranges, knowing that the default and most popular UE3 game port is 7777 and >1 UE3 server on the same IP gets next port in line, e.g. 7778 and so on by default. Thus, they may engage in throttling of 7777 and plus say hundred UDP ports. Of course, if they did this, there would be no incentive to go public about it because it's all bad publicity and they would treat potential complaints with obscure explanations regarding performance issues and overall useless helpdesk stalling etc. Of course, the fact that TWI (and maybe other major hosters) don't use the default ports (BTW, does anyone know the reason why?) ensures that they don't complain.

Coincidentally, in the past 2 or so weeks the players on my servers were complaining about lags and jitter more often than usual and I know for sure it's not because of my servers. Is it a coincidence? Or is this how the aforementioned throttling may manifest itself?

If that was the case, one would do themselves good service by switching their servers from the default ports range up to something that TWI themselves uses. So I'm going to do this right now and report back the results.

Expected results: 1) attack stops at least for a while, until new addresses are rediscovered via Steam API 2) attack may stop altogether if the filtering by port numbers is in use 3) there may be less lags and jitter complaints because of less/no throttling.
Click to expand...

My best guess as to how bot net ddos amplifiers work:

I doubt they are looking at information here or the steamAPI. It's way to much work. They will be looking at CVE's for services and seeing if there are known exploitable vulnerabilities. Then attackers will use something like shodan to get a list of all servers running software they know has vulnerabilities. e.g. https://beta.shodan.io/search?query=port:27015+product:"Steam+Dedicated+Server"&page=2
Once they have a list they do a test run to see what the results are like, then they turn it on. It's is then fine tuned by sending sample packets to there test server during the attack.

In general there are 4 ways ISP's block this traffic.
1) Just having enough bandwidth to absorb the attack as it transits there network.
2) Rate limiting some services. This is normally done with some kindof monitoring box and automation of the edge of there network. This is normally considered a bad idea because you have already paid to have the traffic enter your network.
3) The most common way is by black-holing ip addresses. This is either done on there network or when they get too much traffic they have agreements with there upstreams (other ISP's they interconnect with) to black hole that ip addresses before they arrive in there network. This is normally setup in a automated fashion.
4) With drawing routes from there upstreams. This means to stop advertising the route to an upstream so they stop sending you traffic for that ip or subnet. This is normally last ditch as you are getting to much traffic and it is causing problems for other services.

As for tripwire they appear to be using VOXEL / Internap some kind of hosting company. This is based on the ip addresses used by there servers and a quick look up here https://www.whois.com/whois. Looks like Bare Metal hosting. I assume there is some kind of commercial firewall in front of there servers. The marketing material says DDoS Protection & Mitigation https://www.inap.com/managed-services/managed-security/
 
  • Like
Reactions: o2xVc3UuXp0NyBihrUnu
Upvote 0
#98
baztheallmighty said:
A possible untested solution:
We could then use something like a script or fail2ban with a custom command to scan the log file created by a iptables log rule and put the ip address into an ipset which is part of a iptables accept rule for connecting to the kf2 server port.
Click to expand...
OK, I'm on it, will report back. Basically, from what I understand from your message, it would be a "port knocking" whitelist solution that would allow connects to the game port only after steam query port was accessed by a certain IP. I'll also test how it works with the direct "open" command.

baztheallmighty said:
My best guess as to how bot net ddos amplifiers work:

I doubt they are looking at information here or the steamAPI. It's way to much work. They will be looking at CVE's for services and seeing if there are known exploitable vulnerabilities. Then attackers will use something like shodan to get a list of all servers running software they know has vulnerabilities. e.g. https://beta.shodan.io/search?query=port:27015+product:"Steam+Dedicated+Server"&page=2
Once they have a list they do a test run to see what the results are like, then they turn it on. It's is then fine tuned by sending sample packets to there test server during the attack.
Click to expand...
Shodan is bad for discovering open UDP ports with not well-known services. The reason behind this is that there's no 2-way handshake for UDP. The inquirer has to know what to send and what to expect back to be able to tell if the port is open. To the best of my knowledge shodan is clueless with UE3 protocol and cannot identify it. Notice that the URL you've provided lists only TCP services on 27015 and no UDP services. Are you also aware, that for KF2 27015 UDP is also customizable? I currently run servers that use steam query port at 47000 range, and yet they are visible in the browser and discoverable by players. KF2 client has to retrieve this information from somewhere, specifically, the IP of the game server and its custom ports and the only place it can get it is from the Steam API.

baztheallmighty said:
In general there are 4 ways ISP's block this traffic.
1) Just having enough bandwidth to absorb the attack as it transits there network.
2) Rate limiting some services. This is normally done with some kindof monitoring box and automation of the edge of there network. This is normally considered a bad idea because you have already paid to have the traffic enter your network.
3) The most common way is by black-holing ip addresses. This is either done on there network or when they get too much traffic they have agreements with there upstreams (other ISP's they interconnect with) to black hole that ip addresses before they arrive in there network. This is normally setup in a automated fashion.
4) With drawing routes from there upstreams. This means to stop advertising the route to an upstream so they stop sending you traffic for that ip or subnet. This is normally last ditch as you are getting to much traffic and it is causing problems for other services.
Click to expand...

Agreed, plus BCP 38 for last mile ISPs from which the malicious traffic actually originates.

baztheallmighty said:
As for tripwire they appear to be using VOXEL / Internap some kind of hosting company. This is based on the ip addresses used by there servers and a quick look up here https://www.whois.com/whois. Looks like Bare Metal hosting. I assume there is some kind of commercial firewall in front of there servers. The marketing material says DDoS Protection & Mitigation https://www.inap.com/managed-services/managed-security/
Click to expand...

They use various, seemingly unrelated hosters all over the place. Different hosters in different regions. Other 2 ones: performive.com, velia.net.

Also DDoS marketing is typically complete garbage. They rarely can deal effectively with such rare and obscure things (until recently) as UE3 UDP reflective L3/L4 volumetric DDoS. Typically they can deal well only with well-known threats hardcoded into them.
 
Upvote 0
#99
Upvote 0
You must log in or register to reply here.
Top Bottom
Back
Jump to latest Follow Reply
Jump to latest Follow Reply
Community News
Forums
Merch
Menu