Server Linux Killing Floor 2 server memory leaks?

[o2xVc3UuXp0NyBihrUnu]

This what I thought was an urban legend has been around for a while and one of the reasons many server hosters restart their servers once in a while.

I never liked that, as it's obviously painful for unaware players who are say on wave 10 of their 40+ minutes run and suddenly get disconnected because of a server restart. Very annoying.

So I've been running my servers with no scheduled automatic restarts and they have been just okay even after >1 month of running 24/7. They consume typical 500 MB (at the start) -750 MB (after a match starts and everything gets loaded) for the first KF2 instance and 370 MB (at the start) - 550 MB (after a match starts and everything gets loaded) physical process memory (RES values) for all the rest instances running with separate configs.

Recently though I've rented another VPS, this time with a slightly different flavor of Linux and different hardware, mainly AMD Epyc vCPUs. I thought it made sense as AMD CPUs provide much better performance for lower price. Immediately however I've noticed the memory leak issue. Both use KVM. No difference in KF2 server config except the usual things like ports and server names.

Here are the relevant screenshots.

Too high memory consumption by 3 KF2 server instances after 5 days running, multiple matches each since the beginning.


Right after services restart on the same VM.


OS: Rocky Linux release 8.6 (Green Obsidian)

[root@vm-8GE3gmho Logs]# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              3
On-line CPU(s) list: 0-2
Thread(s) per core:  1
Core(s) per socket:  3
Socket(s):           1
NUMA node(s):        1
Vendor ID:           AuthenticAMD
BIOS Vendor ID:      QEMU
CPU family:          23
Model:               49
Model name:          AMD EPYC Processor
BIOS Model name:     NotSpecified
Stepping:            0
CPU MHz:             2445.406
BogoMIPS:            4890.81
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-2
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat umip rdpid arch_capabilities

I plan to slowly dig into this and conduct cleaner experiments, e.g. obvious things like ruling out the OS interference, CPU interference, making more measurements etc and updating this post.

 

[o2xVc3UuXp0NyBihrUnu]

I've traced the issue down to certain community maps. There must be something about some of community maps that causes the KF2 server process to bloat in size, at least on Linux.

The solution to this is not to run garbage maps, until this issue is recognized, the root cause is found and fixed.

This particular one was caused by KF-Random_Holdout map. Another identified offender is KF-ShoppingMall.

Way to reproduce is to launch the KF2 server with a vanilla map, run top command, notice the res column and notice the currently consumed RAM by a process. Change the map to KF-Random_holdout (have to be subscribed to it). Notice the res value of the top command to roughly double in size.