Server [REQUEST] KF2 map compression support

[bviktor]

Hey everyone,

Currently, my server's custom maps are 3.3G in size. But they're just 975M with LZMA compression. So one has to wonder, why doesn't KF2 support compression?

I speak for redirect servers for the most part here, because telling everyone to go to Steam workshop, search for x map, and click subscribe, and potentially restart the game is irritating compared to just having to do... nothing but wait until the map downloads with a properly set up redirect server. Yes, I know TW deprecated this functionality for whatever reason. But as you'll see, it's also beneficial for Steam map downloads.

Let me explain, it really only has advantages:

1. It saves monthly transfer (and thus potentially reduces the bill) on the redirect server.

2. It also saves a lot of disk space on the server, which is often scarce with a VPS instance.

3. Most importantly, it saves a lot of map download time for players who don't have the map yet.

4. Requires CPU time on the client to decompress, but I can't imagine a rig capable of playing KF2 having a hard time decompressing a few dozen MB file. Compressing the Asgard map (260MB, one of the biggest maps) takes around 17 secs on my comp, decompressing is around 4 secs. The good news is, you only gotta compress once, and decompress several times Now compare these 4 seconds to the several minutes it takes for players with a bad internet sub to download a new map

5. Not only redirect servers, but Steam subscriptions would also benefit from it, because the Steam client downloads the very same .kfm file, except from Valve's CDN instead of a random redirect server. But the mechanism in the background is the same. While Valve's CDN has more bandwidth than your random redirect server, that won't help with people on slow internet. The bottleneck is the client, not the server for the 99%.

6. It doesn't even need to be a breaking change. Say we introduce a new .kfmz extension. New clients would try to download that file first. Not there? Okay, fall back to .kfm. Old client will try .kfm in the first place, you're good to go, both new and old will work just fine. In this case, if we want to maintain compatibility with old clients, you can ignore point 2 of course, coz it will actually increase disk usage on the server (you gotta offer both the .kfm and .kfmz files for download), but that's the smaller problem here imo. Waiting for minutes till everyone downloads the maps is a major PITA compared to a few more gigs on the server. It totally kills the momentum.

7. For whatever reason, KF2 has this habit of crashing if the map download takes too long. Happened to me, also got it reported from other players. After a few attempts it will eventually succeed (I assume it resumes the download instead of restarting from scratch), but it's still a pain, especially when the server becomes full while you're busy struggling with your crashes.

8. Implementing LZMA, or a comparable compression seems relatively easy to implement to me. In fact, we also did this with a mobile benchmark app we're developing, I was responsible for the server side.

9. There's several royalty-free, open source implementations you could go for, LZMA, xz, even Google released some new algo some time ago IIRC.

It really is a win-win for everyone involved. So Tripwire, please consider this. Consider us who love custom maps Think of the children, stop wasting internet bandwidth What do you guys think?

 

[bviktor]

I just realized there's an "Ideas and Suggestions" category, if you think it's a more appropriate place for this discussion, could you please move it there? Thanks and sorry for my ignorance

 

[Slie]

Currently, my server's custom maps are 3.3G in size. But they're just 975M with LZMA compression. So one has to wonder, why doesn't KF2 support compression?

I speak for redirect servers for the most part here, because telling everyone to go to Steam workshop, search for x map, and click subscribe, and potentially restart the game is irritating compared to just having to do... nothing but wait until the map downloads with a properly set up redirect server. Yes, I know TW deprecated this functionality for whatever reason. But as you'll see, it's also beneficial for Steam map downloads.

A) Steam uses compression. I had a map that was ~800MB uncompressed but the steam transfer was only 166MB.
B) You can setup your server so clients automatically download from steam. There's no need to tell anyone to subscribe. Essentially, steam is giant redirect server with higher bandwidth and compression.

All your points are already accomplished via steam..

With that said. KF1 had compression and it would be nice for KF2 to have compression for certain instances with mismatches and the odd mod you don't want to host on workshop. I don't see this ever being a high priority though.

 

[forrestmark9]

UT3 already had a system for this called .uz3 which was a updated version of .uz2. Tripwire either removed the system, disabled it, or it was not included in the version of UE3 they licensed

 

[®omano]

I already asked long time ago for compression for redirect servers , but they didn't reply and I think it is because they want people to use Workshop, they do not want to support the old way of redirect servers, and surely don't want to add more bug with another system added in the game.

As said above steam already uses compression for Workshop.

I didn't understand what you're talking about with old clients and new clients, it doesn't make any sense to me we're all forced to play the same unique version of the game.

edit: actually I didn't open the old thread https://forums.tripwireinteractive.c...ct-compression and we actually had an answer https://forums.tripwireinteractive.c...26#post2275726

Workshop in its current implementation works as a global public redirect (that compresses files as well as much as they can be - maps are already compressed when they are made) and it is what we are supporting moving forward.

So short answer:

 

[bviktor]

Thanks for your replies, folks! I didn't know Steam downloads the maps compressed, that's pretty cool. OTOH I have some maps that cannot be found on Steam, only on Game Banana. Or the Cucumber map from which I removed the music, because it makes the map unplayable imo.

Is there a way to make workshop items download from workshop while the rest download from the specified redirect server? That'd be great, I could definitely settle with that.

 

[®omano]

You can use both, keep the workshop download manager first like the wiki explains.

 

[bviktor]

You can use both, keep the workshop download manager first like the wiki explains.

Thanks, will look it up!

 

[bviktor]

Finally I had time to actually check this out. Not trying to be rude, but workshop downloads seem to have been implemented the dumbest way possible. I imagined workshop downloads like the server just caches the current map, and that's it. I expected at least an option for that, so that my server doesn't have to store several gigs of maps all the time. Because servers usually have tolerable bandwidth, you know. Like 1Gbps. So the server will most likely be the first to get the map before any player actually does anything.

Now, with workshop downloads not only does the server download and save ALL listed maps to disk, it actually does that TWICE.

It downloads the map to KF2Server/Binaries/Win64/steamapps/workshop/content/232090, then copies the file to KF2Server/KFGame/Cache/0. Right? Why have one copy, when you can have two, right?

Now, if you stop the server and remove the KF2Server/KFGame/Cache folder, it copies the .kfm file there again. And here comes the part where it actually gets ridiculous: if you delete KF2Server/Binaries/Win64/steamapps/workshop/content/232090, it actually creates KF2Server/KFGame/Cache/1 right next to KF2Server/KFGame/Cache/0, downloads the map again, and makes another copy. So now you have the same friggin' map THRICE.

Man, I'm baffled. I actually loved the idea of saving disk space, but nope, it's the exact opposite. And I'd add some block storage to my London Vultr instance, except for the fact that block storage's only available for NY instances

Also, I still gotta list all workshop maps in KFGame.ini. On top of KFEngine.ini. So while using the redirect server you had to add them once, now you gotta add them twice. Again, it's not making things easier, but rather the opposite.

Finally, you gotta deal with deleted maps. Which the server and previous players will still have, but not newcomers. And version collisions as well.

While a redirect server just works.

I don't know, man. I'm not convinced.

I still believe that it'd be great to have maps compressed in the first place, instead of relying on compression in a totally different level (i.e. Steam).

ATM I have 300MB free space on a 25GB CentOS server thanks to the ever-growing KF2 updates (does a headless server really need that 12GB KF2Server/KFGame/BrewedPC/Packages folder to run btw? I smell a lazy dev here), but once I manage to grow my disk someway, I'll prolly revisit workshop anyway, coz disk space affects only me, while map download speeds slow everyone down...

 

[®omano]

The only good point you made is that it actually stores workshop maps twice, that's right. It is the same on your client if you didn't know there is the Steam cache and the My Game/ cache.
If you manually mess with files yeah it can create another copy of the file in a +1 folder in KFGame/cache/ so don't mess manually with files unless you do it properly (delete both cache for the item then you'll not have the map 'thrice').

As much as I would like the server to support redirect with compression, I think KF 2 workshop is great for servers.

 

[bviktor]

(Nevermind, I misunderstood your comment at first.)

 

[bviktor]

Alright, I put a good 12 hours of work into resolving this. First off, I replaced the current ext4 /home partition with a Btrfs one so now I have both compression and deduplication OOTB = the dumbness of Steam shouldn't be a problem anymore.

But that meant reinstalling and reconfiguring KF2. Again. So I just said, f*ck it, time to have some proper configuration management. Here it is:

https://github.com/bviktor/kf2-centos/blob/master/conf.sh

This is just a rough "beta", I'll properly integrate this into the install script and update the tutorial as well. Now I can have my own .ini files which contain ONLY my modifications and they're merged into the actual game ini files. So these are basically the overrides that should be changed from the defaults. No more manual backporting after every update.

There's also a maps.txt where you can list the maps you want to subscribe to. Like this:

715740524,KF-A-UnwelcomeGuest
644445723,KF-Antarctic-09b
1110673053,KF-ArcadeMode
747826742,KF-Asgard
643383080,KF-BikiniAtoll
...

The script automatically updates the KFGame and KFEngine ini files accordingly. Better yet, it automatically generates 2 separate map cycles both for the built-in maps and the workshop ones.

Finally, you can have cycles.txt where you can list any custom cycles you want to have, like for xmas I have one like this:

KF-KrampusLair,KF-Antarctic-09b,KF-BurningParis_CD,KF-ChillwindPlateau,KF-SantaWorkshopV4,KF-Santa_to_Xmas,KF-WinterFarm,KF-XmasHoldout,KF-YuletideTown,KF-finlandwinterland,KF-londonSnow,KF-xmasball

I think this will make things way easier for people. Can't wait to have it properly released for ya

 

[bviktor]

Alright, first off, the Btrfs guide:

Migrating your /home to Btrfs

The updated installer at the regular place:

Killing Floor 2 CentOS Installer

I decided to revise the old guide instead of publishing a new one, so that's also available at the same location:

Dedicated Killing Floor 2 Server on Linux

Whew.

 

[®omano]

I suggest you open a thread to have more visibility on the Btrfs tutorial maybe explaining the benefits of doing that and things like performance of KF 2 server regarding starting and map loading, things like that? I feel this is a little off topic and nobody will see it.

 

[bviktor]

Thanks, will try to find time for that eventually. Without actual numbers, I'd say it feels like server startup and map change takes around 1.5x more time. Nothing drastic, but noticeable. Naturally it doesn't affect in-game performance, matches are just as butter smooth as earlier.