Sooner or later you'll run out of memory

[Dave_Scream]

I start with this quote:
Originally Posted by poosh

All resources custom weapons use must be statically loaded into memory.

I rewrote Preload/Unload assets calls in ScrnBalance, so it now supports Refs instead of static links (e.g. MeshRef instead of Mesh etc.). But then every weapon's code must be rewritten too to use this. But if you do so, your weapon won't work without ScrnBalance anymore (its resources won't be loaded, if ScrnBalance isn't used).

Maybe I'll contact with Marco to make ServerPerks support refs of custom weapons by default.

and few shots of how it realized in SCRNBalance:
In PlayerController:

simulated function ClientWeaponSpawned(class<Weapon> WClass, Inventory Inv)
call PreloadAssets for spawned weapon - it run DynamicLoadObject and load meshref to Mesh. by default Mesh==none

simulated function ClientWeaponDestroyed(class<Weapon> WClass)
unloads Mesh by setting it to none;


I want to talk about memory.

community make big amount of weapons and mods. each weapon eats memory. On most modded servers game start to crash on some clients because of not enough virtual memory. + 32bit memory limit.

1. As I understand, the Idea of replacement Mesh to MeshRef based on the problem: in weapon, even if weapon is not spawned in game, all meshes, sounds, textures are loaded in memory, cause they are statically written like Mesh=Mesh'MyMesh'. So even if this weapon will never be spawned, it will eat memory on server and on clients like if it is was spawned.

1. is it rigth?

next.
Garbage collector cant unload them from memory, because it can unload only not referenced(unreferenced) items. But Weapon.default.Mesh and default.Sounds and other defaults always reference to it so it cant be unloaded

2. is it right?

next.
if it right, then why TWI remove weapon pickups on ground and say that it will help to save memory? pickup.defaults, and weapon.defaults continue to reference on resourses and eat memory.

3. Why they think it will help? or I missed somethink?


The idea to avoid this is make default.Mesh default.Sounds default.Textures to none, and Load them only when weapon actually spawned. Apply them to Instance, not default class.


Does it right:
1. we set default.mesh = none - it dont eat memory
2. weapon spawned, on spawned weapon we set Mesh=WeaponMesh (we set mesh on exact instance of weapon, not on default.mesh of it class) - it start to eat memory
3. when weapon destroyed, Garbage collector will look that loaded in memorty Mesh not referenced by any objects (default.mesh we set to none before, and weapon instance is destroyed) - so Garbage collector unload it automatically

will it work?


What you think about this mutator idea?

Spoiler!

- with config we specify dynamic array of "need-to-optimize-weapons"
- mutator look at each "need-to-optimize-weapon", and fill the struct for it:

var string WeaponClass
var string MeshRef
var string Skins
var string Sounds
var string anothersounds...
var string Pickup resources....
var string FireModes resources...

- so we have dynamic array with string representation of all weapon resouses, so we dont need to have it in defaults

- replicate this array to clients (yes, it is very big, but we replicate it initially once in game)

- on client and on server go through all this weapons list, and remove default meshes sounds skins of all this weapons - all this resourses will be unloaded from memory with garbage collector on client and server


here we have trouble when Weapon spawned, it will have no mesh and no skin and no sounds cause defaults are nulled
so:
- for weapons we can modify player controller and as poosh do use ClientWeaponSpawned ClientWeaponDestroyed

or another way not only for weapons:
- on server in CheckReplacement we hook when Weapon or WeaponPickup or other "need-to-optimize" object spawned, and on server fill VERY LIGHTWEIGHT array "InGameObjects" it will be replicated to clients.

- when client will receive new InGameObject from server, it will go througth our big array with string representation of all weapon resources. And will find this spawned ObjectClass in resources-refs-array. And will initialize this weapon/object by like this

Weapon.LinkMesh ( DynamicLoadObject ourarray.MeshRef ). And so on with skins sounds and other resources. the main thing here is that we apply it on instance, not default.class.


so. our weapon defaults are nulled. and if WeaponInstance destroyed - no references to that resources left, so it will be unloaded from memory with garbage collector.

to make it work need hooking and replicate to clients each spawned weapon or pickup object. because CheckReplacement not work on client side (as i know).

bad things:
- some lag because of replication delay. Weapon spawned but mesh and texture will be assigned only when client receive this InGameObject from server and initialize it.

well what you think can in work?

If this scheme can work...
Because of this lag I think this scheme can be used not for all weapons, but for weapons that not often used and eats memory....
AND it can be used everywhere for Sound resources, cause Sound lag is not so bad, it very rare when weapon spawned and start to fire simultaneoucly. Maybe few shots before InGameObject replicated, and SoundsRefs loaded.
----
UPD.
I found that
- Garbage collection starts only when mapchange.
- it can be started with console PlayerController.ConsoleCommand("OBJ GARBAGE")
- it freezes game for a short moment
http://wiki.beyondunreal.com/Legacy:Garbage_Collection
http://wiki.beyondunreal.com/Legacy:Destroying_Objects
http://wiki.beyondunreal.com/Legacy:Console_Commands/Stats_Console_Commands
-----
crosspost to epicgames the link

 

[Clowndoe]

I'm sorry, I can't make heads or tails of this...

If I had to guess what the subject is, I'd say it's part of their commitment to allow people with toasters to continue playing.

Next time: Isolate and simplify.

 

[Dave_Scream]

I'm sorry, I can't make heads or tails of this...

If I had to guess what the subject is, I'd say it's part of their commitment to allow people with toasters to continue playing.

Next time: Isolate and simplify.

community make big amount of weapons and mods. each weapon eats memory. On most modded servers game start to crash on some clients because of not enough virtual memory. + 32bit memory limit.

in this post Im trying to find the way to avoid static load of all objects in memory. Somehow Load weapon resources(sounds skins meshes) only when they needed and unload after.

 

[King Sumo]

I'm sorry, I can't make heads or tails of this...

If I had to guess what the subject is, I'd say it's part of their commitment to allow people with toasters to continue playing.

Next time: Isolate and simplify.

better move this thread to the coding forum to avoid such unnecessary comments.

 

[Guardiannknight]

better move this thread to the coding forum to avoid such unnecessary comments.

^^^^ Agree with King Sumo above ^^^^

I'm still learning coding stuff... but I understood with no problem what Dave Scream was saying.... It's a major problem.... Because on both my servers, with all the zeds and mods i'm running... along with all the modded weapons... i've hit the memory limit on the server... I'm right on the border line... And i know that is why myself and other crash sometimes.... Even some maps won't allow me nor my clients to join because of the 2GB memory limit...(map loads, but won't let anyone connect into the game, meaning server is running and hasn't crashed, just doesn't have the memory available to allow players to join).

What Dave was talking about would help and do wonders. Also, if TWI would compile the exe files to allow the file to access over 2GB, lets say 3GB, that would help... that would give another 1GB for the game... The reason TWI will not I'm sure, is to keep the game able to run on older gaming system, that still run with 2GB of RAM. They will probably hold it at 2GB for this main reason. So the game will be playable by a huge selection of gamers, especially those running lower powering systems. I did a lot of searching and researching to see if it would be possible to decompile the ucc.exe or killingfloor.exe, but.. no such luck... and it's not possible. It must be done by the maker of the game. Which makes perfectly good sense after reading up on it and researching it..

But, since TWI probably won't be upping the 2GB limit up to 3GB, any time soon... What Dave is suggesting is a great idea... Especially if it could be a mod that does it... That would save a lot of headache and trouble from modders.. having go into each weapon (and hopefully this would work for custom zeds also or mod packages) and change the code from a static to a ref.

Hopefully this will be a mod that will make it. This would help server operators big time... allowing even more stuff to be run on the servers

 

[Massacrer]

The specs of my server box has more than 8 GB RAM to spare, and I hope that I can utilize this, or something.

 

[Guardiannknight]

The specs of my server box has more than 8 GB RAM to spare, and I hope that I can utilize this, or something.

Yeah, i have 16GB on the system i have my servers running. Just seems wasteful when there is so much RAM there to use Would be awesome to add a few more things back to my servers... or at least the few things I have had to remove to make room for new weapons... :\

 

[Dave_Scream]

The specs of my server box has more than 8 GB RAM to spare, and I hope that I can utilize this, or something.

my team members have server and last time many crashes appear for their players. Clients crashes with error message "out of virtual memory". No matter how much memory they have. One of them have 16gb of ram and windows 7 - and he got this crash on some maps every 10 seconds!

as they admitted
these crashes often gone if player move from Windows 7 to Windows XP (I have no ideas why)

so no matter how much ram you have, it is because some kind of hardcoded limit.

the problems:
- KillingFloor is 32bit, so game can address only 4gb max in theory
- on practice, as Guardiannknight sayed, TWI compile game that way so it can use only 2gb max.

On 1039 or so they recompiled game dll's (Engine.dll and Core.dll). Dont know what they do exactly but I found some things that they change Object allocate memory routine. maybe they tried to solve problem. or maybe they added some kind of asyncronous garbage collection during memory allocation routines.

 

[forrestmark9]

my team members have server and last time many crashes appear for their players. Clients crashes with error message "out of virtual memory". No matter how much memory they have. One of them have 16gb of ram and windows 7 - and he got this crash on some maps every 10 seconds!

as they admitted
these crashes gone if player move from Windows 7 to Windos XP.

so no matter how much ram you have, it is because some kind of hardcoded limit.

the problems:
- KillingFloor is 32bit, so game can address only 4gb max in theory
- on practice, as Guardiannknight sayed, TWI compile game that way so it can use only 2gb max.

On 1039 or so they recompiled game dll's (Engine.dll and Core.dll). Dont know what they do exactly but I found some things that they change Object allocate memory routine. maybe they tried to solve problem. or some kind of asyncronous garbage collection during memory allocate routines...

Killing Floor does not have the Large Address Aware flag set to true so it can only use 2GB of ram, with Large Address Aware crashes become non existent and increases performance. This is why it is recommended for use on any custom servers cause it will let KF use up to 4GB of ram and not the 32-bit 2GB limit

 

[Guardiannknight]

Killing Floor does not have the Large Address Aware flag set to true so it can only use 2GB of ram, with Large Address Aware crashes become non existent and increases performance. This is why it is recommended for use on any custom servers cause it will let KF use up to 4GB of ram and not the 32-bit 2GB limit

I have made the ucc.exe on the server and the killingfloor.exe file on my gaming system large address aware, but sadly that doesn't help much if any. Whatever is coded into the .exe when it's compiled is what we are sadly stuck with However what Dave is coming up with, would help out a lot

 

[Massacrer]

Killing Floor does not have the Large Address Aware flag set to true so it can only use 2GB of ram, with Large Address Aware crashes become non existent and increases performance. This is why it is recommended for use on any custom servers cause it will let KF use up to 4GB of ram and not the 32-bit 2GB limit

Can you do that on a Linux box?

 

[forrestmark9]

Can you do that on a Linux box?

I have no idea, linux files are different then windows files and may be lacking the flag or it is called something else

 

[Dave_Scream]

allright I got what I wanted to get by posting on epic forums:.

their local guru Wormbo answered my crosspost:

A garbage collector run in pre-UE3 engines is a very expensive operation. It can take up to several seconds to complete and it's a synchronous operation, i.e. it will stop everything until it completes.
Messing with default values is a tricky thing. Defaults will not be explicitly reset on map change, that only happens as a result of package unload and later reload, which is not guaranteed to happen for stock content.

To be honest, I don't think content being loaded without actually being used is such a big problem. Map content and also the actor data itself probably takes up a lot more than you could save by not hard-linking weapon resources. Also note that packets can be loaded partially, i.e. if a class isn't used, it will not be initialized and thus its references will not be resolved. BTW: The server might still be interested in sounds, textures and meshes for their meta data. Sounds have a playback duration, textures have dimensions, meshes may have bones, etc. and the server might need to check any of them.
Long story short: It's more trouble to not load these resources than the reduced memory consumption is worth it.

In my opinion, Wormbo more powerful than even Marco, so I think he knows what he say. But maybe he dont know about how much differences between usual unreal and Killing Floor? 120 weapons and new skins is much more than usual unreal game has. I think need to start testings.

 

[poosh]

You're overstating

 

[Dave_Scream]

You're overstating

 

[Z-MAN]

Server Perks thread
rewrite

I have successfully loaded until the summer was swap memory.
The current version is broken.
Swap is not working properly and memory crush.

The report should be sent to TWI.
They also ignore half a year.

I think so too.
I tested changed to Ref 500 points in weapon mutators.
Mesh and Sound and Texture.
But server failed load.
The current version is meaningless.

No move swap bug problem since the summer.

step 1.
TWI fix bug.
step2.
ref coding.

 

[poosh]

no Im talking not about classes, Im talking about textures sounds and mesh resources to be unloaded.

Sorry, then I misunderstood you. I thought you're suggesting to unload pickup classes to save some memory.

I've done a stress-test just now. I put every pickup from ServerPerks shop on every navigation point on the map. Here's the code (need to be put in PlayerController subclass):

exec function SpamPickups()
{
    local NavigationPoint N;
    local int i, c;
    local ClientPerkRepLink L;
    
    L = SRStatsBase(SteamStatsAndAchievements).Rep;
    for( N=Level.NavigationPointList; N!=None; N=N.NextNavigationPoint ) {
        for ( i = 0; i < L.ShopInventory.Length; ++i ) {
            spawn(L.ShopInventory[i].PC, N, '', N.Location);
            c++;
        }
    }
    ConsoleMessage("Spawned pickup count = " $ string(c));
}

After compile I launched a big map (= more nav. points) and typed in console: "SpamPickups". It took about a minute to execute the code. In result I got map full with pickups: 33756 pickup objects were spawned. Of course, game became unplayable (<1 FPS), but it still ran without crashes or hitting a memory limit.

Then I wrote the following clean-up code:

exec function FreePickups()
{
    local NavigationPoint N;
    local Pickup P;
    
    for( N=Level.NavigationPointList; N!=None; N=N.NextNavigationPoint ) {
        foreach N.ChildActors(class'Pickup', P) {
            P.Destroy();
        }
    }
}

After executing "FreePickups" game started to run smooth again, despite the fact that there were 30k+ extra objects on it a moment ago.
I hope this is a good example how many objects engine can handle.

I think it is hell for TWI to remake Killing Floor to Unreal Engine 3. They have RedOrchestra too. and it must be ported too.
so I think that we are hardly attached for a long time to Unreal Engine 2.5

I'm not saying they should do it for free. They can rename it to Killing Floor HD and sell it as a new game to earn some dosh (giving current KF owners a discount, of course). However, if TWI will keep their current state of balance, nobody will buy it.

So I talking about some universal method to optimize resources. Hand-rewrite all weapons code is bad idea. Need some universal method.
What you think about my mutator idea?

Why do you think manual rewriting weapon code is so bad? I made changes in Russian weapon pack used on my servers in less than an hour. If every weapon author (+ some volunteers from the community) will do the same, all custom weapons can be rewritten in a couple of evenings.
Do you think mutator you suggested can be written in less time? Idea is good, but imo it's too complicated to do.

 

[Dave_Scream]

Spoiler!

Sorry, then I misunderstood you. I thought you're suggesting to unload pickup classes to save some memory.

I've done a stress-test just now. I put every pickup from ServerPerks shop on every navigation point on the map. Here's the code (need to be put in PlayerController subclass):

exec function SpamPickups()
{
    local NavigationPoint N;
    local int i, c;
    local ClientPerkRepLink L;
    
    L = SRStatsBase(SteamStatsAndAchievements).Rep;
    for( N=Level.NavigationPointList; N!=None; N=N.NextNavigationPoint ) {
        for ( i = 0; i < L.ShopInventory.Length; ++i ) {
            spawn(L.ShopInventory[i].PC, N, '', N.Location);
            c++;
        }
    }
    ConsoleMessage("Spawned pickup count = " $ string(c));
}

After compile I launched a big map (= more nav. points) and typed in console: "SpamPickups". It took about a minute to execute the code. In result I got map full with pickups: 33756 pickup objects were spawned. Of course, game became unplayable (<1 FPS), but it still ran without crashes or hitting a memory limit.

Then I wrote the following clean-up code:

exec function FreePickups()
{
    local NavigationPoint N;
    local Pickup P;
    
    for( N=Level.NavigationPointList; N!=None; N=N.NextNavigationPoint ) {
        foreach N.ChildActors(class'Pickup', P) {
            P.Destroy();
        }
    }
}

After executing "FreePickups" game started to run smooth again, despite the fact that there were 30k+ extra objects on it a moment ago.

I hope this is a good example how many objects engine can handle.

very interest testings. that what I waited for.
but I have some explanations why your game not crashed with no memory error. Benjamin talked about it:

Resources (textures, meshes) are shared, so having 100 of the same pickup will use only a little more memory than 1. Also, if I'm not mistaken the textures are the same for held weapons and floor weapons, so by removing the common pickups you aren't really saving much (or any) memory.

probably game dont crashed because resources are really shared and Benjamin was right.

Marco said that TWI already launch garbage collection at each wave end (game freeze).

To more understand how garbage collector work need write some MemoryEATActor with
var array<Material> Skins;
var array<Sound> Sounds;
var array<Mesh> Meshes;

and two functions that will load and unload to this arrays the whole Killing Floor resourses from all Monsters and Weapons. It will be really hard weight actor.

And because of size it will be noticeable with MemStat when it will be spawning/destroying garbage collecting.
-----
another moment.
When I load some simple map with only one room "KF-Die". and Console "Obj Linkers" I have a list of resources.

Many of resources have Lazy=1 or more so, they are Lazy load and will load only when they need.
I exported the list to Excel and look only Lazy=0 resources. in theory all they are in memory already.

so I found some interest "garbage" from other MAPS. I start map KF-Die, but found resources:
../Textures/Krasnyi_Cubemaps.utx - another map
../Textures/KillingFloorManorTextures.utx - another map
../Animations/HillbillyHorror_anim.ukx - another map!
../Sounds/AbusementParkSND.uax - another map
../Animations/KF_RobotDance.ukx - another map
../Textures/MountainPass_T.utx - another map

http://dl.dropbox.com/u/3755765/2013-01-05_204947.jpg

they are all Lazy=0 so in memory. why MountainPass (15mb) in memory when i run completely different map is a question. and maybe problem with crashes is here?

 

[poosh]

Of course, textures are shared between object instances. My example shows that objects weight next to nothing without additional multimedia info load.
Weapons 1-st and 3-rd models are different, cuz no need in hi-res model for first person.

Pickups are auto-deleted at wave start, because weapon data is deleted. Deleting pickup ensures that there will be no need in spawning a weapon. But there is no need to delete all pickups. For example, if player has xbow in inventory, then there is no need to delete another xbow pickup on the ground, because xbow's data must be loaded anyway. Also there could be a weapon cap, e.g. max 32 weapons could be simultaneously loaded into the memory. If limit is reached, block other weapons from buying/picking and delete their pickup from the ground. But it's too complicated for TWI.
Beside that, I really like pickup disappearing. It brings weapon's max ammo value into the balance. Max ammo didn't meant anything before, because player could stockpile whatever amount of weapons he was needed.

Maps reference another maps, because they use some textures/objects from them. I may be wrong, but IIRC only referenced objects are loading into the memory, not the entire package.

 

[Dave_Scream]

thank you poosh for detailed answer.
Wormbo says that you're right. And written another detailed answer if somewho interested

so Wormbo Marco and poosh said that universal resource optimizer is bad idea.

"it's more of a trouble than it's worth it"

"Listen to poosh and go with the "rewrite all weapons" approach. That will be much less of a headache than the mutator you suggested."

---
please explain what need to actually do and what community/volonteers actually must do?

need some detailed description to avoid overjob and mistakes.
As I understand:

Spoiler!

- in each class (weapon, pickup, attachment) need to specify static functions PreloadAssets and UnloadAssets. and make initialization in this functions. defaults are nulled.

questions:
- who will call this functions? in SCRNBalance modifyed playercontroller call it. As I understand TWI not realized standart call to this functions

- if we do this modifications on all weapons and make it default in future works, then it will work only with modified PlayerController?

- so to make it more standartized can we move initializations to PreBeginPlay / Destroyed ?
in PreBeginPlay load to defaults and call LinkMesh and others. In Destroyed LinkDefaultMesh(none), defaults - none.
Why you dont make so? and use modified playercontroller instead?