Request for TWI: Streaming

[-=THOR=-]

------------------
INTRODUCTION
------------------
First, let me introduce streaming, for those who don't know what it is. Streaming allows to separate a level into multiple sublevels. Those sublevels can be reused (streamed) into other levels.

For example, with TWI maps, we have:
ART-Spartanovka (terrain, static meshes, BSP)
SND-Spartanovka (sounds)
FX-Spartanovka (post-processing effects, particle effects)

Those levels are then streamed into top level files. There are two at the moment:
TE-Spartanovka
CD-Spartanovka

TE-Spartanovka, by itself, does not contain the terrain, does not contain sounds, and no post-processing effects. The sublevels ART, SND and FX are streamed into TE-Spartanovka and CD-Spartanovka. Basically, TE and CD only contain the gametype information. When you modify a sublevel, opening TE or CD will display the change, because the modified sublevel is streamed into both of the top levels. Note the the top level file does not embed the streamed sublevels, they are dynamically combined.

If you played CTF-Spartanovka, you probably noticed that the download was only 16 Mb, this is because I used streaming: ART SND and FX are streamed into CTF-Spartanovka, and the latter contains only the gametype-specific data. Since you already have the stock sublevels, you didn't have to download the ART sublevel, and this saved you a few Mbs (ART is generally causing the most important increase in file size)

-----------
REQUEST
-----------
Streaming is part of the engine, and it works. What is the problem then? The redirect does not support streamed sublevels. So if for example, you download TE-MyMap, which has a sublevel ART-MyMap, the redirect will be unable to send ART-MyMap. So my request is simple, please fix the redirect to support streamed sublevels.

EDIT:

Originally Posted by -=THOR=-

Actually, the download *works*, but when the map starts to load, I get an error message: "Environment_Mats" not found (download was already attempted). Could it be that the sublevel doesn't recognize the stock packages? Anyway, whatever the issue is, unfortunately it prevents Streaming from being usable by modders. Try it out, with this test map, TE-Aparments2, which streams a custom sublevel (ART-Apartments2) instead of the stock one (ART-Apartments):
http://bit.ly/11WqjoV

---------------
MOTIVATION
---------------
Atm, I find it very hard to convince mappers to implement custom game types when I explain the impact it will have on their map development process.

Other than that, with streaming, mappers could do a much better job, faster.

Also, it would reduce download sizes (if you play TE-SomeMap and download ART-SomeMap, you would not need to download ART-SomeMap again when playing CD-SomeMap, same story for different map versions [typically combined arms vs infantry only]).

---------
IMPACT
---------
What happens if streaming is not supported by the redirect? It means that mappers have to provide one single file that contains everything that would be in the sublevels (ART + SND + FX + TE/CD/FF/CTF).

When there are multiple gametypes, a mapper has two options:
a) Combine the logic of all gametypes in one level. When releasing, duplicate the file, rename it, and use booleans in Kismet to enable a specific gametype.
Result:
-The file is huge, and will require to download everything (including the ART, inside the single file), as many times as you download different gametypes of the same map. I don't need to explain the negative effect of large downloads on servers.
-The file is a mess, very difficult to understand.
-It is very difficult to diagnose errors, which results in a longer development period.
-Changes are painful, and may cause new errors.
-If a custom gametype is to be used, it means that the 'single' file will contain the Mod (along with the TE/CD/FF stuff for example). As a result, the TE file, for instance, would contain the Mod logic, and thus require a server to install the Mod although he plays the TE version.
-Fixing a simple bug in Kismet requires that everyone re-download the big file, instead of only the top-level file, which would be much smaller.

b) Create multiple versions of the same map, and implement specific gametype logic inside each of them.
-The file is huge, and will require to download everything (including the ART, inside the single file), multiple times.
-Any fix to the terrain, static meshes needs to be replicated in the other files. <= BIG NO NO
-A user that download CD-SomeMap will get no benefit when downloading TE-SomeMap, because they redownload the same art, embedded into the file twice.
-Fixing a simple bug in Kismet requires that everyone re-download the big file, instead of only the top-level file, which would be much smaller.

------------------
USAGE
------------------
How would mapper use streaming? Well, they could have one sublevel, which would contain FX SND and ART stuff, and one top level file per gametype. The top level would contain Kismet, volumes, and gametype-specific actors. There is no need for multiple sublevels in their case, one would be enough.

[danh]

I support this.

[Spaehling]

Originally Posted by danh

I support this.

yeah me too. would be really great for CTF-Climbup and CTF-Stalag 13!

[danh]

By doing this, you can essentially have a sublevel file that contains your static mesh,terrain and sound. Then you can have your gametype levels that contain objectives, spawns, and kismet. These gametype levels will probably be between 10 to 50 mb in size. This also means that you can make frequent balance changes to the game type level more often and be able to tweak much better.

With the current system now, we put everything in one level and it amounts to 150 to 200mb. Any change we want to do means the player has to download another 200mb even if the change is as small as adding another spawn.

With the proposed idea from Thor, we are separating the game from the art. The game part of the map is like I said only about 10 to 50 mb depending on the map. That means better updates and more frequent game updates while keeping the art stable. Then in a month or two there can be a total update. This is a much better model than what we currently have.

[-=}WoLvErInE{=-]

I Approve This Message!

[Catalavos]

+1

[Cat_in_da_Hat]

Anything that reduces the map download for people without losing map quality gets my tip of hat.

[=GG= Mr Moe]

Interesting. If this is the case, I totally agree with Thor.

[Yoshiro]

Are you sure this isn't working? Sub levels will need to be renamed .upk on redirect (like always).

So ART-Maphere.roe would become ART-Maphere.upk on the redirect.

[danh]

Okay I will try it out.

[-=THOR=-]

Originally Posted by Yoshiro

Are you sure this isn't working? Sub levels will need to be renamed .upk on redirect (like always).

So ART-Maphere.roe would become ART-Maphere.upk on the redirect.

Unless it has been recently added, last time I tried, it didn't work, and that is also what you said earlier:

Originally Posted by Yoshiro

Engine doesn't support cooked Sublevels on redirect, just .upk packages. We currently have no plans to support it (or know what it would take to make it happen).

I'll try it out now just to make sure.

[-=THOR=-]

I just tried. Actually, the download *works*, but when the map starts to load, I get an error message: "Environment_Mats" not found (download was already attempted). Could it be that the sublevel doesn't recognize the stock packages? Anyway, whatever the issue is, unfortunately it prevents Streaming from being usable by modders. Try it out, with this test map, TE-Aparments2, which streams a custom sublevel (ART-Apartments2) instead of the stock one (ART-Apartments):
http://bit.ly/11WqjoV

[Twrecks]

Streaming would be nice, as this would also facilitate the use of props packages. Making it easy for server admins is the trick. Yakovlevo initially had a props package, which IMO was the right thing to do, however caused issues with redirects and packages not properly loading. Eventually I placed all assets into the map file, making it bloated and the editor unhappy about being over 300MB. Seperating the sound files (cues, wavs) out did not work at all and had to be in the map file from day 1, same with terrain related items. Streaming allows more efficient workflow and better in game optimization. Unfortunately for custom content, simply doesn't work at this time to my knowledge.

True one could zip/rar the map folder with all the supporting packages and have players unzip/unrar then in the correct folder. This is what the workshop no doubt was intended to be the KISS tool to make this happen, but it has a 200mb limit and no one uses collections to distribute maps cause no one understands it, least of all players. It still doesn't address server issues with players joining that don't have the required files. Maybe there is a solution that I am unaware about?

Guess killing redirects and making maps (map packs) generally available (TWI web page/Workshop), with detailed instructions on how to install them would allow for multiple packages. This would unfortunately reduce the current player population, as from what I gather, players are more complacent and less likely to do anything on their own other than join a server. Then there is of course map updates, which breaks compatability with local files of previous versions. Allowing redirects point to the Workshop files would be the correct, that way everyone updates and no players are left swinging in the breeze.

Uncapping the 200mb limit on the workshop would be the first thing needed. Back in the Unreal1 days a map over 2 mbs was considered huge, and by 56k modem standards it was. But that was 20 years ago, and I wouldn't be suprised if maps for the Unreal engine 4 don't hit 2 gb. Higher poly meshes, larger texture files and more and more detail seem to be the trend. The only way for modders or any content creators to handle the influx is to allow Streaming beyond stock.

So yea +1 OP

[Tummel]

Dis thread....is relevant to my interests!

[Major_Day]

Originally Posted by Yoshiro

Are you sure this isn't working? Sub levels will need to be renamed .upk on redirect (like always).

So ART-Maphere.roe would become ART-Maphere.upk on the redirect.

Yoshiro is correct,any file WILL download from the redirect providing it is a .upk file,problem is that the very same file has a different extension name in the server package.

[-=THOR=-]

Originally Posted by Major_Day

Yoshiro is correct,any file WILL download from the redirect providing it is a .upk file,problem is that the very same file has a different extension name in the server package.

Streamed packages will download, but won't load (at least not on the first attempt), because it cannot find stock packages.