This is what happens when you use mip maps with a transparent background on the texture instead of a solid colour. You'll need to reopen the .dds file in Photoshop and fill the transparent or black background with another colour. It is best to use a colour that matches the original mesh closely because this outer texture does bleed into the texture as it gets mip mapped.
Basically foliage in RO (and most engines) will always have unavoidable problems that must each be weighed before your choice of texture options is made. These options are;
1. Leave off mip maps altogether
Good - No colour bleed. Texture will not disappear. Often
must be used on thin railings (see Odessa) or else the texture will disappear or have the outer texture completely bleed in and fill the gaps causing the railing to look like a solid sheet of metal without any open spaces at longer ranges.
Bad - The texture will not be optimized and will 'shimmer' or pixelate at longer ranges.
2. Mip maps on with transparent or black background.
Good - No colour bleed. Optimized.
Bad - Texture will disappear at longer ranges, although it may not be noticeable for larger objects in the texture such as a thick bush or something similar.
3. Mip maps on with coloured background.
Good - Texture won't disappear. Optimized.
Bad - Texture will completely fill up with the outer colour as it bleeds in. If this is used on a railing texture, for example, at a certain mip level (or distance from the texture) the gaps between the railing will be filled up and become solid.
Remember, for solid textures without transparency, you want to leave them at DXT1 no alpha when saving out of Photoshop and always check the mip maps button when importing to UnrealEd. If you do not tick off the mip maps box your texture will not be optimized and will 'shimmer' / look ugly. We see far away objects blurred in real life which is basically what mip mapping does. It lowers the resolution on the texture based on your distance from it, so always use it for non-transparent objects!
Here's a basic list of the DXT types and their uses. Much more info can be found online. The Nvidia Plug-in for Photoshop that allows you to save textures as .dds can be found
here.
DXT 1 no alpha - Use this for all solid textures that do not have any transparency.
DXT 1 1-bit alpha - This means that each pixel will either be 100% transparent or 100% opaque. Most textures with basic transparency needs will use this.
DXT 3 - Used for textures that need pixels to be partially transparent. Useful for water, glass, etc. or any texture of that nature. I have also used DXT 3 when DXT 1 with 1-bit alpha has caused masking issues (objects behind the texture being rendered through the object in front or similar problems).
DXT 5 - This is used when DXT 3 causes masking issues and is generally left to objects that the player will see through up close (windows, scopes, etc).
Your best bet is to find a similar texture in the editor and see if it is DXT 1/2/3/5 and then do the same.
If this is confusing (and it probably is
) I could explain further or do a mini-tutorial sometime, just let me know.
As an additional note, has anyone tried to set custom mip map levels in Photoshop with the Nvidia .dds plug-in? They don't seem to work. It would be nice to define the rate at which the texture gets mip mapped.