New Orbiter SVN commit (r.71, Oct 14 2017)

[johan]

Texture files

What are the chances there's some kind soul living nearby (Cape Town, South Africa) who have already downloaded the hi-res textures for the beta?

I can get them over time. But it will take a substantial amount of effort and time to do so, because they're so big and I don't have an uncapped connection.

 

[Pipcard]

Would it be possible to have high-resolution clouds like this KSP mod?

 

[kuddel]

@johan: Can't you walk into an internet-cafe with better bandwidth?
You sure have to virus-check the USB stick extensively after pluggin' it into those untrusted computers there (if you are allowed to do so anyway), but that might give you a chance.

A university is always a good point to ask, too. I'm sure they will allow you when you explain them what you need to download
--- it's rocket science, right?

 

[johan]

@johan: Can't you walk into an internet-cafe with better bandwidth?
You sure have to virus-check the USB stick extensively after pluggin' it into those untrusted computers there (if you are allowed to do so anyway), but that might give you a chance.

A university is always a good point to ask, too. I'm sure they will allow you when you explain them what you need to download
--- it's rocket science, right?

I like the university idea! Unfortunately internet cafe's in South Africa are impractical due to high cost (per minute) and low download speeds. But the university is a good idea. Thanks!

 

[martins]

New Orbiter Beta Released (r.56, Jun 30 2016)

Bit of a gap since last beta - sorry for that. Anyway, here is a new one:

Change log:

1. TileLoader: now loads multiple tiles per mutex lock to better utilise the tile loader thread. Should result in faster scene buildup.
2. Bug fix: made GraphicsClient::TexturePath threadsafe (caused visual artefacts in planet surface rendering of D3D7 client)
3. TileManager2: added support for loading tiles from compressed archive files
4. API: added oapiDeflate and oapiInflate
5. texpack: utility for packing the directory tree contents of a given layer into a compressed archive file
6. Subsystem: added clbkConsumeBufferedKey and clbkConsumeDirectKey methods to allow subsystems to process their own key events. Updated DeltaGlider code accordingly.
7. Scenarios: Fixed DeltaGlider/DG and DG-S scenario (moved the two vessels out of the woods)
8. Reduced default ambient level from 5 to 2

New OVP commit to go with this beta: r.51.


The main new feature in this beta is added support for loading planetary tiles from a compressed archive. This was the last component I wanted to implement before a release.

I experimented with various options here, including an interface to a RAR decompressor kindly provided and tested by Doug. In the end in the interest of decompression performance I decided on a more homespun approach. Instead of using a standard multi-file compression format, I am compressing the input files individually (using zlib) and concatenate the result into an achive file after adding a table of contents for the tree structure.

The TOC is essentially a linked list similar to a FAT allocation table, except it's a quadtree list instead of a linear one. The advantage is fast searching through the tree (instead of having to go through a hashtable for 100,000's of file entries), and the TOC is also fairly compact to be kept in memory.

The disadvantage of this approach is the less efficient compression since each file is compressed individually, instead of the input stream as a whole. But I think the tradeoff is worth it.

I will update the texture download packages soon to provide these archive files instead of the individual tile directory trees, but for now you can create the archive files yourself (if you have already installed the planet texture packs for the previous betas):

I have included Utils\texpack.exe which takes the current directory tree for a layer in a planet's texture directory, and converts it into an achive file. Run "texpack -h" for a help page.

For example, the syntax for compressing Earth's surface layer is

cd utils
texpack ..\Textures\Earth Surf

This will produce the compressed archive in ..\Textures\Earth\Archive\Surf.tree, which is the correct location for orbiter to pick it up.

Do the same for the other layers, for all planets supporting the new tile format. Be warned that the larger trees (Earth, Moon, Mars) can take a long time (several hours) to archive.

Note that it is not necessary to generate the archives to use this beta. The old format of individual files is still supported.

You can set Orbiter's behaviour in searching for tiles under Extra | Visualisation parameters | Planet rendering options | Tile sources.

• Load from tile cache only: Ignore the archives, and load from individual files as before
• Load from compressed archive only: Ignore the individual files, and load only from the archive files
• Try cache first, then archive: what it says. This option allows to subsitute individual tiles without the need of repacking the archive, but comes with a performance penalty since Orbiter may have to search both the cache and the archive.

Note that loading from archives only works for the inline client and the D3D7 client at the moment. (The D3D7 client has a similar option in its configuration page). But the D3D9 client will need Jarmo to add support before the archives can be loaded.

The D3D7 code in the new OVP commit demonstrates how to add support.

I would be interested in users' performance comparisons between the three loading options. Do you get similar loading times from the archive as from the individual files? Any other observations/problems with this feature?

 

[Enjo]

The main new feature in this beta is added support for loading planetary tiles from a compressed archive. This was the last component I wanted to implement before a release.

An exemplary project management, since the elevation data is the main feature!

Instead of using a standard multi-file compression format, I am compressing the input files individually (using zlib) and concatenate the result into an achive file after adding a table of contents for the tree structure.
[...]
The disadvantage of this approach is the less efficient compression since each file is compressed individually, instead of the input stream as a whole. But I think the tradeoff is worth it.

I think so too, but perhaps you could try to compress the resulting archive with fastest compression to leverage the common parts of the individual files, and, because it's the fastest compression, it wouldn't take that much overhead. But it might still not be worth it, if the difference between the sum of sizes of individual compressed files and whole set in one archive is small. Have you measured it maybe?

I would be interested in users' performance comparisons between the three loading options. Do you get similar loading times from the archive as from the individual files? Any other observations/problems with this feature?

I will have a look.

A note to other users here: because the textures loaded from your disk are cached, a truly fair test can only be performed if you start your simulation from a different (random?) location each time. I propose not testing it on Earth, because of its vast oceans, not having any height data, but rather on the Moon or Mars.
[EDIT] I think I have a good idea how I could test it myself: by using the Topographic Map MFD, which loads (too) large amounts of data, when in orbit, since it automatically scales as the altitude increases. Also, the loaded data will be quite different each time, as a body revolves underneath a polar orbit.

Martins: thanks for taking into account all of the ideas presented by us in this thread.

 

[jroly]

texpack ..\Textures\Earth Cloud
texpack ..\Textures\Earth Elev
texpack ..\Textures\Earth Elev_mod
texpack ..\Textures\Earth Mask
texpack ..\Textures\Earth Surf
texpack ..\Textures\Mars Elev
texpack ..\Textures\Mars Surf
texpack ..\Textures\Moon Surf
texpack ..\Textures\Moon Elev

I put it in a bat file and is cuirrently processing. The compression will be good, not so much because of the small file sizes, but simply copying/moving Orbiter directory took hours because of all the numerous files.

 

[Urwumpe]

Still compressing Earth here, I hope I get it done before the Windows 10 upgrade :rofl:

 

[jroly]

Earth is still processing here too, been 3 hours. Elev has finished and it was 24.6GB now it is 10GB. I might have room on the SD to put Orbiter Beta on now.

 

[Urwumpe]

Surf went from 18.9 to 14.3 GB over night here. Not a big change, but to torture the English people with a bad translation of a German proverb: Small animals also produce manure.

According to the modification date, it needed about 4 hours to process.

 

[jroly]

Yes I noticed the dds files in Surf do not compress well that is pretty normal for images. It took 8 hours for Earth to compress, Mars is done, the Moon is now processing.

 

[Enjo]

Surf went from 18.9 to 14.3 GB over night here. Not a big change (...)

But now you don't need to keep the original compressed textures (not to have to re-download) and the decompressed ones in the same time.

 

[Urwumpe]

But now you don't need to keep the original compressed textures (not to have to re-download) and the decompressed ones in the same time.

I know, I am already deleting it, since the original files are already on the external HDD. Still, I hope the next files are better compressed than this worst case.

 

[jroly]

It took just under 13 hours to process.

Earth 63.2GB -> 26.2GB
Moon 26.8GB -> 14.4GB
Mars 10.4GB -> 6.20GB

Total 100.4GB -> 46.8GB

Now time to press the del key

 

[martins]

Before you delete the uncompressed directory trees, could you give it a few tests regarding performance? In fact, if your hard disk space permits, it may be good to hang on to the original files for a while, just in case I need to make changes to the compressed format.

 

[Urwumpe]

Does it even make sense to compress "Elev_mod"? Should be pretty disturbing for add-ons, am I right?

---------- Post added at 06:48 PM ---------- Previous post was at 03:12 PM ----------

What I see the whole day long.... "adding node ..\Textures\"

 

[martins]

What I see the whole day long.... "adding node ..\Textures\"

Every node counts ...

In fact, after having spent weeks on hand-painting elevation tiles around KSC, amounting to a pathetic handful of "nodes", I am beginning to appreciate just how much information is contained in those textures.

 

[Urwumpe]

Every node counts ...

In fact, after having spent weeks on hand-painting elevation tiles around KSC, amounting to a pathetic handful of "nodes", I am beginning to appreciate just how much information is contained in those textures.

What does the deflate-phase do?

 

[martins]

The "adding node" phase only builds up the TOC, while the "deflating" phase does the actual work of compressing the input file and appending it to the archive.

To be honest, it's not the most efficient way of building the archive, but I just wanted to have something working. Eventually I might come up with something more efficient (and add additional features such as extract and merge), but it wasn't a priority since most users won't have to do it, if the textures are distributed by archive anyway.

 

[Urwumpe]

The "adding node" phase only builds up the TOC, while the "deflating" phase does the actual work of compressing the input file and appending it to the archive.

To be honest, it's not the most efficient way of building the archive, but I just wanted to have something working. Eventually I might come up with something more efficient (and add additional features such as extract and merge), but it wasn't a priority since most users won't have to do it, if the textures are distributed by archive anyway.

Oh. I always felt like adding node was the most busy phase, because my HDD is making noise, while the deflate phase is almost happening in idle. Was already wondering, what the hell it does so silently.