Orbiter-Forum  

Go Back   Orbiter-Forum > Orbiter Space Flight Simulator > Orbiter Beta
Register Blogs Orbinauts List Social Groups FAQ Projects Mark Forums Read

Orbiter Beta Topics related to Beta releases of Orbiter and Orbiter development.

Closed Thread
 
Thread Tools
Old 08-05-2016, 10:23 AM   #46
Face
Beta Tester
 
Face's Avatar

Default

Quote:
Originally Posted by martins View Post
 I think you are pretty much correct on all points (except byte offs 8, which is a 4-byte bit flag mostly reserved for future use. At the moment, only bit 0 is used which signifies that the contents are compressed). In any case, I'll put more details on the archive format into the next revision of the paper.
Thanks for the info. I'm currently working on a commandline tool to manage tree archives, called "treeman". ATM it has the same syntax as texpack (except for the flags) and lists the TOC: treeman.zip

I'm planning to give it regex filtering and extract/update capability. For the update thing: did you implement the reader in Orbiter with some optimization assumptions in mind? E.g. a certain node sorting, placing higher nodes in a branch always behind lower nodes, every node in the list must be reachable, archive in the same order as nodes, and so on...

I'm asking because I don't want tree manipulation to irritate the runtime extraction mechanism.
Face is offline  
Thanked by:
Old 08-05-2016, 10:58 AM   #47
martins
Orbiter Founder
Default

Quote:
Originally Posted by Face View Post
 Thanks for the info. I'm currently working on a commandline tool to manage tree archives, called "treeman". ATM it has the same syntax as texpack (except for the flags) and lists the TOC: Attachment 14692

I'm planning to give it regex filtering and extract/update capability. For the update thing: did you implement the reader in Orbiter with some optimization assumptions in mind? E.g. a certain node sorting, placing higher nodes in a branch always behind lower nodes, every node in the list must be reachable, archive in the same order as nodes, and so on...

I'm asking because I don't want tree manipulation to irritate the runtime extraction mechanism.
Sounds good. I should mention that the next version of texpack will have an extract function to unpack the entire archive. Essentially the inverse of the packing operation. This was necessary to make the tileedit utility functional. But it sounds like your tool will be much more capable. An update/merge and selective filter option would definitely be useful.

As to your questions:
  • The archive _must_ be in the same order as the TOC nodes, since this is used to calculate the compressed size of each data block (compressed size = node[i+1].blockpos - node[i].blockpos)
  • All nodes must be reachable. The only way for Orbiter to find the level/latidx/lngidx designation of a node is by traversing the quadtree. This is the reason why even nodes without data block have a TOC entry, if they bridge the gap to higher-level nodes, as you noted above.
  • The order of the nodes in the list is given by recursively stepping through the quadtree. I thought that this would provide a natural order that groups related nodes close together, but I didn't do any rigorous performance tests on it. If your tool will support different ordering strategies, maybe we can do more tests on this. In any case, currently it is true that higher nodes in a branch are always after lower ones, and that the 4 branches emanating from a node are stored consecutively.

Edit: another thing: the 4-byte field at position 28 of each TOC entry is currently unused (in fact, the TOC entry size is 28 Byte, but is padded to 32). I was thinking of using those 4 bytes for a timestamp of some sort, so that archives could be selectively updated with new tiles.

Last edited by martins; 08-05-2016 at 01:25 PM. Reason: Fixed wrong byte offsets in last paragraph
martins is offline  
Thanked by:
Old 08-05-2016, 11:43 AM   #48
Face
Beta Tester
 
Face's Avatar

Default

Quote:
Originally Posted by martins View Post
 Sounds good. I should mention that the next version of texpack will have an extract function to unpack the entire archive. Essentially the inverse of the packing operation. This was necessary to make the tileedit utility functional. But it sounds like your tool will be much more capable. An update/merge and selective filter option would definitely be useful.
I think it is important to provide a filter option for developers, because always packing/extracting the whole tree gets cumbersome quickly. I'm aware of the cache/files overloading, but I think having it inside the tree might still be an option (advanced) users want.

"Merge" is an interesting idea, though. Do you think people will come up with whole tree archives bundled in addons? Originally I just thought about filtered extraction, then editing the tiles, then filtered updating.

Quote:
Originally Posted by martins View Post
 
  • The archive _must_ be in the same order as the TOC nodes, since this is used to calculate the compressed size of each data block (compressed size = node[i+1].blockpos - node[i].blockpos)

I've already anticipated that. The only other option would be to unpack until the uncompressed length was reached, which would perhaps need a customized inflate algorithm. Thanks for the clarification.



Quote:
Originally Posted by martins View Post
 
  • All nodes must be reachable. The only way for Orbiter to find the level/latidx/lngidx designation of a node is by traversing the quadtree. This is the reason why even nodes without data block have a TOC entry, if they bridge the gap to higher-level nodes, as you noted above.

Reachability was more meant in the sense of superfluous nodes. I.e. if you want to delete some tiles that make up a sub-tree "leaf", and the corresponding nodes are in the middle of the list, you could just cut it off by setting the root to -1. The nodes in question would then be orphaned, not reachable anymore, but still inside the node list. Is this a problem?



Quote:
Originally Posted by martins View Post
 
  • The order of the nodes in the list is given by recursively stepping through the quadtree. I thought that this would provide a natural order that groups related nodes close together, but I didn't do any rigorous performance tests on it. If your tool will support different ordering strategies, maybe we can do more tests on this. In any case, currently it is true that higher nodes in a branch are always after lower ones, and that the 4 branches emanating from a node are stored consecutively.
The idea here was to add new tiles to the tree by means of appending to the list and archive. I.e. even if a node would be in the middle of the list due to its level/lat/lon order, the entry would be added at the end of the list, with the index in the appropriate root updated from -1 to whatever that n+1 is. The tile data would then also be added at the end of the file.

OTOH, having the index and the archive in one file, the append-only strategy is not a huge win, anyway.

Your idea about alternative pack-strategies is interesting, though.

---------- Post added at 13:43 ---------- Previous post was at 13:24 ----------

Quote:
Originally Posted by martins View Post
 Edit: another thing: the 4-byte field at position 44 of each TOC entry is currently unused (in fact, the TOC entry size is 44 Byte, but is padded to 48). I was thinking of using those 4 bytes for a timestamp of some sort, so that archives could be selectively updated with new tiles.
That's another good idea! As long as you don't use that for Orbiter's loading mechanism, I could add a function to convert trees to this new format by means of setting e.g. Bit1 of the flag block and put a current date timestamp in it (whatever resolution that could be). Another function/option could then update the tiles automatically, if the file system tile has a newer date than the TOC entry.
Face is offline  
Old 08-05-2016, 12:48 PM   #49
martins
Orbiter Founder
Default

Quote:
Originally Posted by Face View Post
 Reachability was more meant in the sense of superfluous nodes. I.e. if you want to delete some tiles that make up a sub-tree "leaf", and the corresponding nodes are in the middle of the list, you could just cut it off by setting the root to -1. The nodes in question would then be orphaned, not reachable anymore, but still inside the node list. Is this a problem?
Not really a problem - just an efficiency issue, since Orbiter loads the entire TOC into memory. If there are too many dead branches, this will start eating up precious memory.

Quote:
"Merge" is an interesting idea, though. Do you think people will come up with whole tree archives bundled in addons? Originally I just thought about filtered extraction, then editing the tiles, then filtered updating.
Yes, I could imagine that local trees archives might become the standard format for distributing local texture updates via addons, as long as users have a way of merging it into their main archives. Another idea I had was to keep these sub-archives separate and have Orbiter merge them on the fly, but this is further down the road. At the moment I can't see how to make this efficient, since you might have to search multiple archives for a file.
martins is offline  
Old 08-05-2016, 06:26 PM   #50
Face
Beta Tester
 
Face's Avatar

Default

Quote:
Originally Posted by martins View Post
 Yes, I could imagine that local trees archives might become the standard format for distributing local texture updates via addons, as long as users have a way of merging it into their main archives. Another idea I had was to keep these sub-archives separate and have Orbiter merge them on the fly, but this is further down the road. At the moment I can't see how to make this efficient, since you might have to search multiple archives for a file.
I see. So a reliable tool that does just that would actually help here, right?

This is the current progress on treeman: treeman.zip

It has now regex filtering, 4 verbosity levels (0-3) and a summary command that shows some statistics.
Face is offline  
Thanked by:
Old 08-05-2016, 10:30 PM   #51
DaveS
Addon Developer
 
DaveS's Avatar


Default

Martin any comments on this: http://www.orbiter-forum.com/showthr...&postcount=201
DaveS is offline  
Old 08-06-2016, 09:51 AM   #52
martins
Orbiter Founder
Default

=== Please continue the discussion in the RC.3 thread ===
martins is offline  
Old 08-06-2016, 03:50 PM   #53
dbeachy1
O-F Administrator
 
dbeachy1's Avatar


Default

Thread closed since RC.3 is now out.
dbeachy1 is offline  
Thanked by:
Closed Thread

  Orbiter-Forum > Orbiter Space Flight Simulator > Orbiter Beta


Thread Tools

Posting Rules
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
Forum Jump


All times are GMT. The time now is 07:27 AM.

Quick Links Need Help?


About Us | Rules & Guidelines | TOS Policy | Privacy Policy

Orbiter-Forum is hosted at Orbithangar.com
Powered by vBulletin® Version 3.8.6
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright 2007 - 2017, Orbiter-Forum.com. All rights reserved.