Discussion Visual Studio 2012 allows 64 bit build of Orbiter

Urwumpe

Not funny anymore
Addon Developer
Donator
Joined
Feb 6, 2008
Messages
37,616
Reaction score
2,337
Points
203
Location
Wolfsburg
Preferred Pronouns
Sire
I'd suggest you to read again what you're responding to...
I take this as insult unless you will explain to me how to fit multiple 2GB+ files within 2Gb of address space. Remember MMF needs address space, and it's shortage is what lead me to consider going 64bit.

Disclaimer: If you take this as insult, I will have no problem with it.

No, MMF does not need address space. MMF needs memory management.

MMF is not "map the whole file into memory". MMF means accessing a more or less large window of your file by mapping it into virtual memory. MMF is made for fast random-access to data in the petabyte range. 4 GB of address space in your .data segment is not that much if you have to map 10 GB of simulation data on a 2 GB geometry - and that was already possible in Windows 2000 and newer. The big advantage is, that the kernel can handle the needed memory handling and I/O much faster than you could in user space.

Address space makes things easier on the developer end - but it also permits a modern form of laziness that I despise. Like having to hold Gigabytes of data ready in RAM without any purpose except that you might need a part of it often and some of the rest only maybe once per hour)
 
Last edited:

RisingFury

OBSP developer
Addon Developer
Joined
Aug 15, 2008
Messages
6,427
Reaction score
492
Points
173
Location
Among bits and Bytes...
Dude, you can't just take > 2 GB of RAM for yourself. There's other stuff going on in the computer. You're likely to run into issues with lack of RAM anyway. Not everyone has 8 GB of RAM and a lot of people (*cough*me*cough*) have the nasty habit of leaving every program ever written running on their computer.

With the greedy browsers and operating systems, you can pretty much subtract 2 GB from the system's total RAM anyway.
 

asmi

Addon Developer
Addon Developer
Joined
Jan 9, 2012
Messages
350
Reaction score
0
Points
0
Location
Ontario
No, MMF does not need address space. MMF needs memory management.

MMF is not "map the whole file into memory". MMF means accessing a more or less large window of your file by mapping it into virtual memory. MMF is made for fast random-access to data in the petabyte range. 4 GB of address space in your .data segment is not that much if you have to map 10 GB of simulation data on a 2 GB geometry - and that was already possible in Windows 2000 and newer. The big advantage is, that the kernel can handle the needed memory handling and I/O much faster than you could in user space.
First of all - I know what MMF are for and how to use them.
Second, I've tried using "smaller" MMF windows (if I remember correctly it was around 100Mb), but the problem is heightmap files layout requires random access, so I ended up remapping MMF way too often. And this forces you to do a memory management yourself instead of relying on Windows memory manager to take care of that for you (as it will automatically unload unused pages when there is shortage of memory). So since the main reason for using MMF was to have automatic page-in/out capability, I resort to loading/uploading data myself through regular File API. And THAT could be avoided alltogether in Win64.
 

Urwumpe

Not funny anymore
Addon Developer
Donator
Joined
Feb 6, 2008
Messages
37,616
Reaction score
2,337
Points
203
Location
Wolfsburg
Preferred Pronouns
Sire
First of all - I know what MMF are for and how to use them.
Second, I've tried using "smaller" MMF windows (if I remember correctly it was around 100Mb), but the problem is heightmap files layout requires random access, so I ended up remapping MMF way too often. And this forces you to do a memory management yourself instead of relying on Windows memory manager to take care of that for you (as it will automatically unload unused pages when there is shortage of memory). So since the main reason for using MMF was to have automatic page-in/out capability, I resort to loading/uploading data myself through regular File API. And THAT could be avoided alltogether in Win64.

sorry, but the first does still not sound like that. Maybe you have heard about it and know approximately what it is meant for. But you see it as tool for the wrong problems. Only because you can open a bottle with a pocket lighter, that does not mean that the lighter is the optimal solution for opening bottles.

The second does not make the first better. No, quite contrary: The height mapping process will not get better by bigger windows. you only delegate your problems to the kernel that way.

Ever considered solving the problem by using a tree structure until the amount of data is small enough? Even with the L12 heightmap (which should still be less data than the full SRTM coverage, which would be about L14, with differences in the coordinate system used), you actually either only want sparse data (high distance) or local data (low distance).

The memory mapped file will not bring you much joy there, if you do calculations in every other time-step, that could be done once in advance. If I am now not completely wrong, you could reduce the heightmap easily into small 2 MB chucks (1024 x 1024 x 16 bit signed, DEM data) and have only a L4 size quad-tree dictionary for producing distant height data and find the needed chucks again in a compiled height file containing the fine data. Which you could then memory map into multiple 2 MB windows.

Slower than completely stuffing the data into physical memory as simple 2D array? Sure. Much slower in practical rendering? Not really. And it leaves you resources for other functions, instead of having topography of a single planet consume 90% of the memory available on an average consumer PC.
 

asmi

Addon Developer
Addon Developer
Joined
Jan 9, 2012
Messages
350
Reaction score
0
Points
0
Location
Ontario
The second does not make the first better. No, quite contrary: The height mapping process will not get better by bigger windows. you only delegate your problems to the kernel that way.
THAT is EXACTLY what I wanted to achieve!

Ever considered solving the problem by using a tree structure until the amount of data is small enough? Even with the L12 heightmap (which should still be less data than the full SRTM coverage, which would be about L14, with differences in the coordinate system used), you actually either only want sparse data (high distance) or local data (low distance).
That's the way it's done now. The problem is that I couldn't even complete a reentry without running out of memory (due to massive amount of "small" heightmap tiles I go through in the process). So right now I just track heightmap usages and remove once that were not accesses for a certain period of time.

Slower than completely stuffing the data into physical memory as simple 2D array? Sure. Much slower in practical rendering? Not really. And it leaves you resources for other functions, instead of having topography of a single planet consume 90% of the memory available on an average consumer PC.
Now you're missing a point there - by mapping the whole file into process's address space doesn't get it loaded up. And Windows memory manager will not let you use up all memory anyways ;)
 

Urwumpe

Not funny anymore
Addon Developer
Donator
Joined
Feb 6, 2008
Messages
37,616
Reaction score
2,337
Points
203
Location
Wolfsburg
Preferred Pronouns
Sire
THAT is EXACTLY what I wanted to achieve!

So you want to go the Wing Commander way. Make the software for making us buy better computers.

That's the way it's done now. The problem is that I couldn't even complete a reentry without running out of memory (due to massive amount of "small" heightmap tiles I go through in the process). So right now I just track heightmap usages and remove once that were not accesses for a certain period of time.

I find this strange - do you use the small tiles even for the high distance case? As in : Showing the whole visible planet at 120 meter resolution, instead of using a kilometer or even 100 km resolution for the higher distances?

Now you're missing a point there - by mapping the whole file into process's address space doesn't get it loaded up. And Windows memory manager will not let you use up all memory anyways ;)

Usually not, but would rapid use most of your memory if you have no data reduction. Windows can't predict what your application does.
 

dumbo2007

Crazy about real time sims
Joined
Nov 29, 2009
Messages
675
Reaction score
0
Points
0
Location
India
I guess for a re-entry, since a ship would gradually approach the surface, one can adjust the level of detail with larger tiles with low resolution for places far away from the camera.

More tiles for places closer to the camera with high resolution. I have no idea how it would even work with hardware tessellation but I am guessing its possible to tessellate parts of a sphere more & others less or something similar.

I am not sure what would happen if the camera is suddenly moved from a vessel in orbit to a vessel on the surface though :p
 
Last edited:

Urwumpe

Not funny anymore
Addon Developer
Donator
Joined
Feb 6, 2008
Messages
37,616
Reaction score
2,337
Points
203
Location
Wolfsburg
Preferred Pronouns
Sire
I am not sure what would happen if the camera is suddenly moved from a vessel in orbit to a vessel on the surface though :p

either nothing or or the height data is slowly updated as data is reloaded in the background
 

dumbo2007

Crazy about real time sims
Joined
Nov 29, 2009
Messages
675
Reaction score
0
Points
0
Location
India
I think what asmi was referring to earlier about random access requiring large amounts of data is that , say you have 2 adjacent tiles. Their data may be stored at locations in the file that are more than 2 GB bytes apart. So then the whole file would need to be in memory. This may happen if the data along 1 entire sweep like say all altitudes between latitudes 20 degrees and 25 degrees are stored sequentially in the file. Then all altitudes between 26 & 30 degrees etc.

Then if the spacecraft is targeting a base at 26 degrees & is quite close to the surface, you will need data from both sweeps & at high resolution, to show the surrounding terrain realistically.

Maybe its possible to resort the file in someway to avoid this.
 
Last edited:

Urwumpe

Not funny anymore
Addon Developer
Donator
Joined
Feb 6, 2008
Messages
37,616
Reaction score
2,337
Points
203
Location
Wolfsburg
Preferred Pronouns
Sire
Not really: first of all, showing surrounding terrain realistically means you need to show things at roughly constant angular resolution. if you have 30 m resolution at the end of your 30 km tile, you need the same resolution for the following 8 tiles. But for the next distance, you are already 3 times further away from the center of your current tile than the edge of your tile, so you can use 3 times lower resolution. instead of 3 x 3 * 8 tiles at 30 m, you can use 8 tiles of 90 meter resolution. so, you have 9 tiles at 30m and 8 tiles at 90 m (all together 17 tiles with 34 MB of data if you use 16 bit data), and cover a distance of about 45 + 90 km = 135 km. That would be enough coverage for flying in about 2000 meter altitude (above the average tile altitude, not MSL).

If you fly in 30 km altitude, you don't really need 30 meter resolution for covering your place, 90 meter would already be more than enough. This permits you to cover 3 times the terrain with the same number of data tiles loaded: about 450 km distance.

What you need is intelligent preloading: which tiles are more likely to be used in the future. These are the tiles in your current flight path, while tiles right behind you would not be needed once out of sight. At higher speeds, you also can't see much detail, so you don't need 30 m resolution around you as well, you only need to be prepared for rendering them as soon as speed drops.

(Of course, the 30 m resolution is only approximate - in reality you would use angle as unit for covering a sphere surface)

By having enough memory available to hold all height data in memory, you can of course have the highest speed with that algorithm. But you could still scale the algorithm down for weaker machines. You could reduce the number of tiles that are shown by having lower base visibility ranges. You could use lower minimum resolution for the tiles.


Now, you could also have the special case of flying with a telescope sight... but then you would need less tiles, since your most likely line of travel produces a very narrow cone of high-resolution tiles.

You could also make better use of multi-cores by such a divide and conquer approach. While I/O would still be limited by the single HDD that your data is on, you could calculate the tile priorities faster if you have a tree-like structure, compared to a pure iterative approach which would not allow good task stealing.

I doubt you will fly at more than 3000 m/s at less than 30 km altitude. so, you would need a new set of full tile data every 10 seconds at 30 meter resolution. ten seconds, for loading 12 tiles. If you reduce resolution by speed to 90 meters, you would only need to load 4 tiles in the same period of time.

Also, you don't need a large overhead of lower resolution tiles on your HD for the process, you can skip some of the lower resolutions. After 270 meter resolution (270 km tile width) and 810 m (810 km tile width) there won't be much demand for Earth or Venus (about 90 810 km tiles would be needed for covering one hemisphere of Earth, or ten 2430 m tiles). Even less on Moon or Mars.

And you could even do more optimization ;) At lower viewing angles, you only see every tile from the side and with low detail. you could replace the terrain in rendering by one of 8 simple rectangles (N,E,S,W, NE, SW, NW, SE) with prerendered textures. And this case could happen at pretty low distances already. if you assume 10° as criteria, you would get the chance for reduction at:

Altitude (km)|Limit(10°, km)|Limit(5°, km)
1| 5.6 | 11.4
2|11.2 | 22.8
5|28 | 57
10|56 | 114
20|112 | 228
30|168 | 342
50|280 | 570
100|560| 1140

Or already at the next or second next ring of tiles, depending on altitude and base resolution.
 
Last edited:
Top