Orbiter 2024 Launch readiness

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,704
Reaction score
1,378
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
I ran a test with these configurations. Frame rate was about 2.1 fps. There is a small constant increase of system memory consumption.
High-res textures installed, 90deg fov down to moon. Resolution bias at max.

In your case how does the memory climb to 1.7G ? Could you give a plot from application start ?

Does the DirectX9 run natively or via wrapper. If yes then DXVK could be tested.
I'll run the test again tonight to gather data. I saw it as high as 1.8GB when it was running out of memory.

DX9 is running natively.

Maybe there's some other hardware weirdness going on? I have ecc memory, but this is on a dual-cpu (2x x5690) system.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,704
Reaction score
1,378
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
Collected a bit more data:

I am using your latest commit:
1710808774905.png

This is heap profiling from as soon as I opened the launchpad. Frame-rate was pretty low, and I wasn't getting a crash, so I went back to the launchpad. That snapshot below shows quite a bit of memory still allocated after returning to the launchpad, which I wouldn't expect.

1710808049278.png

I changed propagator settings so I could get a slightly better frame-rate (mid 20s). Memory climbs until crash.

1710808356523.png

1710808945198.png


Not sure if this is helpful, but most of them are these 41,580 byte objects (tiles at some LOD, I presume)
1710809349357.png




Is it possible that this is the result of a race condition? Some mutex lock/unlock in the wrong place?

Could someone other than me and Jarmo try this out, too.
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
It would be nice to be able to see something in the screen shots but I get the picture. Currently, the system memory data cache doesn't release memory when exiting back to launch pad, it's kept pre-allocated for the next run. It was easier implement that way. But I can make it to release the memory.

Look like it ran out of memory after the second attempt from a launchpad. I'll try to re-produce that.
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
The problem must be the screen size, it's somehow causing surface tile density to become very high. The VEXTEX_2TEX[] in your case appears to be around 258 MB, mine remains below 50 MB. What are the display's pixel counts ?

I ran a successful over night test on 30x30 lunar orbit. Memory climbed slowly during the first 15 mins and settled around 450MB.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,704
Reaction score
1,378
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
The problem must be the screen size, it's somehow causing surface tile density to become very high. The VEXTEX_2TEX[] in your case appears to be around 258 MB, mine remains below 50 MB. What are the display's pixel counts ?

I ran a successful over night test on 30x30 lunar orbit. Memory climbed slowly during the first 15 mins and settled around 450MB.
I'm running at 2560x1080. so not a super high resolution, but definitely a different aspect ratio.

@CaptainSwag101 was able to replicate (partially) this memory problem. check the orbiter development channel on the O-F discord.
 

GLS

Well-known member
Orbiter Contributor
Addon Developer
Joined
Mar 22, 2008
Messages
6,030
Reaction score
3,078
Points
188
Website
github.com
Not wanting to derail the important debugging of the memory issues, but tileedit is not being built...
 

Gondos

Well-known member
Joined
Apr 18, 2022
Messages
245
Reaction score
286
Points
78
Location
On my chair
I'm running at 2560x1080. so not a super high resolution, but definitely a different aspect ratio.

@CaptainSwag101 was able to replicate (partially) this memory problem. check the orbiter development channel on the O-F discord.
It would be good to know if you're hitting a 32bit address space limit with your configuration or if it's a real memory leak. What's the behavior in 64bit? Does it keeps growing endlessly or just peaking at a high value?
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
I'm running at 2560x1080. so not a super high resolution, but definitely a different aspect ratio.

@CaptainSwag101 was able to replicate (partially) this memory problem. check the orbiter development channel on the O-F discord.

There is a new commit, is the problem fixed ?
There is still a memory leak somewhere causing the process memory to jump about 20-40MB each restart from a launchpad.
 

N_Molson

Addon Developer
Addon Developer
Donator
Joined
Mar 5, 2010
Messages
9,296
Reaction score
3,270
Points
203
Location
Toulouse
Hello, sorry if I'm posting in the wrong place.

Using version 240315 I can see the Sun rising through mountains. The DG is in Kathmandu, Nepal, and the camera is looking towards the Himalayas :

DG_Nepal_Sunrise.jpg
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,704
Reaction score
1,378
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
There is a new commit, is the problem fixed ?
There is still a memory leak somewhere causing the process memory to jump about 20-40MB each restart from a launchpad.
This is a significant improvement to the memory usage over time. I've been trying pretty hard for the last two hours.

Eventually, (by switching between 4 vessels in orbit around Vesta, Moon, Mars and Earth) I was able to get the memory to run out.

From Orbiter.log
Code:
007297.875: Failed to create vertex buffer for surface tile. Likely [Out of Video Memory]

1710904104038.png

Unfortunately, because I was trying this in Release, I'm missing some data.

I am going try:
  • capturing the cause of the crash in debug,
  • running with "normal" settings until a crash occurs (days, weeks, however long it takes etc)
With this commit, there has been a drastic improvement so I feel like we're really close to solving this.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,704
Reaction score
1,378
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
It would be good to know if you're hitting a 32bit address space limit with your configuration or if it's a real memory leak. What's the behavior in 64bit? Does it keeps growing endlessly or just peaking at a high value?
I will test this too.

It seems strange that the crash is occurring at 1.7GB, and not closer to the 2GB limit. Or am I thinking of that wrong.
 

misha.physics

Well-known member
Joined
Dec 22, 2021
Messages
475
Reaction score
600
Points
108
Location
Lviv
Preferred Pronouns
he/him
Using version 240315 I can see the Sun rising through mountains.
Seems to be the same known problem: moon (see P.S.), the sun, runways (as objects), DG shadows, etc. I've made a runway for the Ascension Island as a mesh with its own texture that isn't shining through the ground (so there can be a solution), but exhausts (shadows) still is. Local lights are visible through objects as well, maybe the relevant issue should be created on GitHub. Even if it doesn't have a priority right now.
 
Last edited:

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
This is a significant improvement to the memory usage over time. I've been trying pretty hard for the last two hours.
Eventually, (by switching between 4 vessels in orbit around Vesta, Moon, Mars and Earth) I was able to get the memory to run out.

Ok, that should be fixed now. I noticed when the camera was moved to a distant body, previous body becoming inactive the tiles were not unloaded. A second problem much more trickier was found when zooming to a planet's surface from a low orbit (i.e. narrow FOV 10deg). Tile resolution is calculated from a camera-tile distance and FOV. On a low orbit the camera-tile distance doesn't wary much, this caused a high number of high resolution tiles to be loaded. So, to get this working better a reference point for tile distance calculations should be closer to a surface than the camera is. But this gets very tricky when zooming to a horizon instead of straight down. Surface elevation is also a major factor.

There is a "hotfix" that's limits the FOV in tile resolution calculations, and an other safe-guard that starts limiting tile count after reaching 800 tiles. These will reduce a surface sharpness under these extreme conditions.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,704
Reaction score
1,378
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
Ok, that should be fixed now. I noticed when the camera was moved to a distant body, previous body becoming inactive the tiles were not unloaded. A second problem much more trickier was found when zooming to a planet's surface from a low orbit (i.e. narrow FOV 10deg). Tile resolution is calculated from a camera-tile distance and FOV. On a low orbit the camera-tile distance doesn't wary much, this caused a high number of high resolution tiles to be loaded. So, to get this working better a reference point for tile distance calculations should be closer to a surface than the camera is. But this gets very tricky when zooming to a horizon instead of straight down. Surface elevation is also a major factor.

There is a "hotfix" that's limits the FOV in tile resolution calculations, and an other safe-guard that starts limiting tile count after reaching 800 tiles. These will reduce a surface sharpness under these extreme conditions.
Awesome.

I'll pull and build this when I get home tonight.

I did leave a long-term test running overnight (with the previous commit) and this morning it was still going strong with a process memory of just over 1GB (obviously the camera isn't focusing on anything different though).

Hopefully this is the last pass through and we have some good confidence that users running long-term simulations will be safe. once this gets merged I will probably set up some kind of long-term scenerio to run for a week or so. I know with Beta and its associated client. @STS was able to fly the full Apollo 11 mission in realtime without reloading Orbiter. Just trying to make sure we can still do that :)


Thank you for all the hard work that has gone into this.
 
  • Like
Reactions: STS

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,704
Reaction score
1,378
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
@jarmonik

Good news: Memory usage seems to be very stable now.

Bad news:

1710987636009.png

I think this one should be pretty easy to replicate though, on your end.

  1. Load any scenario (DG at Brighton Beach, is usually my go-to).
  2. Select the camera dialog and target a different planet than the one your scenario (e.g. Mars, if you loaded on the moon)
  3. Zoom in on the surface, until you can see textures loading.
  4. Select a random planet in the camera dialog.
  5. Repeat 3 and 4 until the crash happens.
When the crash happens, it happens instantly upon selection of a planet, but the number of times you need to change your selection is not constant. (4-10 switches, revisiting earlier selections, seems to be the norm.)

The stack trace is always:

Code:
>    D3D9Client.dll!QuadTreeNode<SurfTile>::Entry() Line 23    C++
     D3D9Client.dll!SurfTile::GetElevation(double lng, double lat, double * elev, oapi::FVECTOR3 * nrm, SurfTile * * cache, bool bFilter, bool bGet) Line 679    C++
     D3D9Client.dll!SurfTile::GetElevation(double lng, double lat, double * elev, oapi::FVECTOR3 * nrm, SurfTile * * cache, bool bFilter, bool bGet) Line 679    C++
     D3D9Client.dll!TileManager2<SurfTile>::GetElevation(double lng, double lat, double * elev, oapi::FVECTOR3 * nrm, SurfTile * * cache) Line 1590    C++
     D3D9Client.dll!vPlanet::GetElevation(double lng, double lat, double * elv, oapi::FVECTOR3 * nrm) Line 708    C++
     D3D9Client.dll!Scene::SetupInternalCamera(D3DXMATRIX * mNew, VECTOR3 * gpos, double apr, double asp) Line 3503    C++
     D3D9Client.dll!Scene::UpdateCameraFromOrbiter(unsigned long dwPass) Line 3414    C++
     D3D9Client.dll!Scene::UpdateCamVis() Line 1036    C++
     D3D9Client.dll!Scene::RenderMainScene() Line 1250    C++
     D3D9Client.dll!oapi::D3D9Client::clbkRenderScene() Line 1197    C++
     Orbiter.exe!Orbiter::Render3DEnvironment() Line 1040    C++
     Orbiter.exe!Orbiter::Run() Line 1106    C++
     Orbiter.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * __formal, char * strCmdLine, int nCmdShow) Line 245    C++
     [External Code]

I did try this rather naïve solution, and it does seem to fix the issue. I have no idea if this might be introducing some horrible side-effect somewhere. Maybe something else earlier in the stack trace needs to check this?

C++:
    //in SurfTile::GetElevation



    if (state==Active) {
        int i = 0;
        if (lng > (bnd.minlng + bnd.maxlng)*0.5) i++;
        if (lat < (bnd.minlat + bnd.maxlat)*0.5) i += 2;
        if (node->Child(i)) {
            return  node->Child(i)->Entry()->GetElevation(lng, lat, elev, nrm, cache);
        }
    }

1710991220835.png
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
What should we do with the SDK Samples, the code files are in place but there's no project files to compile them ?
Should there be some easy way the compile them without compiling the Orbiter, or are the code files enough ?
 
Top