Orbiter 2024 Launch readiness

[n72.75]

I ran a test with these configurations. Frame rate was about 2.1 fps. There is a small constant increase of system memory consumption.
High-res textures installed, 90deg fov down to moon. Resolution bias at max.

In your case how does the memory climb to 1.7G ? Could you give a plot from application start ?

Does the DirectX9 run natively or via wrapper. If yes then DXVK could be tested.

I'll run the test again tonight to gather data. I saw it as high as 1.8GB when it was running out of memory.

DX9 is running natively.

Maybe there's some other hardware weirdness going on? I have ecc memory, but this is on a dual-cpu (2x x5690) system.

 

[n72.75]

Collected a bit more data:

I am using your latest commit:


This is heap profiling from as soon as I opened the launchpad. Frame-rate was pretty low, and I wasn't getting a crash, so I went back to the launchpad. That snapshot below shows quite a bit of memory still allocated after returning to the launchpad, which I wouldn't expect.



I changed propagator settings so I could get a slightly better frame-rate (mid 20s). Memory climbs until crash.






Not sure if this is helpful, but most of them are these 41,580 byte objects (tiles at some LOD, I presume)





Is it possible that this is the result of a race condition? Some mutex lock/unlock in the wrong place?

Could someone other than me and Jarmo try this out, too.

 

[jarmonik]

It would be nice to be able to see something in the screen shots but I get the picture. Currently, the system memory data cache doesn't release memory when exiting back to launch pad, it's kept pre-allocated for the next run. It was easier implement that way. But I can make it to release the memory.

Look like it ran out of memory after the second attempt from a launchpad. I'll try to re-produce that.

 

[jarmonik]

The problem must be the screen size, it's somehow causing surface tile density to become very high. The VEXTEX_2TEX[] in your case appears to be around 258 MB, mine remains below 50 MB. What are the display's pixel counts ?

I ran a successful over night test on 30x30 lunar orbit. Memory climbed slowly during the first 15 mins and settled around 450MB.

 

[n72.75]

The problem must be the screen size, it's somehow causing surface tile density to become very high. The VEXTEX_2TEX[] in your case appears to be around 258 MB, mine remains below 50 MB. What are the display's pixel counts ?

I ran a successful over night test on 30x30 lunar orbit. Memory climbed slowly during the first 15 mins and settled around 450MB.

I'm running at 2560x1080. so not a super high resolution, but definitely a different aspect ratio.

@CaptainSwag101 was able to replicate (partially) this memory problem. check the orbiter development channel on the O-F discord.

 

[GLS]

Not wanting to derail the important debugging of the memory issues, but tileedit is not being built...

 

[Gondos]

Not wanting to derail the important debugging of the memory issues, but tileedit is not being built...

The Qt dependency may be the reason. It would have to be set up in the CI environment.

 

[Gondos]

I'm running at 2560x1080. so not a super high resolution, but definitely a different aspect ratio.

@CaptainSwag101 was able to replicate (partially) this memory problem. check the orbiter development channel on the O-F discord.

It would be good to know if you're hitting a 32bit address space limit with your configuration or if it's a real memory leak. What's the behavior in 64bit? Does it keeps growing endlessly or just peaking at a high value?

 

[jarmonik]

I'm running at 2560x1080. so not a super high resolution, but definitely a different aspect ratio.

@CaptainSwag101 was able to replicate (partially) this memory problem. check the orbiter development channel on the O-F discord.

There is a new commit, is the problem fixed ?
There is still a memory leak somewhere causing the process memory to jump about 20-40MB each restart from a launchpad.

 

[N_Molson]

Hello, sorry if I'm posting in the wrong place.

Using version 240315 I can see the Sun rising through mountains. The DG is in Kathmandu, Nepal, and the camera is looking towards the Himalayas :

 

[n72.75]

There is a new commit, is the problem fixed ?
There is still a memory leak somewhere causing the process memory to jump about 20-40MB each restart from a launchpad.

This is a significant improvement to the memory usage over time. I've been trying pretty hard for the last two hours.

Eventually, (by switching between 4 vessels in orbit around Vesta, Moon, Mars and Earth) I was able to get the memory to run out.

From Orbiter.log

007297.875: Failed to create vertex buffer for surface tile. Likely [Out of Video Memory]

Unfortunately, because I was trying this in Release, I'm missing some data.

I am going try:

• capturing the cause of the crash in debug,
• running with "normal" settings until a crash occurs (days, weeks, however long it takes etc)

With this commit, there has been a drastic improvement so I feel like we're really close to solving this.

 

[n72.75]

It would be good to know if you're hitting a 32bit address space limit with your configuration or if it's a real memory leak. What's the behavior in 64bit? Does it keeps growing endlessly or just peaking at a high value?

I will test this too.

It seems strange that the crash is occurring at 1.7GB, and not closer to the 2GB limit. Or am I thinking of that wrong.

 

[misha.physics]

Using version 240315 I can see the Sun rising through mountains.

Seems to be the same known problem: moon (see P.S.), the sun, runways (as objects), DG shadows, etc. I've made a runway for the Ascension Island as a mesh with its own texture that isn't shining through the ground (so there can be a solution), but exhausts (shadows) still is. Local lights are visible through objects as well, maybe the relevant issue should be created on GitHub. Even if it doesn't have a priority right now.

 

[jarmonik]

This is a significant improvement to the memory usage over time. I've been trying pretty hard for the last two hours.
Eventually, (by switching between 4 vessels in orbit around Vesta, Moon, Mars and Earth) I was able to get the memory to run out.

Ok, that should be fixed now. I noticed when the camera was moved to a distant body, previous body becoming inactive the tiles were not unloaded. A second problem much more trickier was found when zooming to a planet's surface from a low orbit (i.e. narrow FOV 10deg). Tile resolution is calculated from a camera-tile distance and FOV. On a low orbit the camera-tile distance doesn't wary much, this caused a high number of high resolution tiles to be loaded. So, to get this working better a reference point for tile distance calculations should be closer to a surface than the camera is. But this gets very tricky when zooming to a horizon instead of straight down. Surface elevation is also a major factor.

There is a "hotfix" that's limits the FOV in tile resolution calculations, and an other safe-guard that starts limiting tile count after reaching 800 tiles. These will reduce a surface sharpness under these extreme conditions.

 

[jarmonik]

Hello, sorry if I'm posting in the wrong place.
Using version 240315 I can see the Sun rising through mountains. The DG is in Kathmandu, Nepal, and the camera is looking towards the Himalayas :

I'll try to fix that in some point in the future.

 

[n72.75]

Ok, that should be fixed now. I noticed when the camera was moved to a distant body, previous body becoming inactive the tiles were not unloaded. A second problem much more trickier was found when zooming to a planet's surface from a low orbit (i.e. narrow FOV 10deg). Tile resolution is calculated from a camera-tile distance and FOV. On a low orbit the camera-tile distance doesn't wary much, this caused a high number of high resolution tiles to be loaded. So, to get this working better a reference point for tile distance calculations should be closer to a surface than the camera is. But this gets very tricky when zooming to a horizon instead of straight down. Surface elevation is also a major factor.

There is a "hotfix" that's limits the FOV in tile resolution calculations, and an other safe-guard that starts limiting tile count after reaching 800 tiles. These will reduce a surface sharpness under these extreme conditions.

Awesome.

I'll pull and build this when I get home tonight.

I did leave a long-term test running overnight (with the previous commit) and this morning it was still going strong with a process memory of just over 1GB (obviously the camera isn't focusing on anything different though).

Hopefully this is the last pass through and we have some good confidence that users running long-term simulations will be safe. once this gets merged I will probably set up some kind of long-term scenerio to run for a week or so. I know with Beta and its associated client. @STS was able to fly the full Apollo 11 mission in realtime without reloading Orbiter. Just trying to make sure we can still do that


Thank you for all the hard work that has gone into this.

 

[n72.75]

@jarmonik

Good news: Memory usage seems to be very stable now.

Bad news:



I think this one should be pretty easy to replicate though, on your end.

1. Load any scenario (DG at Brighton Beach, is usually my go-to).
2. Select the camera dialog and target a different planet than the one your scenario (e.g. Mars, if you loaded on the moon)
3. Zoom in on the surface, until you can see textures loading.
4. Select a random planet in the camera dialog.
5. Repeat 3 and 4 until the crash happens.

When the crash happens, it happens instantly upon selection of a planet, but the number of times you need to change your selection is not constant. (4-10 switches, revisiting earlier selections, seems to be the norm.)

The stack trace is always:

>    D3D9Client.dll!QuadTreeNode<SurfTile>::Entry() Line 23    C++
     D3D9Client.dll!SurfTile::GetElevation(double lng, double lat, double * elev, oapi::FVECTOR3 * nrm, SurfTile * * cache, bool bFilter, bool bGet) Line 679    C++
     D3D9Client.dll!SurfTile::GetElevation(double lng, double lat, double * elev, oapi::FVECTOR3 * nrm, SurfTile * * cache, bool bFilter, bool bGet) Line 679    C++
     D3D9Client.dll!TileManager2<SurfTile>::GetElevation(double lng, double lat, double * elev, oapi::FVECTOR3 * nrm, SurfTile * * cache) Line 1590    C++
     D3D9Client.dll!vPlanet::GetElevation(double lng, double lat, double * elv, oapi::FVECTOR3 * nrm) Line 708    C++
     D3D9Client.dll!Scene::SetupInternalCamera(D3DXMATRIX * mNew, VECTOR3 * gpos, double apr, double asp) Line 3503    C++
     D3D9Client.dll!Scene::UpdateCameraFromOrbiter(unsigned long dwPass) Line 3414    C++
     D3D9Client.dll!Scene::UpdateCamVis() Line 1036    C++
     D3D9Client.dll!Scene::RenderMainScene() Line 1250    C++
     D3D9Client.dll!oapi::D3D9Client::clbkRenderScene() Line 1197    C++
     Orbiter.exe!Orbiter::Render3DEnvironment() Line 1040    C++
     Orbiter.exe!Orbiter::Run() Line 1106    C++
     Orbiter.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * __formal, char * strCmdLine, int nCmdShow) Line 245    C++
     [External Code]

I did try this rather naïve solution, and it does seem to fix the issue. I have no idea if this might be introducing some horrible side-effect somewhere. Maybe something else earlier in the stack trace needs to check this?

    //in SurfTile::GetElevation



    if (state==Active) {
        int i = 0;
        if (lng > (bnd.minlng + bnd.maxlng)*0.5) i++;
        if (lat < (bnd.minlat + bnd.maxlat)*0.5) i += 2;
        if (node->Child(i)) {
            return  node->Child(i)->Entry()->GetElevation(lng, lat, elev, nrm, cache);
        }
    }

 

[jarmonik]

Good news: Memory usage seems to be very stable now.
Bad news:

Thanks for the report, problem should be fixed now.

 

[n72.75]

Thanks for the report, problem should be fixed now.

I have approved the PR.

 

[jarmonik]

What should we do with the SDK Samples, the code files are in place but there's no project files to compile them ?
Should there be some easy way the compile them without compiling the Orbiter, or are the code files enough ?