Orbiter 2024 Launch readiness

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
Hmm, do you mean having the code box only in one page, as opposed to having part on one page and the other part on another page?
Yes, exactly that. Especially with short examples. Can't be avoided with long ones.

I'll start working on conversion of Atlantis Manual and I'll make a PR to your documentationrework_part1_conversion There is already one waiting.
 

Marg

Active member
Joined
Mar 20, 2008
Messages
485
Reaction score
68
Points
28
I would like to add an observation that Default Atlantis has graphic anomalies with ET and SRB meshes (february (pre-release?) Orbiter24), and only with them (shuttle-orbiter is OK), they get black when accelerating time, and flicker very little (darker-lighter).
 

GLS

Well-known member
Orbiter Contributor
Addon Developer
Joined
Mar 22, 2008
Messages
6,095
Reaction score
3,178
Points
188
Website
github.com
Yes, exactly that. Especially with short examples. Can't be avoided with long ones.
I'll see what is possible to do.

I'll start working on conversion of Atlantis Manual and I'll make a PR to your documentationrework_part1_conversion There is already one waiting.
A PR on a PR... 🤯 never would have thought of that.
I see you are "fixing and correcting" things... IMO, for speed, that should be left for after the LaTeX conversion is finished. Not having to worry about whether, e.g., parameter x has more possible values has helped progress, as I only have to think about making the text show up with +/- the correct formatting. I'm signaling stuff that needs checking with TODOs as I happen to find them, but that is it, I'm not "derailing" the LaTeX train to go dig into the code, editing a file or running a sim to check something.

Another thing: the Atlantis manual should be inside the Orbiter User Manual, as opposed to being a separate file. Same for DG, ShuttleA, etc.
At the end we would have 3 pdf files:
Orbiter User Manual - explain Orbiter, how to configure and use it, along with the included vessels, MFDs and bases
Orbiter Developer Manual - addons, vessels, bases, planets, textures, meshes, code, lua, etc
Orbiter Technical Reference - collection of papers/articles detailing the theory behind some feature or capability


I'm just finishing another bunch of additions (tried to finish them yesterday, but gave up as I had been up for 21h) and will commit them soon. I'll then will have to leave for a while, but will be back later today.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,712
Reaction score
1,408
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
I think there may be an issue with memory.

I was testing the gravity models in a low lunar orbit (15x15km). With LP165 set to a 165x165 cutoff I'm normally able to get frame-rates in the 120s-180s depending on number of vessels. That's what I got when I added the gravity models, and that's about what I'm getting on initial load now.


1710600243770.png


After an hour or so in orbit around the moon frame-rate drops off to the 20s and orbiter eventually crashes (hard crash, no log). Memory usage steadily climbs as frame-rate falls; it seems to top around around 1.7GB, but that's about where the crash occurs.

1710600053371.png

Attaching the debugger to the release version during the crash gives this:
1710601082212.png

Which is rather unhelpful, unless someone wants to look at an assembly listing of Orbiter.

I'm going to build in debug and see if I can pinpoint exactly what is causing this.

  • Eventually Orbiter will crash (with the settings in the attached configs)
  • Sometimes Orbiter will crash if you shutdown while the frame-rate is low (before it crashes on its own).
    • When you reload, memory usage is back down, and frame-rates are good again.
  • Sometimes after going back to the launchpad when you have the low frame-rate issue, orbiter will crash on startup.
    • This never seems to happen (as far as I can test) on a clean scenario load, after completely closing Orbiter
 

Attachments

  • D3D9Client.cfg.txt
    1.4 KB · Views: 2
  • Orbiter_NG.cfg.txt
    4.5 KB · Views: 1
  • Moon.cfg.txt
    1.6 KB · Views: 3

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,712
Reaction score
1,408
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
I think there may be an issue with memory.

I was testing the gravity models in a low lunar orbit (15x15km). With LP165 set to a 165x165 cutoff I'm normally able to get frame-rates in the 120s-180s depending on number of vessels. That's what I got when I added the gravity models, and that's about what I'm getting on initial load now.


View attachment 37342


After an hour or so in orbit around the moon frame-rate drops off to the 20s and orbiter eventually crashes (hard crash, no log). Memory usage steadily climbs as frame-rate falls; it seems to top around around 1.7GB, but that's about where the crash occurs.

View attachment 37341

Attaching the debugger to the release version during the crash gives this:
View attachment 37346

Which is rather unhelpful, unless someone wants to look at an assembly listing of Orbiter.

I'm going to build in debug and see if I can pinpoint exactly what is causing this.

  • Eventually Orbiter will crash (with the settings in the attached configs)
  • Sometimes Orbiter will crash if you shutdown while the frame-rate is low (before it crashes on its own).
    • When you reload, memory usage is back down, and frame-rates are good again.
  • Sometimes after going back to the launchpad when you have the low frame-rate issue, orbiter will crash on startup.
    • This never seems to happen (as far as I can test) on a clean scenario load, after completely closing Orbiter


This is what the moon looks like as the frame-rates start to get really bad:
1710603461038.png
Closing Orbiter during this, results in an access violation here:
1710603720113.png

EDIT:

Crash during an Orbiter session. This took about an hour (realtime) at 10x in lunar orbit.
1710605940885.png

Code:
Exception thrown: read access violation.
this->**pVB** was nullptr.

>    D3D9Client.dll!MeshBuffer::Map(IDirect3DDevice9 * pDev) Line 155    C++
     D3D9Client.dll!D3D9Mesh::D3D9Mesh(void * hMesh, bool asTemplate, D3DXVECTOR3 * reorig, float * scale) Line 237    C++
     D3D9Client.dll!vBase::vBase(void * _hObj, const Scene * scene, vPlanet * _vP) Line 163    C++
     D3D9Client.dll!vPlanet::Update(bool bMainScene) Line 837    C++
     D3D9Client.dll!Scene::UpdateCameraFromOrbiter(unsigned long dwPass) Line 3411    C++
     D3D9Client.dll!Scene::UpdateCamVis() Line 1036    C++
     D3D9Client.dll!Scene::RenderMainScene() Line 1250    C++
     D3D9Client.dll!oapi::D3D9Client::clbkRenderScene() Line 1174    C++
     Orbiter.exe!Orbiter::Render3DEnvironment() Line 1040    C++
     Orbiter.exe!Orbiter::Run() Line 1106    C++
     Orbiter.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * __formal, char * strCmdLine, int nCmdShow) Line 245    C++
     [External Code]   
     [Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]

Code:
-        &pTgt    0x0112db7c {0x13fbc000}    void * *
            0x13fbc000    void *
        Lock    0    unsigned long
        nVtx    740    unsigned long
-        pSB    0x38c418e0 <Information not available, no symbols loaded for d3d9.dll>    IDirect3DVertexBuffer9 *
+        IDirect3DResource9    {...}    IDirect3DResource9
-        pSBSys    0x4168b2c0 {x=25.0000000 y=0.00000000 z=0.00000000 ...}    SMVERTEX *
        x    25.0000000    float
        y    0.00000000    float
        z    0.00000000    float
        tu    0.00000000    float
        tv    0.00000000    float
        pTgt    0x13fbc000    void *
-        pVB    0x00000000 <NULL>    IDirect3DVertexBuffer9 *
+        IDirect3DResource9    <struct at NULL>    IDirect3DResource9
+        this    0x1b3d0d28 {pVB=0x00000000 <NULL> pGB=0x38c41ca0 <Information not available, no symbols loaded for d3d9.dll> ...}    MeshBuffer *
 
Last edited:

Gondos

Well-known member
Joined
Apr 18, 2022
Messages
267
Reaction score
317
Points
78
Location
On my chair
What moon lvl are you using? Trying to repro this on an x64-debug build but nothing stands out clearly after 70min.
Because of the dynamic tile loading, the memory comes and goes depending on what's on the viewport...
The main memory deltas between snapshots are mostly located around here :
Code:
VERTEX_2TEX *vtx = new VERTEX_2TEX[nvtxbuf];
and
Code:
WORD *idx = new WORD[nidx];
in Tile::CreateMesh_quadpatch
and
Code:
e = new INT16[ndat];
elev = new float[ndat];
in SurfTile::ReadElevationFile
which is to be expected.
Do you have the same behavior?
The heap size looks stable but the process memory increases, maybe it's a memory fragmentation issue?
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
Looks like it makes random crashes after running out of memory. Is this a new problem or has it existed for some time ?
Terrain vertex buffers are already recycled, the same space is re-used over and over again. However, that doesn't apply to the tile loading. I suppose those system memory buffers could also be re-used, since most of them are constant size. That should reduce fragmentation. Will look into it...
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
Visual studio gives a list of unreleased memory allocations but it's difficult to pinpoint the source (i.e. memory allocator). What I have looked they appear to be rather small.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,712
Reaction score
1,408
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
Looks like it makes random crashes after running out of memory. Is this a new problem or has it existed for some time ?
Terrain vertex buffers are already recycled, the same space is re-used over and over again. However, that doesn't apply to the tile loading. I suppose those system memory buffers could also be re-used, since most of them are constant size. That should reduce fragmentation. Will look into it...
I don't think this is new

I've seen this type of elevation issue before (since 2022, but I have a vague memory of seeing this in the Beta/2016 versions of the client too (unconfirmed)):
I should capture a video next time. It usually starts out with some distant z-fighting, then the surface gets jumbled like in the screenshot, then you can't see distinct squares very garbled.

It seems to be most pronounced when flying fast and low, so putting a lot of load on the terrain loading/unloading parts of the code. I had always attributed it to the fact that I have slow mechanical drives, and I'd never tried to debug until now.


What moon lvl are you using? Trying to repro this on an x64-debug build but nothing stands out clearly after 70min.
Because of the dynamic tile loading, the memory comes and goes depending on what's on the viewport...
The main memory deltas between snapshots are mostly located around here :
Code:
VERTEX_2TEX *vtx = new VERTEX_2TEX[nvtxbuf];
and
Code:
WORD *idx = new WORD[nidx];
in Tile::CreateMesh_quadpatch
and
Code:
e = new INT16[ndat];
elev = new float[ndat];
in SurfTile::ReadElevationFile
which is to be expected.
Do you have the same behavior?
The heap size looks stable but the process memory increases, maybe it's a memory fragmentation issue?
level 19

It really seems like to bring this problem out, you have to impact performance in another way. try turning up the gravity coefficient cutoff to 165 for the moon, and putting a 5-10 ships in orbit around the moon.
 

Urwumpe

Not funny anymore
Addon Developer
Donator
Joined
Feb 6, 2008
Messages
37,845
Reaction score
2,581
Points
203
Location
Wolfsburg
Preferred Pronouns
Sire
Visual studio gives a list of unreleased memory allocations but it's difficult to pinpoint the source (i.e. memory allocator). What I have looked they appear to be rather small.

Its always easier to handle large allocations in comparison to small allocations, especially, if you need to pin down memory leaks.
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
I wrote a small memory manager and moved all terrain elevation and vertex based allocation under it. There is a PR for testing. No leaks detected. If it looks good then might be good idea to move some other things under it too.

Those surface tile issues in a screen shots are a result of out of memory. No memory to load elevation data.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,712
Reaction score
1,408
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
I wrote a small memory manager and moved all terrain elevation and vertex based allocation under it. There is a PR for testing. No leaks detected. If it looks good then might be good idea to move some other things under it too.

Those surface tile issues in a screen shots are a result of out of memory. No memory to load elevation data.
This appears to have made the issue significantly better...but it's still present.

It's much more resistant to crashing, and even seems to be able to recover (sometimes).

1710632788408.png

1710633100037.png

upon scenario close:
1710633254396.png

Observation:

I noticed this originally because I was testing the gravity models (again) in a way which results in low frame-rates.

The rate at which heap usage rises seems to be inversely proportional to frame-rate. It seems like there is a threshold (~20FPS +/-5FPS) below which memory usage is guaranteed to climb. Does something doesn't have the time to free memory at low frame-rates?

Setting the option "Preload at Session Start" does not fix the issue. (not that I expected it to.) (it might make it take longer to appear, but that could also be a red herring)
1710634891919.png
 
  • Like
Reactions: GLS

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
If that helps then the issue is as Gondos suggested a memory fragmentation. I'll try to pinpoint more places of constant memory allocation/release.
I used the frame-rate limiter to drop down to 20fps. In a few minutes application size climbed from 300M up to 600M held there during 10 minute test session. Does the D3D9 stats show any climb anywhere Ctrl+Shift+Alt+C ?

How does the memory usage climb as function of time, constant from a beginning ? EDIT: Ok, you got the memory table in your last post.
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
Here's my test session.
 

Attachments

  • cs.png
    cs.png
    362.2 KB · Views: 11

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
The extreme drop in a frame rate in the end (in your screen shot) is likely due to lot of errors printed in Orbiter Log due to out of memory condition. 1.7GB huh. Where does it all go... Is the terrain resolution bias at default ? No, it's at max. Hmm.
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
The Forum still appears to down scale attached images. The original was 99kb forums scaled it and raised it to 360kb. It ain't working.
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,712
Reaction score
1,408
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
If that helps then the issue is as Gondos suggested a memory fragmentation. I'll try to pinpoint more places of constant memory allocation/release.
I used the frame-rate limiter to drop down to 20fps. In a few minutes application size climbed from 300M up to 600M held there during 10 minute test session. Does the D3D9 stats show any climb anywhere Ctrl+Shift+Alt+C ?

How does the memory usage climb as function of time, constant from a beginning ? EDIT: Ok, you got the memory table in your last post.
if you want to exactly replicate my test case:

Nonspherical gravity on
Moon.cfg gravity coeff cutoff, change from 10 to 165
Dynamic state propagation: set RK8 to be the only one at all timeatep lengths.
two vessels in 15x15km lunar orbit.
10x-100x time acceleration
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
Over Night Test

I made over night test with the latest fixes in 15x15km lunar orbit with two docked DGs (v-synced 60fps) 10x acc. Memory consumption remained constant about 500MB, data caches remained constant. However, the frame rate had dropped down to 5fps and the vessels were in nan-space. No errors in the logs. Default settings, non-spherical gravity off.

Could the orbit altitude have degraded and caused a collision ending in nan-space ?

C++:
GL-01:DeltaGlider
  STATUS Orbiting Moon
  RPOS -nan(ind) -nan(ind) -nan(ind)
  RVEL -nan(ind) -nan(ind) -nan(ind)
  AROT 0.000 -0.000 0.000
  PRPLEVEL 0:0.553000
  DOCKINFO 0:0,GL-02
  NAVFREQ 0 0 0 0
  XPDR 0
  HOVERHOLD 0 1 0.0000e+00 0.0000e+00
  HUDMode 3
  COMPARTMENT_TEMP nan nan -nan(ind) nan -nan(ind) nan nan nan nan -nan(ind) -nan(ind) nan -nan(ind)
  COOLANT_STATE 0 0.500 287.000
  NOSECONE 1.0000 0.0000
  AAP 0:0 0:0 0:0
END
GL-02:DeltaGlider
  STATUS Orbiting Moon
  RPOS -nan(ind) -nan(ind) -nan(ind)
  RVEL -nan(ind) -nan(ind) -nan(ind)
  AROT 180.000 -0.000 180.000
  PRPLEVEL 0:0.409000 1:1.000000
  DOCKINFO 0:0,GL-01
  NAVFREQ 0 0 0 0
  XPDR 0
  HOVERHOLD 0 1 0.0000e+00 0.0000e+00
  HUDMode 3
  COMPARTMENT_TEMP nan nan -nan(ind) nan -nan(ind) nan nan nan nan -nan(ind) -nan(ind) nan -nan(ind)
  COOLANT_STATE 0 0.500 287.000
  NOSECONE 1.0000 0.0000
  AAP 0:0 0:0 0:0
END
 

n72.75

Move slow and try not to break too much.
Orbiter Contributor
Addon Developer
Tutorial Publisher
Donator
Joined
Mar 21, 2008
Messages
2,712
Reaction score
1,408
Points
128
Location
Saco, ME
Website
mwhume.space
Preferred Pronouns
he/him
Over Night Test

I made over night test with the latest fixes in 15x15km lunar orbit with two docked DGs (v-synced 60fps) 10x acc. Memory consumption remained constant about 500MB, data caches remained constant. However, the frame rate had dropped down to 5fps and the vessels were in nan-space. No errors in the logs. Default settings, non-spherical gravity off.

Could the orbit altitude have degraded and caused a collision ending in nan-space ?

C++:
GL-01:DeltaGlider
  STATUS Orbiting Moon
  RPOS -nan(ind) -nan(ind) -nan(ind)
  RVEL -nan(ind) -nan(ind) -nan(ind)
  AROT 0.000 -0.000 0.000
  PRPLEVEL 0:0.553000
  DOCKINFO 0:0,GL-02
  NAVFREQ 0 0 0 0
  XPDR 0
  HOVERHOLD 0 1 0.0000e+00 0.0000e+00
  HUDMode 3
  COMPARTMENT_TEMP nan nan -nan(ind) nan -nan(ind) nan nan nan nan -nan(ind) -nan(ind) nan -nan(ind)
  COOLANT_STATE 0 0.500 287.000
  NOSECONE 1.0000 0.0000
  AAP 0:0 0:0 0:0
END
GL-02:DeltaGlider
  STATUS Orbiting Moon
  RPOS -nan(ind) -nan(ind) -nan(ind)
  RVEL -nan(ind) -nan(ind) -nan(ind)
  AROT 180.000 -0.000 180.000
  PRPLEVEL 0:0.409000 1:1.000000
  DOCKINFO 0:0,GL-01
  NAVFREQ 0 0 0 0
  XPDR 0
  HOVERHOLD 0 1 0.0000e+00 0.0000e+00
  HUDMode 3
  COMPARTMENT_TEMP nan nan -nan(ind) nan -nan(ind) nan nan nan nan -nan(ind) -nan(ind) nan -nan(ind)
  COOLANT_STATE 0 0.500 287.000
  NOSECONE 1.0000 0.0000
  AAP 0:0 0:0 0:0
END

If you had nonspherical gravity on, I'd say that's definitely a possibility, but with it off I wouldn't expect that. even at 5fps the propagators, substeps, and orbit stabilization should be able to keep up with that, I'd think.
 

jarmonik

Well-known member
Orbiter Contributor
Addon Developer
Beta Tester
Joined
Mar 28, 2008
Messages
2,670
Reaction score
800
Points
128
if you want to exactly replicate my test case:
Nonspherical gravity on
Moon.cfg gravity coeff cutoff, change from 10 to 165
Dynamic state propagation: set RK8 to be the only one at all timeatep lengths.
two vessels in 15x15km lunar orbit.
10x-100x time acceleration

I ran a test with these configurations. Frame rate was about 2.1 fps. There is a small constant increase of system memory consumption.
High-res textures installed, 90deg fov down to moon. Resolution bias at max.

In your case how does the memory climb to 1.7G ? Could you give a plot from application start ?

Does the DirectX9 run natively or via wrapper. If yes then DXVK could be tested.
 

Attachments

  • mem.png
    mem.png
    174.7 KB · Views: 12
Top