Question Time acceleration management

[Boxx]

For the development of OMX (fork of Orbiter Multiplayer), I am creating a feature to restore a previous state: I unlocked the acceleration rate that was developed by @Face & fellows for OMP to start a scenario in the past and re-propagate until real time with time accelerations higher than x2... up to x1500. Sometimes, not always, Orbiter freezes. I could isolate the problem (bug?) and I'm pretty sure the problem comes from Orbiter's core, not from OMP, but no error message, nowhere.

Hence, my question: are there requirements on the time acceleration and its management from an addon's code? This topic may be related to a past discussion here

Tips:

• for those familiar with OMP code:
  • the thread "DWORD WINAPI statustimer(DWORD sd)" is responsible for the freezing: if I skip the loop based on the acceleration rate, like

    while (threaddata->activation)    {
            if (oapiGetTimeAcceleration() > 2) continue;...

    , the code works smoothly (accelerating and decelarating automagically until real-time) but Orbiter freezes as soon as the loop executes (acceleration < x2)
  • OMP includes a function "get_clock_acceleration(...)" that computes an acceleration, based on a PID control loop, with Clock control parameter: Kp=1 Ki=0.5 Kd=0 Eps=0.5 -values by default) => @Face and fellows, were you motivated by some freezing already at the time? Would adjusting parameters be key?
• When decelerating to real time, if I "help" Orbiter by continuously pressing the "R" key (acc.decrease), time-acceleration keeps decreasing very progressively and Orbiter smoothly decelerates and adjusts to real time, then everything (including the connection to OMX server) goes back fine. Which makes me think that something in the cores' time-step adjustment is impacted by my code.

In the meantime, the restore feature, which is key, is not operational (admin is requested to press "R" during restore!!!)... too bad, really. Thx for any help!

 

[Face]

the thread "DWORD WINAPI statustimer(DWORD sd)" is responsible for the freezing: if I skip the loop based on the acceleration rate, like

while (threaddata->activation)    {
        if (oapiGetTimeAcceleration() > 2) continue;...

, the code works smoothly (accelerating and decelarating automagically until real-time) but Orbiter freezes as soon as the loop executes (acceleration < x2)

If everything works smoothly without that loop, why do you keep it at all?

OMP includes a function "get_clock_acceleration(...)" that computes an acceleration, based on a PID control loop, with Clock control parameter: Kp=1 Ki=0.5 Kd=0 Eps=0.5 -values by default) => @Face and fellows, were you motivated by some freezing already at the time? Would adjusting parameters be key?

OMP always had a 2 stage system for MJD sync: if your time was behind the server and the acceleration was commanded manually to be 10x, 100x, 1000x and so on, the system simply reduced the acceleration by a P-based limiter, meaning the lower the offset, the lower the allowed acceleration. It was no control loop, because if - for whatever reason - the server accelerates again, the system would not follow by means of increasing acceleration again.
The second stage was a digital PID loop, with the parameters you posted. The idea was to keep the acceleration fine-tuned in order to keep the client/server MJD offset as small as possible. The parameters in the default distro should be good enough for starters, but of course could be optimized depending on machine and/or server behavior.

I never had freezes in this system, maybe your changes removed the lock vs. the main Orbiter thread, so the oapi-calls are out of sync. Keep in mind that the Orbiter API is not thread-safe.

 

[Boxx]

At first thank you for your fast reaction!

If everything works smoothly without that loop, why do you keep it at all?

Its content does things that seem to overload other tasks, but I can't say what. (e.g. at start, or GINFO <id>"N..., GINFO <id>"D..., that make the server send a REQST... that are needed...)

Keep in mind that the Orbiter API is not thread-safe.

That's a point. I'm thinking at the many clbk... But then, you give me an idea: if I can locate what makes Orbiter freeze (i.e. accessing same variables from the thread and the core), I could encapsulate the lines within a critical section (&orbiter)... (or maybe some encapsulation in statustimer makes Orbiter freeze? if the core waits for something or loops indefinitely)

Regarding the smoothing with get_clock_acceleration, I tried a lot of combinations (other control laws...) and none of them seem to improve or remove the issue. I was hoping that some time-step management was known to be sensitive.....

 

[Face]

Well, in OMPU (the successor to OMP), I've moved the MJD sync to the main loop, anyway. In the PreStep callback, I do something like this:

double acc = oapiGetTimeAcceleration();
double acceleration = wrapper.GetSyncingAcceleration(SimDT, mjd, acc);
if (acc != acceleration)
{
	oapiSetTimeAcceleration(acceleration);	
}

The wrapper object is the link to .NET code, and the appropriate method does this:

        private double integral = 0;
        private double e1 = 0;
        private double Kp = 1;
        private double Ki = 1;
        private double Kd = 0;
        public double GetSyncingAcceleration(double SimDT, double mjd, double acc)
        {
            this.mjd = mjd;
            double mAcc;
            double mMjd;
            long mTS;
            double freeStart;
            lock (mainBubble)
            {
                mAcc = mainBubble.a;
                mMjd = mainBubble.m;
                mTS = mainBubble.t;
                freeStart = mainBubble.FreeTimerStart;
            }
            double acceleration = acc / Math.Max(mAcc,0.001);
            double currentTime = ntpClient.Now.ToDouble;
            double e = ((mjd - mMjd) * 86400 - ((long) (currentTime * FixedFrequency) - mTS) / FixedFrequency * mAcc) / mAcc;
            if (currentTime - freeStart > 2 && Mode!=BubbleMode.Free)
            {
                if (acceleration <= 10 || e > 0)
                {
                    integral += e * SimDT;
                    if (integral > 1) integral = 1;
                    if (integral < -1) integral = -1;
                    acceleration = -Kp * e - Ki * integral - Kd / SimDT * (e - e1);
                    if (acceleration < 0.001) acceleration = 0.001;
                    if (acceleration > 10) acceleration = 10.0;
                }
                else if (e > -acceleration) acceleration = -e;
                e1 = e;
                if (acceleration < 1.05 && acceleration > 0.95) acceleration = 1;
            }
            acceleration *= mAcc;
            //IntPtr text = Marshal.StringToHGlobalAnsi($"Off:{e:F3}s Acc:{mAcc:F1} Skew:{ntpClient.Skew * 1E6:F2}µs/s");
            //exampleDelegate(1, text);
            //Marshal.FreeHGlobal(text);
            if ((acceleration - acc) / SimDT > 10) // positive Jerk reduction
            {
                acceleration = acc + SimDT * 10;
            }
            return acceleration;
        }

As you can see, nothing changed much there, "e" is still the MJD difference between client and server. The difference is that this system is now taking the bubble-idea into account, so a client can switch space-time bubbles, and a bubble can have a different acceleration than x1. In fact the server bubble in OMPU is always at x20, so a real-time client will always be behind the "universe".

 

[Boxx]

Well, in OMPU (the successor to OMP), I've moved the MJD sync to the main loop, anyway. In the PreStep callback, I do something like this:

I understand and, indeed, I think having the conflicting code (whatever it is) in PreStep callback should secure access to shared variables with Orbiter's core, at a moderate cost in terms of multi-treading.

I also see that you changed the PID loop parameters. I haven't dived into their interpretation yet. Although I am not supporting the space-time bubble concept for OMX (sorry for that), I see that it made you develop a nice approach to sync varying times. I want to look at these parameters more carefully.

 

[n72.75]

Would having better controls/custom limits in Orbiter be useful? If you're checking and setting the time aceleration in clbkPreStep you can run into the issue where you run one timestep at a high aceleration before it gets reset back to what you're code is trying to control....idea for a future release, not O2024.

 

[Boxx]

Would having better controls/custom limits in Orbiter be useful? If you're checking and setting the time aceleration in clbkPreStep you can run into the issue where you run one timestep at a high aceleration before it gets reset back to what you're code is trying to control....idea for a future release, not O2024.

I think definitely yes: for instance, PursuitMFD also warns about acceleration higher than 100x (crashes? I don't know).

Orbiter aims at reproducing a system behavior in an environment via physical laws that must be computed quick enough (or would imply an "acceptable" accumulation of error). Then:

• either it is possible to switch to a simplified control at high rates (maybe addon-specific): the physical values are directly controlled, e.g. the pointing direction, instead of controlling the system (RCS thrusters, in the example, that attempt to modify the pointing)
• or the acceleration rate should be self-limited by the ability to compute the effects (reactions of RCS on the pointing) or by an "acceptable" accumulated error.

For OMX, I would likely prefer the 2nd option as the acceleration is only used to accelerate from a backup status in the past to the re-synchronization in real-time.

 

[Boxx]

I understood where is the bug, but not what kind of bug it is. Actually, the root cause is not the time acceleration. Orbiter freezes at a "LeaveCriticalSection" (of the thread statustimer), but not always which makes the bug hard to catch. Time acceleration makes it easier to have collisions between threads and the core.



Analyzing the thread spider, I realized there isn't only 1, but multiple threads called orbiter.exe (after the simu was started), then many more regarding the C++ libraries (msvcrt...) and, for OMX, 6 threads (statustimer, receiverthread for TCP, synccontrol, hostsendthread and hostrecvthread for UDP, garbagecollector). Surprisingly, Orbiter loops indefinitely at the exit of a C++ Critical Section, not at the entry... then, after some time, it freezes eventually. That may be inspirational for O2024.... at least to prevent from infinitely looping or logg where it happened if so.

For the time being (and for specialists of OMP), I disabled the TCP-sending of the NAV frequencies and DOCK status that can be transferred another way (indeed, they trigger a reply by the server that may explain some collisions with the TCP-thread, but why a collision with orbiter core?... don't know). It runs smoothly now, ready for additional feature very soon