The Future of Lua with Orbiter? Is it at a Dead-End?

[Thunder Chicken]

While I have been in the Orbiter community since the M6 days, and Lua scripting has been a feature since at least Orbiter 2010 if not earlier, I've only recently started tinkering with it in Orbiter 2016 within the last year to make add-ons as it is easier to work than trying to setup C++ on a Linux machine. As Orbiter is now open source, I am wondering if any further development of Lua scripting is being done? Is Lua at a dead-end in Orbiter? Lua does seem to be an excellent way to lower the barrier to entry to add-on development and potentially prevent a lot of add-ons from dying when their C++ .dlls become obsolete in newer versions.

Lua seems to have quite a lot of the functionality of the C++ API. But in my scripting efforts I've been stymied by the following issues which really challenge the use of Lua scripting as a fully viable and robust means to make add-ons (FYI I am using Orbiter 2016 on Linux). I offer these as a running punch-list for anyone who may be working on Lua scripting development:

1. No ability to save or retrieve MFD states in the scenario file. There are MFD callbacks readstatus(scn) and writestatus(scn) which are documented in the Orbiter Help documentation, but there are no examples of their use, and there apparently is no way to actually read or write anything to the scenario file within them. [Github Issue #404]
2. No ability to save or retrieve vessel states in the scenario file. There don't seem to be any vessel callbacks to allow this at all in Lua. It is possible to read and write using standard Lua I/O, but that rather breaks the click-and-go functionality of the scenario files.[Github Issue #405]
3. Need for absolute file paths for loading Lua modules, accessing Orbiter data files, etc. There is a vessel helper function add_mesh to load mesh files which can use relative paths from the main Orbiter Mesh folder, but for anything else an absolute path is needed. I understand that this is a general "feature" of Lua I/O, but it can complicate making a general add-on that would work for anyone. A helper function that would allow relative paths from the main Orbiter folder for general I/O would be very helpful.
4. No analog of callback DrawHud is available in Lua. Text can be written to the HUD using the annotation method, but custom indicators like the DG landing gear status indicators are not possible.
5. Keypress detection in vessel scripts (to activate landing gear, canopy openings, other vessel functions) requires the following additional module and procedure to be implemented in Lua. https://www.orbiter-forum.com/threads/trying-to-implement-keypress-detection-in-lua-solved.41000/ It works, but something more organic like the keypress callbacks in the C++ API would be helpful.

It may be that there are certain technical limitations in Lua that prevent resolution of these issues and further development of Lua scripting in Orbiter, but many in the community would like to know whether that is indeed the case or not.

 

[Urwumpe]

I would say, 1 & 2 depend on the respective generic C++ module you use. There is no standard way to do it.

3 could maybe be adressed.

4 would need to be tested, because Lua is much slower than C++. Maybe a separate display description language would be better.

 

[Thunder Chicken]

I would say, 1 & 2 depend on the respective generic C++ module you use. There is no standard way to do it.

Could you explain what you mean by "respective generic C++ module you use"? Are you referring to the Lua interpreter?

4 would need to be tested, because Lua is much slower than C++. Maybe a separate display description language would be better.

Unsure if it is related, but it seems that the Sketchpad MFD update function is slow as well. It rather seems faster to update text display in the pre- or poststep callbacks. Managing menu trees and pages in the MFDs seems much harder than necessary.

 

[Urwumpe]

Well, LuaVessel and LuaMFD are both C++ modules which just delegate the event handling to Lua. Loading would need to be delegated to Lua as well, but there could not be a 1:1 translation from C++ to Lua.

 

[Thunder Chicken]

Well, LuaVessel and LuaMFD are both C++ modules which just delegate the event handling to Lua. Loading would need to be delegated to Lua as well, but there could not be a 1:1 translation from C++ to Lua.

As a user I still don't quite understand the importance of this. Could you provide an example in user-speak?

 

[jedidia]

The problem I see is that Lua is a dead end in any case in an environment where you can plug in c++ code.
Lua is fine as long as Lua is all you have. Lua is also fine for small, not too complex things, which it can mostly do, I think. Undoubtedly that could be a bit extended.
But at the end of the day, when you're getting complexity, you want to work in C++ (Even if you don't know it yet). Static typing, compile-time syntax checking, and being able to hook in a debugger are indispensible tools once your code starts getting complex, and Lua offers none of them.

So the question really is: Up to what degree of complexity is Lua actually more convenient than setting up visual studio, and does it have the necessary support up to that point? Because anything beyond that point is not only wasted effort, it's an outright trap for add-on developers.

 

[Urwumpe]

As a user I still don't quite understand the importance of this. Could you provide an example in user-speak?

I am not fluent in user-speak, but I can try.

Take a look at the Deltaglider:

// --------------------------------------------------------------
// Read status from scenario file
// --------------------------------------------------------------
void DeltaGlider::clbkLoadStateEx (FILEHANDLE scn, void *vs)
{
    char *line;

    while (oapiReadScenario_nextline (scn, line)) {
        if (!_strnicmp (line, "TANKCONFIG", 10)) {
            if (ssys_scram) sscanf (line+10, "%d", &tankconfig);
        } else if (!_strnicmp (line, "PSNGR", 5)) {
            DWORD i, res, pi[4];
            res = sscanf (line+5, "%d%d%d%d", pi+0, pi+1, pi+2, pi+3);
            for (i = 0; i < res; i++)
                if (pi[i]-1 < 4) psngr[pi[i]-1] = true;
        } else if (!_strnicmp (line, "SKIN", 4)) {
            sscanf (line+4, "%s", skinpath);
            char fname[256];
            strcpy (fname, "DG\\Skins\\");
            strcat (fname, skinpath);
            int n = strlen(fname); fname[n++] = '\\';
            strcpy (fname+n, "dgmk4_1.dds");  skin[0] = oapiLoadTexture (fname);
            strcpy (fname+n, ssys_scram ? "dgmk4_2.dds" : "dgmk4_2_ns.dds");  skin[1] = oapiLoadTexture (fname);
            strcpy (fname+n, "idpanel1.dds"); skin[2] = oapiLoadTexture (fname);
            if (skin[2]) {
                oapiBlt (insignia_tex, skin[2], 0, 0, 0, 0, 256, 256);
                oapiReleaseTexture (skin[2]);
                skin[2] = NULL;
            }
        } else if (!_strnicmp (line, "PANELCOL", 8)) {
            sscanf (line+8, "%d", &panelcol);
        } else {
            // offer the line to all subsystems
            if (!ComponentVessel::clbkParseScenarioLine (line))
                // unrecognised option - pass to Orbiter's generic parser
                ParseScenarioLineEx (line, vs);
        }
    }
}

We would need to delegate this outer for-loop to Lua, if we want to do it 1:1. Not really attractive and I doubt it can work fine.

Also, we would need to go through all these if-else-statements, which could be done better.

The scenario file format is not really standardized, but generally has to fit into a 255 character long line (i think).

 

[Urwumpe]

The problem I see is that Lua is a dead end in any case in an environment where you can plug in c++ code.
Lua is fine as long as Lua is all you have. Lua is also fine for small, not too complex things, which it can mostly do, I think. Undoubtedly that could be a bit extended.
But at the end of the day, when you're getting complexity, you want to work in C++ (Even if you don't know it yet). Static typing, compile-time syntax checking, and being able to hook in a debugger are indispensible tools once your code starts getting complex, and Lua offers none of them.

So the question really is: Up to what degree of complexity is Lua actually more convenient than setting up visual studio, and does it have the necessary support up to that point? Because anything beyond that point is not only wasted effort, it's an outright trap for add-on developers.

DCS also uses Lua for extending scenarios and AI, so I think we could at least do the same. You are right about anything we want to be done fast. But then, having something like Lua for faster prototyping is also not too bad.

 

[Thunder Chicken]

The problem I see is that Lua is a dead end in any case in an environment where you can plug in c++ code.
Lua is fine as long as Lua is all you have. Lua is also fine for small, not too complex things, which it can mostly do, I think. Undoubtedly that could be a bit extended. But at the end of the day, when you're getting complexity, you want to work in C++ (Even if you don't know it yet). Static typing, compile-time syntax checking, and being able to hook in a debugger are indispensible tools once your code starts getting complex, and Lua offers none of them.

I don't think this is a compelling argument. I have my full air-breathing jet engine code converted into Lua, with multiple modules with several thousand lines of code, with its associated vessel add-on, also in Lua. It works as well as the C++ variant. I have an autopilot script with four different autopilot functions and subfunctions with about a thousand lines of code that works well. Perhaps if you are coding something like SSU your argument might be valid, but for enthusiastic beginners Lua scripting has plenty of capability.

I frankly struggled to work in VS Studio and the debugger tools were Greek to me, and every time I put down a project and picked it up again I had to remember what to do with it. A lot of us are amateurs and aren't professional coders and don't code frequently. A lot of the tools and features in the IDE that might help one code an add-on are simply more arcane knowledge that folks would have to learn and remember in addition to sorting out the Orbiter API just to get started. Contrast that to just re-opening a text file and making tweaks it is much easier to pick up projects after an absence.

Years ago when I learned programming as an engineering student we didn't have IDEs. We had to logic through things and use comments and print statements and all the basic tricks to sort out what is going on with the code. Those methods still work.

So the question really is: Up to what degree of complexity is Lua actually more convenient than setting up visual studio, and does it have the necessary support up to that point? Because anything beyond that point is not only wasted effort, it's an outright trap for add-on developers.

For many just getting a compiler setup is the biggest nuisance. To be able to open up a text editor and start coding, and to be able to edit the code and immediately test it without having to recompile is refreshing. Lua basically doesn't beat the enthusiasm out of you when you just want to do something fun in Orbiter.

 

[Thunder Chicken]

I am not fluent in user-speak, but I can try.

Take a look at the Deltaglider:

// --------------------------------------------------------------
// Read status from scenario file
// --------------------------------------------------------------
void DeltaGlider::clbkLoadStateEx (FILEHANDLE scn, void *vs)
{
    char *line;

    while (oapiReadScenario_nextline (scn, line)) {
        if (!_strnicmp (line, "TANKCONFIG", 10)) {
            if (ssys_scram) sscanf (line+10, "%d", &tankconfig);
        } else if (!_strnicmp (line, "PSNGR", 5)) {
            DWORD i, res, pi[4];
            res = sscanf (line+5, "%d%d%d%d", pi+0, pi+1, pi+2, pi+3);
            for (i = 0; i < res; i++)
                if (pi[i]-1 < 4) psngr[pi[i]-1] = true;
        } else if (!_strnicmp (line, "SKIN", 4)) {
            sscanf (line+4, "%s", skinpath);
            char fname[256];
            strcpy (fname, "DG\\Skins\\");
            strcat (fname, skinpath);
            int n = strlen(fname); fname[n++] = '\\';
            strcpy (fname+n, "dgmk4_1.dds");  skin[0] = oapiLoadTexture (fname);
            strcpy (fname+n, ssys_scram ? "dgmk4_2.dds" : "dgmk4_2_ns.dds");  skin[1] = oapiLoadTexture (fname);
            strcpy (fname+n, "idpanel1.dds"); skin[2] = oapiLoadTexture (fname);
            if (skin[2]) {
                oapiBlt (insignia_tex, skin[2], 0, 0, 0, 0, 256, 256);
                oapiReleaseTexture (skin[2]);
                skin[2] = NULL;
            }
        } else if (!_strnicmp (line, "PANELCOL", 8)) {
            sscanf (line+8, "%d", &panelcol);
        } else {
            // offer the line to all subsystems
            if (!ComponentVessel::clbkParseScenarioLine (line))
                // unrecognised option - pass to Orbiter's generic parser
                ParseScenarioLineEx (line, vs);
        }
    }
}

We would need to delegate this outer for-loop to Lua, if we want to do it 1:1. Not really attractive and I doubt it can work fine.

I am still not seeing what the complication is. A complete 1:1 match to C++ callbacks isn't necessary - we just need equivalent functionality. All that is missing in Lua is a way to get the scenario file name that was used to initiate the simulation session. If that can be done, then a read function can be written in pure Lua. If the handle for the scenario file can be properly passed to Lua, the scenario file could potentially be read using only Lua code at the start of the simulation using a run-once flag in the pre-step callback. If simt == 0, read stuff from the scenario file.

I don't know if a similar write could happen automatically at the end of the session, but there the file handling is easier - it gets written to (Current State).scn. What is needed there is a write function that runs last when the simulation is triggered to end.

Also, we would need to go through all these if-else-statements, which could be done better.

There is always a better way. Much of the original Orbiter code could probably be updated with more modern methods available.

The scenario file format is not really standardized, but generally has to fit into a 255 character long line (i think).

It's basically just a config file. As long as it has an expected syntax and both Lua and C++ methods honor it there shouldn't be a problem? I mean, I can get Lua to read and write from the scenario file right now - the only problem is that it can't be done automatically at the start and end of the simulation session.

 

[Thunder Chicken]

Earlier today I posted this in the Orbiter SDK thread on an old post about the problems with the readstatus(scn) and writestatus(scn) methods in Lua, and I did some experiments with the interpretation of the scn handle:

LUA - Store and reload MFD

I want to store a value in my Lua-based MFD when the user exits Orbiter and restore it when the next Orbiter session is started. But I don't understand how the MFD callback functions writestatus(scn) readstatus(scn) are supposed to be used. Do I have to define both functions in my MFD's .cfg...

www.orbiter-forum.com

Is the scenario file path obtainable through the scn handle in any way that I am missing? I don't think so, but I'll ask anyway.

These methods (at least their intended capabilities) were to allow read/write access to the initiating scenario file. They are not identical to the state callback functions, but as I mentioned previously, they don't need to be - they just need to allow us a way to a similar functionality. A function isn't even really necessary - all that is minimally needed is a method for Orbiter to pass the scenario file name to the Lua script or make it available in some way. That's it.

In a forlorn hope, I went rummaging through the Orbiter.log to see if it recorded the scenario file name. It does not, but that illustrates how little is needed to get a toe-hold on that functionality.

 

[n72.75]

I'm fairly certian that LUA in Orbiter, is older than this forum. It wasn't in O2006 iirc, but it was definitely in the betas leading up to 2010. So some of the original discussions about it are probably lost to the sands or time in an old M6 thread.

I seem to remember though, that the purpose of Lua originally was to give users the ability to write scripts for vessels that could interpret commands (e.g. writing custom launch autopilots for a rocket, new dance moves for your manipulator arm, flipping switches, etc)

Question - Raising landing gear via API

I'm using Lua scripting to write a general purpose autopilot. I've looked everywhere in the Orbiter API doc for how to raise the landing gear. I must be missing it somewhere. Does anyone know how to raise the landing gear using the regular Orbiter API. From that I can apply it the Lua...

www.orbiter-forum.com

I've never used it because I've never been clear on why I should. And it seems to me like the options for making orbiter addons without using C++ have been quite successful (SC3,4 etc, similar modules)

NASSP has Lua support for setting/getting switch states: https://github.com/orbiternassp/NAS...amples/ProjectApollo/src_csm/saturn.cpp#L5190



That's the type of stuff that I see Lua being the most useful for.

 

[jedidia]

For many just getting a compiler setup is the biggest nuisance.

Absolutely. It's the same way for me, as I don't work with visual studio often (essentially never these days when I don't write orbiter add-ons...). But at some point of complexity, it just simply becomes worth it.

 

[Thunder Chicken]

Absolutely. It's the same way for me, as I don't work with visual studio often (essentially never these days when I don't write orbiter add-ons...). But at some point of complexity, it just simply becomes worth it.

That's fine. But this is a personal judgement. My take is that Lua can be used to make complete add-ons of astonishing complexity. It has plenty of capability for novice Orbiteers just starting out with programming and add-on development who aren't at a point to use or appreciate a full IDE. A text editor is all that is strictly needed. I use VS Code with a Lua add-on that does basic syntax checking which is lightweight, easy to setup on Windows or Linux, and works out of the box without having to configure an IDE.

 

[jedidia]

My take is that Lua can be used to make complete add-ons of astonishing complexity. It has plenty of capability for novice Orbiteers just starting out with programming and add-on development who aren't at a point to use or appreciate a full IDE

And I do agree with that. Hence the question, to which extent should its capabilities go? Because that's in the end what we're talking about with "dead ends". Unless the full API is supported, you'll always reach a point where you hit a wall with LUA and need to move to C++. That's the dead end. In my opinion, that dead end should not be too far out, because you'll always hit it eventually, and you don't want to have too much effort invested when that point is reached (i.e. I'd rather have people run into an API wall than them eventually realising that their project has become an unmanageable mess).

Whether the dead end is currently in a good place or not I can't really judge, as I've never worked with Lua in orbiter. And there's no doubt that some documentation about the limits would be helpful. But I was really mostly directly addressing your question "is Lua a dead end", which in my opinion it is and always will be, and even should be.

 

[N_Molson]

I'd say that the point with LUA is to have a "plug-and-play" solution somewhere between editing .cfg files and going through VS much dreaded setup.

 

[n72.75]

I will agree that the hardest part of Orbiter addon development is setting up VS. Especially because every year they come out with a new version and the project formats change, so there's a brand new "how to set up a project that you need to know.

I think both Lua and C++ would benefit from some tutorials that are a bit newer than 2005.

 

[kuddel]

Just to add my thoughts abaout this:

1. Do I think Lua is a dead end? No, but I think it is just for basic interaction. Just like BASIC is a very fine language for initial contact with a computer/progrmming-language, it still is not good enough to do all your needs.
2. The problem with Lua is the burden it loads to the add-on developer. Having implemented several API methods for the Orbiter core, I have to say: It's a mess!
   And if some special funtions of an add-on should be available in Lua, the developer needs to step into this mess. Or a standardized data-input-output system needs to be created and that sounds a lot like "too generic". And, still the add-on developer has to do his/her work to comply to that interface anyhow.
   Even if many add-on developers would do this, there will be still some that can't. And there's another "dead end".
3. Lua does not bring enough benefit over C++. If any. Learning C++ is not harder than learning Lua! Setting up a development environment to create an add-on isn't that hard either. Sure, a better documentation/tutorial is always possible, but setting up Visual Studio, changing some parameters in there is not that hard once you understand what you do. The little differences from (Visual Studio) version to version are annoying -yes-, but not something that should hinder anyone getting it done.
4. I am still hoping someone with interest in Lua and willingness to implement the interfaces (in C++ that is) can be found to add some of the things that are worth it.

To be clear: I am not against Lua, I just think getting anything done in the native language (C++) of Orbiter is far better invested time.

 

[Thunder Chicken]

And I do agree with that. Hence the question, to which extent should its capabilities go? Because that's in the end what we're talking about with "dead ends". Unless the full API is supported, you'll always reach a point where you hit a wall with LUA and need to move to C++. That's the dead end. In my opinion, that dead end should not be too far out, because you'll always hit it eventually, and you don't want to have too much effort invested when that point is reached (i.e. I'd rather have people run into an API wall than them eventually realising that their project has become an unmanageable mess).

As far as I can tell, this is the status of vessel API functions that Lua supports, which really is very sufficient with one or two exceptions:

Orbiter BETA Lua development

Finally here's the Lua (sub-)Forum! My current progress and questions regarding the Orbiter Lua API will be posted here. I will update this 1st post with the current state, as soon as it changed. New items (since last edit of this post) will be marked green The attached file represents the...

www.orbiter-forum.com

It obviously doesn't bother with methods deprecated in the Orbiter API. For whatever reason, it is missing some functions for exhaust stream rendering which are just eye-candy, and it is missing the capabilities to read and write to/from the scenario file. If they could be resolved I think Lua could be considered complete.

Lua is really close to being a relatively complete alternative to the C++ API. And it can be used to extend the capabilities of existing vessels. The Lua script documentation in the Help console and in the docs is pretty good.

Whether the dead end is currently in a good place or not I can't really judge, as I've never worked with Lua in orbiter. And there's no doubt that some documentation about the limits would be helpful. But I was really mostly directly addressing your question "is Lua a dead end", which in my opinion it is and always will be, and even should be.

You've never worked with Lua in Orbiter, but pronounce that it is and should be a dead end? Really?

I've made add-ons both in C++ and in Lua, and I tell you Lua is a much easier environment to get started playing with Orbiter add-ons for folks who don't program frequently. I'd like the folks that come to Orbiter out of a deep interest in space flight and who might consider trying to program an add-on for their own interest to have a low-barrier-to-entry means of starting out.

The only compelling arguments I would see in favor of C++ methods might be computational efficiency, and the ability to keep add-ons closed source. If you are just mucking around with add-ons for your own amusement, or aren't really concerned about sharing your code or licensing issues, Lua works great.

 

[jedidia]

You've never worked with Lua in Orbiter, but pronounce that it is and should be a dead end? Really?

I think I sufficiently explained what exactly I understand under "dead end", and why I think it's a feature, not a bug. So yes, really.
I have worked with LUA in other contexts, and I am working with other weakly typed languages like python and javascript in my day job (not too often, thankfully), and the result is always the same: Once things start getting complex, you really wish you were using a strongly-typed language with strict compile-time checking. Even a weakly typed language that has compile-time checking screwed onto it, like typescript, still has an annoying amount of pitfalls and hard to find bugs because the interpreter the result runs in just doesn't know what types are and does unexpected things. And in most of these cases I at least have a decent debugger, which in this case we don't.

Lua is a much easier environment to get started playing with Orbiter add-ons for folks who don't program frequently. I'd like the folks that come to Orbiter out of a deep interest in space flight and who might consider trying to program an add-on for their own interest to have a low-barrier-to-entry means of starting out.

As mentioned, I completely agree with and reaffirm these sentiments.

The only compelling arguments I would see in favor of C++ methods might be computational efficiency, and the ability to keep add-ons closed source.

The first one is compelling only for specific things, and there it comes down to algorithmic efficiency much more than it comes down to the additional drag of the code running in a VM, unless you have something extremely specific. In all other situations, comprehensible code beats efficiency every time. The second argument isn't compelling to me at all...