The Future of Lua with Orbiter? Is it at a Dead-End?

[kuddel]

It's not crazy, noone has needed this "feature" since.
There you see how many acually use Lua. The reason this is not perfect is because noone seems to check and as Orbiter is a complete hobby project (nobody gets any money for it) it's kind or rude what you say.

By the way you can help make things better! You are requesting from others to fulfill your needs, so please be so kind and calm down.
I have absolutely no interest in Lua, still I am trying to debug your issue.

Having the FILEHANDLE as it is in the C++ Orbiter API and having to use it with the other (not yet implemented) functions is good, as it is documented already!
The behaviour is well understood by many C++ add-on developers.
If you like to have another API, go and make it, but be sure to make it 100% and well documented, else someone might think it is "[...]absolutely unusable without helper functions that don't exist and haven't been developed in over 10 years[...]".

Don't get me wrong, I still like to help you out and improve the Lua interface for you, but please be patient.

 

[Thunder Chicken]

It's not crazy, noone has needed this "feature" since.
There you see how many acually use Lua. The reason this is not perfect is because noone seems to check and as Orbiter is a complete hobby project (nobody gets any money for it) it's kind or rude what you say.
By the way you can help make things better! You are requesting from others to fulfill your needs, so please be so kind and calm down.
I have absolutely no interest in Lua, still I am trying to debug your issue.
Having the FILEHANDLE as it is in the C++ Orbiter API and having to use it with the other (not yet implemented) functions is good, as it is documented already!
The behaviour is well understood by many C++ add-on developers.
If you like to have another API, go and make it, but be sure to make it 100% and well documented, else someone might think it is "[...]absolutely unusable without helper functions that don't exist and haven't been developed in over 10 years[...]".

Don't get me wrong, I still like to help you out and improve the Lua interface for you, but please be patient.

I think you are deeply misinterpreting the tone of my posts and are getting offended for no reason. I don't see what you are interpreting as rude. That certainly was not my intent. I am not knocking you or other developers. I am asking lots of questions and looking at the history of Lua in Orbiter trying to sort out why things are the way they are. I see significant potential for Lua going forward and am trying to understand what issues need to be resolved from a user standpoint to make it more usable. It's not even clear to many of us if there are technical matters that make these issues impossible to surmount or not. That is all I am trying to understand.

I would suggest that the causality of the lack of Lua use is in the other direction - Lua is not being used because it is still missing some key functionality that was determined by the community over 10 years ago. I simply want to know if there are issues that prevent any further progress.

 

[kuddel]

O.K. now it's your turn to help me

I have two options to implement the following functions: readscenario_nextline, readitem_string, readitem_float, readitem_int, readitem_bool and readitem_vec
I can implement them to either return one value or two. What is the best way a Lua developer might expect them to be:

• a) Return one value
  Return (string/float/int/bool/vector) value if tag was found in the file, nil if not.
• b) Return two values
  Returning one boolean to indicate success/failure and
  return a second value containing the value (if successful)
  (...or the other way around if it is preferred)

v = readitem_float(scn, "foo")
if v ~= nil then
    -- ...access v
end

vs.

b,v = readitem_float(scn, "foo")
if b then
    -- ...access v
end

What would be the option a regular Lua developer would expect? Or what is considered "best practice"?

Functions returning boolean values might be a thing to think about, although nil, true and false are distinguishable.

 

[kuddel]

By the way, this question goes out to all Lua developers

 

[Gondos]

I'm no lua guru but I think returning nil on error should be enough. It plays nice with the common idiom used to do default initialisation (except if value is a bool of course)

value = value or default_value

Going a bit further you could propose an optional default value to return in case the item is not found (nil being the default value of the default value).

v = readitem_float(scn, "foo", 3.1416)

 

[Thunder Chicken]

O.K. now it's your turn to help me

I have two options to implement the following functions: readscenario_nextline, readitem_string, readitem_float, readitem_int, readitem_bool and readitem_vec
I can implement them to either return one value or two. What is the best way a Lua developer might expect them to be:

• a) Return one value
  Return (string/float/int/bool/vector) value if tag was found in the file, nil if not.
• b) Return two values
  Returning one boolean to indicate success/failure and
  return a second value containing the value (if successful)
  (...or the other way around if it is preferred)

v = readitem_float(scn, "foo")
if v ~= nil then
    -- ...access v
end

vs.

b,v = readitem_float(scn, "foo")
if b then
    -- ...access v
end

What would be the option a regular Lua developer would expect? Or what is considered "best practice"?

Functions returning boolean values might be a thing to think about, although nil, true and false are distinguishable.

Something that might be simpler and more general might be to just make one single function that passes the entire contents of the scenario file as an indexed table of lines as strings to Lua and let the user do the parsing in Lua on the receiving end. No worries about future changes to Orbiter or Lua typing standards that might break compatibility. Note that the version of Lua currently used in Orbiter only supports the number type, while newer versions support float and integer types. Iterating through the table using pairs() and using Lua string functions to parse the necessary information isn't too hard. Having open source Lua function snippets to automate this for inexperienced add-on developers could be done.

EDIT: Such a table could be updated during the simulation, and then when the scenario is closed, a function that takes this table and writes it back out into (Current status).scn would close the loop.

 

[kuddel]

Hold on, I am only implementing plain eqivalents to the OrbiterAPI.
Special Lua things only create special problems. Like there is no-one to help, as this would be very very Lua specific!
The idea might be tempting, but please please please don't make it more complicated.
Having a 1 to 1 relation between Lua and C++ interface is the way Martin decided to do it and I think it's the best way to go. Even people without Lua knowlwdge can help others if the general usage pattern of the API is equal.

If you like to change it and add special features, give us source-code (including documentation) - I am not a Lua guy!
Would you be so kind and answer the question?

 

[kuddel]

I'm no lua guru but I think returning nil on error should be enough. It plays nice with the common idiom used to do default initialisation (except if value is a bool of course)

value = value or default_value

Thanks, that's what I have thought.

Going a bit further you could propose an optional default value to return in case the item is not found (nil being the default value of the default value).

v = readitem_float(scn, "foo", 3.1416)

I get the idea, but I like to keep the API as close as possible to the OrbiterAPI as that helps in documenting it as well.
Shure the above implementation is not 100% equal to the C++ API as it uses pointer values to store the value and return a boolean.
But I have no idea if providing references as parameters to be filled by a fnction is even possible with Lua - if it is, can someone point me to a document how to do this?
This (as an example) is how I do it now:

/***
Read the value of a tag from a configuration file.

@function readitem_int
@tparam FILEHANDLE f file handle
@tparam string item pointer to tag string
@treturn integer value if tag was found in the file, _nil_ if not.
@see readitem_string for more details
*/
int Interpreter::oapi_readitem_int (lua_State* L)
{
    FILEHANDLE f;
    ASSERT_SYNTAX(lua_islightuserdata(L, 1), "Argument 1: invalid type (expected handle)");
    ASSERT_SYNTAX(f = lua_toObject(L, 1), "Argument 1: invalid object");

    ASSERT_STRING(L, 2);
    const char* item = lua_tostringex(L, 2);

    //ASSERT_NUMBER(L, 3); // I don't think readitem_int should have 3 parameters
    int i;

    bool ok = oapiReadItem_int(f, const_cast<char*>(item), i);
    if (ok) {
        lua_pushnumber(L, i);
    } else {
        lua_pushnil(L);
    }
    return 1;
}

 

[Thunder Chicken]

Hold on, I am only implementing plain eqivalents to the OrbiterAPI.
Special Lua things only create special problems. Like there is no-one to help, as this would be very very Lua specific!
The idea might be tempting, but please please please don't make it more complicated.
Having a 1 to 1 relation between Lua and C++ interface is the way Martin decided to do it and I think it's the best way to go. Even people without Lua knowlwdge can help others if the general usage pattern of the API is equal.

If you like to change it and add special features, give us source-code (including documentation) - I am not a Lua guy!
Would you be so kind and answer the question?

I think this is the pathway to madness, and we're just setting up for obsolescence by putting everything behind a C++ module again. Lua can do string parsing very easily, it's one of its strengths. Offload that to Lua.

What Lua users simply need is access to the contents of the initiating scenario file. That's it. Something like the following that could work within the existing callbacks would be ideal:

function readstatus(scn)

    scenario_contents = read_scenario(scn) --this is a mythical oapi function that is needed to get the scenario file contents

    return scenario_contents

end

where scenario_contents is a lua table that holds all the lines of the scenario file as strings. We can parse anything we like out of that very easily. Nothing else is needed. One new oapi function is all that is needed to get those file contents. That's it. It is future-proof and gives all the flexibility in the world. If readstatus(scn) could be generalized for use in vessel as well as MFD scripts that would be ideal.

We will also need to write out to the default scenario file at the end of the session, but that can be done within the writestatus(scn) callback. We just need a function that can write variables to either the MFD or vessel blocks in the scenario file. Again, if this callback could be generalized to both MFDs and vessels, we're done.

 

[kuddel]

I am pretty sure Martin has designed this API to be as good as possible and I would like to keep it that way.

We can discuss optional additional features like that read_scenario function later, ok?
You can request it via github issues (you already did it once). That way it dosn't get lost and it can be discussed/documented there whenever somone feels like doing it.

By the way, why does your example return anything? The API of readstatus requires a function that does not return anything.

 

[Thunder Chicken]

I am pretty sure Martin has designed this API to be as good as possible and I would like to keep it that way.

I'm not questioning Martin's C++ API, or the current Lua API for that matter. If you want to torture yourself trying to match them 1:1, that's fine I guess. But it will just be made obsolete again when something in either Orbiter and Lua changes and breaks something, and it's just a lot of tedium for tedium's sake. It's why Orbiter 2010 add-ons are a graveyard. I don't think that is a great use of your effort. We'll be back here in 5 years trying to unbreak things.

If a single function can just deliver the contents of the originating scenario file to the Lua script, we'll never have to worry about it again and it would give Lua developers everything they need, and we won't be harrassing C++ developers in the future to recompile to fix broken code. I think this is the pragmatic solution vs. trying to dogmatically replicate the C++ API.

We can discuss optional additional features like that read_scenario function later, ok?

I don't think this is an optional function. It's exactly what we need. That's what I am trying to explain.

You can request it via github issues (you already did it once). That way it dosn't get lost and it can be discussed/documented there whenever somone feels like doing it.

Done. Will do.

By the way, why does your example return anything? The API of readstatus requires a function that does not return anything.

Because I am not an Orbiter developer, just a highly interested user that made a guess on the form of the solution.

Whatever delivers the contents of the scenario file to the user, that's what we need.

 

[kuddel]

Hey,

after lots of trouble getting the current OpenOrbiter to build (mainly CMake oddities) i have been able to implement some of the missing Orbiter-API.
This is just the pain 1 to 1 "translation" of C++ API to Lua, so not exactly what you've requested, but as I said: One step after the other

I was able to get a manual test-setup, so I'm pretty sure it works (as far as I could test it of course). Thanks for your example by the way Thunder Chicken, without it I would have had no idea how a ScriptMFD works....

I would like to include automatic tests for the CI, but that might not be possible and is a task for later.

Now my question for you: As I will not create a PR, which is not at least somehow tested, I can provide the code at my github repository[*].

Are you able to compile OpenOrbiter? Posting a complete build here is a bit big I'm afraid.


[*] https://github.com/schnepe2/orbiter/tree/Lua_extension

...which should not be private. If it is, I can post the source files you need (that's not so much).​

 

[Thunder Chicken]

Hey,

after lots of trouble getting the current OpenOrbiter to build (mainly CMake oddities) i have been able to implement some of the missing Orbiter-API.
This is just the pain 1 to 1 "translation" of C++ API to Lua, so not exactly what you've requested, but as I said: One step after the other

I was able to get a manual test-setup, so I'm pretty sure it works (as far as I could test it of course). Thanks for your example by the way Thunder Chicken, without it I would have had no idea how a ScriptMFD works....

If you need an hand with things on the Lua script side I'm happy to help.

I would like to include automatic tests for the CI, but that might not be possible and is a task for later.

Now my question for you: As I will not create a PR, which is not at least somehow tested, I can provide the code at my github repository[*].

Are you able to compile OpenOrbiter? Posting a complete build here is a bit big I'm afraid.


[*] https://github.com/schnepe2/orbiter/tree/Lua_extension

...which should not be private. If it is, I can post the source files you need (that's not so much).​

I was just barely able to understand things well enough to compile C++ modules when I was on a Windows box. I'm on Linux now and it's for these reasons that I have taken such a shine to Lua scripting. I believe it is technically possible for me to compile it, but I would need a lot of hand-holding, or at least a step-by-step procedure geared to noobs to accomplish it.

I have downloaded OpenOrbiter and run it on this machine. Is it possible to transfer only a subset of affected files from the build?

BTW - OpenOrbiter does run on my Linux box, but it has a lot of graphics problems. It renders vessels OK, but Earth terrain is basically a billiard ball no matter what I do, and it runs really slowly. I can run Orbiter2016 unmodified and it works appropriately on my machine in full-screen mode.

 

[kuddel]

I could try to pack the changed binaries (I think only a couple of DLLs, as documentation can not be generated yet ...)
Do you run a 32-bit version or a 64-bit version?

 

[Thunder Chicken]

I could try to pack the changed binaries (I think only a couple of DLLs, as documentation can not be generated yet ...)
Do you run a 32-bit version or a 64-bit version?

I have the 32-bit version installed.

 

[Thunder Chicken]

I do have to say that I am having troubles running Open Orbiter. I can only get graphics on full-screen mode (Orbiter 2016 is like this). But even when it loads, there are no surface textures to anything with the exception of the vessel itself. Basic buildings render, PAPI and approach lights render, but the Earth is otherwise a billiard ball. If I start the DG at KSC scenario, simply tapping the + key once sends me into solar orbit. It is EXTREMELY laggy, 1 fps or less. This doesn't change with all visuals turned off. I've tried all the different video parameters and none of them seem to do any better. It's frankly not usable, but Orbiter2016 runs beautifully.

 

[kuddel]

Here you are.

Notes:

• Backup your original LuaInterpreter.dll - it is overwritten
• The Scenario is based on my unit-test approach, so nothing really usefull here.
  Note however how the key-value pairs are represented in the BEGIN_MFD Right block.
• The ScriptMFD itself (Config\MFD\ScriptMfdApiTest.cfg) is written as a Config/Lua combination - also just as a test ...splitting .cfg and .lua file would be nicer in the end
• In the script itself you might see best what to do or what not to do.
  I'm sure a "good" Lua script would look different - as you know I'm not a Lua developer per se.
• Some things (like for example the trunctuation of "float" values) are strange, but can be tackled later on, or circumnavigated by using "string"
• A "stop-word-string" might also be a good idea to be placed at the end of a BEGIN_MFD ...END_MFD block, as for example TransX does is, too.

I hope this gets you going.
...time to go to bed now.

Good night
/Kuddel

 

[Thunder Chicken]

The LuaInterpreter.dll seems to work. The ScriptMFD test seems to be able to write out to (Current Status).scn appropriately, but it's a bit hard to see if it is reading the originating scenario correctly, but I can work on that.

The vessel script scenario starts, but it is jerky and stutters and then crashes. I can look into that and see if I see anything problematic in the Lua side.

Thanks for doing this! I'll test it out and report back.

 

[Thunder Chicken]

Modified the ScriptMFDAPITest script just for simplicity for testing. The oapi.readscenario_nextline method is actually excellent and in line with what I was hoping for. I had thought there were read functions that would try to parse the lines into vectors, ints, floats, etc.. and I expected a nightmare. Passing the lines as strings and parsing them in Lua works fine.

The code now reads the originating scenario file using readstatus(scn) for the specified variables, parses the lines and formats those variables, and then writestatus(scn) writes them out to (Current state).scn.

function readstatus(scn)
    while true do
        line = oapi.readscenario_nextline(scn)
        if line == nil then
            break
        elseif string.find(line, "VAL_STR") then
            split_string(line)
            VAL_STR = temp_table[2]
         elseif string.find(line, "VAL_INT") then
            split_string(line)
            VAL_INT = tonumber(temp_table[2])
        elseif string.find(line, "VAL_FLT") then
            split_string(line)
            VAL_FLT = tonumber(temp_table[2])
        elseif string.find(line, "VAL_VEC") then
            split_string(line)
            VAL_VEC = {x=tonumber(temp_table[2]), y=tonumber(temp_table[3]), z=tonumber(temp_table[4])}
        else
            oapi.dbg_out(">> got unexpected \"" .. line .. "\"")
        end

        oapi.dbg_out(tostring(type(VAL_VEC)))
       
    end
end

function writestatus(scn)
    assert(scn ~= nil)
    oapi.writescenario_string(scn, "VAL_STR", VAL_STR)
    oapi.writescenario_int   (scn, "VAL_INT", VAL_INT)
    oapi.writescenario_float (scn, "VAL_FLT", VAL_FLT)
    oapi.writescenario_vec   (scn, "VAL_VEC", VAL_VEC)
end

function prestep(simt,simdt,mjd)
end

function poststep(simt,simdt,mjd)
end

function split_string(line)
    temp_table = {}
    i = 0
    for str in string.gmatch(line, "[^%s]+") do
        i = i + 1
        table.insert(temp_table, i, str)
    end
end

This code reads the variables and reproduces them in (Current status).scn, but truncates the floats:
In:

  VAL_STR foo
  VAL_INT 42
  VAL_FLT 3.141592
  VAL_VEC 1.20 -3.40 5.60

Out:

  VAL_STR foo
  VAL_INT 42
  VAL_FLT 3.14
  VAL_VEC 1.20 -3.40 5.60

The debug output prints 3.141592 for VAL_FLT, so the truncation must be occurring inside the oapi.writescenario_float method.

I did one test to see how the write_scenario functions would do with integers and floats if I forced one into the other, namely:

oapi.writescenario_int (scn, "VAL_FLT", VAL_FLT)
oapi.writescenario_float (scn, "VAL_INT", VAL_INT)

Which gave these results:

In:

  VAL_STR foo
  VAL_INT 42
  VAL_FLT 3.141592
  VAL_VEC 1.20 -3.40 5.60

Out

  VAL_STR foo
  VAL_FLT 3
  VAL_INT 42.00
  VAL_VEC 1.20 -3.40 5.60

IIRC numbers are all internally double precision floats in Lua, whether formatted as an integer or not. The oapi.writescenario_int method seems to be writing the integer portion of the number, and again the oapi.writescenario_float seems to be truncating to two decimal places. I don't know if you want to implement some sort of error checking that the number supplied to oapi.writescenario_int is actually formatted as an integer or just leave this as it. If they are accidentally misused like this then it could quietly corrupt MFD status values unbeknownst to the user.

Excellent progress! Great work!

I'll take a look at the vessel script tomorrow.

 

[kuddel]

The vessel script scenario starts, but it is jerky and stutters and then crashes. I can look into that and see if I see anything problematic in the Lua side.

"vessel script scenario"? If you mean .\Scenarios\Tests\VesselApiTest.scn, that's intended to do so. Automatic CI Tests need to be run that way - this has nothing to do with the changes (see Tests\README.md). That's why I said it is another challenge to write those CI "headless tests".

Regarding the truncation of floats: This is something I've already noticed. This also has nothing to do with the changes in Interpreter.dll, this happens with the direct C++ calls as well.
This bug might have been introduced by me lately but I have to investigate in that. It seems like the default precision has somehow dropped to 3 instead of the expected 6.
This looks like a unexpected change in the stream-based I/O library. I will fix that whenever I find the time.