integration/Docs/Our In-House Lua API.md

### Our In-House Lua API

Lua comes with a C API, which makes it possible to write C functions that manipulate Lua data. The standard API is functional, but it has some readability issues and can be difficult to work with. Therefore, we have created our own replacement API, which we call "LuaStack." In our codebase, you are expected to use LuaStack, not the standard Lua API. This paper will explain LuaStack using the following example function in Lua, which compares two tables to see if they are exactly equal:

```lua
function table.equal(table1, table2)

if (type(table1) ~= "table") then

error("table1 must be a table")

end

if (type(table2) ~= "table") then

error("table2 must be a table")

end

local size1 = table.nkeys(table1)

local size2 = table.nkeys(table2)

if (size1 ~= size2) then

return false

end

for key, value1 in pairs(table1) do

local value2 = table2[key]

if value1 ~= value2 then

return false

end

end

return true

end
```

A few notes about that function. You can't just use the expression "table1==table2." That would tell you if table1 and table2 are the same table (ie, pointer equality). The function above, by contrast, checks if two different tables contain the same key-value pairs. The built-in function *table.nkeys* is something that we added to lua, it returns the number of key-value pairs in a table. Once you fully understand the Lua code above, have a look at the following C++ code, which does the same thing. Don't worry if you don't understand everything yet:

```cpp
LuaDefine(table_equal, "(table1, table2)", "Return true if two tables are equal") {

LuaArg table1, table2;

LuaVar size1, size2, key, value1, value2;

LuaRet equalflag;

LuaDefStack LS(L, table1, table2, size1, size2, key, value1, value2, equalflag);

LS.cktable(table1);

LS.cktable(table2);

LS.set(size1, LS.nkeys(table1));

LS.set(size2, LS.nkeys(table2));

if (!LS.equal(size1, size2)) {

LS.set(equalflag, false);

return LS.result();

}

LS.set(key, LuaNil);

while (LS.next(key, value1, table1)) {

LS.rawget(value2, table2, key);

if (!LS.equal(value1, value2)) {

LS.set(equalflag, false);

return LS.result();

}

}

LS.set(equalflag, true);

return LS.result();

}
```

There's a lot to unpack there. You might be wondering what *LuaDefine* is, you might be wondering what *LS* is, you might be wondering what the *LuaDefStack* constructor does, and so forth. The rest of this paper is going to pick that C++ function apart piece by piece. We're going to explain every class, macro, and template used above. Once you're done reading this paper, you'll understand at a deep level how everything works.

## Class LuaArg, Class LuaVar, and Class LuaRet

In C++ example function, the first few lines inside the body of the LuaDefine are:

```cpp
LuaArg table1, table2;

LuaVar size1, size2, key, value1, value2;

LuaRet equalflag;
```

These are variable declarations. You are meant to think of these as variables that can hold any lua data type - lua tables, lua strings, lua numbers, lua closures, lua threads, any data that the lua interpreter can handle. LuaArg are function arguments, LuaVar are local variables, and LuaRet are return values.

That sounds straightforward, but there's a catch. The Lua garbage collector can't allow you to put Lua values into C++ variables. The reason for this is that if you were to put a pointer to a Lua table into a C++ local variable, the Lua garbage collector would not know that this pointer exists. It might think there are no pointers to the table, and it might free the table. In reality, we can't actually put Lua values into C++ variables.

So that raises the question, how is it possible to have LuaArg, LuaRet, and LuaVar variables that seemingly hold lua values? The answer is that there's a hidden level of indirection. The values are actually stored on the Lua stack. In the example function, our variables end up on the lua stack in the following order:

```
stack[1] holds equalflag

stack[2] holds size1

stack[3] holds size2

stack[4] holds key

stack[5] holds value1

stack[6] holds value2

stack[7] holds table1

stack[8] holds table2
```

In our example function, those eight stack positions are permanently reserved to hold the values of those eight variables. We never pop those values off the stack - at least, not until the end of the lua function.

Here are the C++ declarations of class LuaArg, class LuaRet, and class LuaVar, with methods omitted:

```cpp
class LuaSlot { int stack_position; };

class LuaArg : public LuaSlot {};

class LuaVar : public LuaSlot {};

class LuaRet : public LuaSlot {};
```

So as you can see, LuaArg, LuaRet, and LuaVar are all derived from LuaSlot. It really is true that LuaArg, LuaRet, and LuaVar don't add anything at all to class LuaSlot: their class bodies literally are completely empty, as shown above. A LuaSlot contains nothing but an integer stack_position.

In our example function, once the LuaSlots are initialized, the stack_position values are as follows:

```
equalflag.stack_position == 1

size1.stack_position == 2

size2.stack_position == 3

key.stack_position == 4

value1.stack_position == 5

value2.stack_position == 6

table1.stack_position == 7

table2.stack_position == 8
```

Technically, it is not exactly correct to say that the LuaSlots "contain" lua values. It is more accurate to say that the LuaSlots contain stack positions, and those stack positions contain lua values. That is, there's a level of indirection. However, when you're actually using the LuaStack API, you don't need to think about that hidden indirection. Conceptually, you're meant to think of the LuaSlots as effectively containing lua values. For the remainder of this paper, I will often say that LuaSlots "contain" lua values, glossing over the existence of a level of indirection.

## The LuaDefStack Constructor

In the previous section, I mentioned that the LuaSlots are all assigned unique stack positions. You might be wondering: where is the code that assigns unique stack positions to the LuaSlots?

It's not in the LuaSlot constructor. The LuaSlot constructor doesn't do anything but set stack_position to 0. Since everything in Lua is numbered from 1, that's an invalid stack position. It means that the LuaSlot is still not initialized. Any attempt to use the LuaSlot at this stage will generate an error.

The machinery to assign unique stack positions lives inside the LuaDefStack constructor. LuaSlots are useless without a LuaDefStack to set them up. This section will describe the LuaDefStack constructor, and all the various functions it performs.

In our example C++ function, you'll find the LuaDefStack constructor immediately after the LuaSlot declarations:

```cpp
LuaDefStack LS(L, table1, table2, size1, size2, key, value1, value2, equalflag);
```

As you can see, the first argument to the LuaDefStack constructor is a capital L. That's boilerplate: the first argument is always capital L. The declaration of that variable, *lua_State *L,* is concealed inside the *LuaDefine* macro. I'll talk about LuaDefine later. That data type, *lua_State*,* is from the standard Lua API. It's a C-style pointer to the lua stack. The LuaDefStack constructor stores this pointer inside the LuaDefStack.

After the capital L, all the LuaSlots have been passed to the LuaDefStack constructor. They are passed by non-const reference, which means that the LuaDefStack can mutate them.

The LuaDefStack constructor allocates stack space for the LuaSlots. In our example function, it counts that there are eight LuaSlots, so it allocates the first eight positions on the lua stack to hold them. Then, it chooses a stack position in the range 1 to 8 for each of the LuaSlot variables. It mutates each LuaSlot to contain the correct stack_position.

The LuaDefStack constructor notices that two of the LuaSlots are of class LuaArg. It knows that LuaArgs are meant to hold function arguments. So it checks that two function arguments were passed to the function. If that's not the case, it throws a lua error. But if all is well, it moves the function arguments into the stack positions reserved for the LuaArgs.

This is why we have the declaration of LuaArg which derives from LuaSlot, but doesn't add any functionality to class LuaSlot. Even though LuaArg inherits everything from LuaSlot, the LuaDefStack constructor can differentiate it, and treat it as special.

The LuaDefStack initializes the contents of all the LuaSlots. As mentioned above, the LuaArg are initialized from the function arguments. The LuaRet and LuaVar are initialized with nil. After the LuaDefStack constructor, all the LuaSlots have the correct stack_positions, and via that level of indirection, the LuaSlots all contain the correct lua values.

Meanwhile, the LuaDefStack contains the pointer to the lua stack. Note that in the example function, the object of class LuaDefStack is named *LS*. That stands for "Lua Stack." That is a naming convention that we use throughout our codebase - the LuaDefStack is called *LS*. But unlike *lua_State* L*, which is also a pointer to the lua stack, *LuaDefStack LS* is a C++ style smart pointer, which has methods.

## Methods of Class LuaDefStack

In our example function, you'll find this line of code:

```cpp
LS.set(size1, LS.nkeys(table1));
```

Let's break that down into pieces. Here's a simpler statement, using just LS.set:

```cpp
LS.set(size1, 3);
```

Conceptually, you're supposed to think of that in the very obvious way: it sets *size1* to 3. But you also know that there's a hidden level of indirection, so you might be wondering about *how* it sets *size1* to 3.

The function LS.set is a call to the following method:

```cpp
void LuaDefStack::set(const LuaSlot &slot, int value) const;
```

Inside that method, *this* contains the pointer to the lua stack, *slot* contains the necessary stack_position, and *value* contains the value to store. What this method actually does is set stack[stack_position] = value. In our example function we know that *size1.stack_position* == 2, and *value* == 3. So this sets stack[2] = 3. But again, you're meant to ignore the hidden indirection, you're supposed to think of it as just setting *size1* to 3.

OK, now, let's consider this statement from the example function:

```cpp
LS.set(size1, LS.nkeys(table1));
```

That's two method calls. The *nkeys* method call fetches the size of *table1*, returning it as an 'int'. The *set* method stores the int in the appropriate stack slot for *size1*.

The first argument to LuaDefStack::set must be a LuaSlot. The second argument is overloaded, it can be any one of several different data types:

```cpp
LS.set(slota, "hello"); // set to a char*

LS.set(slota, 3.14159); // set to a double

LS.set(slota, true); // set to a bool

LS.set(slota, LuaNil); // set to nil

LS.set(slota, slotb); // copy from another LuaSlot
```

In general, methods of LuaDefStack will automatically convert the following C++ types to Lua: int, int64_t, float, double, char *, string, string_view, and bool. If you want to convert lua values back to C++ values, it's a little harder. More on that in the next section.

For a complete listing of all the operators in LuaDefStack, have a look at the header file luastack.hpp, at the class LuaBaseStack. There, you will find prototypes for all the functions, along with documentation for each one.

## Converting Lua Values to C++ Values

Suppose that you have a function, callable from lua, that is supposed to accept a string as a parameter:

```cpp
LuaDefine(frob, (str), "Frobnicate a string.") {

LuaArg str;

LuaDefStack LS(L, str);

…etc…
```

The variable 'str' is a LuaSlot. Suppose you want to convert that to a C++ string. LuaDefStack provides a method that can do that:

```cpp
eng::string s = LS.ckstring(str);
```

The method *ckstring* verifies that *str* really is a lua string. If it is a string, it converts it to an eng::string and returns it. If it's not a string, ckstring throws a lua error: "value must be a string". That's not an especially good error message, it lacks all context. You can make it slightly better, "str must be a string", by passing the variable's name to ckstring:

```cpp
std::string s = LS.ckstring(str, "str");
```

Class LuaDefStack provides a full catalog of conversion functions that return different C++ data types:

```cpp
bool ckboolean(LuaSlot s, const char *argname = "value") const;

lua_Integer ckinteger(LuaSlot s, const char *argname = "value") const;

int ckint(LuaSlot s, const char *argname = "value") const;

lua_Number cknumber(LuaSlot s, const char *argname = "value") const;

eng::string ckstring(LuaSlot s, const char *argname = "value") const;

std::string_view ckstringview(LuaSlot s, const char *argname = "value") const;

lua_State * ckthread(LuaSlot s, const char *argname = "value") const;

LuaToken cktoken(LuaSlot s, const char *argname = "value") const;

util::DXYZ ckxyz(LuaSlot s, const char *argname = "value") const;

void cktable(LuaSlot s, const char *argname = "value") const;

void cknil(LuaSlot s, const char *argname = "value") const;

void ckfunction(LuaSlot s, const char *argname = "value") const;

void ckcfunction(LuaSlot s, const char *argname = "value") const;

void cktangible(LuaSlot s, const char *argname = "value") const;
```

Notice that cktable returns void. That's because you can't convert a lua table to a C++ anything. So cktable just checks the type and if it's not a table, it throws a lua error, "value must be a table". If it is a table, cktable doesn't do anything at all. Note that we used *LS.cktable* in the example table_equal function, to verify that the caller passed in two tables.

OK, so what if you want to convert the LuaSlot str to a C++ string, but you want to handle the non-string exception case yourself? Use this code:

```cpp
std::optional<eng::string> s = LS.trystring(str);
```

You may not be familiar with C++ std::optional, so I'll summarize here. An optional<string> can either contain a string, or it can contain nothing. There are methods that can check if the optional is empty, and methods to extract the string value, if present.

What LS.trystring(str) does is: if str is a string, trystring returns an optional<string> containing the string. If str is not a string, trystring returns an optional<string> that is empty.

Unlike the ck-functions functions which throw errors on type mismatch, the try-functions never throw an error. Using the try-functions is ideal in code where throwing errors would be bad.

What if you just want to know if it's a string, but you don't need the actual string? In that case, use *isstring:*

```cpp
bool b = LS.isstring(str);
```

You can also use LS.type to fetch the type of a lua value as an enum.

## Dealing with Return Values

Our sample function contains this code:

```cpp
LuaRet equalflag;

…

if (!LS.equal(size1, size2)) {

LS.set(equalflag, false);

return LS.result();

}
```

The variable declaration, "LuaRet equalflag", declares that this function has a return value whose name is equalflag. In our API, return values are just variables. You have to assign a value to the variable before you return from the function. In the code above, when size1 != size2, it stores *false* into equalflag. Then, there is a return-statement.

Lua calling conventions dictate that at the end of a function, there shouldn't be anything on the lua stack except for the return values. That is accomplished by the method LS.result. This takes everything off the stack except for the LuaRet values. Another lua calling convention is that the function must return the number of return values that are on the lua stack. Therefore, LS.result returns the number of LuaRet values that were declared.

So, in every LuaDefine, the process of returning from a function always consists of two steps:

1. You must set the LuaRet slots to whatever values you want to return.
2. There must be an explicit return-statement that must always be *return LS.result()*

## The LuaDefine Macro

Our sample function starts with this line of code:

```cpp
LuaDefine(table_equal, "(table1, table2)", "Return true if two tables are equal")
```

The two strings in this line of code are just documentation. Neither has any effect on how the function works. They are collected automatically by our documentation system, which can print them out on demand.

LuaDefine is a C preprocessor macro. Before I tell you what it macro expands into, I'd like to tell you how Lua functions are normally written using the standard Lua API. When using the standard API, lua functions look like this:

```cpp
int table_equal(lua_State *L) {

…

}
```

The C prototype is always the same: the return value is always *int,* and the only argument is *lua_State*L*, a pointer to the lua stack. A C function must have this prototype to be compatible with the lua runtime.

Next, I will show you the macro expansion of a LuaDefine to help you understand what it does. Here is the LuaDefine that I will macro expand:

```cpp
LuaDefine(table_equal, "(table1, table2)", "Return true if two tables are equal")
```

Here is the expansion:

```cpp
int lfn_table_equal(lua_State *L);

const char *lfnarg_table_equal = "(table1, table2)";

const char *lfndoc_table_equal = "Return true if two tables are equal";

LuaFunctionReg reg_table_equal

("table_equal", lfnarg_table_equal, lfndoc_table_equal, false, lfn_table_equal);

int lfn_table_equal(lua_State *L)
```

So first, you can see that there's a prototype for the function lfn_table_equal. As you can see, it returns *int* and accepts *lua_State *L,* which makes it compatible with the lua runtime. This is a forward declaration.

Next, there are the two documentation strings. I've given these documentation strings two names: lfnarg_table_equal and lfndoc_table_equal. That way, it's possible to access these documentation strings in code.

Next, there is the "registry entry" reg_table_equal, of class LuaFunctionReg. The constructor of reg_table_equal runs at program initialization time. The constructor stores its arguments: the function name, the documentation strings, the flags, and the C function pointer. These are all stored inside reg_table_equal. Then, the constructor links reg_table_equal into a global linked list of LuaFunctionReg objects. This linked list makes up the registry of all LuaDefines.

Finally, we have a second copy of the prototype for lfn_table_equal. This time, it's not a forward declaration: there's no semicolon. This prototype is meant to be followed by the function body.

The upshot of all this is that LuaDefine doesn't just declare a function. It also links that function into a global registry of all the LuaDefines. The registry contains not only the function name and function pointer, but also the function documentation. Remember also that in C++, static objects like reg_table_equal are constructed before *main*. So by the time *main* starts, the registry is ready-to-use.

The registry of all LuaDefines has two primary uses:

1. The routine source_load_cfunctions traverses the registry and inserts every LuaDefine into the lua environment. It's fully automatic: simply writing a LuaDefine causes the function to appear in Lua.

2. The documentation system traverses the registry to find the documentation strings. It uses these strings to auto-generate the manual.

## Class LuaExtStack

What if you want to write C++ code that manipulates Lua data, but which *isn't* called from inside Lua? In that case, LuaDefStack won't work, because LuaDefStack is hardwired to assume that a caller *exists*:

- It checks that the caller passed the right number of arguments.

- It transfers the arguments from the caller to the LuaArg slots.
- It transfers the LuaRet slots back to the caller at function end.

For situations where there isn't any caller, we have created an alternative to LuaDefStack called LuaExtStack. There are several things that LuaExtStack does differently:

- LuaExtStack doesn't accept LuaArg or LuaRet parameters - it only allows LuaVar parameters. That only makes sense, since it should be used in contexts where there's no caller.

- There is no code in LuaExtStack to check that the caller passed the right number of arguments, since again, there is no caller.

- Class LuaExtStack doesn't have a method LS.result, since there's nobody to return a result to.

- Class LuaDefStack assumes that anything already on the stack must be a function argument. Class LuaExtStack assumes that anything already on the stack is data that it is not supposed to touch in any way: it leaves it alone. It allocates stack space above whatever is already there.

- Class LuaExtStack has a destructor that automatically pops the stack back to the level where it was when the LuaExtStack was constructed. This is convenient. LuaDefStack, on the other hand, doesn't restore the stack top: it can't, because return values must be left on the stack.

Aside from those differences, LuaExtStack and LuaDefStack are quite similar. For example, all the methods are the same: LS.set, LS.ckstring, LS.nkeys, and so forth all work the same with LuaDefStack and LuaExtStack.

## Using Multiple Lua Stacks

In 95% of the code you'll write, there is only one lua stack. However, there are two big exceptions to this rule.

In the difference-transmission code, you need to compare the master world model to the synchronous world model. Each world model has its own instance of the lua interpreter. Each instance of the lua interpreter has its own lua stack. So when comparing master to synchronous, you're manipulating two lua stacks at the same time.

In that situation, it is typical to use two LuaExtStacks: one for the master model, one for the synchronous model. The naming convention is as follows:

```cpp
LuaVar mtable; // A variable meant to hold a master model table.

LuaVar stable; // A variable meant to hold a synchronous model table.

LuaExtStack MLS(ML, mtable); // Master world model stack object

LuaExtStack SLS(SL, stable); // Synchronous world model stack object
```

In cases like that, it is essential to be careful - every LuaSlot must be used with the correct stack. For example, this will cause a crash:

```cpp
MLS.set(stable, 3);
```

That's because stable contains a stack_position on the SLS stack, not a stack_position on the MLS stack. Every LuaSlot is associated with one and only one lua stack, and you must pair them up correctly.

The other place we use two stacks is inside World::spawn. This does some initial calculation using the default stack of the world model. It creates a LuaExtStack to manipulate the default stack. Then, after the initial calculations are done, it actually creates the new thread, which has its own stack. In the current codebase, the new thread doesn't get its own LuaExtStack. Instead, it's simpler to just use the standard API to set up the new thread and move the data onto it. Therefore, we avoid the risk of using LuaSlot objects with the wrong LuaExtStack.

## Mixing LuaStack with the Standard API

It is rarely necessary to call functions in the standard API - the LuaStack API is meant to be a complete replacement. But every now and then, you encounter a case that LuaStack doesn't handle, or doesn't handle easily. In cases like that, it's possible to mix the standard API with the LuaStack API.

Here's an example. When you type some lua code into the luprex console, that code has to be compiled into a closure, after which the closure gets called. This is all handled by the function World::invoke_lua.

Ideally, our LuaStack API should provide a method to compile lua code into a closure. But I haven't gotten around to it yet. Therefore, World::invoke_lua has to use the standard lua API to compile the code. This is an excerpt from World::invoke_lua:

```cpp
LuaVar func;

LuaExtStack LS(L, func);

int status = luaL_loadbuffer(L, datapack.data(), datapack.size(), "=invoke");

lua_replace(L, func.stack_position());
```

So the first two lines use the LuaStack API. We've defined a LuaVar *func* to hold the compiled closure. The LuaExtStack constructor allocates one space on the lua stack to hold *func*. At this point, the lua stack contains:

```
stack[1] contains func, whose value is nil
```

Then, we call the function luaL_loadbuffer. This is a function from the standard lua API, it compiles a source code string and pushes the compiled closure on the stack. The C++ variable "datapack" contains the lua source code that was typed in at the console. After calling luaL_loadbuffer, the stack contains:

```
stack[1] contains func, whose value is nil

stack[2] contains the compiled closure
```

Finally, our snippet calls lua_replace. This is a function from the standard lua API, it pops the stack and stores the popped value at a specified position on the lua stack. We use this to pop the compiled closure off the stack, and transfer it into the LuaVar *func*. After calling lua_replace, the stack contains:

```
stack[1] contains func, whose value is the compiled closure
```

From this point forward, we are done using the standard API. The rest of World::invoke_lua uses only the LuaStack API. So as you see, it is often possible to write most of the code using the LuaStack API, switching to the standard API for only two or three lines of code.

The standard API generally manipulates values that are on top of the stack. Generally, when you're done using the standard API, you want to move the values from the top of the stack to a LuaSlot. As demonstrated above, that is done using lua_replace.

There's a mirror image of this function: lua_pushvalue. This is useful for copying a value from a LuaSlot to the top of the stack. This is generally useful when you want to take a value from a LuaSlot, and then manipulate it using the standard API.