### A Summary of Our Lua Patches

This document lists the set of patches we need to make to the lua interpreter. I'm doing my best to keep the set of patches to a minimum, and to keep the patches as simple as possible.

## Error Line Number Patch

Standard Lua is pretty inconsistent about how it reports the line number and filename of errors:

- Some errors have the line number and filename right in the error message. Some don't.
- Some errors generate stack tracebacks, some don't.
- When there's a traceback, the file and line number in the error message is redundant.

The goal of this patch is to achieve a more consistent state of affairs. First, we follow the rule that every error must include a stack traceback. Doing this doesn't require any patches to lua. It just requires consistent use of Lua's traceback facilities at every entry point into lua.

Second, since the traceback will always contain the file and line number, and since there will always be a traceback, there's no reason to put the file and line number into the error message itself. We have taken steps to remove that feature. To accomplish this, the following patches to lua were made:

- removed calls to 'lua_where' (replacing them with 'lua_no_where')
- removed calls to 'addinfo' (replacing them with 'no_addinfo')

This patch is live and functioning.

## Thread NextID Patch

We had the idea that Lua threads should contain an int64 field, "nextid", which theoretically was meant to help our system's unique ID allocator. We implemented a patch to store this field. When a Lua thread is created, the nextid field is initialized to zero. The following accessors were added to the Lua API:

```cpp
lua_Integer lua_getnextid(lua_State *L);

void lua_setnextid(lua_State *L, lua_Integer id);
```

This patch is dead code: as it turns out, we ended up not using this feature. These two accessors are never used anywhere in the Luprex codebase.

## Print Integers Patch

Lua numbers are actually double-precision floating point. There is no separate "int" data type. If you try to store an integer in lua, it gets stored as a double-precision float. That's usually just fine. Double precision can store integers up to 53 bits losslessly and precisely.

When you use the builtin lua function "tostring" that converts a number to a string, this is actually implemented using 'sprintf', using the format directive in LUA_NUMBER_FMT, which by default is "%.14d". As you can see, that's a number format for double-precision numbers, which makes sense, because lua numbers *are* double-precision floats.

That particular format has the effect of printing certain large integers in scientific notation. I found that to be problematic, because you can't see all the digits. It's a very bad format for printing, say, unique ID numbers.

So, I changed the default format from "%.14d" to "%.16d". That may seem like a trivial change, but it makes it so that every large integer up to 53 bits, which is the largest integer that can be losslessly stored in a double, is printed as a simple integer, showing all the digits.

The 'print', 'pprint,' and 'tostring' routines built into our lua interpreter do not use this builtin functionality. We have rewritten all of those functions from scratch, to take precise control over what is printed. None of those functions rely on the builtin *sprintf*.

The patch is probably still live, though. The builtin *sprintf* probably does get called implicitly from inside certain lua error message generating routines. It's just of very minor importance these days.

## The Generalized Less-Than Patch

The standard lua less-than operator will throw an error if you try to compare two objects of different types, or if you try to compare two tables, two functions, or two threads.

This patch adds the lua function *genlt* and the C function *lua_genlt*. This is a generalized less-than operator to compare any two lua objects. It always returns true or false, and never generates an error. Here is how comparison works:

- boolean: false is less than true.
- number: ordered by ascending value.
- string: ordered lexically.
- table: all tables are equal (ie, not less than).
- functions: all functions are equal (ie, not less than).
- threads: all threads are equal (ie, not less than).
- two different types: they are ordered nil, bool, number, string, table, function, thread.

This patch is live and functioning. The generalized less-than operator is quite useful as the second parameter to lua's builtin *table.sort* function. We have also provided an iterator *table.sortedpairs* that is similar to the lua builtin *table.pairs* that iterates over a table in sorted order. This implicitly uses the *genlt* comparison operator.

We originally designed this patch to help with determinism.
But we eventually realized it was the wrong design, and we
ended up not needing it for determinism.  It's still a
very useful function, though.

## The Table Iterator Patch

This patch is designed to address the nondeterminism of the lua 'next' iterator. In the original Lua design, table iteration was nondeterministic. By that, I mean that in the original lua, I can create two empty tables, I can then perform the same sequence of insertions and deletions on those two tables. The two tables are identical: they have the same keys, and they've had the same same sequence of operations applied to them. But I can iterate over them using "table.pairs", and they produce their keys in two different orders. That's nondeterminism.

We can't allow nondeterminism in our version of Lua. To fix this was a big deal: we had to change the internal representation of lua tables. Internally, a lua table now contains a separate "sequence" vector which works as follows:

- When you insert a new key into a table (by mapping that key to non-nil), the new key is appended to the sequence.

- When you remove a key from a table (by mapping that key to nil) that key is removed from the sequence, creating a gap in the sequence. The last key in the sequence is moved to fill the gap in the sequence.

- When you call the iterator, next(table, k), it finds k in the sequence, and then returns the key immediately after k in the sequence.

Importantly, the sequence is deterministic - given a fixed set of operations on a table, it will always be in the same order. Therefore the *next* iterator and the *lua_next* function are now both deterministic, as is the *pairs* iterator.

There's a very special case. The standard lua 'next' iterator can find a successor to a key that was *recently deleted*. This functionality is necessary in order to be able to iterate over a table and delete elements while iterating. We support this same functionality, but only on the very most recently deleted key. It's still sufficient to allow deleting table elements while iterating.

This patch is live and is used implicitly whenever you iterate over a lua table.

## The Table Length Patch

The builtin lua function lua_len is nondeterministic. By that, I mean that two tables with the exact same keys might return different values for lua_len. We can't allow nondeterministic anything in our version of Lua. We have altered the implementation of lua_len so that it is deterministic. Two tables with the same keys will always return the same lua_len, that is now guaranteed.

Our new implementation of lua_len conforms to the specification in the documentation. I'm not sure that's the right thing to do.

It's obvious how this specification got written: they implemented an algorithm to find the length of a vector as efficiently as possible. By "vector," I mean a table whose keys are 1,2,3,4,5 and so forth. After they wrote this vector-length algorithm, somebody asked, "what happens if you apply that algorithm to a table that's not a vector?" The implementor replied, "it wasn't meant for non-vectors." "Ok, but what *does* it do if you apply it to a non-vector?" They puzzled it out, and they wrote down what it does as "the specification." But that specification when applied to non-vectors isn't *useful*. It's just what their vector-length algorithm happens to spit out when you feed it an input that it wasn't designed to handle.

So why did they use an algorithm that only works on vectors? Why not use a better algorithm, one that can return the number of keys in the table regardless of whether the table is a vector? The answer is that given the lua table internal representation, returning the number of keys in the table is O(N), whereas the vector-only implementation is usually O(1).

However, I had to change the table internal representation for the table iterator patch (above). With the modified table representation, returning the number of keys in the table can be done in constant time, whether it's a vector or not.

Now, I'm seriously tempted to have lua_len just return the number of keys in the table. That would be so straightforward and self-explanatory, and faster than the current algorithm. The only reason I haven't done this is that it wouldn't conform to the specification! My new lua_len algorithm is similar to the original algorithm, in that it fails in exactly the same way on non-vectors, in order to be compliant with the specification.

Since this feels insane, I have also provided a totally new API function: lua_nkeys. This returns the number of keys in the table, full stop. It's constant-time.

This patch also includes a function lua_nthkey, to get the Nth item in the table iteration, random-access style. I am not certain that this is a good idea, and I have deliberately avoided the use of this function for now, until I am convinced that it's wise.

This patch is live, and is necessary to the determinism of the system.

## The Table Flag Bits Patch

Our difference transmission algorithm does a recursive walk of the tables in a given tangible. That recursive walk requires a *visited* bit in each table. Of course, the lua way of doing this would be to store the set of visited tables as a separate table. But that would be a lot slower than just setting a bit, and difference transmission is the core of our system's performance bottleneck.

We also need to store, in each table, a *table-type* enum. We have several subtypes of tables: general tables, tangible tables, class tables, and so forth. The difference transmitter treats different types of tables differently.

This patch adds a 16-bit "flagbits" field to every Lua table. We have added these lua API functions to access these flag bits:

```cpp
uint16_t lua_getflagbits(lua_State *L, int index);

void lua_setflagbits(lua_State *L, int index, uint16_t flagbits);

void lua_modflagbits(lua_State *L, int index, uint16_t clearbits, uint16_t setbits);
```

The eris code for serializing the lua data structures has been modified to save and restore the flagbits. Aside from simply storing them, and saving and restoring them with eris, the lua runtime doesn't do anything at all with the flagbits.

The Luprex engine has set aside four of the bits to store the table-type enum. It has set aside one of the bits for the 'visited' bit of the difference transmission algorithm. The rest of the bits are currently unused.

This patch is live and in-use.

## The Insert Frame Patch

When we write C functions for Lua, we allocate a "stack frame" on the lua stack. This is accomplished by class LuaDefStack. See the document "Our In-House Lua API" for more information.

This function has to insert some "nils" into the base of the stack. The lua API does have a function that can do this, but using it would be O(N^2). Since this functionality is used in every single C function for Lua, we decided to optimize things a little. We added a function to the lua API that can do it in O(N) time. The name of the function is lua_insert_frame, which sounds fancy, but all it does is insert N "nils" at the bottom of the stack.

This patch is live and is used in class LuaDefStack.

## The C++ Exceptions Patch

We've compiled lua to use C++ exceptions instead of longjmp. The advantage of this is that if you do a lua_yield or lua_error, any C++ destructors on the stack will get called.

Although lua_yield and lua_error both throw C++ exceptions, Lua cannot *deal with* C++ exceptions except for those it generates itself. Therefore:

- Never call the lua interpreter inside a C++ catch-block!
- Never throw an exception from inside a LuaDefine!

Exception 1: If you throw an uncaught exception, all that does is terminate the program. It's always legal to terminate the program.

Exception 2: If you throw an exception inside a LuaDefine and then catch it inside the same LuaDefine, that's OK, because the lua interpreter is not getting unwound.

Using C++ exceptions in lua_yield and lua_error means that C++ destructors get called. Normally, calling destructors is a good thing. However, there is one known case where this causes issues: class LuaExtStack. Class LuaExtStack pushes values onto the lua stack in its constructor, and later, in its destructor, it pops those values back off. Straightforward enough. But if you throw an error using the lua_error function, then the error message is pushed on top of the lua stack. If the throw triggers the LuaExtStack destructor, then LuaExtStack will pop the stack, and in doing so, it will unintentionally throw out the error message. Oops.

To fix this, we had to add a lua patch, which adds a new API function "lua_isthrowing." This API function is used by the LuaExtStack destructor, to decide whether to clean up the stack or not. This new API function is not used anywhere else in Luprex, and I do not expect it will ever be needed anywhere else.

This patch is live and is needed to keep lua error messages working.

## The Object-Oriented Lua Patch

Lua has a colon operator, for method lookup:

```lua
obj:method(arg1, arg2, arg3)
```

This looks for the method closure in *obj*. Instead of looking for the method in *obj*, I would like lua to look for the method in a separate table, the object's *class* table. One way to accomplish that is to use the *index* metamethod:

```lua
setmetatable(obj, { __index = class } )
```

That's not bad, but it puts both values and methods into the same namespace:

- Looking up data (eg, *obj.value)* will look for value in *obj*, and if it's not found, it will look for value in the *class* table. Looking for a value in the *class* table is pointless and inefficient.

- Looking up a method (eg, *obj:method*) will look for the closure in *obj* first, before looking in the *class* table. Looking for a closure in the data table *obj* is again pointless and inefficient.

- By putting both values and methods into the same namespace, we create the possibility of unintended mistakes.

I am thinking about implementing a new metatable entry: __METHODS = true. If this flag is present, then the colon operator *obj:method* looks for the method in the metatable, instead of looking for it in the object. With this new metamethod, the way to create a class would be to make a table full of methods, and then in that table, also put __METHODS = true. Then you would do this:

```lua
setmetatable(obj, class)
```

The class table would *be* the metatable. The result of this patch is that method lookup is done in the class table, value lookup is done in the object, and the two namespaces are kept separate.

Implementing this, as it turns out, is quite simple. The lua virtual machine contains an opcode, OP_SELF. This opcode is only used when the lua scripter uses the colon operator for method invocation. The opcode has two inputs: *obj*, and *method* (where method is a string, a method name). It returns the closure to call. Currently, OP_SELF just calls luaV_getttable to fetch the method from *obj*. We could easily replace the call to luaV_gettable with a function that we write ourselves, luaV_getmethod.

## The Print No Address Patch (Unimplemented)

The lua library function luaL_tolstring converts a lua object into a string, no matter what the type of the lua object is. In the case of tables, the string it generates looks like "table: 0123456701234567", with the address of the table being part of the string. That would make any code that uses that string dependent on the precise address of the table. That's considered nondeterminism for our purposes, it's not allowed.

We don't use luaL_tolstring in the Luprex codebase. However, the luaL_tolstring function is used from within the lua runtime in three different places:

- In the lua function *string.format*. This is still a serious issue. If you use the "%s" formatting directive, and then pass in a table, it will format it as above, which is against the determinism rules.
- In the builtin function *tostring*. This is no longer an issue. We wrote our own version of tostring which replaces the built-in version. Our implementation doesn't use luaL_tolstring.
- In the eris runtime. It is using luaL_tolstring to generate keys that become part of an associative map.

So the obvious thing to do would be to just change the code for luaL_tolstring to not include the table address. But that would totally break the eris code.

Another possibility would be to patch or override the code for string.format. That fixes the problem, but it leaves luaL_tolstring in there as a time-bomb where somebody else might use it not realizing that it's an issue.

There's no obvious approach to fixing this, so I haven't patched it yet.

## The GC Finalizer Patch and the Weak Table Patch (Unimplemented)

GC Finalizers and weak tables both introduce nondeterminism into Lua execution. We can't allow that. It may be necessary to patch the lua interpreter to simply disable these functions. Alternately, we could simply ask the scripters not to use these features, and declare "undefined behavior" if they do.

Update 1: I'm using GC finalizers in some cases to clean up userdata objects. I think it's safe as long as the only thing the finalizer does is free memory. (NOTE: WHERE?)

Update 2: I don't remember using userdata objects at all. I am not sure that Update 1 is the truth any more.
