Add markdown documentation

2026-02-05 12:40:27 -05:00
parent f0228083c8
commit 12f8062d9a
22 changed files with 2853 additions and 0 deletions
--- a/Patches.md
+++ b/Patches.md
@@ -0,0 +1,205 @@
 ### A Summary of Our Lua Patches
 This document lists the set of patches we need to make to the lua interpreter. I'm doing my best to keep the set of patches to a minimum, and to keep the patches as simple as possible.
 ## Error Line Number Patch
 Standard Lua is pretty inconsistent about how it reports the line number and filename of errors:
 - Some errors have the line number and filename right in the error message. Some don't.
 - Some errors generate stack tracebacks, some don't.
 - When there's a traceback, the file and line number in the error message is redundant.
 The goal of this patch is to achieve a more consistent state of affairs. First, we follow the rule that every error must include a stack traceback. Doing this doesn't require any patches to lua. It just requires consistent use of Lua's traceback facilities at every entry point into lua.
 Second, since the traceback will always contain the file and line number, and since there will always be a traceback, there's no reason to put the file and line number into the error message itself. We have taken steps to remove that feature. To accomplish this, the following patches to lua were made:
 - removed calls to 'lua_where' (replacing them with 'lua_no_where')
 - removed calls to 'addinfo' (replacing them with 'no_addinfo')
 This patch is live and functioning.
 ## Thread NextID Patch
 We had the idea that Lua threads should contain an int64 field, "nextid", which theoretically was meant to help our system's unique ID allocator. We implemented a patch to store this field. When a Lua thread is created, the nextid field is initialized to zero. The following accessors were added to the Lua API:
 ```cpp
 lua_Integer lua_getnextid(lua_State *L);
 void lua_setnextid(lua_State *L, lua_Integer id);
 ```
 This patch is dead code: as it turns out, we ended up not using this feature. These two accessors are never used anywhere in the Luprex codebase.
 ## Print Integers Patch
 Lua numbers are actually double-precision floating point. There is no separate "int" data type. If you try to store an integer in lua, it gets stored as a double-precision float. That's usually just fine. Double precision can store integers up to 53 bits losslessly and precisely.
 When you use the builtin lua function "tostring" that converts a number to a string, this is actually implemented using 'sprintf', using the format directive in LUA_NUMBER_FMT, which by default is "%.14d". As you can see, that's a number format for double-precision numbers, which makes sense, because lua numbers *are* double-precision floats.
 That particular format has the effect of printing certain large integers in scientific notation. I found that to be problematic, because you can't see all the digits. It's a very bad format for printing, say, unique ID numbers.
 So, I changed the default format from "%.14d" to "%.16d". That may seem like a trivial change, but it makes it so that every large integer up to 53 bits, which is the largest integer that can be losslessly stored in a double, is printed as a simple integer, showing all the digits.
 The 'print', 'pprint,' and 'tostring' routines built into our lua interpreter do not use this builtin functionality. We have rewritten all of those functions from scratch, to take precise control over what is printed. None of those functions rely on the builtin *sprintf*.
 The patch is probably still live, though. The builtin *sprintf* probably does get called implicitly from inside certain lua error message generating routines. It's just of very minor importance these days.
 ## The Generalized Less-Than Patch
 The standard lua less-than operator will throw an error if you try to compare two objects of different types, or if you try to compare two tables, two functions, or two threads.
 This patch adds the lua function *genlt* and the C function *lua_genlt*. This is a generalized less-than operator to compare any two lua objects. It always returns true or false, and never generates an error. Here is how comparison works:
 - boolean: false is less than true.
 - number: ordered by ascending value.
 - string: ordered lexically.
 - table: all tables are equal (ie, not less than).
 - functions: all functions are equal (ie, not less than).
 - threads: all threads are equal (ie, not less than).
 - two different types: they are ordered nil, bool, number, string, table, function, thread.
 This patch is live and functioning. The generalized less-than operator is quite useful as the second parameter to lua's builtin *table.sort* function. We have also provided an iterator *table.sortedpairs* that is similar to the lua builtin *table.pairs* that iterates over a table in sorted order. This implicitly uses the *genlt* comparison operator.
 ## The Table Iterator Patch
 This patch is designed to address the nondeterminism of the lua 'next' iterator. In the original Lua design, table iteration was nondeterministic. By that, I mean that in the original lua, I can create two empty tables, I can then perform the same sequence of insertions and deletions on those two tables. The two tables are identical: they have the same keys, and they've had the same same sequence of operations applied to them. But I can iterate over them using "table.pairs", and they produce their keys in two different orders. That's nondeterminism.
 We can't allow nondeterminism in our version of Lua. To fix this was a big deal: we had to change the internal representation of lua tables. Internally, a lua table now contains a separate "sequence" vector which works as follows:
 - When you insert a new key into a table (by mapping that key to non-nil), the new key is appended to the sequence.
 - When you remove a key from a table (by mapping that key to nil) that key is removed from the sequence, creating a gap in the sequence. The last key in the sequence is moved to fill the gap in the sequence.
 - When you call the iterator, next(table, k), it finds k in the sequence, and then returns the key immediately after k in the sequence.
 Importantly, the sequence is deterministic - given a fixed set of operations on a table, it will always be in the same order. Therefore the *next* iterator and the *lua_next* function are now both deterministic, as is the *pairs* iterator.
 There's a very special case. The standard lua 'next' iterator can find a successor to a key that was *recently deleted*. This functionality is necessary in order to be able to iterate over a table and delete elements while iterating. We support this same functionality, but only on the very most recently deleted key. It's still sufficient to allow deleting table elements while iterating.
 This patch is live and is used implicitly whenever you iterate over a lua table.
 ## The Table Length Patch
 The builtin lua function lua_len is nondeterministic. By that, I mean that two tables with the exact same keys might return different values for lua_len. We can't allow nondeterministic anything in our version of Lua. We have altered the implementation of lua_len so that it is deterministic. Two tables with the same keys will always return the same lua_len, that is now guaranteed.
 Our new implementation of lua_len conforms to the specification in the documentation. I'm not sure that's the right thing to do.
 It's obvious how this specification got written: they implemented an algorithm to find the length of a vector as efficiently as possible. By "vector," I mean a table whose keys are 1,2,3,4,5 and so forth. After they wrote this vector-length algorithm, somebody asked, "what happens if you apply that algorithm to a table that's not a vector?" The implementor replied, "it wasn't meant for non-vectors." "Ok, but what *does* it do if you apply it to a non-vector?" They puzzled it out, and they wrote down what it does as "the specification." But that specification when applied to non-vectors isn't *useful*. It's just what their vector-length algorithm happens to spit out when you feed it an input that it wasn't designed to handle.
 So why did they use an algorithm that only works on vectors? Why not use a better algorithm, one that can return the number of keys in the table regardless of whether the table is a vector? The answer is that given the lua table internal representation, returning the number of keys in the table is O(N), whereas the vector-only implementation is usually O(1).
 However, I had to change the table internal representation for the table iterator patch (above). With the modified table representation, returning the number of keys in the table can be done in constant time, whether it's a vector or not.
 Now, I'm seriously tempted to have lua_len just return the number of keys in the table. That would be so straightforward and self-explanatory, and faster than the current algorithm. The only reason I haven't done this is that it wouldn't conform to the specification! My new lua_len algorithm is similar to the original algorithm, in that it fails in exactly the same way on non-vectors, in order to be compliant with the specification.
 Since this feels insane, I have also provided a totally new API function: lua_nkeys. This returns the number of keys in the table, full stop. It's constant-time.
 This patch also includes a function lua_nthkey, to get the Nth item in the table iteration, random-access style. I am not certain that this is a good idea, and I have deliberately avoided the use of this function for now, until I am convinced that it's wise.
 This patch is live, and is necessary to the determinism of the system.
 ## The Table Flag Bits Patch
 Our difference transmission algorithm does a recursive walk of the tables in a given tangible. That recursive walk requires a *visited* bit in each table. Of course, the lua way of doing this would be to store the set of visited tables as a separate table. But that would be a lot slower than just setting a bit, and difference transmission is the core of our system's performance bottleneck.
 We also need to store, in each table, a *table-type* enum. We have several subtypes of tables: general tables, tangible tables, class tables, and so forth. The difference transmitter treats different types of tables differently.
 This patch adds a 16-bit "flagbits" field to every Lua table. We have added these lua API functions to access these flag bits:
 ```cpp
 uint16_t lua_getflagbits(lua_State *L, int index);
 void lua_setflagbits(lua_State *L, int index, uint16_t flagbits);
 void lua_modflagbits(lua_State *L, int index, uint16_t clearbits, uint16_t setbits);
 ```
 The eris code for serializing the lua data structures has been modified to save and restore the flagbits. Aside from simply storing them, and saving and restoring them with eris, the lua runtime doesn't do anything at all with the flagbits.
 The Luprex engine has set aside four of the bits to store the table-type enum. It has set aside one of the bits for the 'visited' bit of the difference transmission algorithm. The rest of the bits are currently unused.
 This patch is live and in-use.
 ## The Insert Frame Patch
 When we write C functions for Lua, we allocate a "stack frame" on the lua stack. This is accomplished by class LuaDefStack. See the document "Our In-House Lua API" for more information.
 This function has to insert some "nils" into the base of the stack. The lua API does have a function that can do this, but using it would be O(N^2). Since this functionality is used in every single C function for Lua, we decided to optimize things a little. We added a function to the lua API that can do it in O(N) time. The name of the function is lua_insert_frame, which sounds fancy, but all it does is insert N "nils" at the bottom of the stack.
 This patch is live and is used in class LuaDefStack.
 ## The C++ Exceptions Patch
 We've compiled lua to use C++ exceptions instead of longjmp. The advantage of this is that if you do a lua_yield or lua_error, any C++ destructors on the stack will get called.
 Although lua_yield and lua_error both throw C++ exceptions, Lua cannot *deal with* C++ exceptions except for those it generates itself. Therefore:
 - Never call the lua interpreter inside a C++ catch-block!
 - Never throw an exception from inside a LuaDefine!
 Exception 1: If you throw an uncaught exception, all that does is terminate the program. It's always legal to terminate the program.
 Exception 2: If you throw an exception inside a LuaDefine and then catch it inside the same LuaDefine, that's OK, because the lua interpreter is not getting unwound.
 Using C++ exceptions in lua_yield and lua_error means that C++ destructors get called. Normally, calling destructors is a good thing. However, there is one known case where this causes issues: class LuaExtStack. Class LuaExtStack pushes values onto the lua stack in its constructor, and later, in its destructor, it pops those values back off. Straightforward enough. But if you throw an error using the lua_error function, then the error message is pushed on top of the lua stack. If the throw triggers the LuaExtStack destructor, then LuaExtStack will pop the stack, and in doing so, it will unintentionally throw out the error message. Oops.
 To fix this, we had to add a lua patch, which adds a new API function "lua_isthrowing." This API function is used by the LuaExtStack destructor, to decide whether to clean up the stack or not. This new API function is not used anywhere else in Luprex, and I do not expect it will ever be needed anywhere else.
 This patch is live and is needed to keep lua error messages working.
 ## The Object-Oriented Lua Patch
 Lua has a colon operator, for method lookup:
 ```lua
 obj:method(arg1, arg2, arg3)
 ```
 This looks for the method closure in *obj*. Instead of looking for the method in *obj*, I would like lua to look for the method in a separate table, the object's *class* table. One way to accomplish that is to use the *index* metamethod:
 ```lua
 setmetatable(obj, { __index = class } )
 ```
 That's not bad, but it puts both values and methods into the same namespace:
 - Looking up data (eg, *obj.value)* will look for value in *obj*, and if it's not found, it will look for value in the *class* table. Looking for a value in the *class* table is pointless and inefficient.
 - Looking up a method (eg, *obj:method*) will look for the closure in *obj* first, before looking in the *class* table. Looking for a closure in the data table *obj* is again pointless and inefficient.
 - By putting both values and methods into the same namespace, we create the possibility of unintended mistakes.
 I am thinking about implementing a new metatable entry: __METHODS = true. If this flag is present, then the colon operator *obj:method* looks for the method in the metatable, instead of looking for it in the object. With this new metamethod, the way to create a class would be to make a table full of methods, and then in the class table, put __METHODS = true. Then you would do this:
 ```lua
 setmetatable(obj, class)
 ```
 The class table would *be* the metatable. The result of this patch is that method lookup is done in the class table, value lookup is done in the object, and the two namespaces are kept separate.
 Implementing this, as it turns out, is quite simple. The lua virtual machine contains an opcode, OP_SELF. This opcode is only used when the lua scripter uses the colon operator for method invocation. The opcode has two inputs: *obj*, and *method* (where method is a string, a method name). It returns the closure to call. Currently, OP_SELF just calls luaV_getttable to fetch the method from *obj*. We could easily replace the call to luaV_gettable with a function that we write ourselves, luaV_getmethod.
 ## The Print No Address Patch (Unimplemented)
 The lua library function luaL_tolstring converts a lua object into a string, no matter what the type of the lua object is. In the case of tables, the string it generates looks like "table: 0123456701234567", with the address of the table being part of the string. That would make any code that uses that string dependent on the precise address of the table. That's considered nondeterminism for our purposes, it's not allowed.
 We don't use luaL_tolstring in the Luprex codebase. However, the luaL_tolstring function is used from within the lua runtime in three different places:
 - In the lua function *string.format*. This is still a serious issue. If you use the "%s" formatting directive, and then pass in a table, it will format it as above, which is against the determinism rules.
 - In the builtin function *tostring*. This is no longer an issue. We wrote our own version of tostring which replaces the built-in version. Our implementation doesn't use luaL_tolstring.
 - In the eris runtime. It is using luaL_tolstring to generate keys that become part of an associative map.
 So the obvious thing to do would be to just change the code for luaL_tolstring to not include the table address. But that would totally break the eris code.
 Another possibility would be to patch or override the code for string.format. That fixes the problem, but it leaves luaL_tolstring in there as a time-bomb where somebody else might use it not realizing that it's an issue.
 There's no obvious approach to fixing this, so I haven't patched it yet.
 ## The GC Finalizer Patch and the Weak Table Patch (Unimplemented)
 GC Finalizers and weak tables both introduce nondeterminism into Lua execution. We can't allow that. It may be necessary to patch the lua interpreter to simply disable these functions. Alternately, we could simply ask the scripters not to use these features, and declare "undefined behavior" if they do.
 Update 1: I'm using GC finalizers in some cases to clean up userdata objects. I think it's safe as long as the only thing the finalizer does is free memory.
 Update 2: I don't remember using userdata objects at all. I am not sure that Update 1 is the truth any more.
--- a/Docs/Animation
+++ b/Docs/Animation
@@ -0,0 +1,414 @@
 ## []()Animation Queues and Tangible Actors
 This document contains several things:
 - Overview
  - Summary of what Animation Steps and Animation Queues are.
  - Reasons that Animation Queues can Change in Unexpected Ways.
  - How Animation Queues get transferred from Luprex to Unreal.
  - What the Tangible Actor Blueprint Classes Are
  - What Unreal Adds to the Animation Queue
  - How Tangible Actor Blueprints Read the Animation Queue
 - Blueprint Class TangibleStaticMesh:
  - How it Gets Notified when the Animation Queue Changes
  - How Animations are Initiated
  - How Some Animations involve Tick Functions
  - What TangibleStaticMesh does when Animations Finish
  - How Tangible Actors handle Warping to Another Plane
 - Advanced Topics:
  - How TangibleCharacter implements Interactive Movement
  - How TangibleCharacter interfaces to its Animation Blueprint
  - Interpreting the Animation Queue in Nontraditional Ways
 ## Summary of Animation Steps and an Animation Queues
 Suppose you want a character to jump up and down in-place. Typically, you would use the lua function *tangible.animate* to achieve this:
 ```lua
 tangible.animate(actor, nil, {action='jump', height=3.0}))"
 ```
 This function adds an *animation step* to the character's *animation queue*. An animation queue is just a sequence of animation steps. Its length is set to a fixed value, perhaps 10 steps. So if you add a step 11, then the oldest step is automatically discarded.
 An animation step is a list of key-value pairs. The tangible.animate above would most likely create an animation step with the following key-value pairs in it:
 ```
 action: "jump"
 height: 3.0
 xyz= (100,100,0)
 plane= "earth"
 facing= 50
 ```
 So obviously, the "action" and "height" key-value pairs were taken directly from the *tangible.animate* command. But where did "xyz", "plane", and "facing" come from? Notice that in the key-value pairs above, I used a colon (:) next to action and height, but I used an equal-sign (=) next to xyz, facing, and plane, to make it clear that they are different from each other.
 The values with the equal-sign are persistent values. All persistent values are automatically copied from one animation step to the next, unless they are manually altered in the *tangible.animate* command. Each time you do a tangible.animate, those persistent values will get propagated forward. They will never change unless you explicitly change them in the *tangible.animate* command. In contrast, the values with the colon are transient values: they do not get propagated to the next animation step.
 Normally, the persistent values are xyz, facing, and plane. However, it is possible to create a tangible with additional persistent values. For example, you could make a chest of gold, with a persistent state "open=true" to indicate that the chest is open. The chest would persist the value open=true until you issue a tangible.animate with open=false in it. After that, it would persist the open=false.
 When you create a new tangible, the animation queue is initialized with a single animation step containing initial values for all of the persistent key-value pairs.
 Unreal monitors the animation queue of every tangible. When it sees a new animation get added to the animation queue, it plays that animation. The rest of this document will explain how we have written unreal blueprints to handle animation queues and animation steps.
 ## Reasons that Animation Queues can Change in Unexpected Ways
 The Lua function *tangible.animate* appends a step to the animation queue. The majority of updates to the animation queue consist of appending. So mostly, Unreal can expect animation queues to grow monotonically.
 However, there are certain cases when something other than an append happens to an animation queue. There are three primary reasons that this can happen:
 - For interactive walking, if we were to append a new animation step every time somebody moves a few inches, then we would be appending a dozen steps per second. Since animation queues have fixed lengths, this would very rapidly push even fairly recent animations out the back end of the queue. This is undesirable. So the interactive walking code sometimes updates the XYZ of the most recent step, rather than appending a new step.
 - It is possible for the lua code to reinitialize the animation queue of a tangible. It might do that, for example, when changing the class of that tangible. Or, it might do that if it decides that the animation queue of the tangible is "messed up." When reinitializing the animation queue, all existing animations are discarded.
 - The difference transmitter can make corrections to the animation queue of the client. When it does, anything can change in any way. Animations can appear, disappear, and reappear. Animations can be altered in subtle ways. Animations can be inserted between existing animations. Anything can happen.
 Unreal monitors the animation queue of every tangible. Unreal must be prepared to deal with all of these atypical changes to the animation queue. Unreal can assume that *usually*, modifications to the animation queue are append-only, but occasionally, there will be exceptions.
 ## How Animation Queues get Transferred from Luprex to Unreal
 Animation queues are stored inside of Luprex. They are part of the world model. Unreal needs to see the animation queues, in order to be able to perform the animations. Therefore, an API is needed that lets us transfer animation queues from Luprex to Unreal.
 Animation queues can be serialized. We made the design decision that the API that transfers animation queues from Luprex to Unreal would use the serialized representation of the animation queue. So when we transfer the animation queue of one tangible from Luprex to Unreal, we're not transferring a complicated data structure: we're just transferring a string. This greatly simplifies the API.
 So, what follows is the actual API that transfers the animation queues, this is part of enginewrapper.hpp. You provide a list of tangible IDs, it returns a list of strings, one for each specified tangible. Each string is the serialized representation of an animation queue:
 ```cpp
 // Get the animation queues for the specified tangibles.
 //
 // You must supply an array of tangible IDs. For each tangible, returns the
 // animation queue as a serialized string. The serialized format is
 // documented in header file animqueue.hpp.
 //
 // You must also supply a buffer for the string lengths, and a buffer for
 // the string pointers. Both buffers must have space for number of
 // tangibles specified in the call.
 //
 // The returned character pointers remain valid until the next call to
 // get_animation_queues.
 //
 void (*get_animation_queues)(EngineWrapper *w, uint32_t count,
 const int64_t *ids, uint32_t *lengths, const char **strings);
 ```
 Notice that in this API, no string copies are being made. This is important, because this API is used every single frame. Unreal is being given a pointer to strings that reside within the world model.
 But think about that a little more: in order to be able to transfer the string without making a string-copy, the serialized string representation must already exist within the world model: it must already be there. So, in order to facilitate this API, we made another design decision: that animation queues would be stored in Luprex in serialized format. That means that whenever Luprex wants to manipulate an animation queue, it has to deserialize it, manipulate the deserialized representation, and then reserialize it. That's OK. We very likely will only do a handful of tangible.animate commands per second, and animation queues are fairly small data structures anyway (maybe a few kilobytes). Optimizing the API above, on the other hand, is extremely important because we use it every frame at 60fps, on every tangible.
 If Unreal had to deserialize every animation queue at every frame, that would be expensive. Fortunately, that's not necessary, because of another trick we use: animation step hash values.
 Each animation step has a hash value, which is computed from the content of the animation step mixed with the hash value of the previous step. The hash function we use is very good, the likelihood of a collision is very low. In the serialized representation of the animation queue, the most recent animation step is serialized first. And when serializing an animation step, the hash value is written first. So that means that the first 8 bytes of the serialized animation queue are always the hash value of the most recent animation step. Unreal can check whether the animation queue has changed by simply comparing the first 8 bytes of the serialized animation queue to the hash value it received the previous frame. If the hash value is the same, then the animation queue hasn't changed. In that event, unreal doesn't need to decode anything. Only when the animation queue actually changes does unreal need to decode it.
 For each tangible, Unreal stores a decoded, deserialized form of the animation queue. When the serialized representation retrieved from Luprex *does* change, then Unreal decodes the serialized representation and updates its own decoded, deserialized representation.
 This decoded, deserialized representation is a C++ data structure that lives within our Unreal project. It is updated by our Unreal-based C++ code. From there, it can be read by our Blueprints. Much of the rest of this paper discusses how our Blueprints fetch data from this animation queue representation.
 ## What the Tangible Actor Blueprint Classes Are
 We have written a Blueprint Class TangibleStaticMesh. By "Blueprint Class," I mean that it is not written in C++. It is written in Unreal's visual scripting language, blueprint. TangibleStaticMesh's main job is to execute the animation queue. It reads the queue, and then performs the animations specified in the queue.
 TangibleStaticMesh only supports tangibles that are static meshes. They can "moveto", they can "warpto", and they can switch from one static mesh to a different one, but they don't have much more behavior than that. They can't play recorded animations because they're static meshes.
 We also provide other classes, TangibleSkeletalMesh and TangibleCharacter. Collectively, TangibleStaticMesh, TangibleSkeletalMesh, and TangibleCharacter are called the "Tangible Actor" classes.
 These three classes provide a lot of common functionality. But eventually, the scripter will have to write his own Tangible Actor class. For example, suppose the game scripter wants a cannon that can fire cannonballs at a target specified in the *tangible.animate* command. Nothing in our code can do anything like that. So the scripter will have to write his own Tangible Actor class. He may be able to derive from one of our classes, however.
 TangibleStaticMesh is the simplest of the Tangible Actor classes. It follows "conventional" rules:
 - Animations are played one at a time.
 - Animations are played in the order that they appear in the queue.
 - When an animation is done, the next one begins.
 Those are the rules used by TangibleStaticMesh. Most other tangible actors will follow the same rules, but some tangible actors use different rules. It is possible to write a tangible actor that executes animations out of order, or that executes more than one animation at a time. In fact, there is no stipulation about how a Tangible Actor interprets the queue. "Tangible Actor" is a very open-ended job description: "a blueprint class that does whatever it wants to in response to the animation queue."
 Reasoning about that is difficult, so we're going to start by explaining TangibleStaticMesh, the simplest of the Tangible Actor classes. Later, we'll talk about Tangible Actors that do unconventional things.
 ## What Unreal Adds to the Animation Queue
 Unreal's copy of the animation queue is mostly a direct copy of what's in Luprex. But there are a few additions that Unreal makes to the animation queue.
 Unreal adds a finished bit to every step of the animation queue. This bit will be used to record which animation steps have already been played, so that our blueprints don't try to play them again. Again, the finished bit is not present in Luprex's copy of the animation queue, it only exists within Unreal's copy. Naturally, when a new animation step shows up in the animation queue, Unreal sets the finished bit for that animation step to false.
 Some tangibles have an idle animation. They play the idle animation when there's nothing else to do. In order to make this easier for tangible actors, the Unreal C++ code synthesizes an "idle" step which is always present at the end of the animation queue. Doing this makes it possible for the tangible actor to treat the "idle" animation as just another step in the animation queue. The blueprint doesn't need to have any special "idle state" that it keeps track of. It just keeps playing the steps in the animation queue, forever. The idle step cannot be marked *finished*, any attempt to mark it as finished will simply be ignored. That ensures the invariant that there's always an idle step to play, if there's nothing else to play.
 ## How Tangible Actor Blueprints Read the Animation Queue
 Tangible actor blueprints will be examining the animation queue. This section explains the blueprint functions that allow blueprints to fetch animation steps from the animation queue, and then fetch Key→Value pairs from the animation steps.
 The most important function to fetch an animation step from the queue is *GetCurrentAnimation*. This scans the animation queue, and returns the first animation step whose *finished* bit is not set. That's usually the animation step which is most important for a tangible actor's purposes.
 The animation step is returned in the form of a *struct lxAnimationStep*. This is a C++ struct which is also a valid type in blueprint language. It contains the Key→Value pairs of the animation step, and also the animation step's hash value. In blueprint language, structs are passed by value, not by reference, therefore, the struct which is returned is a *copy* of some step in the animation queue. You can make blueprint variables of type *lxAnimationStep*, so animation steps are first class values in blueprint language.
 The function *GetCurrentAnimation* is one of many accessors that can fetch animation steps from the animation queue. We plan to write a full library of accessors: functions to get the Nth step, functions to scan the queue for steps with particular attributes, and so forth. The intent is that the blueprint is meant to have unlimited access to the entire animation queue. Most of this is not implemented yet, but it is important that blueprints can read the entire queue, and they need to be able to find what they want to find as efficiently as possible.
 Once the blueprint has an animation step in the form of a lxAnimationStep, the next thing it needs to do is read the Key→Value pairs in it. The blueprint programmer can do that using accessors like these:
 ```
 Action = AnimationStepGetString(step, "action")
 XYZ = AnimationStepGetVector(step, "xyz")
 Plane = AnimationStepGetString(step, "plane")
 Model = AnimationStepGetString(step, "model")
 Facing = AnimationStepGetFloat(step, "facing")
 ```
 Of course, blueprint language is a visual scripting language. The code above looks pretty simple when written in a textual form, but it takes fifteen little rectangles in the event graph just to do the five assignment statements above. That's half a screen full of code. In order to make that more concise, we've provided the accessor *UnpackAnimationStep*. Here's how it works. You add variables like this to your blueprint:
 ```
 Aq Action (string)
 Aq XYZ (vector)
 Aq Plane (string)
 Aq Model (string)
 Aq Facing (float)
 ```
 Notice that all the variable names have a prefix, "Aq". Then, you use a single command, UnpackAnimationStep, to assign values to those variables:
 ```
 UnpackAnimationStep(step, prefix="Aq", into=self)
 ```
 This pulls the Key→Value pairs out of the animation step, and stuffs them into the blueprint variables whose name begins with "Aq." Using a variable name prefix ensures that *UnpackAnimationStep* won't overwrite variables that it isn't supposed to be touching.
 Using *UnpackAnimationStep* is entirely equivalent to fetching all the values one by one. The two versions of the code above fetch exactly the same values, and the results are exactly the same.
 Here's an animation step that I mentioned earlier in this paper:
 ```
 action: "jump"
 height: 3.0
 xyz= (100,100,0)
 plane= "earth"
 facing= 50
 ```
 This contains a pair "height: 3.0". But in code above which uses simple accessors, there's no call to *AnimationStepGetFloat(step, "height").* So that block of code would ignore the height. Meanwhile, in the version of the code that uses *UnpackAnimationStep*, there's no variable "Aq Height." Either way, height gets ignored.
 But what if the blueprint code that implements the "jump" animation needs to know the height? The blueprint author has two options. He can add a variable "Aq Height," and let *UnpackAnimationStep* populate it. His other option is to add a call to *AnimationStepGetFloat(step, "height")* at the point in the code where the height is needed. Either option is perfectly valid.
 You might also notice that there's a variable "Aq Model," but the animation step above doesn't contain a "model=value" pair. So *UnpackAnimationStep* will set variable "Aq Model" to its default value, which for string variables is the empty string. Likewise, the accessor *AnimationStepGetString(step, "model")* will return empty string.
 So now you know how a blueprint can read the animation queue. It has unlimited access to the queue, it can read the whole queue. Next, we'll tell you what a TangibleStaticMesh, our simplest tangible actor, does with the queue.
 ## How TangibleStaticMesh gets Notified when the Animation Queue Changes
 TangibleStaticMesh needs to examine the animation queue to play the animations in it. But we don't want TangibleStaticMesh to examine the animation queue every "tick." Remember, TangibleStaticMesh is written in blueprint language, which is an interpreted language that is not particularly fast. Therefore, we only want the blueprint to look at the animation queue when the animation queue changes.
 To that end, class TangibleStaticMesh implements a function *AnimationQueueChanged*. This function is written in blueprint, and it is automatically called by the C++ code whenever the animation queue changes in any way. It is called immediately after Unreal fetches an animation queue from Luprex and detects a change.
 In Unreal, to call from C++ code into Blueprint code, you need to use what is called a "blueprint interface." Therefore, *AnimationQueueChanged* is part of a blueprint interface called *lxTangibleInterface*. All Tangible Actor classes implement the interface, and our C++ code that calls *AnimationQueueChanged* uses the interface to call it. When looking for the function *AnimationQueueChanged* in the blueprint editor, you will find it in the left hand panel under *Interfaces*, under *Tangible Functionality*.
 ## How TangibleStaticMesh Initiates Animations
 When the C++ code calls *AnimationQueueChanged*, the first thing that *AnimationQueueChanged* does is to call the function *GetCurrentAnimation*. This scans the animation queue for the first animation step whose *finished* bit is not set. This is the animation step that TangibleStaticMesh wants to be playing.
 *GetCurrentAnimation* returns the animation step in the form of a *struct LxAnimationStep*.
 Then, TangibleStaticMesh checks to see if it is already playing that animation step. TangibleStaticMesh accomplishes this by storing the currently-playing animation step in a variable *CurrentAnimation*, of type *LxAnimationStep*. If the animation step returned by *GetCurrentAnimation* matches the one stored in *CurrentAnimation*, then *AnimationQueueChanged* doesn't need to do anything. It can return immediately.
 If *GetCurrentAnimation* returns an animation step that doesn't match the variable *CurrentAnimation*, then TangibleStaticMesh theoretically has to abort the old animation. In reality, you won't find any code to abort anything, because TangibleStaticMesh is so trivial: it can't actually play animations other than "moveto" and "warpto," so there's nothing to clean up or abort. If you were a scripter writing a more complicated Tangible Actor, then it might be necessary to do some cleanup steps here.
 After aborting the old animation, *AnimationQueueChanged* stores the new animation's *LxAnimationStep* in the variable *CurrentAnimation*. Then, it unpacks the step using *UnpackAnimationStep*. The Key→Value pairs are transferred into blueprint variables, to make it easy for the blueprint to access them. The variables used have the variable name prefix "Aq." The Aq variables contain the same animation step as *CurrentAnimation*, in an unpacked form.
 Next, AnimationQueueChanged checks if "Aq Model" has changed. If so, it switches the static mesh to a different mesh.
 Finally, *AnimationQueueChanged* actually initiates the new animation. In TangibleStaticMesh, you will find a "Call Function By Name" that calls *Initiate Action: Moveto*, *Initiate Action: Turnto*, or some other tick-function depending on whether *Aq Action* contains "moveto", "turnto", or some other value.
 In TangibleStaticMesh, the routines that initiate different actions are extremely simple. For example, to initiate a "moveto," all we have to do is store the target XYZ in a variable. But that has already been done: it's in the variable "Aq XYZ." The actual movement will be handled by a tick-function, which I'll explain in a moment. So the routine *Initiate Action: Moveto* is actually a no-op, because everything that needs to be done has already been done. In TangibleStaticMesh, the other routines that initiate animations are all equally trivial.
 This is the process by which TangibleStaticMesh initiates new animation steps.
 ## How Some Animations involve Tick Functions
 After TangibleStaticMesh initiates a "moveto" action, the next thing that has to happen is that every frame, the actor location needs to move a little bit toward its target. This movement is handled by a tick-function.
 The event graph of TangibleStaticMesh has a "tick" event. That tick-event uses "Call Function by Name" to call *Tick Action: Moveto*, *Tick Action: Turnto,* or some other tick function depending on whether *Aq Action* contains "moveto", "turnto", or some other value.
 The function *Tick Action: Moveto* adds a small increment to actor location, moving it a tiny bit at a time toward Aq XYZ. At the end of *Tick Action: Moveto* it checks whether the tangible has reached its destination. If so, it calls the function *FinishedAnimation.* In the next section, we will explain what *FinishedAnimation* does, and how it leads to the next animation getting played.
 That's how TangibleStaticMesh uses tick functions to update the state of ongoing animations.
 In blueprint code, tick functions are discouraged, because blueprint code is interpreted code and it isn't that fast. It is conceivably possible that you could write a version of "moveto" that doesn't require a tick-function. Perhaps there's some Unreal builtin functionality that can move an actor toward a destination automatically: I don't know. If so, you would have to tell Unreal to do the movement automatically, and you would have to tell Unreal to invoke a callback when the movement is done. The callback would be responsible for calling *FinishedAnimation*.
 When TangibleStaticMesh initiates a new animation, it first has to abort any already-playing animation. Currently, there's not anything that TangibleStaticMesh has to do when it "aborts" the existing animations. But if you implemented "moveto" by telling Unreal to perform the movement automatically, then any code to abort already-playing animations would need to be able to abort these automatic movements.
 ## What TangibleStaticMesh does when Animations Finish
 In the code that implements the "moveto" animation, in the function *Tick Action: Moveto* there is an if-statement that checks if the tangible has reached its destination. If so, it calls the function *FinishedAnimation*.
 All *FinishedAnimation* does is set the finished bit in a specified step in the animation queue. Setting the finished bit counts as a modification of the animation queue. This, in turn, guarantees that next frame, the C++ code will call *AnimationQueueChanged*.
 In TangibleStaticMesh, the first thing that *AnimationQueueChanged* does is call *GetCurrentAnimation*. GetCurrentAnimation returns the first animation in the queue whose *finished* bit is not set. So it won't return the step that was just finished. It will return the step after that.
 So here's the chain of events that gets triggered, summarized:
 - Something in the blueprint calls *FinishedAnimation*. This sets a *finished* bit.
 - The *finished* bit is a change to the animation queue. This triggers *AnimationQueueChanged*.
 - *AnimationQueueChanged* calls *GetCurrentAnimation*, as it always does.
 - *GetCurrentAnimation* returns a new animation, because the old one's *finished* bit is set.
 - The new animation gets initiated.
 Through this chain of events, the *FinishedAnimation* in *Tick Action: Moveto* triggers the playing of the next animation in the queue.
 It is perfectly safe to call FinishedAnimation from anywhere. You can call it from a tick-function, you can call it from a callback, whenever you figure out that an animation has finished, you can call it from wherever you are in the blueprint code.
 Some animations are instantaneous, such as "warpto." They take zero frames to complete. The following sequence of events occurs: the lua code pushes the "warpto" animation into the queue. The Unreal C++ code sees this and calls *AnimationQueueChanged*. This, in turn, calls *Init Action: Warpto*. That routine actually performs the warpto, and then calls *FinishedAnimation* immediately. So it's even fine to call *FinishedAnimation* directly from the routine that initiates an animation, if you want the animation to be finished as soon as it starts.
 The routine FinishAnimation takes an *lxAnimationStep* as a parameter. This is so that it knows which animation step to mark as finished. TangibleStaticMesh always passes in the variable *CurrentAnimation*.
 *FinishAnimation* also takes three boolean parameters: Auto Update XYZ, Auto Update Plane, and Auto Update Facing. These require some explanation. Suppose that the lua programmer pushes an animation step that looks like this:
 ```lua
 tangible.animate(actor, nil, {action="emote", animation="dance", xyz={1,2,3}})
 ```
 Since "play an emote" isn't a travel command like "moveto" or "warpto", the lua scripter shouldn't have specified a destination XYZ, but he did so anyway, and now we have to recover from the mistake. Specifically, the tangible actor really does need to move to the new XYZ coordinate, otherwise bad things happen.
 The convention that we have adopted is that to recover from this type of mistake, it is considered acceptable to just play the emote in-place (ie, without moving the actor), then, when the emote is fully finished, the blueprint warps the player to the specified XYZ coordinate. In other words, every animation step is treated as if it has an *implicit* "warpto" at the end of it. This rule guarantees that if the lua programmer sets the xyz, facing, or plane in an animation step, the character will end up at the desired xyz, facing, and plane, no matter what the animation step is.
 That's why *FinishAnimation* has those three boolean flags: Auto Update XYZ, Auto Update Plane, Auto Update Facing. If those are all true – and they almost always should be – then *FinishAnimation* will implement the implicit "warpto" for you. I cannot think of a situation where you would want these flags to be false, but I have left the option, in case somebody wants to do something odd in a Tangible Actor.
 ## How Tangible Actors handle Warping Away
 Suppose that in your game, players can teleport from one place to another by casting a "teleport" spell. Suppose that there is a spellcasting animation with lots of sparkles whenever somebody teleports. You would think that this would be the same as any other animation, but in fact, there is some special-case logic to deal with this case.
 Here is the tricky scenario: player T, the teleporter, is standing in front of player V, the viewer. Player V can see player T. Then, player T casts the teleport spell, and executes this lua code:
 ```lua
 tangible.animate(actor, nil, {action="sparkle", particles=999})
 tangible.animate(actor, nil, {action="warpto", xyz={1000000, 0, 0}})
 ```
 Luprex has no idea how long these animations take to execute. As far as Luprex is concerned, they are instantaneous. As soon as player T executes that code, Luprex thinks that player T has moved to {1000000, 0, 0}, a million miles away from player V.
 Every frame, Unreal asks Luprex what tangibles are within the view radius. When player V's Unreal client does a *scanradius*, the scan no longer returns player T, because Luprex thinks that player T is a million miles away. Yet, player V's Unreal client hasn't even begun to play the sparkle animation. How is it supposed to play the sparkle animation if Luprex is no longer even reporting the existence of tangible T?
 The answer is that Tangible Actor T was already visible to player V the previous frame, and our Unreal code doesn't allow Tangible Actors to blink out of existence that easily. Once Tangible Actor T exists in Unreal, that Tangible Actor will continue to exist in Unreal, and Unreal will continue to query its animation queue from Luprex. Luprex can still report the animation queue: even though T isn't in player V's scanradius any more, T still exists and still has an animation queue. So Luprex can still report the animation queue to Unreal.
 Since Tangible Actor T hasn't played the sparkle animation or the warpto animation yet, Tangible Actor T has an "actor location" that is still in front of player V. That means there's a disconnect: the Luprex tangible is already at {1000000, 0, 0}, far away. But the Unreal Tangible Actor T is still located in front of player V. It will remain there for however long it takes to play the sparkle animation.
 When Tangible Actor T is done with the sparkle animation, only then does it play the "warpto" animation. At that point, it updates its actor location to {1000000, 0, 0}, using Unreal's built-in function, "Set Actor Location" or something similar. Once it has done that, the Tangible Actor T is finally at the same location as Luprex's tangible T.
 In order for Unreal to actually discard a Tangible Actor, two conditions must be met. First, the tangible has to no longer be appearing in the Luprex *scanradius*. In other words, it has to be "far away according to Luprex." Second, it has to have an actor location that is far away from the viewer. In other words, it has to be "far away according to Unreal." When both of these are true, the tangible actor is garbage collected. So on Player V's client, Tangible Actor T will get automatically deleted by the C++ code one frame after the "warpto" animation is played.
 Now, let's consider a small variation of the lua commands:
 ```lua
 tangible.animate(actor, nil, {action="sparkle", particles=999})
 tangible.animate(actor, nil, {action="warpto", plane="somewhere-else"})
 ```
 Here, instead of changing the XYZ of the player, the warpto changes the plane of the player. Unreal engine doesn't have the concept of "planes." What is the blueprint supposed to do? The answer is that we've added the concept of "plane" to Unreal at the C++ level. We have provided an API function *SetTangiblePlane*, that takes a string. When the tangible actor executes the "warpto" animation, that's all it has to do. The C++ code will make sure to only render tangibles that are on the same plane as the player.
 ## How TangibleCharacter implements Interactive Movement
 Blueprint class TangibleCharacter is meant to implement bipedal characters, which can be interactively controlled using a gamepad or keyboard. TangibleCharacter is derived from Unreal's built-in "Character" class.
 TangibleCharacter has two modes that it switches between: scripted mode, and interactive mode. When lua uses *tangible.animate* to push an animation onto the queue, TangibleCharacter switches into scripted mode, and plays the scripted animation. But when there are no more *tangible.animate* commands left to play, TangibleCharacter switches into interactive mode.
 In scripted mode, TangibleCharacter works just like TangibleStaticMesh. There's a function *AnimationQueueChanged*, which calls *GetCurrentAnimation*. Just like TangibleStaticMesh, there's a function *Initiate Action: Moveto*, there's a function *Tick Action: Moveto*, and so forth. The code is a little more complex because we have to control the animation blueprint (more on that later), but it's all structured in exactly the same way.
 Interactive mode is activated when there are no more tangible.animate commands left to play. Specifically, it is activated when *AnimationQueueChanged* calls *GetCurrentAnimation*, and *GetCurrentAnimation* returns the idle animation. Interactive mode is implemented entirely by the functions *Initiate Action: Idle* and *Tick Action: Idle*.
 When we enter into interactive mode (in *Initiate Action: Idle*), we enable Unreal's built-in "Character Movement Component." This is the part of Unreal that implements interactive movement for characters. Enabling the movement component causes the gamepad and keyboard to become active. Pushing the gamepad stick will cause the actor to move. This movement follows all of Unreal's built-in movement rules: you can't walk through walls, you stay on the ground, you move at the speed specified in the character blueprint parameters, and so forth. When we switch to scripted movement, the Character Movement Component is disabled: it is only used for interactive movement.
 When you move interactively, Luprex needs to be notified. In *Tick Action: Idle*, the code checks to see whether the character has moved a significant distance from where Luprex thinks it is. If the distance is large enough, *Tick Action: Idle* invokes the Lua function *engio.move*. This Lua function calls *tangible.animate* to add a moveto animation to the queue, to reflect the interactive movement that recently occurred.
 The next time Unreal fetches the animation queues from Luprex, it will see the animation that *engio.move* just added to the queue. We don't want Unreal to switch into scripted mode and play the animation that just appeared in the queue, because it *already* performed that movement. In order to prevent this from happening, after invoking *engio.move*, the blueprint calls *SetAutoFinish*. This anticipates the appearance of the "moveto" in the animation queue, and asks our Unreal C++ code to automatically mark that "moveto" as *finished* as soon as it appears. This will prevent the switch to scripted mode.
 TangibleCharacter is used for the player character, and also other players and non-player characters. For characters other than the player, interactive mode is disabled. Instead, we just play an idle animation.
 ## How TangibleCharacter interfaces to its Animation Blueprint
 The Unreal distribution includes a sample character, Manny, which uses an animation blueprint, ABP_Manny. We use a slightly-modified version of ABP_Manny for TangibleCharacter. I will explain how ABP_Manny works, and what we have altered.
 Every frame, ABP_Manny fetches four parameters from Manny's Character Movement Component:
 - Actor Location
 - Velocity Vector
 - "Is Falling" flag
 - "Is Accelerating" flag
 Using these four parameters, ABP_Manny automatically switches between walking, running, falling, jumping, and idling animations. These are roughly the rules it uses:
 - If Manny "is Falling" and his velocity is downward, transition into the "falling" pose.
 - If Manny "is Falling" and his velocity is upward, transition into the "jumping" pose.
 - If Manny is not falling and his velocity is zero, play the "idle" animation.
 - If Manny is not falling and his velocity is nonzero, put him in the "walking/running" blend.
 - The "is Accelerating" flag tweaks the walking/running blend.
 - If Manny's feet are close to the ground, use IK to snap his feet to the ground.
 I may have gotten some details wrong. The important thing to know, however, is that ABP_Manny takes care of all of this. That means that the Manny Blueprint doesn't have to implement animation. It can leave it all up to ABP_Manny.
 For example, when Manny is walking along flat ground, Manny's Character Movement Component is using "Set Actor Location" every frame to continuously move Manny along a trajectory. It does not have to reason about his body shape, his skeleton, or his footsteps. It just moves him along a linear, straight path in a smooth, continuous way.
 Manny's Character Movement Component does have to pay attention to collisions: it thinks of Manny as a capsule, it makes sure the capsule does not penetrate a wall. It also makes sure the capsule doesn't pass through the floor, or rise off the ground. This functionality is all built-in to the Character Movement Component, so no blueprint code is needed to implement it.
 What TangibleCharacter does differently from Manny is as follows. TangibleCharacter has two modes, scripted and interactive. By contrast, Manny only has interactive mode. TangibleCharacter's two movement modes are implemented separately: interactive mode uses the Character Movement Component (like Manny), scripted mode does not. In both modes, the four key parameters must be provided to the animation blueprint:
 - Actor Location
 - Velocity Vector
 - "Is Falling" flag
 - "Is Accelerating" flag
 In interactive mode, these four parameters are taken directly from the Character Movement Component. But in scripted mode, TangibleCharacter has to calculate them. In the original code for ABP_Manny, it used to reach directly into the Character Movement Component to fetch these values. In our updated version of the code, the animation blueprint fetches these values using a formal interface. That's the only change we have made to ABP_Manny.
 In scripted mode, we ignore collisions, because we want the character to do what the script says, no questions asked. If a *tangible.animate* says that the TangibleCharacter should walk through a wall, he walks right through. So really all that the scripted "moveto" routine in TangibleCharacter has to do is move the Actor Location in a smooth, continuous line toward the destination location. Of course, it also has to report the velocity vector (trivial), and that he's not falling. That's it. Our modified version of ABP_Manny handles the rest.
 ## Interpreting the Animation Queue in Nontraditional Ways
 In TangibleStaticMesh, animations are played one at a time. But you can imagine a Tangible Actor that plays more than one animation at a time.
 For example, imagine a "fireworks launcher" object that launches rockets, and which can launch many rockets simultaneously. The rockets are owned by the launcher: it's actually just one Tangible Actor, the launcher and its rockets. Now imagine that the fireworks launcher is controlled using animate commands like this:
 ```lua
 tangible.animate(actor, nil, {action="launch", type="burst", color="blue"})
 tangible.animate(actor, nil, {action="launch", type="burst", color="green"})
 tangible.animate(actor, nil, {action="wait"})
 tangible.animate(actor, nil, {action="launch", type="sparkle", color="blue"})
 tangible.animate(actor, nil, {action="launch", type="sparkle", color="green"})
 tangible.animate(actor, nil, {action="wait"})
 ```
 Our rocket launcher has the following unusual behavior: when it sees a sequence of animation steps with action="launch", it launches them all at the same time. So in response to the *tangible.animate* commands above, our launcher would launch two "burst" rockets simultaneously. Then, it would wait for both of those rockets to finish, however long that takes. Then, it launches two "sparkle" rockets simultaneously. Then it waits for the two sparkles to finish.
 This illustrates the principle that a Tangible Actor is allowed to interpret its animation queue in whatever way it wishes. Most Tangible Actors will play animations one at a time, but as you can see from the fireworks launcher, there's no rule against playing multiple animations at once.
 You can think of a Tangible Actor as a sort of a script interpreter, and the animation queue is the script that it interprets. This is a flexible way of doing things: if there's something that one of our existing Tangible Actors can't do, you can write a new one, and you can design a simple "scripting language" to control it that makes it possible to achieve the behaviors you want.
 It's interesting to consider how the rocket launcher class would be implemented. A lot of the code structure would be the same as in TangibleStaticMesh. But there are a few things that would have to be done differently.
 For example, in class TangibleStaticMesh, aborting animations is simple: it just aborts the old animation whenever a new animation begins. But in the fireworks launcher, it needs to do something more complicated, because there is more than one animation in flight at a time. It must keep a list of the rockets that are in-flight, and for each one, it needs to record the *lxAnimationStep* that launched the rocket. In *AnimationQueueChanged*, it should run over this list and for each rocket in-flight, it should make sure that the animation step that launched the rocket is still present in the animation queue. If not, it needs to abort that rocket.
 One interesting question is what to do with the *finished* bit in the animation queue. Should a launch step be marked "finished" as soon as it is launched, or should it be marked "finished" when the rocket is completely done exploding? The answer is that there is no right answer. In fact, the fireworks launcher isn't required to *ever* set the finished bit. The *finished* bit is just a convenience designed to make it easy to implement "the usual." If the blueprint wants to keep its own records of what it has played, and ignore the *finished* bit, that's valid. In practice, it's probably most convenient for the launcher to mark a rocket-launch step finished as soon as the launcher has launched the rocket. That way, it can immediately use *GetCurrentAnimation* to get the next rocket in the sequence.
 In general, we don't expect the game developer to have to write custom blueprints like this all the time. The provided Tangible Actor classes (TangibleStaticMesh, TangibleSkeletalMesh, and TangibleCharacter) should be general enough for many of the objects in the game. But we expect that there will occasionally be an object which needs more interesting behavior, so we expect that the game developer will from time to time need to write his own blueprint classes that do things that our Tangible Actors don't do.
 When the game developer writes his own Tangible Actor classes, it may be possible to derive from our Tangible Actor classes. Right now, that isn't very feasible, because our Tangible Actor classes are spaghetti code. That's the case because I'm not very good at blueprint programming yet. My plan is that in a few months, when I'm better at blueprint programming, I'll refactor the Tangible Actors to make them good classes to derive from.
 But, even then, it may be simpler to write new Tangible Actors from scratch. That will especially be the case if I provide a well-designed library of utility functions to make it easy to write Tangible Actors. I think this may be the best approach: rather than encouraging people to derive from our classes, encourage them to study our classes and then write their own from scratch.
--- a/NoPredict.md
+++ b/NoPredict.md
@@ -0,0 +1,47 @@
 ## IMPLEMENTING A BLOCKING OPERATION
 We're going to use 'http.get' as a prototypical example of a blocking operation.
 Before you do any blocking operation, you need to use a guard to see if it is actually safe to do a blocking operation. The guard always has three cases:
 **Case 1. Probe.** Check for !lua_isyieldable. If you're not in a yieldable context, then you're in a probe. You can't do any blocking operation at all. Throw an error.
 If you accidentally fail to check this, and yield anyway, the yield will generate an "attempt to yield from outside a coroutine" error (a lua built-in error). This, in turn, should trigger an assert fail.
 **Case 2.** **Nonauthoritative**. Check for !World::is_authoritative. Blocking operations in a nonauthoritative model should trigger a nopredict. In this case, you should call 'lua_yield(L, 0)', this will automatically be converted to a nopredict by run_scheduled_threads.
 In this case, you must remember that you killed the thread, and therefore, you shouldn't schedule any kind of thread wakeup - there is no thread to wake up.
 **Case 3.** **Authoritative.** You're in a yieldable context, in an authoritative model. You can actually do the blocking operation.
 These are the steps of performing a blocking operation:
 - Initiate the operation. For example, you might store an HTTP request in a queue of outgoing HTTP requests.
 - Store your place ID and thread ID so that your thread can be awakened when the operation completes. For example, you might store it in the HTTP request data structure. You can get your place and thread ID from lthread_place_id_ and lthread_thread_id_.
 - Execute the code 'return lua_yield(L, 0)'. This ends the blocking subroutine while pausing the thread.
 - When the system code determines that the blocking is complete, it must generate the return value for the blocking operation. It stores this return value by pushing it onto the stack of the sleeping thread. To find the sleeping thread, it must look up the thread's tangible, and then find the thread in the thread table of the tangible. If for some reason there's no such thread (eg, tangible has been bulldozed), simply drop the operation.
 - Finally, store the place ID and thread ID in the scheduler queue with time zero, and call run_scheduled_threads.
 ## NOPREDICT
 NoPredict follows a similar pattern to blocking: there are three cases.
 **Case 1. Probe.** Check for !lua_isyieldable. If you're not in a yieldable context, then you're in a probe. NoPredict should have no effect in this case. Do nothing.
 **Case 2. Authoritative.** Check for World::is_authoritative. If you're in an authoritative context, then nopredict should not do anything. Do nothing.
 **Case 3. Nonauthoritative.** You're in a yieldable context, in a nonauthoritative model. Call 'lua_yield(L, 0)'. This will automatically be converted to a nopredict by run_scheduled_threads.
 In other words, the correct code snippet for 'nopredict' is:
 ```cpp
 if (lua_isyieldable(L) && !w->is_authoritative())
    return lua_yield(L, 0);
 }
 ```
 When doing a nopredict, Instead of checking is_authoritative, you can check for the presence of the registry key "globaldb", or the registry key "oncedb." This avoids having to include "world.hpp"
--- a/Docs/Difference
+++ b/Docs/Difference
@@ -0,0 +1,155 @@
 ### Concurrent Difference Transmission
 Our system difference transmits sequentially. First, it transmits to client 1. Then, it transmits to client 2. It keeps going until it has done all the clients. We came up with the idea that it would be awesome if the system could use threads to difference transmit to all clients simultaneously.
 Suppose you were to create one thread per client. Each thread would be responsible for comparing the master model to a single client model. In that case, all of the threads would be reading from the master model at the same time. What if we were to try to do this without any mutexes?
 In the abstract, you would not expect that to be a problem. Almost all data structures in C++ are thread-safe for multiple readers at the same time. For example, if I create a std::map, then I can have an unlimited number of threads reading from the map at the same time, and there's no conflict. The only time you need a mutex is if somebody is mutating the data structure. In difference transmission, nobody needs to mutate the master model, so you would think it would be possible for all those threads to read from the master model at the same time, with no mutex.
 The following sections will explain why this idea falls apart, and what the problems are. Most of the problems have to do with the Lua runtime, which simply wasn't designed with threads in mind.
 Then, the sections after that will tell you some solutions that solve some of the problems, but each solution has its own issues. Currently, there isn't any one obvious solution that would solve everything at the same time.
 ## Using Threads: The Stack Problem
 Let's say you want to read from a Lua table, you want to get the value of table[5]. You would use lua_rawget to do this, lua_rawget reads a key-value pair from a table. It doesn't mutate the table. But to do a lua_rawget, you have to push the key onto the stack. Pushing a value onto a stack is mutating that stack. Then, lua_rawget pops the key from the stack and pushes the result on the stack. So lua_rawget doesn't mutate the table it's reading from, but it *does* mutate the stack.
 All of the functions in the Lua API mutate the lua stack. If you're using the Lua API to read parts of the master model, then you're mutating the master model. There's simply no way to "read only" from the master model, because every function in the lua API mutates the lua stack.
 However, there is a potential solution for the stack problem. We could create a lua stack for each client thread. Each thread would indeed be mutating its own private lua stack, but since there is only one thread per stack, there's no race conditions. That eliminates the contention over the lua stack.
 Each lua stack would have to be sized to make room for "big" recursion. It would not be feasible to enlarge the stacks after the fact.
 ## Using Threads: The Garbage Collector Problem
 The lua garbage collector is, I believe, an incremental garbage collector. It does a little bit of garbage collection work every time you call a Lua API function. So that, unfortunately, makes every single API function a mutator of the lua heap.
 I believe that lua has the means to shut down the garbage collector entirely for a time. We could just plain turn it off during difference transmission.
 I'm not sure this is as simple as it sounds. Incremental garbage collectors have interesting things they do with marking objects 'white' and 'black', and these flag-setting operations are scattered all over the code.
 ## Using Threads: The Visited Bit Problem
 Our difference transmission code traverses tables recursively. Because tables can contain cycles, ie, two tables can point to each other, the recursive traversal algorithm needs to use a *visited* bit to avoid getting into an infinite loop. Currently, we store the bit in the table itself, because that's fast.
 However, it's also mutation of the table. It's not compatible with a read-only approach to scanning the master world model. We could change the code to store the visited set in some other way, it's solvable, but we need to be aware of this.
 ## Using Threads: The String Problem
 Suppose you want to read a table: you want to find out the value of table["foo"]. To do that with the Lua API, you have to do a lua_pushstring("foo"). That, of course, mutates the lua stack, but we've solved the stack problem. Unfortunately, it also allocates dynamic memory in the lua runtime to hold the string "foo", and it also puts that string into Lua's central string table. All of the threads would be mutating the lua memory heap and the central string table every time they do a lua_pushstring.
 We could solve this problem with a lua patch. Currently, lua's central string table only contains short strings. We could modify it so that all strings go into the central string table, even long ones. That way, every string used anywhere in the lua runtime will be in the central string table.
 Then, we could create an API function lua_pushexistingstring. This pushes a string onto the stack, but only if the string is already present in the lua central string table. This function doesn't ever allocate any new strings. This function would be sufficient for table lookups with string keys.
 The routine internshrstr in the lua runtime contains an 'isdead' check. If isdead is true, then I think we need to treat the string as if it's not already present in the central string table.
 ## Using Threads: The Cache Problem
 It is entirely possible that something within the lua runtime uses caching.
 In fact, I believe I have found an example of caching: metamethod lookup. When you do a lua_gettable, it has to check to see if the table has a metatable, and if it does, it has to check if the metatable has an "INDEX" metamethod. This is a lot of extra work for every single table lookup. I believe that the lua runtime uses a fairly simple trick to optimize this. After it looks for the metamethod "INDEX", if there is no such metamethod, it stores a flag in the table to indicate, "don't bother looking for an INDEX, there isn't one." The next time you do a table lookup on that table, it skips the whole INDEX check.
 Unfortunately, that means that when you do a lua_rawget on a table, you may be mutating the flag bit. The flag bit is probably not a show-stopper, because it's just a bit. If two threads try to set it to "true" at the same time, then it still gets set to true, no problem. There's also no reason that two threads would disagree about whether it should be true or false: either INDEX is there, or it's not. The threads should agree.
 The real problem is: what if something else in the lua runtime uses caching in a more complicated way? The Lua codebase is too large to reliably search for examples. I think this is just a significant risk and I don't see a simple solution.
 ## Using Threads: The Nondeterminism Problem
 The code to do difference transmission uses the memory allocator. Of course, there are lots of thread-safe memory allocators that we could use. But even if the memory allocator were thread-safe, having multiple threads accessing it concurrently would lead to a situation where the memory addresses of things in the heap become unpredictable and nondeterministic.
 We have worked hard to ensure the determinism of the Luprex library: see the document "The Event-Driven Structure of the Engine" for explanations of why we want determinism, and how threads screw up the determinism. I don't see a simple solution to the determinism problem, other than to not use threads, or to give up on determinism.
 ## Using Threads: The Unknown Problem
 So I've done my damndest to find things in the lua runtime that cause "unexpected mutation." I found the stack problem, the garbage collector problem, the string problem, and the cache problem. But what if there's a fifth problem that I didn't find? We could write a big complicated multithreaded difference transmission system, and it might just crash randomly, and we wouldn't know why.
 I don't see any simple solution to this problem. Lua just wasn't written with multiple threads in mind.
 ## Solution: Using Mutexes
 One solution to all this would be to use a mutex to control access to the master model. Obviously, if there was only one thread inside the master lua at a time, then most of the problems above would be solved (not all: the determinism problem would still be an issue).
 The thing I worry about is that if we do this, then pretty much, all the threads are just going to do nothing until they get their turn at the master model. I'm afraid that using a mutex isn't a solution, it just turns it back into sequential difference transmission.
 We could try to do very fine-grained locking, where we lock the mutex, read one value from one table, and then unlock the mutex again. I'm afraid that the overhead of the locking would exceed the amount of time saved.
 ## Solution: Writing a Brand-New Lua API
 Could we add a brand new API to lua, one that is designed from the ground up to not mutate anything? I think it's conceivably possible, but it would be a LOT of code.
 So first of all, to solve the stack problem, the new API would not be passing parameters by pushing them on the lua stack. Instead, we could just pass parameters the regular way, as… well, C parameters. Normally, the garbage collector can't allow us to put lua pointers into C variables, but if we shut down the garbage collector when using this API, that would not be a problem. So this API would be set up so that it only works when the garbage collector is off.
 So that part is pretty straightforward. But then, we'd have to actually write the code for the getters. For example, you want to write a replacement for lua_gettable. Our replacement might look like this:
 ```cpp
 LuaValue our_lua_gettable(LuaValue table, LuaValue key);
 ```
 We could write that… but we'd have to totally rewrite the code for table lookup, otherwise we might trigger the caching problem I told you about, or the unknown problem I told you about.
 So what I'm saying is: this is certainly possible, but it's a lot of work.
 ## Solution: Writing a Brand-New Lua Implementation
 We could just rewrite lua from scratch. If we did so, we could make it entirely thread-safe for multiple readers. That sounds crazy, but lua's a simple language. I don't think it would take that long to write a new implementation.
 If we did that, it might make sense to write a non-incremental garbage collector. That would hugely simplify the whole garbage collection issue.
 The big problem, though, is that we've implemented *tons* of C++ functions that hook into lua. All of that code would have to be rewritten.
 ## Solution: Serializing the Master World Model
 Another solution to the contention over the master world model is that before the difference transmission process begins, we could serialize the entire master world model, turning it into the "master world string". Then, we could start a thread for each client. Each thread would deserialize the master world string, so each thread would then have a copy of the master world model all to itself. This solves all of the issues having to do with contention for the master world model.
 In fact, you could optimize it further: since it's really just the lua interpreter where the problems reside, you could just serialize the lua part of the master world model. The threads could concurrently read from the C++ parts of the master world model, but they would each have their own copy of the lua parts.
 The problem with this is that it requires copying the entire lua interpreter of the master world model using the eris serialization code. I don't think eris is designed to be particularly fast.
 ## Solution: Memcpy of the Master Lua Interpreter
 Suppose that we were to arrange for the master world models' lua interpreter to reside in a custom memory allocator, so that all the lua data structures live in a contiguous region of memory. This is quite feasible, lua allows us to hook the memory allocator. In that case, each thread could make a memcpy of the master world model's lua interpreter by simply block-copying the lua interpreter's memory region. Then, each thread would have its own copy of the master world model's lua interpreter. This would solve all the issues of contention within Lua.
 Unfortunately, the copies wouldn't be valid, because all the pointers in the copy would point back at the original. To make it work, you'd have to fix up the pointers. Fixing up the copy would consist of finding all the pointers in the copy, and adding the correct offset to all of them.
 The fixup code would be a recursive routine to traverse all of lua's data structures. It would not be especially complicated, but it would be a lot of code. It would also have to be a lua interpreter patch. It would have to be 100% thorough, you would have to make sure you didn't miss a single pointer. Despite these difficulties, this actually seems relatively feasible to me, even if it is a lot of work.
 One problem with this approach is that it still takes time proportional to the size of the master world model, though I believe it would be a pretty fast traversal. The other problem is that it still suffers from the thread nondeterminism problem.
 Despite these two problems, I think this is one of the most promising approaches.
 ## Solution: Memcpy of the Master Lua Interpreter with Segmentation Protection
 This solution is similar to the previous one: you store the master world model's lua interpreter in a contiguous region of memory, then when it's time for the clients to difference transmit, you make a memcpy for each client. Again, the copies would not be valid because all the pointers in the copies would point back to the original.
 But this solution differs in that instead of fixing up the pointers in the copy, we allow them to remain as-is. We then apply an *mprotect* to the original master world model. When any code tries to follow a pointer that's in one of the copies, they will try to read the original master world model, and they will be blocked by the *mprotect*: a segmentation fault will occur. We trap that segmentation fault, and the segmentation fault handler fixes up the access by adding the correct offset at that time.
 This could be slow because of a lot of invocations of the segmentation handler, but it might still work.
 ## Solution: Using Linux Fork
 Instead of creating a separate thread for each client, call linux *fork* once per connected client. The child processes are all clones of the parent process, therefore, each child process has a copy of the master world model. The child process would compare one client model to the master model, and generate a difference transmission in the form of a string. It would then hand the difference string back to the parent process, and then the child would shut itself down.
 The simplicity of this approach is amazing. We're talking about a couple dozen lines of code. This solution is so clean that it even solves the determinism problem, as long as the master process forks all the children in a deterministic order, and then reads back the solutions in a deterministic order.
 But, we have to think a little about what's happening behind the scenes. When you do a fork, Linux sets up a page tables of the parent and child so that they both are sharing 100% of their pages: there's only one copy of everything. But the page-table entries are all flagged as "copy-on-write". If either the parent or child tries to write to any of those pages, then linux makes two copies of the page, one for the parent, one for the child, and then the write is allowed to happen.
 It's kind of interesting to think about what happens when difference transmitting the master model. All those problems: the string problem, the stack problem, the garbage collector problem, etc – all of them trigger mutations, and those mutations in turn activate Linux's copy-on-write mechanism. If the size of the mutations are small - just writing a few bytes - then just one or two pages get copied-on-write.
 The problem with this approach is that *fork* can be slow. Fork has to set up the page tables for the child process, which takes an amount of time proportional to the memory size of the parent process. Likely, *fork* is too slow for our purposes, but I haven't measured it.
 You could improve the situation by just not forking so many children. For example, you could always fork two children, and assign each child half of the clients. That would give about a factor of two speedup, in theory. And two forks might not be so expensive.
 It is possible, in Linux, to reconfigure the page size from 4KB to 2MB. Those are the only two sizes that x86-64 supports. Doing this vastly reduces the size of the page tables, with the effect that fork is much faster. But, it also makes the copy-on-write operation much slower. This would dramatically alter the performance of the forking approach, but would it be faster or slower? I don't know.
 Of all the solutions here, I think the fork solution is the one most likely to be fruitful.
 ## Solution: Using Linux Posix_Spawn and MMap
 Linux *fork* is slow because it has to create a copy of the parent process. This is usually a waste of time, because 99% of the time that somebody calls *fork*, they then immediately call *exec*, which throws out the copy that took so long to create. To solve this problem, Linux has a newer function, *posix_spawn*, which acts a lot like *fork-and-exec* in one step, but it entirely avoids the copying overhead.
 Unfortunately, that doesn't seem very useful for us: we wanted the child to be a copy of the parent process, so that it would have a copy of the master and client world models. But Linux also has ways for two processes to manually share parts of their memory. You can even arrange for two processes to share copy-on-write regions of memory. This is all done through the *mmap* system call.
 If we could somehow get the master world model to reside in a contiguous block of memory, we could spawn a child process using the fast posix_spawn, then we could manually hand the child process a copy-on-write copy of that region of memory holding the master world model. This would of course, require the copying of certain page-table pages, but it wouldn't involve copying the entire parent process - only the part of the parent process that holds the master world model. This would probably be a vastly smaller block of memory.
 The big problem here is that each model would have to have its own memory heap. That, in turn, would lead to a new type of bug where you might accidentally allocate something using one memory allocator, and free it using a different memory allocator, causing heap corruption. This sounds very hard to navigate to me. I think this is just too error-prone.
--- a/Docs/Displaying
+++ b/Docs/Displaying
@@ -0,0 +1,444 @@
 ### []() Displaying Widget Blueprints
 Unreal provides a class, "Widget Blueprint," which is intended for GUI elements. The Unreal Editor includes an editor for widget blueprints, in which you can position and align buttons, check boxes, text boxes, and other widgets. We intend to use widget blueprints for our GUI elements.
 A widget blueprint can represent a window full of buttons and checkboxes and the like, but it doesn't necessarily need to be that complicated. It could also be a menu, or even just a label.
 Widget blueprints, like all Unreal blueprints, contain an event graph. Therefore, you can write Unreal blueprint code to get any behavior you want from them. They can update their contents periodically, and they can respond to mouse clicks or hotkeys.
 This paper explains how we use widget blueprints as the foundation for Luprex's GUI system.
 ## How Blueprint Code calls Lua Functions
 Before we explain how gui elements are made, I need to tell you about a general-purpose pathway that we have created that allows Unreal blueprints to make calls into Lua. There are two kinds of calls: "invokes", and "probes."
 An "invoke" is when a blueprint calls into Lua for the purpose of changing the state of the world. An invocation needs to be transmitted to the server, so that it may affect the master copy of the world. Invocations are queued, serialized, and executed in a precise order according to the rules of "predictive reexecution" - we have a separate paper on this topic. Because of this, the "invoke" is not executed the instant that the blueprint calls an invoke-function. The invoke therefore doesn't return anything to the blueprint.
 A "probe", on the other hand, does have a return value: that's the purpose of a probe, to return information to the blueprint. A probe is not allowed to alter the state of the world. Ideally, probes will be executed in a sandbox that prevents them from changing the state of the world. Probes are only executed by the world model on the client - they are not forwarded to the server, because the don't need to be.
 It is possible for a malicious hacker to alter his blueprints, he can therefore cause any function to be invoked, with any arguments. Therefore, when an invocation comes into the server, the server must account for the possibility that the invocation is an attempted hack.
 Because of this, we have added a security mechanism: a permit_invoke function that allows you to name a class and a function, to enable blueprints to invoke that particular function. Without the permit_invoke, the invocation will be rejected. However, that still leaves the possibility of the hacker deliberately passing invalid arguments. Because of this, any function that is invocable must error-check its arguments extensively.
 Probes do not have the same security issues. The worst a probe can do is crash the hacker's client. So we have no security mechanism for probes.
 When you do a probe or an invoke, the blueprint can pass parameters to the Lua code. Actor and place are always passed automatically. After that, the blueprint can pass four types of parameters: floating point, strings, vectors, and booleans. The blueprint may pass as many of these as desired.
 When the blueprint calls a probe-function, the probe must return a value. The return value is always an array containing floating point, strings, vectors, and booleans.
 To call a Lua function, use the blueprint node 'Invoke Lua Function' or 'Probe Lua Function.'
 ## Look-At Widgets
 By default, luprex puts a crosshair on the screen. When you aim the crosshair at an object, you are said to be "looking at" that object. Sometimes, looking at an object causes a "look-at widget" to pop up. Here's a typical example of a look-at widget, which displays the hotkeys for that object:
 ![](Displaying%20Widget%20Blueprints_html_fc056206a4099c85.jpg)
 Luprex contains all the following mechanisms to support the idea of look-at widgets:
 - *Event Calculate Look At*. Once per frame, luprex will call event *Calculate Look At* to determine which object you are looking at. The event is written in blueprint.
 - *Current Look-At Object*. The luprex gamemode contains a global variable that stores a pointer to the object you're currently looking at.
 - *Event Look-At Changed*. When the value of the *Current Look-At Object* changes, event *Look-At Changed* gets triggered. This event is responsible for popping up the look-at widget, if any. The event is written in blueprint.
 - *The Lua Function engio.getlookat*. When the crosshair moves to a new object, the blueprint code asks Lua what blueprint to pop up. It does so by probing the Lua function *engio.getlookat*.
 - *Current Look-At Widget*. The luprex gamemode contains a global variable that stores a pointer to the current look-at widget. When the look-at changed event creates a look-at widget, it stores a pointer to the widget in this global variable.
 The global variables are provided by our C++ code. The code that invokes the *Calculate Look-At* and *Look-At Changed* are also written in C++. Everything else is either blueprint or Lua. What that means is that there's very little about this system that can't be overridden or redefined by the scripter.
 The following sections give more information about event *Calculate Look-At*, event *Look-At Changed*, and the Lua function *engio.getlookat*.
 ## Event Calculate Look-At
 Part of the Luprex system is class lxGameMode. It is expected that all games that use Luprex will either use lxGameMode, or some other blueprint class derived from lxGameModeBase.
 lxGameMode contains a custom event, *Calculate Look At*. This event gets called once per frame to figure out which object the player is looking at. Open the Unreal Editor now, open class lxGameMode, and take a look at the code for *Calculate Look-At*. The following is an explanation:
 Event *Calculate Look-At* fetches the XY position of the crosshair. It does a *Line Trace Through Pixel* to cast a ray from the camera, through the crosshair, and out into the world. The line trace returns a *HitResult* struct. The *HitResult* contains the ID of the object that was hit by the ray. The *HitResult* also contains other potentially useful information, such as the world XYZ coordinate where the ray hit the object.
 The final step in Event *Calculate Look-At* is to call *Set Look-At*, which stores the *HitResult* in a global variable. Once the *HitResult* has been stored, you can fetch it from anywhere using the function *Get Look-At*. You can also use the function *Get Look-At Actor* if you don't need the whole *HitResult*, and you just want to know which Actor was hit.
 The *HitResult* might contain an Actor that is known to Lua - ie, a Luprex tangible. It might also hit an object which is just scene decoration, something that was put into the scene using the Unreal Editor without any involvement of Lua. It could also hit the ground plane. If the ray didn't hit anything at all, then the *HitResult* might be empty. All of these possibilities are valid. *Get Look-At* can return any of these.
 ## Overriding Event Calculate Look-At
 The normal implementation of *Calculate Look-At* traces a ray through the crosshair. But what if you don't want a crosshair in your game? What if you want to write a point-and-click game that uses a visible mouse pointer instead? In that case, it would probably make sense to change *Calculate Look-At* so that instead of casting a ray through the crosshair, it would cast a ray through the mouse pointer instead.
 Of course, you could just open up the Unreal Editor and edit the blueprint code for *Calculate Look-At*. However, I don't recommend editing code provided by the Luprex system. Instead, I would suggest creating a new GameMode class that derives from lxGameMode: perhaps you could call it PointAndClickGameMode. Then, you can override event *Calculate Look-At* using inheritance.
 Actually writing the new version of Calculate Look-At would be simple: just copy the code for the existing routine, then replace Crosshair XY with Mouse XY.
 Imagine a slightly different game design - a game design where to "look at" an object, you just bump into it while walking. In this game design, there's no crosshair or mouse pointer. In such a design, you might entirely disable event *Calculate Look-At* by overriding it with blank code. Instead, you might set up collision detection events, and in those collision detection events, call *Set Look-At*. This is entirely legitimate. You can call *Set Look-At* from anywhere.
 The routine *Set Look-At* doesn't directly call event *Look-At Changed*. Instead, if it detects that the look-at object changed, it sets a flag. Later, the flag causes event *Look-At Changed* to be called.
 ## About The Line Tracing Code
 The standard implementation of *Calculate Look-At* uses a routine *Line Trace Through Pixel* to cast a ray from the camera, through the crosshair, and out into the world. *Line Trace Through Pixel* is a routine provided by Luprex. It is a variation of *Line Trace by Channel*, which is a standard Unreal built-in. It would be entirely possible to accomplish the same thing using *Line Trace by Channel*, but it would require more blueprint code.
 In the default implementation of *Calculate Look-At*, the line trace is configured to use *Trace Channel: LookAtDetection*. You can use the Unreal Editor to edit the blueprint for an actor, and specify whether that actor collides with *LookAtDetection* rays. You will find this checkbox in the blueprint by selecting the mesh component, and then looking in the *Details* panel for *Collision/Collision Presets*. If you don't touch this checkbox, then by default, all actors will collide with the *LookAtDetection* ray. If you mark this checkbox false, then the crosshair line trace will pass right through that actor.
 The trace channel *LookAtDetection* is a trace channel that we added to the project in the *Collision* section of the *Project Settings*. Initially, we tried using trace channel *Visibility*, which is an Unreal built-in trace channel that (according to the documentation) collides with any visible object. However, some visible objects are omitted, your player character in particular. That is not what we want. We eventually decided to have a trace channel specifically for *LookAtDetection* in order to make it possible to configure it precisely without interfering with anything else.
 ## Event Look-At Changed and engio.getlookat
 After *Calculate Look-At* figures out which object you're looking at, it calls *Set Look-At,* to store the object in a global variable. *Set Look-At* also checks whether the look-at object has changed, and if so, it sets a flag. The flag eventually causes event *Look-At Changed* to be called. Open the Unreal Editor now, open class lxGameMode, and take a look at the code for event *Look-At Changed*. The following is an explanation:
 The first thing that *Look-At Changed* does is log an informational message. You can therefore see the look-at object changing in the Unreal debugging window.
 Then, *Look-At Changed* calls *Clear Look-At Widget*, which deletes any previous look-at widget. If this were not here, then looking away from an object would not cause the look-at widget to go away.
 Next, if the object you're looking at is a Lua tangible, *Look-At Changed* probes the Lua function *engio.getlookat*. The purpose of this call is to ask Lua what widget blueprint to display. The function *engio.getlookat* is expected to return a string (the blueprint name), and an array of additional values that will be passed through to the widget blueprint's constructor.
 Once we have a blueprint name, event *Look-At Changed* calls *Create Look-At Widget by Name*. This function creates a new look-at widget. *Create Look-At Widget by Name* also adds the widget to the viewport, and it calls *Set Look-At Widget* to store the widget as the current global look-at widget.
 The blueprint language doesn't really have "constructors." Instead, to give the widget a chance to initialize itself, event *Look-At Changed* calls the custom event *Read Lua Configuration*. The widget can then use data returned from Lua to set itself up.
 It is theoretically possible to override event *Look-At Changed.* Suppose, for example, that you want to add a glowing highlight to whatever object the player is looking at. You could add code to *Look-At Changed* to modify the material of the look-at object.
 ## The Standard Hotkey Widget
 The hotkey widget for an object stays on the screen until you turn the camera, moving the crosshairs off of that object. As soon as you point the crosshairs away, the hotkey widget is automatically destroyed.
 The look-at widget is highly scriptable and configurable. Almost every part of its behavior can be altered by editing either lua code or blueprint code. In order to explain it in a comprehensible way, we're going to start by explaining the *default behavior*.
 By default, the look-at widget is implemented by blueprint class *lxLookAtWidgetBP*, which is a built-in part of the luprex codebase. When *lxLookAtWidgetBP* pops up, it probes a builtin lua function *gui.gethotkeys.* This returns a vector, which might look like this:
 ```lua
 { "PX", "Close" }
 ```
 PX stands for gamepad X. When the player presses the gamepad X button, the *lxLookAtWidgetBP* captures that hotkey. It invokes the builtin lua function *gui.presshotkey(place, "Close").*
 The built-in system supports an enumeration of hotkeys, including these:
 M1 - the left mouse button
 M2 - the middle mouse button
 M3 - the right mouse button
 PX - the X button on the xbox gamepad, or the square button on the PS gamepad
 PY - the Y button on the xbox gamepad, or the triangle button on the PS gamepad
 PB - the B button on the xbox gamepad, or the circle button on the PS gamepad
 PA - the A button on the xbox gamepad, or the cross button on the PS gamepad
 There are many more. The full list will be included elsewhere. In addition to the hotkeys listed above, you can use single letters to represent keyboard keys.
 To add a hotkey to a lua class, the scripter must write lua code similar to this:
 ```lua
 makeclass("brickoven")
 gui.addhotkeys(brickoven, "Close", "PX")
 function brickoven.do_close(place)
 … implement the hotkey here …
 end
 function brickoven.do_show_menu(place)
 ```
 The builtin lua function *gui.addhotkeys(place…)* stores one or more hotkeys in the class table. If you were to pretty-print the class table for *brickoven*, you would see this:
 ```lua
 {
 __class = "brickoven",
 __hotkeys = { PX = "Close" },
 hotkey_close = <function>
 }
 ```
 If you're playing a game using keyboard and mouse, you won't be able to press gamepad keys. Likewise, if you're playing with gamepad, you won't be able to press mouse buttons. Therefore, the lua script should include hotkeys for both, like this:
 ```lua
 gui.addhotkeys(brickoven, "Close", "PX", "M3")
 ```
 That means that you can invoke the close function by pressing either gamepad X, or the right mouse button. Class *lxLookAtWidgetBP* will attempt to determine which input device you're using, and it will show the correct hotkeys for the input device you're currently using.
 The implementation of *gui.gethotkeys(place)* is very straightforward: it gets the class from the tangible, then it gets the hotkey list from the class. The implementation of *gui.presshotkey(place, hotkey, action)* is equally straightforward: it figures out which function to call based on the action string. It does so by simplifying the action string: it lowercases it, it replaces whitespace with underscore, and it removes punctuation.
 ## Modifying the Look-At Widget Blueprint
 If the scripter doesn't like the appearance of the provided *lxLookAtWidgetBP*, one option is to edit the blueprint. By editing the blueprint, it is possible to change the appearance in any way desired.
 One possible modification is to actually make the widget entirely invisible. This might be a reasonable thing to do in a point-and-click game: when you aim the mouse pointer at an object, a look-at widget pops up, but it's invisible. But the widget is there, and it responds to the "hotkeys" left-mouse and right-mouse.
 If you examine *lxLookAtWidgetBP*, you will find the calls to *gui.gethotkeys* and *gui.presshotkey.* In theory, it is possible to remove these calls and replace them with some completely different lua calls. You can completely change the way look-at widgets communicate with lua. I don't recommend that, but it is possible if for some reason the current design is not sufficient.
 In general, it's not ideal for the scripter to actually edit our provided classes. Instead, it's better to write a class that derives from *lxLookAtWidgetBP*, and then edit the derived class.
 ## Using Multiple Look-At Widget Blueprints
 What if you want to use *lxLookAtWidgetBP* for some of the objects in your game, but you have a few objects for which you want to use a different blueprint? In that case, you will need to tell luprex which blueprint to pop up for which object.
 When unreal is about to pop up a look-at widget, it first probes the lua function *gui.getlookwidget(place).* The built-in version of this function always returns nil, which in turn causes unreal to use the default blueprint, *lxLookAtWidgetBP*. If you wish, you can edit the code for *gui.getlookwidget*. For example, you might put this in your lua script:
 ```lua
 function gui.getlookwidget(place)
 local class = getclass(place)
 return { class.lookwidget }
 end
 ```
 Then, having done that, you could write code that looks like this:
 ```lua
 makeclass("brickoven")
 brickoven.lookwidget = "BrickOvenLookWidgetBP"
 ```
 Any class that doesn't have a look widget specified will end up falling back to *lxLookAtWidgetBP*.
 ## Adding a Menu System
 ## More Complex Widgets
 Currently, there is no way to pop up any widget other than a look-at widget. The following section is for future reference.
 The configuration for a widget blueprint could be complicated. Consider this menu:
 ![](Displaying%20Widget%20Blueprints_html_24d3706fb604c218.jpg)
 This menu could be a single widget blueprint, *MenuWidgetBP*. In its *BeginPlay* event, it could probe *gui.getmenu(actor, place),* which might return { "Resume", nil, "Save Game", "Load Game", nil, "Session", "Difficulty", "Options", nil, "Main Menu", "Quit Game" }. This design would make it possible for one widget blueprint to represent all the menus in the game.
 We will provide common blueprints like *HotkeyWidgetBP* and *MenuWidgetBP* in our distribution to help the scripter get started. However, these won't be a privileged part of the system: they're really just example blueprints. If the user doesn't like the appearance of the menus rendered by our *MenuWidgetBP*, then he can alter the blueprint's code, or write his own *MyMenuWidgetBP*. If he wants to have several different types of menus in his game, he can have multiple menu widget blueprints.
 Here's an interesting window from a role-playing game:
 ![](Displaying%20Widget%20Blueprints_html_ae2696951c4fbe2a.png)
 Since there are hundreds of spells in this game, it would not be good to have a separate widget blueprint for each one. Instead, it's probably best to have a *SpellWidgetBP* that can render any spell. This might probe *spell.getinfo(actor, place, spell),* which might return a vector like this:
 ```
 {
 "Heading", "Ice Storm", 4, "Evocation",
 "Damage Type", "2D8", "Bludgeoning",
 "Bonus Damage", "4D6", "Cold",
 "Description", "A hail of rock-hard ice pounds …",
 "Saving Throw", "DEX", "Targets still take half damage",
 "Range", 300, "feet",
 }
 ```
 However, instead of making a *SpellWidgetBP*, there's another option. The window above is pretty simple: just text and images. There's nothing interactive. There are no resizable elements. There's nothing that scrolls. One could consider making a widget blueprint that can render any simple window: text, images, and buttons only. Such a widget blueprint might be called *SimpleDialogWidgetBP.* This widget blueprint might probe a lua function, gui.getlayout(actor, place), which might return a vector like this:
 ```
 {
 "window", "WindowStyle3", 0, 0, 500, 500,
 "text", "Ice Storm", "12pt", "white", 20, 20, 200, 40,
 "text", "Level 4 Evocation Spell", "9pt", "grey", 20, 45, 200, 65,
 …
 }
 ```
 There's no right answer as to whether *SimpleDialogWidgetBP* or *SpellWidgetBP* is the best choice for this window. *SpellWidgetBP* has the disadvantage that you have to create a separate widget blueprint just for spells. On the other hand, having a separate widget for spells might make some things easier. Typing coordinates into Lua is awkward, having a SpellWidgetBP would make it possible to use the Unreal Editor to lay out the window. It might also be possible to have the window resize to fit the length of the description. It's also possible to implement it both ways: start by using *SimpleDialogWidgetBP*, and then when you decide you want to add a little polish, switch it over.
 It may even be desirable to make a full-blown *ScriptableWidgetBP*, that takes a little language that is even more sophisticated. If the scripter wants to do so, he can create his own little languages to control the windows in his game.
 ## Building a Menu System
 Let's walk through the design of a menu system.
 This menu system will only contain one blueprint: *MenuWidgetBP.* When this blueprint pops up, it *probes* a Lua function *getmenu* to get the menu configuration. Then later, when the user clicks a menu item, it *invokes* a Lua function *execmenu* to execute the menu item, passing the selected menu item as a string. Here's what the two functions would look like, with the example of a brick oven:
 ```lua
 makeclass("brickoven")
 brickoven.onclick = "MenuWidgetBP"
 function brickoven.getmenu(actor, place)
 return { "Add Clay", "Add Straw", "Add 1 Fuel", "Add 10 Fuel", "Ignite the Oven" }
 end
 function brickoven.execmenu(actor, place, selection)
 if (toofaraway(actor,place)) then return end
 if (selection == "Add Clay") then place.addclay(actor, place) end
 if (selection == "Add Straw") then place.addstraw(actor, place) end
 if (selection == "Add 1 Fuel") then place.addfuel(actor, place, 1) end
 if (selection == "Add 10 Fuel") then place.addfuel(actor, place, 10) end
 if (selection == "Ignite the Oven") then place.ignitetheoven(actor, place) end
 end
 permit_invoke("brickoven", "execmenu")
 ```
 That's easy to understand, and it would absolutely be possible to implement *MenuWidgetBP* to work like that. But it's not very convenient for the Lua scripter. We can do significantly better. We're going to make a series of improvements to that. The first step is adding a level of indirection:
 ```lua
 makeclass("gui")
 function gui.getmenu(actor, place)
 local class = getclass(place)
 return class.getmenu(actor, place)
 end
 function gui.execmenu(actor, place, selection)
 local class = getclass(place)
 class.execmenu(actor, place, selection)
 end
 ```
 The idea here is that instead of *MenuWidgetBP* calling *brickoven.getmenu*, it calls *gui.getmenu*. Then, *gui.getmenu* fetches the class of the place (brickoven), and then it calls *brickoven.getmenu*. Effectively, *MenuWidgetBP* is still probing *brickoven.getmenu* and invoking *brickoven.execmenu*, but it's doing so through our two dispatchers. The functionality is exactly the same, so far.
 The advantage of having the centralized dispatchers, *gui.getmenu* and *gui.execmenu*, is that we can now move certain things into the dispatcher. First, we're going to move the *permit_invoke* to the dispatcher, and also the security check *toofaraway(actor, place):*
 ```lua
 makeclass("gui")
 function gui.getmenu(actor, place)
 local class = getclass(place)
 return class.getmenu(actor, place)
 end
 function gui.execmenu(actor, place, selection)
 if (toofaraway(actor, place)) then return end
 local class = getclass(place)
 class.execmenu(actor, place, selection)
 end
 permit_invoke("gui", "execmenu")
 ```
 So now, instead of needing a *permit_invoke* in class *brickoven*, and another in class *furnace*, and another in class *kiln*, we only need one in class *gui*. Likewise, instead of needing the *toofaraway* check in every class, we only need one in class *gui*.
 The next improvement we can make is to eliminate the need for separate *getmenu* and *execmenu* functions, by writing a single *makemenu* function that operates in two modes (*getmenu* mode or *execmenu* mode). This is the updated code: we've combined *getmenu* and *execmenu*, we've eliminated the *toofaraway* check (because it's in the dispatcher), and we've eliminated the *permit_invoke* (again, because it's in the dispatcher):
 ```lua
 makeclass("brickoven")
 brickoven.onclick = "MenuWidgetBP"
 function brickoven.makemenu(actor, place, menu)
 gui.menuitem(menu, "Add Clay", function () place.addclay(actor,place) end)
 gui.menuitem(menu, "Add Straw", function () place.addstraw(actor, place) end)
 gui.menuitem(menu, "Add 1 Fuel", function () place.addfuel(actor, place, 1) end)
 gui.menuitem(menu, "Add 10 Fuel", function () place.addfuel(actor, place, 10) end)
 gui.menuitem(menu, "Ignite the Oven", function () place.ignitetheoven(actor, place) end)
 end
 ```
 This function takes a parameter "menu". This is an object that indicates whether *makemenu* is running in *getmenu* mode or *execmenu* mode. In *getmenu* mode, *gui.menuitem* adds the menu item to a list of menu items. In *execmenu* mode, *gui.menuitem* conditionally calls the lambda. To make this work, we have to alter our centralized dispatchers. We will still have separate *getmenu* and *execmenu* dispatchers, both of which will call *makemenu*:
 ```lua
 makeclass("gui")
 function gui.getmenu(actor, place)
 local menu = { "mode":"getmenu", "config": {} }
 place.makemenu(actor, place, menu)
 return menu.config
 end
 function gui.execmenu(actor, place, selection)
 if toofaraway(actor, place) then return end
 local menu = { "mode":"execmenu", "selection" : selection }
 place.makemenu(actor, place, menu)
 end
 permit_invoke("gui", "execmenu")
 function gui.menuitem(menu, item, lambda)
 if menu.mode == "getmenu" then
 table.insert(menu.config, item)
 end
 if menu.mode == "execmenu" then
 if menu.selection == item then lambda() end
 end
 end
 ```
 So far, we've simplified the following: we only need to write one *makemenu* function per class like *brickoven*, we don't need to write security checks in the *makemenu* function, and we don't ever need to write *permit_invoke* other than the one in class gui.
 There is one last thing left to simplify: we can eliminate the need to write *brickoven.onclick = "MenuWidgetBP".* We will make "MenuWidgetBP" the default *onclick* whenever a *makemenu* function exists. We can accomplish that by overriding the built-in function *gui.get_onclick*:
 ```lua
 function gui.get_onclick(actor, place)
 local class = getclass(place)
 if class.onclick == nil and class.makemenu ~= nil then
 return { "MenuWidgetBP" }
 else
 return { class.onclick }
 end
 end
 ```
 There is one more thing we could simplify. The *gui.menuitem* function requires us to pass in a lambda. We could imagine modifying *gui.menuitem* so that if you pass nil for the lambda, it will automatically search for a function whose name matches the menu-item. For example, it could take the menu item "Add Clay", strip out the white space and uppercase yielding "addclay", and then it could look for a function brickoven.addclay. This would allow us to optionally omit the lambda and just write a function whose name matches the menu item.
 The most important thing about this menu system is: it's all implemented in "user space." It was created by writing non-privileged Lua code and non-privileged blueprint code – no C++. If we were to ship this menu system with our distribution, the scripter could modify it at will. Or, he could build his own menu system using similar or different principles, and his menu system could coexist peacefully with ours.
--- a/Blueprints_html_24d3706fb604c218.jpg
+++ b/Blueprints_html_24d3706fb604c218.jpg
--- a/Blueprints_html_ae2696951c4fbe2a.png
+++ b/Blueprints_html_ae2696951c4fbe2a.png
--- a/Blueprints_html_fc056206a4099c85.jpg
+++ b/Blueprints_html_fc056206a4099c85.jpg
--- a/Variables.md
+++ b/Variables.md
@@ -0,0 +1,167 @@
 ## Global Variables in Luprex
 Normally, lua programs store global variables in the global environment table. In luprex, however, global data needs to be differentiated into several different kinds:
 - Some global data needs to be difference transmitted, some doesn't.
 - Some global data needs special garbage collection strategies.
 - Some global data needs to be instanced per-user, some doesn't.
 Simply throwing all these different types of data into a single table causes a problem, because it doesn't give us the means to treat these different types of data differently. So instead, we have different tables for different types of global data.
 This document explains all the different things that would normally go into a lua global environment table, and it describes what we do with each of those different kinds of data.
 ## Function Definitions, Class Definitions, and Constant Definitions
 In a normal lua interpreter, function definitions are stored in the global environment table. Luprex sticks with that convention. Classes and constants also get stored in the global environment table. For example, suppose you write this code in a lua source file:
 ```lua
 makeclass("deque")
 function deque.push_right(deque, value)
 …
 end
 deque.default_size = 10
 ```
 The three statements above are all actually storing data into _G, the global environment table. The code above could be rewritten to explicitly mention the global environment table:
 ```lua
 _G["deque"] = {} – this is an oversimplification of makeclass
 _G["deque"]["push_right"] = function(deque, value) … end
 _G["deque"]["default_size"] = 10
 ```
 When you think about it that way, a typical lua source file is really a long list of statements that insert closures, and sometimes classes and constants, into the global environment table. That's how it works in normal lua, and that's also how it works in luprex.
 We need to transmit the lua code to all connected clients. In lua, closures are opaque objects. There is no API to do anything to them, other than call them: they can't be examined. So we simply can't transmit closures.
 That means we need a different strategy. Instead of transmitting the closures, we transmit the source code. When we transmit the source code to a client, that client compiles the source code itself. When the client loads the source code, the client inserts class definitions, function definitions, and constant definitions into its own copy of the global environment table.
 For this reason, the class definitions, function definitions, and constant definitions that are present in the global environment table don't need to be difference transmitted. Instead, we transmit the source code, and that's sufficient.
 Suppose that you edit a lua source file, and remove the definition of the function deque.push_right. Removing the function from the source doesn't automatically remove it from the global environment table. Even if you reload the source file, the function deque.push_right will linger in the global environment. We added a garbage collection strategy to handle this problem: before reloading the lua source, all the function and constant definitions in the global environment are removed. The global environment is rebuilt cleanly from scratch.
 The global environment table has these properties:
 - **Table**: The Global Environment Table
 - **Contents**: Function definitions, class definitions, and constant definitions from the lua source.
 - **Difference Transmission**: The source is transmitted, the client makes its own copy from source.
 - **Per-User Instancing**: There's only one such table, shared by all users.
 - **Garbage Collection**: We purge the table before reloading the source.
 ## Interactive Temporary Variables
 When you're typing commands into a lua interpreter interactively, you inevitably will need to create temporary variables. For example, suppose that you're an administrator and you want to teleport yourself somewhere. You might type these commands into the interactive lua prompt:
 ```lua
 a = tangible.actor() — get a pointer to my actor
 tangible.animate(a, nil, {action="warpto", xyz={100,100,100}})
 ```
 You just created a temporary variable, *a*. In a normal lua interactive shell, these temporary variables go into the global environment table.
 But consider what would happen if the temporary variables from hundreds of interactive sessions were to just accumulate in the luprex global environment table. It would be a mess.
 Another problem is that two administrators might be typing commands at the same time. They might both use the temporary variable "x" to mean different things, and they would interfere with each other.
 For this reason, when you log into a luprex server with an administrative account that has permission to type interactive lua commands, it creates a new *interactive environment table* which is private to you. When you log out again, the table is discarded. Lua treats the interactive environment table as if it were the global environment table for any commands you type. The interactive environment table is initially empty, but it has a metatable that points to the true global environment table, so typing the name of a function or class will work - you can "read through."
 The interactive environment table is a useful tool in that it makes it feasible to use temporary variables. But it is important to remember that it is not a sandbox. It is very easy to unintentionally inject junk into the real lua environment table.
 For example, suppose you type this into the interactive prompt:
 ```lua
 function foo(x) … end
 ```
 This is fine. It stores a function *foo* in your interactive environment table. Nobody else can see it, and it gets garbage collected when you log out. But if you type this:
 ```lua
 function foo.bar(x) … end
 ```
 This inserts a function *bar* into the already-existing class *foo*. If *class foo* is a class defined in a lua source file, and therefore, *class foo* is visible to all users, then the new *function foo.bar* will be visible to all users. Doing this would be a bad idea. Remember, the system has a mechanism to garbage collect any function that has been removed from the lua source files - the function *foo.bar* is not in a lua source file. It will get removed from the global environment as soon as somebody reloads the lua source. It also won't get transmitted to clients, again, because it's not in a lua source file.
 The upshot is this: don't define functions, classes, or constants in interactive sessions. Put them into source files, and reload the source files.
 The interactive environment table has these properties:
 - **Table**: The Interactive Environment Table
 - **Contents**: Temporary variables created in interactive shells.
 - **Difference Transmission**: Transmitted as part of the actor tangible.
 - **Per-User Instancing**: Each logged-in administrator gets his own.
 - **Garbage Collection**: We purge the table when the administrator logs out.
 Note: this table is not implemented yet.
 ## Broadcast Globals
 By "broadcast globals," I mean global variables that can be updated during the course of the program. When you do update one, the new value gets broadcast to all connected clients. When a new user logs in, he receives the current values of all broadcast globals. When using these, it is important to be thoughtful about the bandwidth requirements of using them.
 Broadcast globals are stored in their own table. This table is accessed using these two functions:
 ```lua
 global.set("x", value)
 value = global.get("x")
 ```
 When you call global.set("x", value), the value is immediately serialized. It is this serialized value that is stored. When you call value = global.get("x"), the stored value is deserialized and returned.
 It is important to remember those semantics, for performance reasons. Both the *global.set* and *global.get* operations are O(N) in the size of the data being stored. It is also important to remember that both functions have copy semantics: if you store an object in the table, any mutations you apply to the object don't affect what's in the table: the table contains a serialized copy of the object, not the original.
 Simple values can be stored in a very straightforward way: int, string, float, boolean, tokens, nil - all of these can be stored and retrieved without thinking about it. But more complex values require a little thought.
 Tables can be stored, even if they recursively contain more tables, and even if the table references contain cycles. The entire data structure will be packaged up. Be aware that if the data structure contains a pointer to another data structure, that other data structure will get traversed and packaged up too. Storing a large data structure will take time proportional to the size of the data structure.
 Tangibles are a special case. Storing a tangible in a broadcast global does not serialize the contents of the tangible. Instead, only the tangible's ID is serialized. When a client uses *global.get* to fetch the value, the the function returns the client's copy of the tangible. Tangible contents are difference transmitted separately.
 Classes are also a special case. You can store a class in a broadcast global, but only the class name is actually serialized. Since every client has a copy of the source code, using global.get will fetch the client's copy of the class.
 Closures can not be stored in broadcast globals at all. Our system is not capable of transmitting closures across the network. Attempting to store one will cause an error to be thrown in the routine *global.set*.
 For more details about object serialization, read the documentation of the *table.serialize* function.
 The Broadcast Globals Table has these properties:
 - **Table**: The Broadcast Globals Table
 - **Contents**: Anything you store using the function *global.set*.
 - **Difference Transmission**: Difference transmitted to all clients, immediately upon update.
 - **Per-User Instancing**: None, all users share this data.
 - **Garbage Collection**: None. This data must be manually cleaned up.
 ## Global Once Flags
 Sometimes, in a source file, it is useful to say "execute this code only if it has never been executed before." You can accomplish this in luprex using code that looks like this:
 ```lua
 if global.once("initialize-database") then
 initialize_database()
 end
 ```
 When the source file is loaded, if the *global.once* expression has never been evaluated before, then the database will get initialized.
 Sometimes you want to trigger the 'once' code to execute again. In the example above, for example, you might want to reinitialize the database. In that case you can do this:
 ```lua
 global.clearonce("initialize-database")
 ```
 After doing that, the next time you reload the source file, the database will get initialized again.
 Currently, the global once flags are implemented on top of broadcast globals. It is not clear whether this is the right choice. I am not sure that it is necessary to difference transmit global once flags. I may reimplement these to have their own table which only exists on the master, and which is not predicted on the client. But for now, they work on top of broadcast globals. That should work okay for now.
 Since these are implemented on top of broadcast globals, they don't have their own table yet.
--- a/Structures.md
+++ b/Structures.md
@@ -0,0 +1,235 @@
 ## Overview
 This document gives a grand overview of the major data structures used by the engine.
 ## The World Model
 There will be a C++ class World.
 Each World will own its own lua interpreter. The World constructor will call *lua_newstate* to create a new lua interpreter. The World will retain a pointer to the lua_State returned by lua_newstate. When the World is destroyed, the associated lua_State will be closed down.
 When you call *lua_newstate* to create a new lua interpreter, it automatically creates:
 - One lua registry table
 - One global environment table
 We think of these two tables as "owned" by the World.
 We're going to put a pointer to the C++ World object into the lua registry. Any C++ code that takes a lua_State can access the registry, so therefore, any C++ code that takes a lua_State can access the World object.
 ## The Global Environment Table
 When you create a lua interpreter, it automatically creates one global environment table. We use the one global environment table created by lua as the global environment for all functions, methods, and threads inside that lua interpreter. In short: there's only one global environment table per interpreter. Or to put it differently, one global environment table per World.
 The global environment table will contain the class database, which are described in the next section. We will take steps to ensure that the global environment doesn't contain anything else other than classes.
 ## Class Database
 We will have a database of classes. Classes have these properties:
 - A class is a lua table, typically full of functions and methods.
 - Classes have names, which are strings. Class names are flat, not heirarchical.
 - Classes are stored in the global environment table, by name.
 - Has "__class" key which is the class name.
 - Has "__index" key that points back to itself.
 - May have metamethods in it.
 - Can be used directly as a metatable.
 - Can be used as the __index part of a metatable.
 In addition, if the class is meant to be used with tangible objects, the class may have these things inside it:
 - An 'action' subtable containing plans.
 We provide an operator *makeclass(name).*  This operator does the following steps:
 - Creates the table if it doesn't already exist.
 - If the table already exists, it is retained.
 - Inserts the key __class=classname into the table.
 - Inserts the key __index=(self) into the table.
 - Does *not* remove any existing methods or functions.
 We also provide an operator *maketangible(name).* This is a variant of makeclass. In addition to the steps above, it also does these:
 - Creates an 'action' subtable within the class table, for plans.
 ## Source Database
 The game's script will consist of a collection of lua source files. The lua files will be stored in a source database. The source database is a table inside the lua registry, under the key "sourcedb." The table contains keys which are filenames. Each filename maps to the following:
 ```
 filename : {
 "name" : the filename again,
 "fingerprint" : a hash of the file's modification date and size,
 "code" : entire source code of the lua file as a string,
 "hash": a 128-bit hash of the code,
 "loadresult": result of calling 'load' - either a closure or an error message,
 "sequence": an integer indicating order in which files are to be loaded
 }
 ```
 The source database is loaded from the lua subdirectory. This subdirectory contains a file "control.lst" which governs which lua files are to be loaded, and in what order.
 The source database is considered part of the World.
 ## Class Rebuilds
 Frequently, we will do "class rebuilds." That means we reconstruct the entire class database, from scratch, from the source database. Here's how that works. Suppose you write this source file:
 ```lua
 makeclass("furnace")
 function furnace.burn() …
 ```
 That source file gets loaded into the source database. As part of this process, the code gets passed to lua's *load* function, which returns a *loadresult*, which is a closure (if there's no error). When we call that closure, it creates the class 'furnace' (if it's not already there), and inserts the function *burn* into that class. In summary: calling a source database *loadresult* typically has the effect of inserting functions into the class database.
 Therefore, a class rebuild consists of these steps: first, empty out all the classes in the class database. Then, for every source file in the source database, call the *loadresult*. This will repopulate the class database.
 The class rebuild is an extremely cheap operation. It doesn't need to load the source from disk, the source is already in RAM. It doesn't need to compile the source - the loaded closures are already present in the source database. The only actual thing happening is that the loaded closures are reinstalling the pointers into the class database. It should take less than a millisecond.
 ## Reloadable Source
 We need our lua source files to be reloadable, for two reasons. First, the class rebuild procedure described above reloads the source files over and over. Second, in order to support on-the-fly code testing, we're going to be editing, loading, editing, loading, and so forth. So for both reasons, we need our lua source files to be reloadable.
 Function and method definitions tend to be reloadable without any extra effort. Let's look at that example source file again:
 ```lua
 makeclass("furnace")
 function furnace.burn() … end
 ```
 If you load that source file twice, no harm is done. The first time you load it, you fetch the class and insert the function burn. The second time, you re-fetch the class, and re-insert the function burn. It works fine. In general, when you insert functions into the class database, you don't have to do anything special to make the code reloadable.
 But other code may not be reloadable. For example, suppose we want to implement a user database module, and let's say that module has some initialization code:
 ```lua
 local user_database = global("user database")
 user_database.hashed_passwords = {}
 ```
 That's not going to work - it's going to clear the hashed passwords table every time we reload the source file. Lua source files will sometimes need explicit code to achieve reloadability:
 ```lua
 local user_database = global("user database")
 if user_database.hashed_passwords == nil then
 user_database.hashed_passwords = {}
 end
 ```
 We will try to provide builtin functions to make this sort of thing as easy as possible, but we can't completely eliminate the burden of thinking about this. Lua source code will need to be carefully written to be reloadable.
 ## Tangibles
 Tangibles are what the legacy engine used to call "sprites."
 Every tangible consists of two parts: a "lua tangible" and a "C++ tangible". The tangible has an ID number which is a 64-bit integer. ID numbers are small enough to fit inside a lua_Number.
 The C++ Tangible is a class that contains these things:
 - The tangible's ID number.
 - The animation queue.
 - The positional tracker (which helps us do "scanradius" searches).
 - If it's a player, the IdPlayerPool (which helps with unique ID allocation).
 The C++ World contains a table that maps tangible ID to C++ tangible.
 The Lua Tangible is a table. The lua tangible is used by the script to store arbitrary data. All keys are available to the scripter, none are reserved. However, the lua tangible always has a metatable which *is* reserved. The metatable of a lua tangible contains:
 - __id -- the tangible's ID number.
 - __metatable -- helps protect the metatable.
 - __index -- points to a class from the class database.
 - __threads -- the thread table, see the next section
 The registry contains a key, "tangibles", which is a table that maps ID number to lua tangible.
 Although the scripter cannot replace the metatable for a tangible, the scripter *can* use an operator that we provide, *setclass(tangible, class)* to change the tangible's class.
 ## Threads
 Every tangible has a table of threads. A thread has a numeric ID which is a 64-bit number. The table of threads has the following contents:
 ```
 tangible metatable : {
 "threads" : {
 tid = { thread info table },
 tid = { thread info table },
 }
 }
 ```
 The thread info table contains the following information:
 ```
 thread info table : {
 "thread" = coroutine handle,
 "actorid" = actor ID,
 "print" = if true, thread return values should be printed on thread completion.
 "useppool" = if true, the thread should use the actor's ID allocator.
 "new" = if true, the thread is newly-created, if false, it's a yielded thread.
 }
 ```
 The "new" thread flag helps us understand what's on the thread's stack. In the newly-created state, the thread's stack contains a function, followed by function arguments. In the yielded state, the thread's stack contains values that should be passed to the lua_yieldk continuation function.
 The World contains a table of scheduled wakeups. This is an ordered list of tuples: <time, thread id, tangible id>. The scheduled wakeups also contain threads that want to wake up immediately, with time=0.
 When a thread is created, it is inserted into the appropriate thread table right away. It is also inserted into the schedule right away, with a time of zero. Then, the scheduler is called to run the thread. The thread stays in the thread table until it terminates.
 ## Snapshot
 Predictive reexecution asks the client to permanently maintain a synchronous model. The client is also supposed to generate transient asynchronous models, apply predictions to those asynchronous models, and then moments later, throw those asynchronous models away.
 To implement this, we're not actually going to create separate synchronous and asynchronous models. Instead, we're going to have one combined "client" model which plays both roles. It will accomplish this using a snapshot-and-rollback ability.
 The lua interpreter's state is snapshotted via a memory trick: we keep track of the memory used by Lua, and we just snapshot the memory without understanding what's in it. When we're ready, we restore all that memory to its previous state. This works quite well to restore a lua interpreter to a previous state.
 Snapshot and rollback is used as follows. The client model initially contains no predictions, which means it's a synchronous model. We snapshot this synchronous state. Then, we apply predictions. Now it's an asynchronous model. When we're ready, we roll back to the snapshotted state, which means the predictions have been removed and it's back to being a synchronous model.
 The snapshot and rollback functionality will be built right into C++ class World.
 ## The World Model Summary
 Here is a summary of all the items we've listed as being part of the world model:
 - The lua_State pointer - the lua main thread
 - The lua registry
 - The lua global environment
 - The source database (in registry key "sourcedb")
 - The class database (in the global environment)
 - The script globals (in registry key "globaldb")
 - The C++ tangibles (stored by ID in the World model)
 - The Lua tangibles (in registry key "tangibles")
 - Possibly, a snapshot of a synchronous state.
 ## What Gets Difference Transmitted
 Here's a summary of what gets difference transmitted:
 **Source database:** transmitted. Source code is easy to difference transmit, it's just strings. If you want to obfuscate it a little, we could serialize the loaded closures and transmit those instead. Either way works fine.
 **Class database**: not transmitted in itself, but it gets rebuilt from the source database whenever the source database gets difference transmitted.
 **Globals database**: not transmitted at all. Only the master world model is allowed to access global data structures. Any attempt to access these data structures in the synchronous models is deliberately trapped.
 **Tangibles**: difference transmitted when nearby.
 FUTURE STUFF - Client / Server
 Code privacy
 Data privacy
 Walkability (Pancake Data Structure?)
 Global Variables
 Good PRNG functions
 FUTURE STUFF - Game / Library
 Doors to other servers / Blockchain stuff
--- a/Transmission.md
+++ b/Transmission.md
@@ -0,0 +1,221 @@
 ### Multipass Difference Transmission
 To difference transmit, we have to compare a tangible in the master world model to the corresponding tangible in a client world model. For example, a tangible has an XYZ coordinate. You compare the XYZ of the tangible in the master model, to the XYZ of the corresponding tangible in the client model. This is so straightforward that it hardly merits any explanation. Where it stops being straightforward is when it comes time to compare the Lua tables of the two tangibles.
 Suppose that in the master world model, tangibles T1 and T2 both point to lua table A, which points to lua table B, which points back to lua table A. Also, suppose that the client model doesn't contain any of this. We would like for the difference transmitter to be powerful enough to recreate the entire graph, including the two tangibles both pointing to the same table, and including the cycle.
 In order for that to be possible, the difference transmitter can't treat each tangible as a separate entity. Instead, it needs to view lua tables as a big interconnected graph with cycles, and it needs to try to reproduce the graph as a whole.
 Let's complicate matters further. Suppose that the player is standing next to tangible T1, but tangible T2 is far away. We want the algorithm to reproduce not the whole graph, but only the portion that is reachable from tangible T1. So, if the player is next to T1, but T2 is far away, then the difference transmitter should transmit tangible T1, which points to table A, which points to B, which points back to A. But it should omit tangible T2.
 Next, let's complicate matters even further. Suppose the player is standing next to tangible T1, and his client model contains the subset of the graph rooted at T1. But then, he warps from where he is, to where tangible T2 is. At this point, the difference transmitter will try to recreate the portion of the graph rooted at T2.
 When it does this, it's OK if the difference transmitter allows tangible T1 to get "out of date", because T1 isn't visible any more. Our difference transmitter creates two new tables, "updated A" and "updated B." T2 points to updated A, which points to updated B, which points to updated A. Meanwhile, T1 points to obsolete A, which points to obsolete B, which points to obsolete A. In short, tangible T1's portion of the graph is now out of date and T1 doesn't point to the updated portion of the graph which is visible from T2.
 Next, suppose the player warps from where he is, to a point halfway between T1 and T2, such that both T1 and T2 are within his sight radius. Only now will the difference transmitter create the entire graph I described: when it is done, both T1 and T2 will point to updated A, updated A will point to updated B, and updated B will point to updated A.
 The difference transmission algorithm has to do a fair amount of analysis. In order to make the algorithm comprehensible, we have divided it into multiple passes, each of which does something fairly simple. We call this algorithm "multipass difference transmission." This paper describes the multipass difference transmission algorithm.
 Not every pass generates server-to-client messages. Some passes are purely done in a local manner, on either the server, the client, or both. When a pass does generate messages, it is expected that those messages get applied to the client models before the next pass begins. In other words, each pass begins after the previous pass has completed fully.
 This paper contains a section for each pass of the algorithm.
 ## Pass: Update Player Position
 This pass compares and transmits the animation queue of the player.
 The reason this step is first is that the difference transmitter wants to transmit tangibles that are near where the player is standing. So therefore, both client and server need to know where the player is standing. The player's position is actually stored in the animation queue: the XYZ coordinate of the last step in the animation queue is the player's position. So to make sure both client and server know the player's position, we must difference transmit the player's animation queue.
 This pass also difference transmits all the other non-Lua data in the player tangible. That includes the player's print-buffer, the player's ID allocator, and possibly some other things. The only reason we do this is that we're transmitting one part of the player tangible, so we might as well transmit all of it.
 This pass *only* applies to the player tangible. No other tangible is updated during this pass.
 ## Pass: Create and Delete Tangibles
 The server does a *scanradius* around the player in the master model, and another *scanradius* around the player in the client model. These both return sets of tangible ID numbers. The server calculates the set-union of these two scans. This is the *visible tangible list*. It contains the tangible IDs of all the tangibles that are near the player in either the master or client model. Then, the following steps are taken to update the client:
 For every tangible TAN in the visible tangible list:
 If TAN exists in both master and client, then compare the animation queues of the two. If necessary, send an animation queue correction to the client.
 If TAN does not exist in the client model, transmit a "create tangible" message to the client. The animation queue of the new tangible is transmitted along with the create tangible message. Nothing else about the new tangible is transmitted yet.
 If TAN does not exist in the master model, then send a "delete tangible" message to the client.
 Once this pass is applied, the set of tangibles in the visible area is guaranteed to match between master and client models, and the tangibles in this radius will all have correct animation queues. Since they have matching animation queues, and since the tangible's XYZ position is stored in the animation queue, they all have matching XYZ positions as well.
 ## Pass: Generate the Close Tangible List
 The server has a *visibility radius*, which is fairly large, and and a *close radius*, which is smaller. Only tangibles within the *close radius* will have their Lua data updated.
 The server does a scanradius around the player, using the close radius and the master model. This yields the *close tangible list*.
 The invariant from the "Create and Delete Tangibles" pass (above) guarantees that doing the same scanradius using the client model yields exactly the same result. For safety, we do verifications to ensure that the invariant hasn't been broken. The first verification step is that the master does the same scanradius using its own copy of the client model. It checks that the two lists are identical. Then, it sends the hash-value of the *close tangible list* to the client. When the client receives the hash-value, it does its own scanradius, and it verifies that the hash matches.
 So at this point, both client and server have a copy of the *close tangible list*, and we are absolutely certain that they are in agreement. Several of the following passes use the *close tangible list*.
 ## Pass: Number the Tables in the Client Model
 We want to give every reachable table in the client model a unique ID number.
 This numbering pass is done both on the server's copy of the client model, and the client's copy of the client model. Because of the invariant that both these models are identical at all times, the numbering pass should produce identical numbering on both the server and client. No messages are transmitted during this pass. Client and server both independently compute the exact same result.
 Numbering is accomplished by doing a graph traversal algorithm. Since there can be cycles in the graph, we need to use a *visited* bit to ensure that we don't get into an infinite loop. When we visit a table for the first time, we assign it a unique ID number. ID numbers start at 1 and increment from there.
 Only "ordinary tables" are numbered and recursed into by the graph traversal. The following are some examples of tables that are not ordinary tables, and therefore are not traversed:
 - Tangible databases.
 - Classes created by the "makeclass" function.
 - The global environment table.
 - The lua registry.
 We avoid all of these for various reasons. When the graph traversal finds one of these special tables, it just ignores it, it doesn't number it or recurse into it.
 In this graph traversal, when we encounter a table entry Key→Value where the Key is not a string, number, or boolean, that table entry is ignored. Strings, numbers, and booleans can are called "sortable" values, because they can be compared less-than. Nowhere in the difference transmission code do we actually do any sorting, but we still call these "sortable keys." Throughout the difference transmission code, we ignore table entries Key→Value if the Key is not sortable.
 Metatables are treated as if they were just another Key→Value table entry, METATABLE→Value. They are traversed just like any other Key→Value pair. Throughout the remainder of this paper, we usually don't mention metatables explicitly. But whenever you see a loop over a table's Key→Value pairs, remember that one of the pairs might be METATABLE→Value. We consider METATABLE to be a sortable key.
 Graph traversal algorithms need to have a set of "roots" to start at. In this algorithm, we loop over all the close tangibles, and then, within each close tangible, we find all ordinary tables pointed to by the tangible database. It is these ordinary tables that make up the roots of the graph traversal.
 The numbers that are assigned by this pass aren't stored in the tables that are being traversed. They are stored in separate maps:
 - The *tnmap*, the table-to-number map. This is a lua table containing T→N, where T is a table that was traversed, and N is the number that was assigned.
 - The *ntmap*, the number-to-table map. This is a lua table containing N→T, where T is a table that was traversed, and N is the number that was assigned.
 These tables get stored in the Lua registry of the client model, in the keys *registry.tnmap* and *registry.ntmap*. These maps get used by a lot of the following difference transmission passes. When difference transmission is done, these two tables are removed from the Lua registry.
 It probably would be a good idea to write some sort of verification step involving the transmission of a hash-value, to ensure that the server and client computed the same result. We haven't done that yet.
 ## Pass: Pair Up Tables in the Master and Client Models
 We attempt to "pair up" tables in the master model to their equivalent tables in the client model.
 The pairing algorithm runs only on the server. No messages are sent to the client. The server analyzes the client and master model, and tries to find a correspondence between the tables.
 When this pairing process begins, the tables in the client model have already been numbered, by the previous pass. The client model contains a *tnmap* and an *ntmap*. But the tables in the master model have no ID numbers. There is no *ntmap* or *tnmap* in the master model. The pairing algorithm creates a *tnmap* and an *ntmap* in the master model, and it initializes them to be empty.
 When the pairing algorithm decides to pair a table TC in the client model to an equivalent table TM in the master model, it does so by copying the table number from TC to TM. Copying the table number consists of the following steps: it fetches the entry TC→N from the *tnmap* in the client model. Then, it stores the entry TM→N in the *tnmap* of the master model, and N→TM in the *ntmap* of the master model. Now the two tables TC and TM both have the same table number.
 When the pairing algorithm is done, the master model has a *tnmap* and an *ntmap* similar to the one in the client model, but there is a difference: the maps in the master model may be incomplete. There may be a reachable table in the master model that didn't have an equivalent in the client model. In that case, that particular table will still not have any table number. There will not be an entry for it in the master *tnmap* or *ntmap*.
 The algorithm to do the pairing is another graph traversal. This time, it traverses the client model and the master model in parallel, in lockstep. The actual traversal algorithm is a little complicated, so bear with me.
 The traversal algorithm maintains a stack containing pairs of tables - one from the master, one from the client. If a pair of tables is on the stack, it means that we have strong evidence that the tables should be paired. However, we still have to do some sanity checks before actually pairing the tables. If they pass the sanity checks, they will be paired.
 Initially, the stack is empty. Then, we populate it with the "roots" of the graph traversal algorithm. Here's how we find all the roots:
 ```
 for every tangible ID in the *close tangible list*:
    lookup the two tangibles TANCLIENT and TANMASTER with the specified ID
    for every table entry Key→Value in TANMASTER:
        if Key is Sortable AND TANCLIENT[Key] and TANMASTER[Key] are both tables:
            push the likely pair TANCLIENT[Key], TANMASTER[Key] onto the stack
 ```
 So now the stack contains all the roots of the traversal. Next, we come to the code that pops the stack and does the sanity checks:
 ```
 while the stack is not empty:
    pop a pair TABLECLIENT and TABLEMASTER from the stack
    if either of the tables is not an ordinary table, then don't pair them
    if TABLEMASTER already has a table number, then don't pair them
    if TABLECLIENT doesn't have a table number, then don't pair them
    if all the checks above pass, then pair the two tables
 ```
 And finally, we have the code that is used when the final decision is made to pair two tables. After pairing the new tables, it generates new possible pairings and pushes them onto the stack:
 ```
 Pair TABLECLIENT, TABLEMASTER:
    Copy the number from TABLECLIENT to TABLEMASTER using *tnmap* and *ntmap*.
    For every table entry Key→Value in TABLEMASTER:
        if Key is Sortable AND TABLECLIENT[Key] and TABLEMASTER[Key] are both tables:
            push the likely pair TABLECLIENT[Key], TABLEMASTER[Key] onto the stack
 ```
 When the algorithm is done, the master's *tnmap* and *ntmap* exist and are populated. There may be tables in the master model that still aren't paired, however. These do not have entries in the *tnmap* and *ntmap*.
 No messages are transmitted during this pass. This pass happens on the server only.
 ## Pass: Number the Remaining Tables in the Master Model
 The goal of this pass is to assign numbers to any reachable tables in the master model that don't already have numbers.
 The tables in the client model were already numbered from 1 to N by a previous pass. We're going to start our numbering from N+1. In other words, the table numbers we're using in this pass pick up where the client numbers left off.
 Numbering is accomplished by doing another graph traversal algorithm in the master model. It visits only ordinary tables, ignoring special tables. It again ignores table entries where the key isn't sortable. It uses the same roots: the ordinary tables pointed to by the close tangibles. When the traversal finds a table that doesn't already have a number, it assigns a value starting from N+1 and incrementing from there. When a table number is assigned, it is stored in the master *tnmap* and *ntmap*.
 After this traversal is done, all the reachable tables in the master model have numbers. The table numbers in the master model now fall into two different ranges:
 - Tables 1 to N: these numbers were assigned by the pass, "Pair Up Tables in the Master and Client Models." Therefore, these tables are all paired to tables in the client model.
 - Tables N+1 to M: these numbers were assigned by this pass, "Number the Remaining Tables in the Master Model." Therefore, these tables aren't paired up with anything in the client model.
 The graph traversal is done only on the server. But after the graph traversal is done, the final step of this pass is to send a message to the client, instructing it to create empty tables numbered N+1 to M. After the client has done this, *all* the tables in the master model have pairings to tables in the client model.
 ## Pass: Transmit Table Contents Non Recursively
 This is the step that actually sends the table corrections to the ordinary tables. In this pass, the server will be sending the client messages of this form:
 ```
 in table [table number], store the entry [Key]→[Value]
 ```
 The client will receive these messages and will just follow the instructions. The algorithm to send the messages is a loop over all the tables:
 ```
 for every N → TABLEMASTER in the master *ntmap*:
    look up the paired table N → TABLECLIENT in the client *ntmap*
    for every table entry Key→Value in TABLEMASTER:
        if Key is Sortable AND TABLECLIENT[Key] != TABLEMASTER[Key]:
            send "in table number N, store the entry Key→TABLEMASTER[Key]"
    for every table entry Key→Value in TABLECLIENT:
        if Key is Sortable AND TABLEMASTER[Key] == nil:
            send "in table number N, store the entry Key→Nil"
 ```
 The equality comparison used in the algorithm above has certain rules:
 - Strings, Numbers, and Booleans, and Nil are compared in the usual way.
 - Two tables are considered equal if they have the same table number.
 - Two tangibles are considered equal if they have the same tangible ID.
 - Two class tables are considered equal if they have the same class name.
 - The client global environment is considered equal to the master global environment.
 The content of the *Value* part of the message depends on the type of *Value*:
 - Strings: send the string.
 - Numbers: send the number.
 - Booleans: send the boolean.
 - Nil: send "nil"
 - Tables: send "table [table number]"
 - Tangibles: send "tangible [tangible ID]"
 - Class tables: send "class [classname]"
 - The global environment: send "globalenv"
 - Functions: cannot be transmitted, send "nil" instead.
 - Threads: cannot be transmitted, send "nil" instead.
 This is all packed into an efficient binary representation. When this pass is done, the contents of all the ordinary tables have been updated. Only the tangible databases have not been updated.
 ## Pass: Transmit Tangible Differences
 For each master tangible database in the *close tangibles list*, compare it to the corresponding tangible in the client model.
 This pass is exactly the same the previous pass, "Transmit Table Differences," except that instead of comparing the ordinary tables, we compare the tangible databases. Aside from that difference, the code and logic are the same. The subroutine used by both passes is the same. We will therefore omit the detailed explanation.
 There is one small difference with the previous pass. Throughout the difference transmission code, we have treated metatables as if they were just another Key→Value pair, METATABLE→Value. We traversed them the same as any other Key→Value pair. But in this pass, we skip over metatables. Tangible metatables are handled in a separate pass.
 ## Pass: Compare Tangible Metatables
 A tangible's metatable is a place where the engine hides a great deal of information about the tangible, including for example its threads. Most of this data should not be difference transmitted at all. So therefore, we have exempted tangible metatables from the general-purpose table transmission passes above. Instead, we handle them separately.
 Currently the only piece of data in the tangible metatable which is transmittable is the tangible's classname. The comparison process is simple, we just compare the class name as a string, and if there's a mismatch, we send the class name as a string.
 ## Afterword
 Currently, that is all the passes of the difference transmission algorithm.
 ## About Determinism
 In order to ensure that the table numbering is identical on both client and server, we need for the graph traversal algorithm to visit Key→Value table entries in a deterministic order. We had to make certain patches to the Lua runtime to achieve this. See the document "A Summary of our Lua Patches" for more information about deterministic table iteration.
--- a/Docs/OLD/Attention
+++ b/Docs/OLD/Attention
@@ -0,0 +1,34 @@
 ### Attention Lock
 Note: at this time, none of the following is implemented.
 Attention lock is a new mechanism that is used when you want to force the player to give his *undivided* attention to a single activity. Attention lock is particularly relevant for turn-based combat. Let’s say you have a turn-based combat in progress. You don’t want the player to wander away and grow flax when it’s his turn to attack.
 The attention lock mechanism can affect the following aspects of the game:
 - Stops the player from walking around (or walking away).
 - Stops the player from clicking on unrelated sprites.
 - Hides the gui interfaces of unrelated sprites.
 - Forces the gui of a sprite to appear even if the user didn’t click on that sprite.
 - Directs the camera to point at a specific place, or a specific object.
 ## Enabling Attention Lock
 The mechanism is as follows: the player sprite has a single field, *actor.attention*, which is a pointer to a sprite that is trying to hold the player’s attention. There is a second field, *actor.attentionID*, which must be equal to *place.attentionID* of the attention-holding sprite. If *attentionID* doesn’t match, then *actor.attention* field is considered no longer valid, and the attention is considered to be not held. With that protocol, it is easy to implement these functions:
 - function attentionClear(place)
  - Release the attention of all sprites paying attention to place.
  - Implementation: set attentionID to a new unique ID.
 - function attentionHolder(actor)
  - Return the ID of the sprite holding actor’s attention.
  - Implementation: returns actor.attention after verifying attentionID match.
 - function attentionGrab(place, actor):
  - Cause place to grab the attention of actor.
  - Implementation: sets actor.attention to place, copies attentionID to actor.
 This is designed to be resilient to bugs: let’s say a battleground is holding the attention of some actor. If the battleground gets bulldozed, attention is implicitly released. Or if the battleground starts a new battle, attention of previous combatants would be implicitly released. In short, it should be pretty easy to clean up messes caused by buggy sprites grabbing attention.
--- a/Docs/OLD/Combat
+++ b/Docs/OLD/Combat
@@ -0,0 +1,34 @@
 ## Mechanisms Discussed:
 - Battlegrounds - using a sprite to implement combat.
 - Battle friends - how the friend system (separate) might designate “battle friends.”
 - Camera lock - a means where a sprite can force the camera to look in a specific place.
 - Attention lock - a means where a sprite can prevent a player from doing anything else.
 Battlegrounds:
 The JRPG combat system expects the god-mode player to build “battleground” in various places throughout the world. A “battleground” marks a physical place where combat can take place, and it defines the parameters of that combat. The battleground sprite contains all the combat-related scripts.
 Every NPC that can fight names a single combat arena. If a player picks a fight with that NPC, then the battle takes place in the specified arena. The players automatically run (or get teleported) to the left side of the arena, and the NPC runs to the right side of the arena, and the battle takes place.
 Combatant Selection:
 The initial step of the combat is that a *single* player picks a fight with a *single* NPC. But the battleground object then drags additional NPCs and players into the fight.
 NPCs: the battleground scans the vicinity for other NPCs that are linked to the battleground. It brings in any other NPC linked to that battleground. There may be additional rules that govern this.
 Players: the battleground scans the area for other players that are “battle friends” with the player. It picks the closest ones, and brings them into combat. If that isn’t enough people on the player side, the battleground scans for non-friend players and brings them in instead.
 Sometimes, the battleground might want to draw another player into the combat, but it might not be able to, because that other player is “too busy.” For example, the other player might already be involved in a different combat.
 The friend system is also a plugin. The friend system plugin should be required to export a function “is_battle_friend” to make it possible for the combat plugin to interface with the friend system plugin.
 Attention Lock:
 When the player enters a combat, the battleground puts the player into “attention lock.” Attention lock is intended for activities where the player is supposed to give his full attention to the activity, and not do other things at the same time. There is a separate document that describes attention lock in greater detail. The attention lock mechanism is used to prevent the player from walking away from the combat, and prevents him from growing flax or doing other irrelevant activities while combat is in progress.
 Turn-Based Combat:
 In this combat system, all players move at the same time, then, all NPCs go at the same time. The players have 10 seconds to specify which attack they want to use. If a player fails to specify an attack in 10 seconds, then a simple combat AI takes over and makes a choice for them. The goal of the combat AI is to give players whose connection is lagging a fair chance.
 When the 10 seconds are up, the battleground animates the combat round: it shows the players attacking one by one, then it shows the monsters attacking. The battleground sprite controls all the animation during this period.
--- a/Docs/OLD/Configurable
+++ b/Docs/OLD/Configurable
@@ -0,0 +1,17 @@
 This document lists all the configurable elements in the game.
 The engine has a list of plugins that *must* be provided in order for the game to be operational. For example. you *must* choose a combat system plugin. The plugin is a lua file. When you select a lua file to fulfill the combat system requirement, the lua file is checked to verify that it defines all the functions that a combat system must define. For most plugin slots, a null plugin is available - for example, a “no combat” combat system plugin.
 Combat System: a Plugin.
 Friend System: a Plugin.
 - Must export a function “battle_partner(A,B)” that returns true if player A has volunteered to help player B win fights. This exported function may be used by the combat system plugin.
 Camera Control: Manually controlled, but can be overridden by sprites
 - The camera system is an interactive camera control system like in the original Tale engine. (Not necessarily the same interactive control system). It’s mostly unscripted, but sprites can take control when they want to. This will involve some set of flags that I haven’t figured out yet.
 Skill Tree: a Database of Skills
 - Skill trees are a hardwired part of the game. Any particular game can choose not to use the skill tree system, or to have an empty skill tree, but the skill tree system’s code is always loaded.
--- a/Docs/OLD/Design
+++ b/Docs/OLD/Design
@@ -0,0 +1,43 @@
 ## Login System
 When you first connect to the server, the server calls *World::create_login_actor()* on the master world model. This creates a tangible of class *login*. This login tangible becomes the actor until further notice. On the server side, the ID of this actor is associated with the connection. This ID is also forwarded to the client so that the client may know its actor ID.
 The *login* is an actor like any other actor. In theory, it’s perfectly possible for the *login* to walk around and interact with objects. However, it’s more commonly the case that the login script prompts the player for a username and password. The login script then uses the lua *redirect(old_actor_id, new_actor_id)* command [not implemented yet].
 In standalone mode, the standalone client mimics this sequence.
 ## Current Actor, Current Place, Current ID Player Pool flag.
 Before a thread is awakened (by calling ‘lua_resume’ or ‘lua_pcall’ from C), the actorid and placeid are is stored in World::lthread_actor_id and World::lthread_place_id. As soon as lua_resume or lua_pcall returns, these two variables are cleared. That way, whenever any thread is running, those two variables contain the current actor and place. These two variables can be fetched using the lua functions tangible.actor and tangible.place.
 Of course, the lua thread scheduling code (which is responsible for calling lua_resume) needs to know the actorid and placeid in order to be able to put them into World::lthread_actor_id and World::lthread_place_id. That is possible because the thread info table stores the actor and implicitly the place.
 In addition to “current actor” and “current place,” we also store “current ID player pool flag.” This flag determines whether a thread uses the player’s ID pool, or the global ID pool to allocate IDs. In general, a thread uses the player pool if it’s just been awakened by “invoke_plan”. It uses the global pool if it’s been awakened by the system clock (ie, after a “wait” statement). This flag is in World::lthread_use_ppool, and in the thread info table.
 ## Deleted Tangibles and Tangible Garbage Collection
 Tangibles can be in three states: “created”, “deleted,” and “nothing”. When a tangible is in the created state,, the following things are true:
 - There’s a lua portion of the tangible.
 - There’s a C++ portion of the tangible.
 - The lua portion has a class, a metatable, and several important fields.
 When a tangible is in the deleted state, the following things are true:
 - There’s still a lua portion, but it’s minimal.
 - There’s no C++ portion of the tangible.
 - The lua portion still has a class and a metatable, but it’s mostly blank and writes are forbidden.
 When a tangible is in the nothing state, nothing is stored. No lua portion, no C++ portion.
 The function LuaStack::maketan retrieves a tangible. If that tangible doesn’t exist, it creates a tangible in the deleted state and returns that.
 The function World::tangible_make creates a tangible. If a deleted tangible with the same ID already exists, that deleted tangible is converted into a created tangible. The function World::tangible_delete transforms a created tangible into a deleted tangible.
 We’re going to have to write our own garbage collector to convert deleted tangibles into nothing tangibles only when there are no references to the deleted tangible.
 ## Makeclass
 We’re going to store a table in the registry, “classes”, that maps names to class tables. When you call makeclass, it creates a class in this table and copies it into the global environment. If for some reason the global environment gets trashed, the “classes” table in the registry will still be good. In this way, we can guarantee that the class of a given name is really a unique table.
 If we want class tables to be garbage collected (as opposed to just cleared), we’re going to have to write our own garbage collector to handle that.
--- a/Docs/OLD/Hard
+++ b/Docs/OLD/Hard
@@ -0,0 +1,25 @@
 ### Solutions to Hard Problems
 **Saving the Game:** This is accomplished by serializing the lua state using eris, and serializing the rest of the C++ data structures the usual way.
 **Snapshotting for Client-Side Prediction:** we serialize the synchronous world model (using the save-game serialization code) to save it. Then we apply predictive commands, and it becomes the asynchronous model. When necessary, we roll it back to its synchronous state.
 ### []() Solutions we Didn’t Use
 **Snapshotting Lua:** the function lua_newstate accepts a memory allocation function as a parameter. Using this hook, we can keep track of every block of memory that lua allocates. To snapshot lua’s state, it’s sufficient to memcpy all its memory blocks into a temporary buffer. To restore lua’s state, memcpy everything back, and also restore the heap to the state it was in. We built a proof-of-concept of this and it works. We could use this on the client side for asynchronous/synchronous work, instead of using eris serialization.
 **Collision Detection:** We could have a function that is a bit different on the client vs. server. On the server:
 ```cpp
 bool TryEnter(Location OldLoc, Location NewLoc)
 ```
 And on the client:
 ```cpp
 void TryEnter(Location OldLoc, Location NewLoc, Location &NewLocMod)
 // NewLocMod will always be set to a value where the server version of this call will
 // succeed, or to OldLoc.
 ```
 We could have a data structure that is a pancake stack of heightfields. The area between odd and even heightfields is enterable. Given an x,y,z coordinate, it is easy to find the x,y,z’ coordinate that represents the floor. The resolution on the client will be higher of course.
--- a/Docs/OLD/Recursion-Free
+++ b/Docs/OLD/Recursion-Free
@@ -0,0 +1,34 @@
 ## Definitions
 - Synchronous Models and Connections have a one-to-one mapping. The terms are used interchangeably.
 - Sprites are objects in the 3D world. Each has a primary table associated with it.
 - Tables are Lua Tables. The runtime assigns each a two-part ID consisting of a Sprite ID and a Table Index that starts at 1 for a Sprite’s Primary Table.
 **Synchronous Models** - Each has a few values within it, which are not directly accessible from Lua:
 - Position (Plane, X, Y, Z)
 - SpriteID - If present, the connection’s Position gets updated periodically based on the Position of the Sprite
 **Sprites** - These also have a few values which are not directly accessible from Lua:
 - Position (Plane, X, Y, Z)
 - Sprite ID
 - NextTableID
 **Tables** - Each table has a mode with hints to the difference transmitter when it’s OK to suppress it from a synchronous model. Each Table contains a two part ID consisting of a Sprite ID and a Table Index. A Table has an inferred position based on it’s Sprite ID. Some modes are:
 - A: Never Suppress (A Global Table)
 - B: Always Suppress (A Secret Table)
 - C: Suppress from SM’s that are far away (A Sprite’s Primary Table, a Player’s publicly viewable data)
 - D: Suppress when a connection’s SpriteID differs from the Table’s Sprite ID (A player’s sensitive data)
 - E: Call the Table.Transmit(OtherSpriteID) to determine whether to transmit. (Does Lua have a way to make a const call that prohibits storing data in any table fields?)
 **Table Creation** - I am assuming that whichever version of GUI->MethodCall we use there will be the notion of “Actor” and “Place” as in our old engine. The default mode of a table is:
 - When Place is the same as Actor: D
 - When Place is different from Actor: C
 **Difference Transmitter Manipulation** - Designers can explicitly override the default mode of a Table with an intrinsic function:
 - SetDiffMode(Table,Mode)
 - SetOwner(Table,Sprite)
--- a/Docs/OLD/Skills
+++ b/Docs/OLD/Skills
@@ -0,0 +1,15 @@
 Every game has a database of possible skills. The database can be edited on the fly.
 Skills have:
 - Name: eg, “handgun accuracy”
 - Point cost: how many skill points do you need to increment this skill. Note that the game can bypass the point system by procedurally incrementing the player’s skill.
 - Maximum skill level: how high can this skill be incremented. Again, the game can bypass this by procedurally incrementing the player’s skill.
 - A graphical icon representing the skill.
 - Prerequisites list: other skills you have to have before incrementing this skill.
 The skill tree isn’t hardwired into code. Instead, there’s an in-game skill tree editor which you can use when you’re in god mode. We should also provide facilities so that two different games can easily share a skill tree.
 The engine should provide a built-in character progression screen, so that the player can spend skill points to advance along the skill tree.
 When a player comes into a game, he brings his skills with him. There are two preliminary checks: first, that he hasn’t spent too many skill points. Second, every skill he has is checked against the skill tree. If he has acquired a skill in violation of the prerequisites, then that skill is “greyed out” until he buys the prerequisites.
--- a/Docs/Our
+++ b/Docs/Our
@@ -0,0 +1,513 @@
 ### Our In-House Lua API
 Lua comes with a C API, which makes it possible to write C functions that manipulate Lua data. The standard API is functional, but it has some readability issues and can be difficult to work with. Therefore, we have created our own replacement API, which we call "LuaStack." In our codebase, you are expected to use LuaStack, not the standard Lua API. This paper will explain LuaStack using the following example function in Lua, which compares two tables to see if they are exactly equal:
 ```lua
 function table.equal(table1, table2)
 if (type(table1) ~= "table") then
 error("table1 must be a table")
 end
 if (type(table2) ~= "table") then
 error("table2 must be a table")
 end
 local size1 = table.nkeys(table1)
 local size2 = table.nkeys(table2)
 if (size1 ~= size2) then
 return false
 end
 for key, value1 in pairs(table1) do
 local value2 = table2[key]
 if value1 ~= value2 then
 return false
 end
 end
 return true
 end
 ```
 A few notes about that function. You can't just use the expression "table1==table2." That would tell you if table1 and table2 are the same table (ie, pointer equality). The function above, by contrast, checks if two different tables contain the same key-value pairs. The built-in function *table.nkeys* is something that we added to lua, it returns the number of key-value pairs in a table. Once you fully understand the Lua code above, have a look at the following C++ code, which does the same thing. Don't worry if you don't understand everything yet:
 ```cpp
 LuaDefine(table_equal, "(table1, table2)", "Return true if two tables are equal") {
 LuaArg table1, table2;
 LuaVar size1, size2, key, value1, value2;
 LuaRet equalflag;
 LuaDefStack LS(L, table1, table2, size1, size2, key, value1, value2, equalflag);
 LS.cktable(table1);
 LS.cktable(table2);
 LS.set(size1, LS.nkeys(table1));
 LS.set(size2, LS.nkeys(table2));
 if (!LS.equal(size1, size2)) {
 LS.set(equalflag, false);
 return LS.result();
 }
 LS.set(key, LuaNil);
 while (LS.next(key, value1, table1)) {
 LS.rawget(value2, table2, key);
 if (!LS.equal(value1, value2)) {
 LS.set(equalflag, false);
 return LS.result();
 }
 }
 LS.set(equalflag, true);
 return LS.result();
 }
 ```
 There's a lot to unpack there. You might be wondering what *LuaDefine* is, you might be wondering what *LS* is, you might be wondering what the *LuaDefStack* constructor does, and so forth. The rest of this paper is going to pick that C++ function apart piece by piece. We're going to explain every class, macro, and template used above. Once you're done reading this paper, you'll understand at a deep level how everything works.
 ## Class LuaArg, Class LuaVar, and Class LuaRet
 In C++ example function, the first few lines inside the body of the LuaDefine are:
 ```cpp
 LuaArg table1, table2;
 LuaVar size1, size2, key, value1, value2;
 LuaRet equalflag;
 ```
 These are variable declarations. You are meant to think of these as variables that can hold any lua data type - lua tables, lua strings, lua numbers, lua closures, lua threads, any data that the lua interpreter can handle. LuaArg are function arguments, LuaVar are local variables, and LuaRet are return values.
 That sounds straightforward, but there's a catch. The Lua garbage collector can't allow you to put Lua values into C++ variables. The reason for this is that if you were to put a pointer to a Lua table into a C++ local variable, the Lua garbage collector would not know that this pointer exists. It might think there are no pointers to the table, and it might free the table. In reality, we can't actually put Lua values into C++ variables.
 So that raises the question, how is it possible to have LuaArg, LuaRet, and LuaVar variables that seemingly hold lua values? The answer is that there's a hidden level of indirection. The values are actually stored on the Lua stack. In the example function, our variables end up on the lua stack in the following order:
 ```
 stack[1] holds equalflag
 stack[2] holds size1
 stack[3] holds size2
 stack[4] holds key
 stack[5] holds value1
 stack[6] holds value2
 stack[7] holds table1
 stack[8] holds table2
 ```
 In our example function, those eight stack positions are permanently reserved to hold the values of those eight variables. We never pop those values off the stack - at least, not until the end of the lua function.
 Here are the C++ declarations of class LuaArg, class LuaRet, and class LuaVar, with methods omitted:
 ```cpp
 class LuaSlot { int stack_position; };
 class LuaArg : public LuaSlot {};
 class LuaVar : public LuaSlot {};
 class LuaRet : public LuaSlot {};
 ```
 So as you can see, LuaArg, LuaRet, and LuaVar are all derived from LuaSlot. It really is true that LuaArg, LuaRet, and LuaVar don't add anything at all to class LuaSlot: their class bodies literally are completely empty, as shown above. A LuaSlot contains nothing but an integer stack_position.
 In our example function, once the LuaSlots are initialized, the stack_position values are as follows:
 ```
 equalflag.stack_position == 1
 size1.stack_position == 2
 size2.stack_position == 3
 key.stack_position == 4
 value1.stack_position == 5
 value2.stack_position == 6
 table1.stack_position == 7
 table2.stack_position == 8
 ```
 Technically, it is not exactly correct to say that the LuaSlots "contain" lua values. It is more accurate to say that the LuaSlots contain stack positions, and those stack positions contain lua values. That is, there's a level of indirection. However, when you're actually using the LuaStack API, you don't need to think about that hidden indirection. Conceptually, you're meant to think of the LuaSlots as effectively containing lua values. For the remainder of this paper, I will often say that LuaSlots "contain" lua values, glossing over the existence of a level of indirection.
 ## The LuaDefStack Constructor
 In the previous section, I mentioned that the LuaSlots are all assigned unique stack positions. You might be wondering: where is the code that assigns unique stack positions to the LuaSlots?
 It's not in the LuaSlot constructor. The LuaSlot constructor doesn't do anything but set stack_position to 0. Since everything in Lua is numbered from 1, that's an invalid stack position. It means that the LuaSlot is still not initialized. Any attempt to use the LuaSlot at this stage will generate an error.
 The machinery to assign unique stack positions lives inside the LuaDefStack constructor. LuaSlots are useless without a LuaDefStack to set them up. This section will describe the LuaDefStack constructor, and all the various functions it performs.
 In our example C++ function, you'll find the LuaDefStack constructor immediately after the LuaSlot declarations:
 ```cpp
 LuaDefStack LS(L, table1, table2, size1, size2, key, value1, value2, equalflag);
 ```
 As you can see, the first argument to the LuaDefStack constructor is a capital L. That's boilerplate: the first argument is always capital L. The declaration of that variable, *lua_State *L,* is concealed inside the *LuaDefine* macro. I'll talk about LuaDefine later. That data type, *lua_State*,* is from the standard Lua API. It's a C-style pointer to the lua stack. The LuaDefStack constructor stores this pointer inside the LuaDefStack.
 After the capital L, all the LuaSlots have been passed to the LuaDefStack constructor. They are passed by non-const reference, which means that the LuaDefStack can mutate them.
 The LuaDefStack constructor allocates stack space for the LuaSlots. In our example function, it counts that there are eight LuaSlots, so it allocates the first eight positions on the lua stack to hold them. Then, it chooses a stack position in the range 1 to 8 for each of the LuaSlot variables. It mutates each LuaSlot to contain the correct stack_position.
 The LuaDefStack constructor notices that two of the LuaSlots are of class LuaArg. It knows that LuaArgs are meant to hold function arguments. So it checks that two function arguments were passed to the function. If that's not the case, it throws a lua error. But if all is well, it moves the function arguments into the stack positions reserved for the LuaArgs.
 This is why we have the declaration of LuaArg which derives from LuaSlot, but doesn't add any functionality to class LuaSlot. Even though LuaArg inherits everything from LuaSlot, the LuaDefStack constructor can differentiate it, and treat it as special.
 The LuaDefStack initializes the contents of all the LuaSlots. As mentioned above, the LuaArg are initialized from the function arguments. The LuaRet and LuaVar are initialized with nil. After the LuaDefStack constructor, all the LuaSlots have the correct stack_positions, and via that level of indirection, the LuaSlots all contain the correct lua values.
 Meanwhile, the LuaDefStack contains the pointer to the lua stack. Note that in the example function, the object of class LuaDefStack is named *LS*. That stands for "Lua Stack." That is a naming convention that we use throughout our codebase - the LuaDefStack is called *LS*. But unlike *lua_State* L*, which is also a pointer to the lua stack, *LuaDefStack LS* is a C++ style smart pointer, which has methods.
 ## Methods of Class LuaDefStack
 In our example function, you'll find this line of code:
 ```cpp
 LS.set(size1, LS.nkeys(table1));
 ```
 Let's break that down into pieces. Here's a simpler statement, using just LS.set:
 ```cpp
 LS.set(size1, 3);
 ```
 Conceptually, you're supposed to think of that in the very obvious way: it sets *size1* to 3. But you also know that there's a hidden level of indirection, so you might be wondering about *how* it sets *size1* to 3.
 The function LS.set is a call to the following method:
 ```cpp
 void LuaDefStack::set(const LuaSlot &slot, int value) const;
 ```
 Inside that method, *this* contains the pointer to the lua stack, *slot* contains the necessary stack_position, and *value* contains the value to store. What this method actually does is set stack[stack_position] = value. In our example function we know that *size1.stack_position* == 2, and *value* == 3. So this sets stack[2] = 3. But again, you're meant to ignore the hidden indirection, you're supposed to think of it as just setting *size1* to 3.
 OK, now, let's consider this statement from the example function:
 ```cpp
 LS.set(size1, LS.nkeys(table1));
 ```
 That's two method calls. The *nkeys* method call fetches the size of *table1*, returning it as an 'int'. The *set* method stores the int in the appropriate stack slot for *size1*.
 The first argument to LuaDefStack::set must be a LuaSlot. The second argument is overloaded, it can be any one of several different data types:
 ```cpp
 LS.set(slota, "hello"); // set to a char*
 LS.set(slota, 3.14159); // set to a double
 LS.set(slota, true); // set to a bool
 LS.set(slota, LuaNil); // set to nil
 LS.set(slota, slotb); // copy from another LuaSlot
 ```
 In general, methods of LuaDefStack will automatically convert the following C++ types to Lua: int, int64_t, float, double, char *, string, string_view, and bool. If you want to convert lua values back to C++ values, it's a little harder. More on that in the next section.
 For a complete listing of all the operators in LuaDefStack, have a look at the header file luastack.hpp, at the class LuaBaseStack. There, you will find prototypes for all the functions, along with documentation for each one.
 ## Converting Lua Values to C++ Values
 Suppose that you have a function, callable from lua, that is supposed to accept a string as a parameter:
 ```cpp
 LuaDefine(frob, (str), "Frobnicate a string.") {
 LuaArg str;
 LuaDefStack LS(L, str);
 …etc…
 ```
 The variable 'str' is a LuaSlot. Suppose you want to convert that to a C++ string. LuaDefStack provides a method that can do that:
 ```cpp
 eng::string s = LS.ckstring(str);
 ```
 The method *ckstring* verifies that *str* really is a lua string. If it is a string, it converts it to an eng::string and returns it. If it's not a string, ckstring throws a lua error: "value must be a string". That's not an especially good error message, it lacks all context. You can make it slightly better, "str must be a string", by passing the variable's name to ckstring:
 ```cpp
 std::string s = LS.ckstring(str, "str");
 ```
 Class LuaDefStack provides a full catalog of conversion functions that return different C++ data types:
 ```cpp
 bool ckboolean(LuaSlot s, const char *argname = "value") const;
 lua_Integer ckinteger(LuaSlot s, const char *argname = "value") const;
 int ckint(LuaSlot s, const char *argname = "value") const;
 lua_Number cknumber(LuaSlot s, const char *argname = "value") const;
 eng::string ckstring(LuaSlot s, const char *argname = "value") const;
 std::string_view ckstringview(LuaSlot s, const char *argname = "value") const;
 lua_State * ckthread(LuaSlot s, const char *argname = "value") const;
 LuaToken cktoken(LuaSlot s, const char *argname = "value") const;
 util::DXYZ ckxyz(LuaSlot s, const char *argname = "value") const;
 void cktable(LuaSlot s, const char *argname = "value") const;
 void cknil(LuaSlot s, const char *argname = "value") const;
 void ckfunction(LuaSlot s, const char *argname = "value") const;
 void ckcfunction(LuaSlot s, const char *argname = "value") const;
 void cktangible(LuaSlot s, const char *argname = "value") const;
 ```
 Notice that cktable returns void. That's because you can't convert a lua table to a C++ anything. So cktable just checks the type and if it's not a table, it throws a lua error, "value must be a table". If it is a table, cktable doesn't do anything at all. Note that we used *LS.cktable* in the example table_equal function, to verify that the caller passed in two tables.
 OK, so what if you want to convert the LuaSlot str to a C++ string, but you want to handle the non-string exception case yourself? Use this code:
 ```cpp
 std::optional<eng::string> s = LS.trystring(str);
 ```
 You may not be familiar with C++ std::optional, so I'll summarize here. An optional<string> can either contain a string, or it can contain nothing. There are methods that can check if the optional is empty, and methods to extract the string value, if present.
 What LS.trystring(str) does is: if str is a string, trystring returns an optional<string> containing the string. If str is not a string, trystring returns an optional<string> that is empty.
 Unlike the ck-functions functions which throw errors on type mismatch, the try-functions never throw an error. Using the try-functions is ideal in code where throwing errors would be bad.
 What if you just want to know if it's a string, but you don't need the actual string? In that case, use *isstring:*
 ```cpp
 bool b = LS.isstring(str);
 ```
 You can also use LS.type to fetch the type of a lua value as an enum.
 ## Dealing with Return Values
 Our sample function contains this code:
 ```cpp
 LuaRet equalflag;
 …
 if (!LS.equal(size1, size2)) {
 LS.set(equalflag, false);
 return LS.result();
 }
 ```
 The variable declaration, "LuaRet equalflag", declares that this function has a return value whose name is equalflag. In our API, return values are just variables. You have to assign a value to the variable before you return from the function. In the code above, when size1 != size2, it stores *false* into equalflag. Then, there is a return-statement.
 Lua calling conventions dictate that at the end of a function, there shouldn't be anything on the lua stack except for the return values. That is accomplished by the method LS.result. This takes everything off the stack except for the LuaRet values. Another lua calling convention is that the function must return the number of return values that are on the lua stack. Therefore, LS.result returns the number of LuaRet values that were declared.
 So, in every LuaDefine, the process of returning from a function always consists of two steps:
 1. You must set the LuaRet slots to whatever values you want to return.
 2. There must be an explicit return-statement that must always be *return LS.result()*
 ## The LuaDefine Macro
 Our sample function starts with this line of code:
 ```cpp
 LuaDefine(table_equal, "(table1, table2)", "Return true if two tables are equal")
 ```
 The two strings in this line of code are just documentation. Neither has any effect on how the function works. They are collected automatically by our documentation system, which can print them out on demand.
 LuaDefine is a C preprocessor macro. Before I tell you what it macro expands into, I'd like to tell you how Lua functions are normally written using the standard Lua API. When using the standard API, lua functions look like this:
 ```cpp
 int table_equal(lua_State *L) {
 …
 }
 ```
 The C prototype is always the same: the return value is always *int,* and the only argument is *lua_State*L*, a pointer to the lua stack. A C function must have this prototype to be compatible with the lua runtime.
 Next, I will show you the macro expansion of a LuaDefine to help you understand what it does. Here is the LuaDefine that I will macro expand:
 ```cpp
 LuaDefine(table_equal, "(table1, table2)", "Return true if two tables are equal")
 ```
 Here is the expansion:
 ```cpp
 int lfn_table_equal(lua_State *L);
 const char *lfnarg_table_equal = "(table1, table2)";
 const char *lfndoc_table_equal = "Return true if two tables are equal";
 LuaFunctionReg reg_table_equal
 ("table_equal", lfnarg_table_equal, lfndoc_table_equal, false, lfn_table_equal);
 int lfn_table_equal(lua_State *L)
 ```
 So first, you can see that there's a prototype for the function lfn_table_equal. As you can see, it returns *int* and accepts *lua_State *L,* which makes it compatible with the lua runtime. This is a forward declaration.
 Next, there are the two documentation strings. I've given these documentation strings two names: lfnarg_table_equal and lfndoc_table_equal. That way, it's possible to access these documentation strings in code.
 Next, there is the "registry entry" reg_table_equal, of class LuaFunctionReg. The constructor of reg_table_equal runs at program initialization time. The constructor stores its arguments: the function name, the documentation strings, the flags, and the C function pointer. These are all stored inside reg_table_equal. Then, the constructor links reg_table_equal into a global linked list of LuaFunctionReg objects. This linked list makes up the registry of all LuaDefines.
 Finally, we have a second copy of the prototype for lfn_table_equal. This time, it's not a forward declaration: there's no semicolon. This prototype is meant to be followed by the function body.
 The upshot of all this is that LuaDefine doesn't just declare a function. It also links that function into a global registry of all the LuaDefines. The registry contains not only the function name and function pointer, but also the function documentation. Remember also that in C++, static objects like reg_table_equal are constructed before *main*. So by the time *main* starts, the registry is ready-to-use.
 The registry of all LuaDefines has two primary uses:
 1. The routine source_load_cfunctions traverses the registry and inserts every LuaDefine into the lua environment. It's fully automatic: simply writing a LuaDefine causes the function to appear in Lua.
 2. The documentation system traverses the registry to find the documentation strings. It uses these strings to auto-generate the manual.
 ## Class LuaExtStack
 What if you want to write C++ code that manipulates Lua data, but which *isn't* called from inside Lua? In that case, LuaDefStack won't work, because LuaDefStack is hardwired to assume that a caller *exists*:
 - It checks that the caller passed the right number of arguments.
 - It transfers the arguments from the caller to the LuaArg slots.
 - It transfers the LuaRet slots back to the caller at function end.
 For situations where there isn't any caller, we have created an alternative to LuaDefStack called LuaExtStack. There are several things that LuaExtStack does differently:
 - LuaExtStack doesn't accept LuaArg or LuaRet parameters - it only allows LuaVar parameters. That only makes sense, since it should be used in contexts where there's no caller.
 - There is no code in LuaExtStack to check that the caller passed the right number of arguments, since again, there is no caller.
 - Class LuaExtStack doesn't have a method LS.result, since there's nobody to return a result to.
 - Class LuaDefStack assumes that anything already on the stack must be a function argument. Class LuaExtStack assumes that anything already on the stack is data that it is not supposed to touch in any way: it leaves it alone. It allocates stack space above whatever is already there.
 - Class LuaExtStack has a destructor that automatically pops the stack back to the level where it was when the LuaExtStack was constructed. This is convenient. LuaDefStack, on the other hand, doesn't restore the stack top: it can't, because return values must be left on the stack.
 Aside from those differences, LuaExtStack and LuaDefStack are quite similar. For example, all the methods are the same: LS.set, LS.ckstring, LS.nkeys, and so forth all work the same with LuaDefStack and LuaExtStack.
 ## Using Multiple Lua Stacks
 In 95% of the code you'll write, there is only one lua stack. However, there are two big exceptions to this rule.
 In the difference-transmission code, you need to compare the master world model to the synchronous world model. Each world model has its own instance of the lua interpreter. Each instance of the lua interpreter has its own lua stack. So when comparing master to synchronous, you're manipulating two lua stacks at the same time.
 In that situation, it is typical to use two LuaExtStacks: one for the master model, one for the synchronous model. The naming convention is as follows:
 ```cpp
 LuaVar mtable; // A variable meant to hold a master model table.
 LuaVar stable; // A variable meant to hold a synchronous model table.
 LuaExtStack MLS(ML, mtable); // Master world model stack object
 LuaExtStack SLS(SL, stable); // Synchronous world model stack object
 ```
 In cases like that, it is essential to be careful - every LuaSlot must be used with the correct stack. For example, this will cause a crash:
 ```cpp
 MLS.set(stable, 3);
 ```
 That's because stable contains a stack_position on the SLS stack, not a stack_position on the MLS stack. Every LuaSlot is associated with one and only one lua stack, and you must pair them up correctly.
 The other place we use two stacks is inside World::spawn. This does some initial calculation using the default stack of the world model. It creates a LuaExtStack to manipulate the default stack. Then, after the initial calculations are done, it actually creates the new thread, which has its own stack. In the current codebase, the new thread doesn't get its own LuaExtStack. Instead, it's simpler to just use the standard API to set up the new thread and move the data onto it. Therefore, we avoid the risk of using LuaSlot objects with the wrong LuaExtStack.
 ## Mixing LuaStack with the Standard API
 It is rarely necessary to call functions in the standard API - the LuaStack API is meant to be a complete replacement. But every now and then, you encounter a case that LuaStack doesn't handle, or doesn't handle easily. In cases like that, it's possible to mix the standard API with the LuaStack API.
 Here's an example. When you type some lua code into the luprex console, that code has to be compiled into a closure, after which the closure gets called. This is all handled by the function World::invoke_lua.
 Ideally, our LuaStack API should provide a method to compile lua code into a closure. But I haven't gotten around to it yet. Therefore, World::invoke_lua has to use the standard lua API to compile the code. This is an excerpt from World::invoke_lua:
 ```cpp
 LuaVar func;
 LuaExtStack LS(L, func);
 int status = luaL_loadbuffer(L, datapack.data(), datapack.size(), "=invoke");
 lua_replace(L, func.stack_position());
 ```
 So the first two lines use the LuaStack API. We've defined a LuaVar *func* to hold the compiled closure. The LuaExtStack constructor allocates one space on the lua stack to hold *func*. At this point, the lua stack contains:
 ```
 stack[1] contains func, whose value is nil
 ```
 Then, we call the function luaL_loadbuffer. This is a function from the standard lua API, it compiles a source code string and pushes the compiled closure on the stack. The C++ variable "datapack" contains the lua source code that was typed in at the console. After calling luaL_loadbuffer, the stack contains:
 ```
 stack[1] contains func, whose value is nil
 stack[2] contains the compiled closure
 ```
 Finally, our snippet calls lua_replace. This is a function from the standard lua API, it pops the stack and stores the popped value at a specified position on the lua stack. We use this to pop the compiled closure off the stack, and transfer it into the LuaVar *func*. After calling lua_replace, the stack contains:
 ```
 stack[1] contains func, whose value is the compiled closure
 ```
 From this point forward, we are done using the standard API. The rest of World::invoke_lua uses only the LuaStack API. So as you see, it is often possible to write most of the code using the LuaStack API, switching to the standard API for only two or three lines of code.
 The standard API generally manipulates values that are on top of the stack. Generally, when you're done using the standard API, you want to move the values from the top of the stack to a LuaSlot. As demonstrated above, that is done using lua_replace.
 There's a mirror image of this function: lua_pushvalue. This is useful for copying a value from a LuaSlot to the top of the stack. This is generally useful when you want to take a value from a LuaSlot, and then manipulate it using the standard API.
--- a/Docs/Roadmap
+++ b/Docs/Roadmap
@@ -0,0 +1,51 @@
 In no particular order:
 Secret / semi-secret variables
 Secret functions
 Global variables
 Unicode support for the console
 Blockchain integration
 Closures instead of a quoted string in menus
 GUI that looks for verbs
 More sophisticated passage of time
 GUI that runs in realtime
 The Graphics Engine
 Little game demos:
 The monopoly game (Errands where you pass thru land owned by others with voronoi diagram)
 The Spectra game
 Heroes of Rock,Paper,Scissors
 Come up with a plan for 'redirect'.
 Modify World::scan_near to not allocate any memory, so it can be accessed as a 'getter' in the DrivenEngine.
 probes should never modify the state of the world. Currently, there is no protection to guarantee this, except for something in lpxclient where it does a snapshot and rollback before and after the probe. We should come up with a more comprehensive mechanism.
 ### []()Questionable Design Choices
 When we delete a tangible, we currently leave the LUA_TT_TANGIBLE flag on the table, and we leave the metatable in place with the tangible ID in it. Is this the right thing to do? Should we turn it into a truly blank normal table?
 ### []()Hard Problems without Solutions
 **Using Multi-Core CPUs**: We want to do difference transmission in parallel (one thread per connected client). This may be possible because each synchronous model is a separate lua interpreter. However, the master world model is shared. It may be possible to access it in a read-only manner in multiple threads.
 **Nonsortable Keys:** what to do about table keys that are other tables, closures, or coroutines?
 **Collision Detection**: In the old engine, people could walk through walls. I don't know if that is okay in 2020. But the separation of world-model and graphics engine makes this hard.
 **Dealing with Infinite Loops:** if the script goes into an infinite loop, there's no easy way to deal with that. The lua interpreter does have hooks where you theoretically could count cycles.
 **Module-Level Locals:** module-level locals are useful to store constants. However, when they're used as variables, they create pseudoglobal state that gets reset every time you edit a source file, and which is unfindable by the difference transmitter.
--- a/Engine.md
+++ b/Engine.md
@@ -0,0 +1,187 @@
 ### The Event-Driven Structure of the Engine
 Most of the Luprex engine is 'event-driven'. The event-driven design makes certain things easier, and certain things harder. This document goes over all the various consequences of this design decision.
 ## What does "Event-Driven" Mean?
 To be clear about what I mean by "event driven," think of a traditional Finite State Machine (FSM). Here's an example FSM, for a coin-operated turnstile:
 ![](The%20Event-Driven%20Structure%20of%20the%20Engine_html_f481ae616a8b35e1.png)
 So this example FSM has two *events* that it can process: a person can insert a coin, and a person can push through the turnstile.
 So the important thing for our purposes is how traditional FSMs deal with I/O. In a traditional FSM, every bit of I/O has to be somehow expressed as an *event*. The event is fed into the FSM, and it drives the FSM forward to the next state.
 When writing GUI software for the Win32 API, you have to write a 'WinMain' function. This is an infinite loop that calls GetMessage, and then processes the message. "Message" is their word for "event." So at least partially, Windows programs are event-driven. However, in Windows, it is possible to do some I/O in a non-event-driven manner. For example, in Windows, you can just call a function to read a file. That's I/O, and it's not event-driven. So the windows API is only partially event-driven.
 Our system is *fully* event-driven. That means that all I/O, no matter where or what kind, is expressed as an event.
 ## The Driver and the Driven Portion
 We have carefully separated the Luprex code into two halves: the *driver*, and the *driven portion*. The driven portion is the part of the system that is event-driven (naturally). The driver is the part that is responsible for generating events and feeding them into the driven portion.
 The driven portion of the system is organized as a library of subroutines. The top-level API of the driven portion is class DrivenEngine. The driver is responsible for constructing the object of class DrivenEngine. Then, the driver feeds events into the driven portion by calling various event methods in class DrivenEngine. Finally, when there's nothing left to do, the driver calls DrivenEngine's destructor. The driven portion of the system is not allowed to call functions in the driver. The driven portion is also not allowed to do I/O. All I/O is done in the driver, and it's always the driver calling into the driven portion.
 Here's an example of how input typically works in this system: if a TCP socket is open, and bytes have arrived, then the driver reads the bytes. Then, the driver calls DrivenEngine::drv_recv_incoming to pass the bytes into the driven portion: this is an input event.
 So how does the driven portion *send* bytes to a TCP socket? The driven portion isn't allowed to do any I/O, and it's not allowed to call into the driver. But what it can do is leave the bytes in a buffer. The driver polls the buffer, by calling DrivenEngine::drv_get_outgoing, and it sees that the bytes are there. By the way, DrivenEngine::drv_get_outgoing doesn't count as an event, because it never changes the state of the driven portion. The driver then sends the bytes to the TCP socket. Finally, the driver calls DrivenEngine::drv_sent_outgoing, to notify the driven portion that it should remove the bytes from the buffer. That *does* count as an event, because it does modify the state of the driven portion. In this way, the driven portion can do output.
 If the driven portion wants to *open* a TCP connection, how does it do that? Again, it can't do I/O, and it can't call into the driver. So instead, here's what it does. It creates a "channel" data structure representing the new communication channel. This channel lives within the driven portion. The "channel" data structure has buffers to store incoming and outgoing bytes, but it doesn't have any actual socket in it: remember, the driven portion is not allowed to use raw I/O primitives. The driver polls for the existence of this channel, using DrivenEngine::drv_get_new_outgoing. Like all the other drv_get methods that don't change the state of the driven portion, this is not an event. When the driver sees the new channel, it opens the TCP socket. TCP sockets live in the driver, whereas channels and their buffers live in the driven portion. The driver maintains a one-to-one connection between the two.
 File I/O is the most awkward thing to do with an event-driven design. Currently, the only file I/O needed by Luprex is to read the lua source code. So the driven portion simply sets a flag indicating that it needs an updated copy of the lua source. The driver polls the flag, and if it is set, the driver reads the lua source code and feeds it into the driven portion using DrivenEngine::drv_invoke_lua_source. It works, but it's awkward for the driven portion of the engine. In all likelihood, this will eventually get replaced by a more sophisticated interface.
 The driver is a very small piece of code. Its only function is to do socket I/O, file I/O, and a few other kinds of I/O. All the complexity of the game engine is inside the driven portion.
 ## Different Drivers for Different Operating Systems
 Currently, we have two versions of the driver: one that runs from the command line using raw operating system primitives to do I/O, and one that runs within the Unreal Engine using Unreal primitives to do I/O. The command-line version uses ifdefs in order to support both Windows and Linux.
 The driven portion, on the other hand, is entirely operating-system independent. There is only one version of the driven portion. The driven portion doesn't need any ifdefs for operating systems because there is no I/O code in the driven portion. It's pure standard-compliant C++.
 The driver is actually a very small piece of code. For example, the command-line version of the driver is only about a thousand lines of code for the whole thing, including both the Windows and Linux versions of certain subroutines.
 The event-driven design has the effect of concentrating all the operating-dependent code into one place, the driver.
 ## How Event-Driven Design is Useful for Debugging
 When our server runs, it can maintain a log of every *event* that the driver feeds into the driven portion. Later, it is possible to replay the log: during replay, the driver creates a DrivenEngine. Then, it feeds that DrivenEngine the exact same events, in the exact same order as when the game was live. If we designed things correctly, then the driven portion will perform the exact same computations, in the exact same way as when the game was running live.
 The advantage of this is that if the server crashes, you should be able to replay the crash a second time, this time inside a debugger. You can single-step the code right up to the point of the crash, examining variables and data structures using the debugger, as many times as necessary to figure out the cause of the crash.
 In order for this to work, the driven portion needs to be *deterministic*. That means that if I create two identical DrivenEngines, and call the same event methods in the same order on both of them, they must both perform the same exact computations, and must both end up in the exact same state. Ideally, this should be the case even if the two DrivenEngines are running on different machines!
 The replay code lives in the driver. The driver knows it is running in replay mode. In replay mode, the driver is generating events from a logfile, not from TCP sockets. So therefore, the driver is *not* in the exact same state as when the system was running live. However, the driven portion has no idea that the system is in replay mode. As far as the driven portion knows, when the driver calls DrivenEngine::drv_recv_incoming, it's because a TCP socket received some bytes. It has no idea that the event is actually coming from a logfile.
 So the driven portion needs to be deterministic, it needs to be running exactly the same during recording and during replay. The driver, on the other hand, does not need to be deterministic, since what it's doing at recording time (receiving bytes from TCP sockets) and what it's doing at replay time (generating events from a logfile) are completely different.
 ## The Driven Portion is a DLL, the Driver is an EXE
 The driven portion of the engine lives in a DLL. The driver is an EXE file. The driver loads the DLL. If running inside Unreal Engine, then the driver and the Unreal Engine are integrated, the EXE contains both. The driven portion is the exact same DLL regardless of whether it's loaded by an Unreal Engine driver or a command-line driver.
 Putting the driver in an EXE and the driven portion into a DLL helps enforce the rule that the driven portion is not supposed to call into the driver. It literally cannot, because the driver is not in the DLL.
 The driven portion DLL is built using the Luprex build system. The command-line driver is also built using the Luprex build system. However, the driver that is integrated into the Unreal Engine is built using the Unreal build system.
 Since it is possible that the two different build systems might use different C++ compilers, it is theoretically possible for the C++ in the DLL to not be entirely compatible with the C++ in the EXE. For example, virtual tables might not be laid out the same by the two compilers. Name mangling might also not be done the same by the two compilers. To avoid problems, we have given the DLL a pure C interface - in general, a C interface is compatible between all compilers.
 The pure C interface of the DLL is implemented by *struct EngineWrapper*. Notice that it's a struct, not a class, because it's pure C, not C++. EngineWrapper is a very thin wrapper around the methods of class DrivenEngine. For example, DrivenEngine contains a method DrivenEngine::drv_recv_incoming. Meanwhile, struct EngineWrapper contains EngineWrapper::play_recv_incoming, which is a C function that passes its arguments to the C++ function DrivenEngine::drv_recv_incoming.
 The wrappers in class EngineWrapper, in addition to simply translating C calling conventions to C++ calling conventions, are also responsible for the logging code. For example, when the driver calls EngineWrapper::play_recv_incoming, the EngineWrapper checks if it is in logging mode. If so, it writes the event to the logfile before it calls DrivenEngine::drv_recv_incoming.
 Struct EngineWrapper is considered part of the driver, not the driven portion. That's the case for two reasons. First, struct EngineWrapper passes events into class DrivenEngine. That's the driver's job, passing events into the DrivenEngine. Second, EngineWrapper does logging. That's I/O, so it must be part of the driver.
 Struct EngineWrapper is a violation of the rule that the driver lives in the EXE, and the driven portion lives in the DLL. EngineWrapper is a piece of the driver, but it lives in the DLL. This is the only violation of the separation rules.
 EngineWrapper does file I/O for logging. Fortunately, the C++ standard provides operating-system independent file I/O primitives. Class EngineWrapper does violate the rule that the driver code lives entirely in the EXE, but it doesn't violate the rule that the DLL is operating-system independent.
 ## Struct EngineWrapper looks a lot like it Has Methods
 In our driver code, you'll see a lot of code that looks like it calls methods in struct EngineWapper. Here's a typical excerpt from the driver:
 ```cpp
 wrapper.play_recv_incoming(&wrapper, chid, len, data);
 ```
 It looks like a method invocation, but struct EngineWrapper is supposed to be pure C. There are no 'methods' in pure C. What is going on here?
 The trick is that it's not actually a C++ method invocation at all. It's a pure C function call, using a pure C function pointer. Look at this excerpt from struct EngineWrapper:
 ```cpp
 struct EngineWrapper {
 …
 void (*play_recv_incoming)
 (EngineWrapper *w, uint32_t chid, uint32_t len, const char *data);
 ```
 Putting a pointer to a C function into struct EngineWrapper gives the illusion of a "method" that you can call. It's an illusion that we deliberately encourage. We want you to be able to pretend that EngineWrapper is a class, full of methods. That makes working with EngineWrapper more intuitive. EngineWrapper contains a long list of function pointers.
 However, there is one important way that C function calls differ from true C++ methods: when you call a C++ method, there is an implicit parameter called *this*. In other words, the address of the object automatically gets passed into the method. In pure C, that's not the case. Because of this, all the pseudo-methods in struct EngineWrapper require you to manually pass the wrapper itself. As you can see in both of the snippets above, the first parameter to play_recv_incoming is a pointer to the EngineWrapper.
 ## Separation of the C++ Code
 In the Luprex codebase, the driven portion lives in the directory "cpp". The command-line driver lives in the directory "drv". We intentionally keep them separate. The Unreal Engine driver lives in a totally separate git repository, with our Unreal code.
 The driven portion does not include any header files from the driver portion. The driver portion only includes one header file from the driven portion: "enginewrapper.hpp".
 This header file, enginewrapper.hpp, only contains the definition of struct EngineWrapper. This header very deliberately does not include any other header file: not even operating-system files like <stdio>, because we don't want to create any operating-system dependencies. It's just the definition of struct EngineWrapper, and nothing else.
 Therefore, when the driver includes enginewrapper.hpp, it's only getting the definition of struct EngineWrapper, and absolutely nothing else. The driver that is integrated with Unreal Engine also includes "enginewrapper.hpp", and it also does not include anything else from the driven portion.
 ## The Eng-Malloc Heap
 It sure would be nice if the malloc heap were in *exactly* the same state during a replay as during logging. By the "exact same state," I mean that every malloc'ed data structure is at exactly the same memory address during replay as during logging. If that were the case, then any buffer overrun or memory overrun in the logging phase would be duplicated exactly during the replay.
 Unfortunately, that's not going to happen. The malloc heap is used by Unreal Engine, the C++ standard library, and the driver, none of which are designed to be deterministic. Like it or not, the addresses of malloc'ed data structures are not going to be the same during replay as during logging.
 Therefore, we have created a new memory allocator specifically for the use of the driven portion. This memory allocator is called "eng::malloc". It is based on Doug Lea's malloc, which is a very fast and well-trusted implementation of malloc. We have arranged that the eng::malloc heap is always positioned at the exact same memory address. And, since it is used exclusively by the driven portion, the sequence of 'malloc' and 'free' operations should be exactly the same during replay as during logging. Therefore, anything allocated by eng::malloc should be at deterministic addresses.
 If a C++ class derives from eng::opnew, then that C++ class will inherit an *operator new* and an *operator delete*. This will make any 'new' operation on that class go to the eng::malloc heap.
 The STL function std::make_shared allocates both the data structure and the reference counts in the malloc heap. Overriding operator new doesn't change this behavior, unfortunately. Therefore, we must provide eng::make_shared, which allocates the data structure and the reference counts in the eng::malloc heap.
 By default, STL classes like *std::map*, *std::vector*, and so forth use the malloc heap to allocate their data. However, using template parameters, it is possible to force these classes to use alternate heaps. We have done exactly this. We have created thin wrappers around these classes, called *eng::map*, *eng::vector* and so forth that allocate their data in the eng::malloc heap. When coding for the driven portion, please use these classes instead of the std ones. These wrappers live in the header files "wrap-map.hpp", "wrap-set.hpp", and so forth.
 Some STL classes, like *std::pair* and *std::string_view* are so simple that they don't ever allocate anything on any heap. We have not wrapped these classes. But that means if you say "new std::pair", it will end up in the malloc heap. Don't do that.
 If you happen to run into an STL class or a third-party library that can't be configured to use the eng::malloc heap, that's not a disaster. It just means that *one* data structure won't be at a predictable address. As long as 99% of the data structures in the driven portion are at predictable addresses, it's okay if the occasional one isn't. Crashes are still very likely to be deterministic. Be careful, though, if you use third-party libraries. They may not be deterministic.
 Driver code must not put anything into the eng::malloc heap, because that would inject nondeterminism into the eng::malloc heap. The driver should use plain old malloc. As long as the driver obeys the rules and doesn't include any header files other than "enginewrapper.hpp," then it won't have a declaration of eng::malloc, which helps keep things properly separate.
 The eng::malloc heap maintains a running hash of all the addresses returned by eng::malloc. This hash value can be fetched using eng::memhash. As long as two eng::malloc heaps are in the exact same state, this hash should be exactly the same. The record and replay code uses eng::memhash as a check to verify the determinism of a replay.
 On Windows, none of this has been implemented yet. It is all stubbed out: on Windows, eng::malloc just calls regular malloc. Therefore, on Windows, determinism does not extend to the addresses of data structures. The driven portion should still run the same code in the same order, but memory overruns might not have predictable effects.
 ## Maintaining Determinism in the Driven Portion
 Computers are naturally deterministic, but there are a few common sources of randomness that break the determinism. These must be avoided inside the driven portion:
 *No genuinely random numbers*. It is legal to use pseudorandom numbers, on the condition that the pseudorandom state is privately owned by the driven portion and never touched, by, say, Unreal. The pseudorandom state must be initialized from a seed that is the same during replay as during the original logging.
 *No iterating over unordered maps*. Unordered maps produce their elements in what is effectively a random order. It's not technically random, but it is usually determined by the exact addresses of the data structures, which may vary between runs. There is an exception to the rule: if you iterate over an unordered map, and then immediately sort the results into a predictable order, then it's OK. In this case, you have to very carefully sandbox the randomness, to ensure that the randomness doesn't influence anything else.
 *No use of genuine real-time clocks*. The amount of time it takes to execute the exact same piece of code can vary randomly. Of course, this is already prohibited based on the prohibition on I/O in the driven portion, but it's worth mentioning separately. There are two exceptions. First, the driver occasionally passes the current time into the driven portion as an event. This is a real-time clock, but it's one that will have the exact same value during replay as during the original logging. Using that clock is fine anywhere. The other exception is for performance profiling. You can measure the amount of time it takes to execute a subroutine, as long as the "random" value (the amount of time) is printed and then immediately forgotten. Properly sandboxing the randomness can be tricky.
 *No use of threads.* When two threads execute at the same time, then the order in which various operations occur is effectively random. The rule against using multithreading is probably the most problematic part of this whole determinism thing: it really is unfortunate from a performance perspective. However, you can bend this rule: you can allow threads if you can somehow sandbox the threads so that they live in their own little worlds, apart from the "deterministic portion," then it can be allowed. However, sandboxing is hard.
 *You don't have to worry when writing Lua code.* It would seem like all of these things are potentially an issue when writing lua code, but we've carefully sandboxed lua from doing anything truly random. For example, the random number generators exposed to Lua are actually seeded pseudorandom generators where the seed is privately owned by the driven portion. Lua tables seem like a kind of unordered map, so you would think that maybe you would have to worry. But we've patched the lua runtime so that table iteration is actually ordered. The order isn't anything straightforward, but it is deterministic. Also, note that the rule against using threads doesn't apply to lua threads, because lua threads aren't really threads at all: they don't run concurrently.
 I feel like I've forgotten a few other sources of randomness. I'll add to this list as I think of things.
 ## On Sandboxing Randomness in the Driven Portion
 The rule against using true randomness in the driven portion has exceptions if you can "sandbox" the randomness. To successfully sandbox randomness, you must do two things. First, you must contain the random-like values to an "infected" region in memory. Second, the program must eventually reach a point where it is done using the random-like values, and it must wipe the "infected area," so that the randomness cannot affect the remainder of the program. Contain the randomness, then get rid of it. That is easier said than done.
 Here's an example you can think about: profiling some expensive code.
 ```cpp
 double t1 = get_current_real_time();
 // Expensive Code Goes Here
 double t2 = get_current_real_time();
 util::dprint("Elapsed time:", t2 - t1);
 t1 = t2 = 0;
 ```
 That uses real-time clocks, which is normally against the rules. But it would be pretty hard to do performance profiling without real-time clocks. So we try to contain the randomness, and then get rid of it.
 By the way, util::dprint is our debugging print routine that sends its output to someplace visible. For example, if you're running inside Unreal, it goes to the unreal console.
 In the first few lines of this example, we fetch the time and store it in the local variables t1 and t2. Now the "infected region" consists of variables t1 and t2, but the randomness hasn't infected anywhere else in memory. So it would appear that we've met the first criterion: we've contained the randomness to a known region of memory. Then, in the last line of code, we set both t1 and t2 to zero. So it would appear that we've met the second criterion: after using the random-like values, we have wiped the randomness out of memory so that it can't affect the remainder of the program.
 However, it's not quite right. The print routine util::dprint allocates a temporary string for the message, in this case, "Elapsed time: 3075.287". The length of the string depends on the actual amount of time elapsed. So therefore, the amount of time elapsed affects how many bytes get allocated to hold the string. So therefore, the amount of time affects the layout of the heap.
 In our old implementation of util::dprint, the string was allocated in the eng::malloc heap. Therefore, in the old implementation, the randomness ends up infecting the layout of the eng::malloc heap. When we realized this, we changed the implementation of util::dprint to use the malloc heap. The malloc heap is already known to be infected by randomness, because Unreal uses the malloc heap. If randomness escapes into the malloc heap, then that's not a problem, because we always treat the malloc heap as an "infected area." Now "util::dprint" contains an explicit guarantee that any randomness you pass to it will remain sandboxed. So now that we've fixed util::dprint, the code above is correct.
 ## Passing Binary Blobs Back and Forth
 Sometimes, the driver sends binary encoded data into the driven portion, and the driven portion sometimes passes binary encoded data to the driver (by leaving binary data in a buffer and letting the driver poll it.) In order for this to be possible, the binary data has to be in a format that is known to both driver and driven portion.
 To help the two different codebases share binary data, we have created a C++ library "base-buffer.hpp". This library defines standardized ways to store ints, floats, doubles, strings, and so forth into a binary blob. It also has ways of storing dynamically typed data into a binary blob. It ensures that both the driver and driven portion use the same endianness, that they both store string lengths in the same ways, and that they both use the same enum for dynamic types. In general, it ensures that the binary blobs created by one can be parsed by the other.
 The base-buffer library is not considered part of either the driver or the driven portion. Instead, it's considered a "third party library" that is included by both. As a third-party library, it goes in the directory for third-party libraries, "ext." It is a pure header-only C++ library, meaning that there's no need for an associated cpp file. It's a single source file. It includes some STL header files, but no other includes, to avoid creating dependencies. It's operating-system independent code.
--- a/Engine_html_f481ae616a8b35e1.png
+++ b/Engine_html_f481ae616a8b35e1.png