Improved Docs, AnimationStepApplyMesh+Materials, some other minor tweaks

Docs/About Determinism.md

@@ -0,0 +1,58 @@
+### About Determinism
+
+The driven portion of the Luprex engine is deterministic. This document explains what that means and why it matters. For the specific rules you must follow to maintain determinism, see "The Event-Driven Structure of the Engine."
+
+## Two Degrees of Determinism
+
+There are two distinct degrees of determinism in the engine, each serving a different purpose.
+
+**Value-level determinism** is the property that the server-synchronous and client-synchronous models stay in the same logical state. These two models run on different machines and receive the same command acknowledgements in the same order. Value-level determinism guarantees that they end up containing the same Lua tables with the same keys and values. If you go into both models and print things out, everything looks the same. However, a Lua table in one model is not necessarily at the same memory address as the corresponding table in the other, because they are running on different machines with different memory layouts. The two models are *equal* in the Lisp sense of `equal`, but not in the sense of `eq`.
+
+**Bit-exact determinism** is the property that a recorded event log, when replayed into a fresh DrivenEngine, reproduces the original execution right down to the memory level: every data structure is at the same address, every byte of memory is the same. This is the `eq` level of equivalence.
+
+The engine currently aims for both, but they serve different purposes and impose different costs.
+
+## Value-Level Determinism: Synchronous Model Pairing
+
+Value-level determinism is what makes the multiplayer architecture work. It is non-negotiable.
+
+Luprex uses four types of world models to handle multiplayer networking (see "Predictive Reexecution" for the full explanation). Two of these models are critical to understand here:
+
+- The **server-synchronous** model runs on the server.
+- The **client-synchronous** model runs on the client.
+
+These two models receive the same command acknowledgements in the same order. Because the driven portion is deterministic at the value level, the two models always end up in the same logical state: the same Lua tables, the same values, the same game state. They never need to exchange full state to stay in sync.
+
+The two models are running on different machines, so naturally they have different memory layouts and different pointer addresses. That's fine. All that matters is that the values match. This is why value-level determinism is sufficient for synchronous model pairing.
+
+The constraints that maintain value-level determinism are:
+
+- **Deterministic Lua table iteration.** We patch the Lua runtime so that iterating over a table always produces keys in the same order, regardless of memory layout. Without this, two engines processing the same commands could iterate tables in different orders and produce different results.
+- **No iterating over unordered maps.** Unordered maps produce elements in an order that depends on memory addresses. Since addresses differ between the two models, iteration order would differ, breaking value-level determinism. (An exception: iterating an unordered map and then immediately sorting the results into a predictable order is allowed, because the randomness is sandboxed.)
+- **No genuinely random numbers.** Pseudorandom numbers are fine as long as the state is privately owned by the driven portion and seeded deterministically.
+- **Controlled use of real-time clocks.** The driven portion (the Luprex DLL) cannot call system functions to obtain the current time, because the result would differ between runs and between machines. However, the driver can feed the current time into the driven portion as an event. Since events are the same during paired execution and during replay, the time value is deterministic from the driven portion's perspective.
+
+## Bit-Exact Determinism: Replay Debugging
+
+Bit-exact determinism enables replay debugging. It is valuable but expensive, and its cost-benefit tradeoff is an open question.
+
+As the server runs, the driver can write a log of every event it feeds into the driven portion. Later, a new DrivenEngine can be created and fed those same events from the log file. The goal of bit-exact determinism is that during this replay, the DrivenEngine does the *exact* same thing it did during the live run, right down to every data structure being at the same memory address.
+
+Why does this matter? If the server crashed during the live run, the replay will crash in exactly the same way. You can run the replay inside a debugger, single-step right up to the crash, and examine the exact same pointers and memory layout that existed during the original crash.
+
+Value-level determinism alone is not sufficient for this. If the replay produces the same logical state but at different memory addresses, then pointer-related bugs (buffer overruns, use-after-free, etc.) might not reproduce. Bit-exact determinism ensures they do.
+
+The additional constraints that maintain bit-exact determinism, beyond those needed for value-level determinism, are:
+
+- **The eng::malloc heap.** A custom memory allocator positioned at a fixed address, used exclusively by the driven portion. Because the driven portion is deterministic, the sequence of allocations and frees is identical between the live run and the replay, so every data structure ends up at the same address. See "The Event-Driven Structure of the Engine" for details.
+- **No threads in the driven portion.** Thread scheduling is nondeterministic at the OS level. Even if two threaded programs produce the same final values, the interleaving of operations differs between runs, which would cause memory allocations to occur in different orders and at different addresses.
+
+Note that the constraints for value-level determinism (deterministic table iteration, no unordered maps, etc.) also contribute to bit-exact determinism. But they are *required* for value-level determinism regardless. The eng::malloc heap and the no-threads rule are the additional cost imposed specifically by the bit-exact guarantee.
+
+## The Practical Distinction
+
+If the engine ever relaxed its determinism requirements, the value-level constraints would remain because they are essential to the multiplayer architecture. The bit-exact constraints (eng::malloc, no threads) could theoretically be dropped if replay debugging were deemed not worth the cost. Dropping the no-threads rule in particular would be a significant performance benefit.
+
+## Lua Scripters Don't Need to Worry
+
+The Lua environment is carefully sandboxed to be deterministic at both levels without any effort from the scripter. Lua's random number generators are seeded pseudorandom generators owned by the driven portion. Table iteration is patched to be deterministic. Lua "threads" (coroutines) are not real OS threads and don't run concurrently. The scripter writes ordinary Lua code and gets determinism for free.

Docs/Animation Queues and Tangible Actors.md

@@ -30,7 +30,7 @@ This document contains several things:
 Suppose you want a character to jump up and down in-place. Typically, you would use the lua function *tangible.animate* to achieve this:
 
 ```lua
-tangible.animate(actor, nil, {action='jump', height=3.0}))"
+tangible.animate{tan=actor, anim={action='jump', height=3.0}}
 ```
 
 This function adds an *animation step* to the character's *animation queue*. An animation queue is just a sequence of animation steps. Its length is set to a fixed value, perhaps 10 steps. So if you add a step 11, then the oldest step is automatically discarded.
@@ -277,7 +277,7 @@ The routine FinishAnimation takes an *lxAnimationStep* as a parameter. This is s
 *FinishAnimation* also takes three boolean parameters: Auto Update XYZ, Auto Update Plane, and Auto Update Facing. These require some explanation. Suppose that the lua programmer pushes an animation step that looks like this:
 
 ```lua
-tangible.animate(actor, nil, {action="emote", animation="dance", xyz={1,2,3}})
+tangible.animate{tan=actor, anim={action="emote", animation="dance", xyz={1,2,3}}}
 ```
 
 Since "play an emote" isn't a travel command like "moveto" or "warpto", the lua scripter shouldn't have specified a destination XYZ, but he did so anyway, and now we have to recover from the mistake. Specifically, the tangible actor really does need to move to the new XYZ coordinate, otherwise bad things happen.
@@ -293,9 +293,9 @@ Suppose that in your game, players can teleport from one place to another by cas
 Here is the tricky scenario: player T, the teleporter, is standing in front of player V, the viewer. Player V can see player T. Then, player T casts the teleport spell, and executes this lua code:
 
 ```lua
-tangible.animate(actor, nil, {action="sparkle", particles=999})
+tangible.animate{tan=actor, anim={action="sparkle", particles=999}}
 
-tangible.animate(actor, nil, {action="warpto", xyz={1000000, 0, 0}})
+tangible.animate{tan=actor, anim={action="warpto", xyz={1000000, 0, 0}}}
 ```
 
 Luprex has no idea how long these animations take to execute. As far as Luprex is concerned, they are instantaneous. As soon as player T executes that code, Luprex thinks that player T has moved to {1000000, 0, 0}, a million miles away from player V.
@@ -313,9 +313,9 @@ In order for Unreal to actually discard a Tangible Actor, two conditions must be
 Now, let's consider a small variation of the lua commands:
 
 ```lua
-tangible.animate(actor, nil, {action="sparkle", particles=999})
+tangible.animate{tan=actor, anim={action="sparkle", particles=999}}
 
-tangible.animate(actor, nil, {action="warpto", plane="somewhere-else"})
+tangible.animate{tan=actor, anim={action="warpto", plane="somewhere-else"}}
 ```
 
 Here, instead of changing the XYZ of the player, the warpto changes the plane of the player. Unreal engine doesn't have the concept of "planes." What is the blueprint supposed to do? The answer is that we've added the concept of "plane" to Unreal at the C++ level. We have provided an API function *SetTangiblePlane*, that takes a string. When the tangible actor executes the "warpto" animation, that's all it has to do. The C++ code will make sure to only render tangibles that are on the same plane as the player.
@@ -382,17 +382,17 @@ In TangibleStaticMesh, animations are played one at a time. But you can imagine
 For example, imagine a "fireworks launcher" object that launches rockets, and which can launch many rockets simultaneously. The rockets are owned by the launcher: it's actually just one Tangible Actor, the launcher and its rockets. Now imagine that the fireworks launcher is controlled using animate commands like this:
 
 ```lua
-tangible.animate(actor, nil, {action="launch", type="burst", color="blue"})
+tangible.animate{tan=actor, anim={action="launch", type="burst", color="blue"}}
 
-tangible.animate(actor, nil, {action="launch", type="burst", color="green"})
+tangible.animate{tan=actor, anim={action="launch", type="burst", color="green"}}
 
-tangible.animate(actor, nil, {action="wait"})
+tangible.animate{tan=actor, anim={action="wait"}}
 
-tangible.animate(actor, nil, {action="launch", type="sparkle", color="blue"})
+tangible.animate{tan=actor, anim={action="launch", type="sparkle", color="blue"}}
 
-tangible.animate(actor, nil, {action="launch", type="sparkle", color="green"})
+tangible.animate{tan=actor, anim={action="launch", type="sparkle", color="green"}}
 
-tangible.animate(actor, nil, {action="wait"})
+tangible.animate{tan=actor, anim={action="wait"}}
 ```
 
 Our rocket launcher has the following unusual behavior: when it sees a sequence of animation steps with action="launch", it launches them all at the same time. So in response to the *tangible.animate* commands above, our launcher would launch two "burst" rockets simultaneously. Then, it would wait for both of those rockets to finish, however long that takes. Then, it launches two "sparkle" rockets simultaneously. Then it waits for the two sparkles to finish.

Docs/Displaying Widget Blueprints.md

@@ -32,7 +32,7 @@ To call a Lua function, use the blueprint node 'Invoke Lua Function' or 'Probe L
 
 By default, luprex puts a crosshair on the screen. When you aim the crosshair at an object, you are said to be "looking at" that object. Sometimes, looking at an object causes a "look-at widget" to pop up. Here's a typical example of a look-at widget, which displays the hotkeys for that object:
 
-![](Displaying%20Widget%20Blueprints_html_fc056206a4099c85.jpg)
+![](Images/Displaying%20Widget%20Blueprints_html_fc056206a4099c85.jpg)
 
 Luprex contains all the following mechanisms to support the idea of look-at widgets:
 
@@ -213,7 +213,7 @@ Currently, there is no way to pop up any widget other than a look-at widget. The
 
 The configuration for a widget blueprint could be complicated. Consider this menu:
 
-![](Displaying%20Widget%20Blueprints_html_24d3706fb604c218.jpg)
+![](Images/Displaying%20Widget%20Blueprints_html_24d3706fb604c218.jpg)
 
 This menu could be a single widget blueprint, *MenuWidgetBP*. In its *BeginPlay* event, it could probe *gui.getmenu(actor, place),* which might return { "Resume", nil, "Save Game", "Load Game", nil, "Session", "Difficulty", "Options", nil, "Main Menu", "Quit Game" }. This design would make it possible for one widget blueprint to represent all the menus in the game.
 
@@ -221,7 +221,7 @@ We will provide common blueprints like *HotkeyWidgetBP* and *MenuWidgetBP* in ou
 
 Here's an interesting window from a role-playing game:
 
-![](Displaying%20Widget%20Blueprints_html_ae2696951c4fbe2a.png)
+![](Images/Displaying%20Widget%20Blueprints_html_ae2696951c4fbe2a.png)
 
 Since there are hundreds of spells in this game, it would not be good to have a separate widget blueprint for each one. Instead, it's probably best to have a *SpellWidgetBP* that can render any spell. This might probe *spell.getinfo(actor, place, spell),* which might return a vector like this:
 

Docs/Eris and Snapshot-Rollback.md

@@ -0,0 +1,45 @@
+### Eris and Snapshot/Rollback
+
+## What Eris Is
+
+Eris is a third-party Lua serialization library. It can serialize almost anything in a Lua VM — tables, closures, yielded coroutines — into a binary string, and deserialize it later, even in a different VM. It is essentially a rewrite of Pluto (a Lua 5.1 serialization library) targeting Lua 5.2. Eris ships as a custom distribution of Lua: the Lua runtime in our `luprex/ext/eris-master/` directory *is* Eris, which includes Lua plus the serialization code.
+
+The two core operations are:
+
+- `eris_dump` — serialize a Lua value (and everything it references) into a binary stream.
+- `eris_undump` — deserialize a binary stream back into a Lua value.
+
+## The Permanent Object Table
+
+C functions (like those registered via `LuaDefine`) cannot be serialized — they are function pointers that may differ between runs. Eris handles this through a "permanent object table": a lookup table that maps C functions to string keys. During serialization, when Eris encounters a C function, it writes the string key instead. During deserialization, it looks up the string key and substitutes the C function.
+
+The `LuaSnap` class maintains both a `persist` table (value-to-key, for serialization) and an `unpersist` table (key-to-value, for deserialization) in the Lua registry. Whenever the source module registers a C function into the Lua environment, it also registers it in these tables.
+
+## How Snapshot/Rollback Works
+
+The class `LuaSnap` (in luasnap.hpp/cpp) wraps Eris to provide snapshot and rollback for the Lua interpreter. It owns a `lua_State` and exposes two operations:
+
+- `serialize(StreamBuffer*)` — uses `eris_dump` to serialize the contents of the Lua registry into a binary buffer. Certain registry entries that don't belong in the snapshot (the main thread, the persist/unpersist tables, the world pointer, diff transmission maps) are excluded.
+- `deserialize(StreamBuffer*)` — uses `eris_undump` to restore the Lua registry from a previously serialized buffer.
+
+## Role in Predictive Reexecution
+
+Snapshot/rollback is used to implement the asynchronous model on the client (see "Predictive Reexecution"). There is only one client model, but it alternates between two forms:
+
+- **Synchronous form** — the confirmed state, in lockstep with the server-synchronous model.
+- **Asynchronous form** — the synchronous state plus unacknowledged user commands applied on top. This is what the player sees.
+
+To get from synchronous to asynchronous form, apply any unacknowledged user commands. To get from asynchronous back to synchronous form, roll back to the snapshotted state. The model switches between these forms on an as-needed basis. The typical cycle is:
+
+1. The model is in synchronous form. Serialize it (take a snapshot).
+2. Apply unacknowledged user commands to produce the asynchronous form.
+3. Render from the asynchronous state.
+4. When new acknowledgements or difference transmissions arrive, deserialize (roll back) to restore the synchronous form.
+5. Apply the updates to the synchronous state.
+6. Repeat.
+
+Because this uses Eris serialization rather than raw memory copying, snapshot/rollback only requires value-level determinism, not bit-exact determinism. The deserialized state has the same values but not necessarily the same memory addresses.
+
+## An Alternative Design
+
+A comment in luasnap.cpp describes an alternative approach that was prototyped and found to work: instead of using Eris serialization, use a custom memory allocator that tracks all memory blocks used by Lua, then snapshot by copying those blocks and rollback by restoring them. This would likely be faster than Eris serialization/deserialization, but would require maintaining additional code. Since Eris serialize/deserialize is also needed for save-game functionality, the current design avoids duplicating effort.

Docs/Global Variables.md

@@ -63,7 +63,7 @@ When you're typing commands into a lua interpreter interactively, you inevitably
 ```lua
 a = tangible.actor() — get a pointer to my actor
 
-tangible.animate(a, nil, {action="warpto", xyz={100,100,100}})
+tangible.animate{tan=a, anim={action="warpto", xyz={100,100,100}}}
 ```
 
 You just created a temporary variable, *a*. In a normal lua interactive shell, these temporary variables go into the global environment table.

Docs/Predictive Reexecution.md

@@ -0,0 +1,89 @@
+### Predictive Reexecution
+
+This document explains the general principles of predictive reexecution, the mechanism by which Luprex keeps all connected clients in sync with the server while hiding network latency from the player.
+
+## The Problem: Network Latency
+
+In a multiplayer game, the server is the authority on the state of the world. When a player clicks a button or types a command, the command must travel to the server, the server must process it, and the result must travel back. This round trip takes time. If we made the player wait for the round trip before showing anything on screen, the game would feel sluggish and unresponsive.
+
+Predictive reexecution solves this problem. The client predicts the outcome of the player's action and shows it immediately, without waiting for the server. Later, when the server's authoritative result arrives, the client corrects itself if the prediction was wrong.
+
+## The Four World Models
+
+The system uses four types of world models. Each is an instance of class World, with its own Lua interpreter.
+
+**Master** (one, on the server). This is the authoritative state of the entire game. When a command arrives from any client, the master executes it immediately. The master contains all tangibles, all globals, and all source code. It is the single source of truth.
+
+**Server-synchronous** (one per connected client, on the server). This is the server's record of what a particular client knows. It does not execute commands when they first arrive at the server. Instead, it executes them later, when the server issues an acknowledgement. The server needs this model in order to compute difference transmissions: it compares the master to the server-synchronous model to figure out what updates to send.
+
+**Client-synchronous** (one, on the client). This is the client's confirmed view of the world. It receives the exact same command acknowledgements as its corresponding server-synchronous model, and because the driven portion of the engine is fully deterministic, the two remain in perfect sync at all times.
+
+**Asynchronous** (one, on the client). This is the client-synchronous model with predictions applied on top. It is what the player actually sees on screen. It is created by taking a snapshot of the client-synchronous model and then applying predicted actions to the snapshot. It is transient: when corrections arrive, the client rolls back to the client-synchronous state and reapplies predictions as needed.
+
+## What Happens When the Player Issues a Command
+
+Suppose the player clicks a button that invokes a Lua function. Here is the sequence of events:
+
+1. The client sends the command to the server.
+
+2. The client immediately applies the command to the asynchronous model, so the player sees an instant response. This is the prediction.
+
+3. The server receives the command and executes it on the master model.
+
+4. The server issues an acknowledgement for the command.
+
+5. The server-synchronous model for that client executes the command upon receiving the acknowledgement.
+
+6. The acknowledgement travels to the client. The client-synchronous model executes the command upon receiving the acknowledgement.
+
+7. The client can now roll back the asynchronous model to the (updated) client-synchronous state and reapply any predictions that haven't been acknowledged yet.
+
+Because the server-synchronous and client-synchronous models always receive the same acknowledgements in the same order, and because the driven portion is deterministic, they always end up in the exact same state. They never need to exchange full state to stay in sync.
+
+## Two Update Channels
+
+The synchronous models receive updates through two channels:
+
+**Command acknowledgements.** These carry the client's own actions. When an acknowledgement arrives, both the server-synchronous and client-synchronous models execute the command. This is how a client's own actions propagate into the synchronous models.
+
+**Difference transmission.** This carries everything else: other players' actions, server-side events, tangibles entering or leaving the visibility radius, and any other changes to the master that aren't caused by this client's commands. The server compares the master model to the server-synchronous model and sends corrections. The client applies those corrections to the client-synchronous model.
+
+Between these two channels, the synchronous models are kept up to date with the master.
+
+## Snapshot and Rollback
+
+The asynchronous model is not a separate, persistent world model. It is implemented using the snapshot-and-rollback mechanism built into class World.
+
+The client-synchronous model is the persistent model on the client. When the client needs to show the player a predicted state, it takes a snapshot of the client-synchronous model's memory, applies predictions to it, and renders from the resulting state. When it is time to incorporate new acknowledgements or difference transmissions, it rolls back to the snapshot, applies the updates to the client-synchronous model, and then reapplies any predictions that are still pending.
+
+The snapshot mechanism works at the memory level: it saves and restores the raw memory used by the Lua interpreter, without needing to understand the contents. This makes it fast.
+
+## NoPredict and Blocking Operations
+
+Not all operations can be predicted. Consider an HTTP request: the client cannot predict what the server will return. For operations like these, the system uses a mechanism called NoPredict.
+
+When a Lua thread attempts a blocking operation in a nonauthoritative (predicted) context, the thread is killed rather than allowed to proceed. This prevents the asynchronous model from going down a code path whose outcome cannot be predicted. The thread will execute for real when the command reaches the master model, and the results will propagate to the client through the normal acknowledgement and difference transmission channels.
+
+The rules for handling this are described in the document "Correct Implementation of Blocking Operations and NoPredict." In summary:
+
+- In an authoritative context (the master model), blocking operations proceed normally.
+- In a nonauthoritative context (a predicted model), blocking operations trigger a NoPredict, which kills the thread via lua_yield.
+- In a probe context (not yieldable), blocking operations throw an error, because probes are not allowed to change the world state at all.
+
+## The Scripter's View
+
+One of the most important consequences of this design is that the Lua scripter does not write any networking code. There are no "send message" or "receive message" calls. The scripter writes Lua code as if the game were running in a single Lua state with coroutines, which is normal for Lua anyway.
+
+The networking is handled entirely by the engine. Command acknowledgements and difference transmission keep all the models in sync automatically. The scripter occasionally needs to provide hints to the system, such as:
+
+- Using `permit_invoke` to mark which functions may be called by blueprints, for security.
+- Being aware that blocking operations will trigger NoPredict in predicted contexts.
+- Using `global.set` and `global.get` for data that needs to be broadcast to all clients.
+
+But the core game logic is written without any awareness of networking. This is a clean separation that makes scripting much simpler.
+
+## Why Determinism Matters
+
+The entire scheme depends on the determinism of the driven portion of the engine. If the server-synchronous and client-synchronous models processed the same acknowledgements but arrived at different states, the system would break down.
+
+This is why the engine goes to great lengths to ensure determinism: patching Lua for deterministic table iteration and table length, using the eng::malloc heap for reproducible memory addresses, avoiding threads, avoiding real-time clocks, and avoiding unordered maps. These constraints are described in detail in the document "The Event-Driven Structure of the Engine."

Docs/The Event-Driven Structure of the Engine.md

@@ -6,7 +6,7 @@ Most of the Luprex engine is 'event-driven'. The event-driven design makes certa
 
 To be clear about what I mean by "event driven," think of a traditional Finite State Machine (FSM). Here's an example FSM, for a coin-operated turnstile:
 
-![](The%20Event-Driven%20Structure%20of%20the%20Engine_html_f481ae616a8b35e1.png)
+![](Images/The%20Event-Driven%20Structure%20of%20the%20Engine_html_f481ae616a8b35e1.png)
 
 So this example FSM has two *events* that it can process: a person can insert a coin, and a person can push through the turnstile.
 

Docs/Things to document.md

@@ -0,0 +1,5 @@
+### Things to Document
+
+1. **The Lua-facing API calling convention.** Lua-facing C++ functions use either positional parameters or a `{keyword=value}` single-table pattern, but never a mix of the two. The codebase is still being migrated toward this convention — some older functions don't follow it yet. Need a doc (or at least a section) explaining the convention and when each style is used.
+
+5. **The Unreal-side integration layer.** How the Unreal driver ticks, fetches animation queues, manages tangible actors — the C++ code in `Source/Integration/`. The docs cover the driver/driven separation conceptually and the blueprint side, but not this middle layer.

Docs/Tokens - A New Lua Type.md

@@ -0,0 +1,71 @@
+### A New Lua Type: Tokens
+
+Tokens are a custom Lua data type built on top of Lua's lightuserdata. They are mainly intended for use as sentinels and special reserved values.
+
+## Motivation
+
+Tokens were invented when we were developing a JSON-to-LUA converter.  Such a converter is mostly straightforward: json tables and lua tables are very similar.  However, we did encounter a stumbling block.  Consider this JSON:
+
+```json
+{ "foo": null }
+```
+
+In Lua, setting a table key to nil deletes the key. There is no way to represent "foo is present with value null" in a Lua table. You might try `{foo = 0}` or `{foo = "null"}`, but both are lossy: you can no longer distinguish JSON null from the number 0 or the string "null". Any sentinel value drawn from an existing Lua type collides with legitimate values of that type.
+
+The solution is to use lightuserdata. A lightuserdata is a distinct Lua type — it cannot be confused with a string, number, boolean, or nil, and unlike nil, it can be stored in a table. The Luprex engine does not use lightuserdata for any other purpose, so all lightuserdata values are available for use as tokens.
+
+## What a Token Is
+
+A token is a short string encoded as a base36 number and stored in the 8-byte lightuserdata value. The lightuserdata is not actually a pointer to anything — it holds the base36-encoded integer directly. Tokens may only contain the characters a-z and 0-9. Since 36^12 fits in 64 bits but 36^13 does not, the maximum token length is 12 characters. That is sufficient for most natural identifiers.
+
+Since lightuserdata is not used for anything else, it is safe to assume that any lightuserdata in our engine represents a token.
+
+## The C++ Side: struct LuaToken
+
+On the C++ side, tokens are represented by `struct LuaToken` (in luastack.hpp). You can construct one from a string or from the raw integer:
+
+```cpp
+LuaToken("null")       // parsed at compile time via consteval — becomes 0x10FAA9
+LuaToken(0x10FAA9)     // equivalent raw value
+```
+
+The string form is preferred — it is readable, and because the constructor is `consteval`, it compiles down to the same constant as the raw integer. There is zero runtime cost. If the string contains invalid characters (anything outside a-z, 0-9) or is too long, the error is caught at compile time.
+
+There is also a runtime constructor that accepts `std::string_view`, for cases where the token string is not known at compile time.
+
+The LuaStack API provides the usual accessors for tokens:
+
+```cpp
+LS.set(slot, LuaToken("null"))     // store a token in a LuaSlot
+LuaToken t = LS.cktoken(slot)      // extract a token (error if not lightuserdata)
+auto t = LS.trytoken(slot)          // extract a token (returns empty optional on mismatch)
+```
+
+Named token constants can be auto-registered into the Lua environment using the `LuaTokenConstant` macro, which works the same way `LuaDefine` auto-registers functions:
+
+```cpp
+LuaTokenConstant(null, "null", "Represents JSON null")
+```
+
+## Properties
+
+- **Distinct type.** Tokens are lightuserdata, a separate Lua type. They cannot collide with strings, numbers, booleans, tables, or nil.
+- **Storable in tables.** Unlike nil, tokens can be used as both table keys and table values.
+- **No allocation.** Tokens are 8 bytes inline. There is no heap allocation and no string interning.
+- **Fast comparison.** Comparing two tokens is just an integer comparison.
+
+## Limitation: No Token Literals in Lua
+
+Lua's parser has no syntax for token literals. In C++, you can write `LuaToken("null")` and it's clean and compile-time. In Lua, there is no equivalent — you cannot write a token literal the way you write `"hello"` or `42`.
+
+Currently, the way tokens are made available to Lua is that C++ code uses `LuaTokenConstant` to insert specific token values into global tables. Lua scripts can then reference these pre-registered constants by name.
+
+Modifying the Lua parser to add token literal syntax has been considered but is unappealing — it would be a significant and invasive patch. Adding a Lua function like `token("null")` to construct tokens at runtime is also possible and not off the table, but there hasn't been a need for it yet.
+
+## Passing Tokens to Unreal
+
+Tokens can get passed to Unreal in a variety of ways.  For example, in animation step key-value pairs, the value can be a token.  When animation queues are passed to Unreal, tokens are converted to FNames. Since both tokens and FNames are short identifier-like strings with fast comparison, the mapping is natural.
+
+## Usage
+
+Tokens are mainly intended as sentinels and special reserved values. The JSON null example above is the motivating case, but tokens can represent any short reserved constant the engine needs.