diff --git a/CLAUDE.md b/CLAUDE.md index c0d749ef..433c225c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,7 +44,7 @@ Two update channels flow into the synchronous models: 1. **Command acknowledgements** — for the client's own actions, keeping the two synchronous models in lockstep. 2. **Difference transmission** — for everything else (other players' actions, server-side events, tangibles entering/leaving visibility). -See `Docs/Predictive Reexecution.md` for the full explanation. +See `Docs/Predictive-Reexecution.md` for the full explanation. ## Architecture: Lua / Unreal Separation @@ -69,14 +69,14 @@ On the Unreal side, **Tangible Actor blueprints** (TangibleStaticMesh, TangibleS ## Architecture: Lua Environment -- **Patched Lua runtime** — deterministic table iteration, deterministic table length, flag bits on tables, generalized less-than, C++ exceptions instead of longjmp, and more. See `Docs/A Summary of our Lua Patches.md`. -- **LuaStack API** — custom C++ API replacing the standard Lua C API. Uses `LuaDefStack`/`LuaExtStack` with `LuaArg`/`LuaVar`/`LuaRet` slots mapped to stack positions. See `Docs/Our In-House Lua API.md`. +- **Patched Lua runtime** — deterministic table iteration, deterministic table length, flag bits on tables, generalized less-than, C++ exceptions instead of longjmp, and more. See `Docs/A-Summary-of-our-Lua-Patches.md`. +- **LuaStack API** — custom C++ API replacing the standard Lua C API. Uses `LuaDefStack`/`LuaExtStack` with `LuaArg`/`LuaVar`/`LuaRet` slots mapped to stack positions. See `Docs/Our-In-House-Lua-API.md`. - **LuaDefine macro** — declares Lua-callable C++ functions and auto-registers them in a global registry for automatic insertion into the Lua environment. - **eng::malloc heap** — custom deterministic memory allocator for the driven portion, ensuring reproducible addresses during replay. ## Architecture: Determinism -The driven portion must be fully deterministic so that synchronous models stay in lockstep and event replay works. Rules: no true random numbers, no iterating unordered maps, no real-time clocks, no threads (with carefully sandboxed exceptions). See `Docs/The Event-Driven Structure of the Engine.md`. +The driven portion must be fully deterministic so that synchronous models stay in lockstep and event replay works. Rules: no true random numbers, no iterating unordered maps, no real-time clocks, no threads (with carefully sandboxed exceptions). See `Docs/The-Event-Driven-Structure-of-the-Engine.md`. ## Architecture: GUI System @@ -84,21 +84,21 @@ Blueprints call into Lua via two mechanisms: - **Invokes** — change world state, forwarded to server, executed in order per predictive reexecution rules. - **Probes** — read-only, return data to blueprints, run locally on client. -Look-at widgets, hotkeys, and menus are built on top of this. The menu system is implemented entirely in "user space" Lua and blueprint code. See `Docs/Displaying Widget Blueprints.md`. +Look-at widgets, hotkeys, and menus are built on top of this. The menu system is implemented entirely in "user space" Lua and blueprint code. See `Docs/Displaying-Widget-Blueprints.md`. ## Key Documentation -- `Docs/Predictive Reexecution.md` — how the four world models stay in sync -- `Docs/The Event-Driven Structure of the Engine.md` — driver/driven separation, determinism, replay -- `Docs/Multipass Difference Transmission.md` — the algorithm for syncing Lua table graphs -- `Docs/Animation Queues and Tangible Actors.md` — how blueprints interpret animation queues -- `Docs/Our In-House Lua API.md` — the LuaStack API (LuaDefStack, LuaExtStack) -- `Docs/A Summary of our Lua Patches.md` — all modifications to the Lua runtime -- `Docs/Major Data Structures.md` — World, tangibles, threads, classes, source database -- `Docs/Displaying Widget Blueprints.md` — GUI system (invokes, probes, look-at widgets, menus) -- `Docs/Global Variables.md` — different types of global data and their transmission rules -- `Docs/Correct Implementation of Blocking Operations and NoPredict.md` — how to handle blocking ops -- `Docs/Difference Transmission with Threads.md` — why concurrent diff transmission is hard +- `Docs/Predictive-Reexecution.md` — how the four world models stay in sync +- `Docs/The-Event-Driven-Structure-of-the-Engine.md` — driver/driven separation, determinism, replay +- `Docs/Multipass-Difference-Transmission.md` — the algorithm for syncing Lua table graphs +- `Docs/Animation-Queues-and-Tangible-Actors.md` — how blueprints interpret animation queues +- `Docs/Our-In-House-Lua-API.md` — the LuaStack API (LuaDefStack, LuaExtStack) +- `Docs/A-Summary-of-our-Lua-Patches.md` — all modifications to the Lua runtime +- `Docs/Major-Data-Structures.md` — World, tangibles, threads, classes, source database +- `Docs/Displaying-Widget-Blueprints.md` — GUI system (invokes, probes, look-at widgets, menus) +- `Docs/Global-Variables.md` — different types of global data and their transmission rules +- `Docs/Correct-Implementation-of-Blocking-Operations-and-NoPredict.md` — how to handle blocking ops +- `Docs/Difference-Transmission-with-Threads.md` — why concurrent diff transmission is hard ## Session Startup diff --git a/Docs/A Summary of our Lua Patches.md b/Docs/A-Summary-of-our-Lua-Patches.md similarity index 100% rename from Docs/A Summary of our Lua Patches.md rename to Docs/A-Summary-of-our-Lua-Patches.md diff --git a/Docs/About Determinism.md b/Docs/About-Determinism.md similarity index 100% rename from Docs/About Determinism.md rename to Docs/About-Determinism.md diff --git a/Docs/Animation Queues and Tangible Actors.md b/Docs/Animation-Queues-and-Tangible-Actors.md similarity index 100% rename from Docs/Animation Queues and Tangible Actors.md rename to Docs/Animation-Queues-and-Tangible-Actors.md diff --git a/Docs/Correct Implementation of Blocking Operations and NoPredict.md b/Docs/Correct-Implementation-of-Blocking-Operations-and-NoPredict.md similarity index 100% rename from Docs/Correct Implementation of Blocking Operations and NoPredict.md rename to Docs/Correct-Implementation-of-Blocking-Operations-and-NoPredict.md diff --git a/Docs/Difference Transmission with Threads.md b/Docs/Difference-Transmission-with-Threads.md similarity index 100% rename from Docs/Difference Transmission with Threads.md rename to Docs/Difference-Transmission-with-Threads.md diff --git a/Docs/Displaying Widget Blueprints.md b/Docs/Displaying-Widget-Blueprints.md similarity index 100% rename from Docs/Displaying Widget Blueprints.md rename to Docs/Displaying-Widget-Blueprints.md diff --git a/Docs/Eris and Snapshot-Rollback.md b/Docs/Eris-and-Snapshot-Rollback.md similarity index 100% rename from Docs/Eris and Snapshot-Rollback.md rename to Docs/Eris-and-Snapshot-Rollback.md diff --git a/Docs/Global Variables.md b/Docs/Global-Variables.md similarity index 100% rename from Docs/Global Variables.md rename to Docs/Global-Variables.md diff --git a/Docs/Major Data Structures.md b/Docs/Major-Data-Structures.md similarity index 100% rename from Docs/Major Data Structures.md rename to Docs/Major-Data-Structures.md diff --git a/Docs/Multipass Difference Transmission.md b/Docs/Multipass-Difference-Transmission.md similarity index 100% rename from Docs/Multipass Difference Transmission.md rename to Docs/Multipass-Difference-Transmission.md diff --git a/Docs/Our In-House Lua API.md b/Docs/Our-In-House-Lua-API.md similarity index 100% rename from Docs/Our In-House Lua API.md rename to Docs/Our-In-House-Lua-API.md diff --git a/Docs/Predictive Reexecution.md b/Docs/Predictive-Reexecution.md similarity index 100% rename from Docs/Predictive Reexecution.md rename to Docs/Predictive-Reexecution.md diff --git a/Docs/Print-Statement-Handling.md b/Docs/Print-Statement-Handling.md new file mode 100644 index 00000000..80c57a4d --- /dev/null +++ b/Docs/Print-Statement-Handling.md @@ -0,0 +1,142 @@ +### Handling Print Statements + +This document describes how "print" statements are handled within +Luprex. It documents the full path by which print statements +originate with Lua, and gradually travel to the Unreal virtual +console and debug log. There are two types of print statements: + +- *dprint*, for messages that go to the debug logs. +- *print*, for messages that go to a window in the user's GUI. + +The following sections explain the differences, and how each +of these is implemented. + +## dprint, for messages that go to the debug logs + +The lua code can use 'dprint' to print a message into Unreal's +debugging logs. A 'dprint' goes to the same places +that a debugging message within Unreal goes. That includes: + +- Visual studio's debug message window. +- The Unreal Editor's debug message window. +- Unreal's debug message log file. + +In the lua server, currently, dprints just go to stderr. We're +probably going to enhance that by adding a log file as well. + +The unreal logs are not managed by predictive reexecution. +To put it differently, once a message goes into the unreal logs, +it can't be corrected by difference transmission. Because of +this, if a dprint is reexecuted, you will get multiple possibly +conflicting messages in the unreal logs. + +## Implementation of dprint + +The luprex DLL contains a function util::dprintview, along with +a couple of convenience wrappers like util::dprintf. This +function is used to make a debugging print. It is used in a +number of places throughout Luprex. It can also be called from +lua, via the 'dprint' function. + +The Luprex DLL exports an API: EngineWrapper::hook_dprint. This +function accepts a pointer to a C callback function, which takes +a string as a parameter. The driver is expected to call +hook_dprint early in 'main', to set the dprint callback. The +pointer to the callback function is stored in the global +variable util::dprint_hook. The function util::dprintview +breaks the string into lines, and calls util::dprint_hook +once per line. + +In the Unreal client, the dprint callback is set to a function +that calls UE_LOG, the unreal error logging macro, with a log +category of LogLuprex. So therefore, in the Unreal client, +calling dprint is equivalent to calling UE_LOG. + +In the server, the dprint callback is set to a function that +outputs the string to the console (ie, stdout). + + +The print-callback is the *only* place where the Luprex DLL actually +calls into the driver (via a callback). Normally, the driver is +supposed to call into the Luprex DLL, but not the other way around. +This one exception is necessary to handle the case where the +Luprex DLL is about to crash and wants to print a message before it +does. + +## print, for messages that go to a window in the user's GUI + +The client is expected to have a GUI of some sort +that includes a text console, where messages can be +displayed. The print statement puts a message into this GUI +console. + +Every person who is logged in has their own GUI text console. +Therefore, it is possible to print a message onto any one +of those. When the lua code executes a print statement, +it sends its output to the GUI console of the *actor*. + +The contents of the GUI text console is part of the world model. +Therefore, it can be corrected by difference transmission. +If a print-statement is predicted incorrectly, the user's +GUI text console will temporarily contain the wrong text, but +it will get fixed by difference transmission. + +## Implementation of print + +Inside the Luprex DLL, class PrintBuffer is used to store +the contents of the GUI console. Every logged in player has +a PrintBuffer as part of their character tangible. +Difference transmission is capable of amending the contents +of the PrintBuffer. + +When the luprex thread scheduler starts executing a thread, +it sets up an ostringstream to collect any print statements. +When the thread pauses or ends, the contents of the ostringstream +are copied into the PrintBuffer. The code to create the +stringstream and to copy it into the PrintBuffer are both +in the thread scheduling code in world-core.cpp. + +The stringstream isn't *always* copied into the PrintBuffer. +There are exceptions: for example, in a 'probe', +print statements don't go into the PrintBuffer, they get +rerouted to dprint instead. + +The difference transmitter handles PrintBuffers specially. +It doesn't just fix the contents of the PrintBuffer. It also +leaves an 'authoritative' bit inside the PrintBuffer to +indicate which print statements are confirmed as authoritative, +and conversely, to indicate which ones are still just +predictions. Because of this, the PrintBuffer knows the +difference between a "final" print and a "tentative" print. + +The client and server in lpxclient.cpp and lpxserver.cpp both +contain objects of class PrintChanneler. This class is designed +to monitor a PrintBuffer to see if anything has changed. +Specifically, it is capable of answering the question: does +the PrintBuffer contain any new authoritative print statements? + +Periodically, lpxserver.cpp and lpxclient.cpp will check +their PrintChannelers to see if there are any new authoritative +print statements. If so, they will set a flag called "have_prints" +in the DrivenEngine. Unreal periodically polls this flag using +EngineWrapper::get_have_prints. + +If the flag is set, Unreal then calls +EngineWrapper::play_access(... CHANNEL_PRINTS ...). +This asks the PrintChanneler to fetch all new authoritative prints. +Then, CHANNEL_PRINTS will send an invoke via the standard +predictive reexecution channels to FLUSH_PRINTS - ie, to forget +the print statements that it just channeled. + +When the PrintChanneler returns prints to Unreal, Unreal +passes those prints to the LuprexGameMode blueprint by calling +LuprexGameMode::AddConsoleOutput. The LuprexGameMode is +currently responsible for implementing the GUI text console. +That will probably change at some point. + +The LuprexGameMode maintains an object of class UlxConsoleOutput +which keeps a record of what's in the GUI Console. When +AddConsoleOutput feeds new prints into the LuprexGameMode, +those prints get added to the UlxConsoleOutput. This stores +the contents of the console as one big string. From there, +the string is copied into a text widget. diff --git a/Docs/Roadmap and TODO List.md b/Docs/Roadmap-and-TODO-List.md similarity index 100% rename from Docs/Roadmap and TODO List.md rename to Docs/Roadmap-and-TODO-List.md diff --git a/Docs/The Event-Driven Structure of the Engine.md b/Docs/The-Event-Driven-Structure-of-the-Engine.md similarity index 100% rename from Docs/The Event-Driven Structure of the Engine.md rename to Docs/The-Event-Driven-Structure-of-the-Engine.md diff --git a/Docs/Things to document.md b/Docs/Things-to-document.md similarity index 100% rename from Docs/Things to document.md rename to Docs/Things-to-document.md diff --git a/Docs/Tokens - A New Lua Type.md b/Docs/Tokens-A-New-Lua-Type.md similarity index 100% rename from Docs/Tokens - A New Lua Type.md rename to Docs/Tokens-A-New-Lua-Type.md diff --git a/Integration.code-workspace.tpl.json b/Integration.code-workspace.tpl.json index 1c33f25f..c65294e8 100644 --- a/Integration.code-workspace.tpl.json +++ b/Integration.code-workspace.tpl.json @@ -54,7 +54,8 @@ "**/Binaries": true, "**/DerivedDataCache": true, "**/*.generated.h": true, - "**/*.gen.cpp": true + "**/*.gen.cpp": true, + "**/*.d": true } }, "extensions": { diff --git a/Source/Integration/AnimQueue.h b/Source/Integration/AnimQueue.h index 6323c7b4..3f34a341 100644 --- a/Source/Integration/AnimQueue.h +++ b/Source/Integration/AnimQueue.h @@ -125,11 +125,9 @@ public: UFUNCTION(BlueprintPure, meta = (BlueprintAutocast), Category = "Luprex|Animation Step") static int64 AnimationStepID(const FlxAnimationStep& step) { return step.Hash; } - // Scan an animation step for key-value pairs of the form mat_XXXX={x,y,z}. - // For each match, create a dynamic material instance on the actor's mesh - // components and set the vector parameter XXXX. Materials are restored to - // their base (non-dynamic) state before applying, so parameters from - // previous calls do not persist. + // Using mat_xxxx values from the animation step, update the actor's + // material parameters. Doing this may involve creating or replacing + // dynamic material instances for the actor. // UFUNCTION(BlueprintCallable, Meta = (DefaultToSelf = "actor"), Category = "Luprex|Animation Step") static void AnimationStepApplyMaterials(const FlxAnimationStep& step, AActor* actor); @@ -137,6 +135,8 @@ public: // Look for a mesh=name key-value pair in the animation step. // If found, load the named mesh and apply it to the actor's // mesh component. The actor must have exactly one mesh component. + // If FallbackToBP is true, and mesh=name is not present, looks + // for a bp=name pair instead. // UFUNCTION(BlueprintCallable, Meta = (DefaultToSelf = "actor"), Category = "Luprex|Animation Step") static void AnimationStepApplyMesh(const FlxAnimationStep& step, bool FallbackToBP, AActor* actor); @@ -144,6 +144,8 @@ public: //////////////////////////////////////////////// // +// An animation step that doesn't actually store the step, +// it just contains a pointer to the string. // //////////////////////////////////////////////// @@ -243,7 +245,7 @@ private: FlxStreamBuffer Decoder; public: - // Initialize the FlxAnimationStepDecoder from the FlxAnimationStepView. + // Initialize the FlxAnimationStepDecoder. // FlxAnimationStepDecoder(std::string_view body) : Decoder(body) {} diff --git a/Source/Integration/ConsoleOutput.h b/Source/Integration/ConsoleOutput.h index 8a7cdf61..5926154f 100644 --- a/Source/Integration/ConsoleOutput.h +++ b/Source/Integration/ConsoleOutput.h @@ -4,21 +4,34 @@ #include "ConsoleOutput.generated.h" - ////////////////////////////////////////////////////////////// // // ConsoleOutput // -// This class stores the text that's in the unreal console. -// It stores it as one great big string, which contains -// newlines to denote line breaks. +// When the lua code executes a print statement, the text +// eventually gets passed to the GameMode blueprint: see +// Docs/Print-Statement-Handling.md for more information. +// +// The GameMode blueprint is expected to create a virtual +// console of some sort to display the print statements. +// This class, ConsoleOutput, is a class that the GameMode +// can optionally use to help implement that virtual console. +// +// This class just collects the print statements and keeps +// a record of what text is in the virtual console. The +// text is stored as one big string. // // This class also contains a 'dirty' bit. Each time somebody // appends a line of text to the console, the dirty bit is // automatically set. The bit can be checked using 'IsDirty' -// and cleared using 'ClearDirty'. This makes it so that -// you don't have to update the unreal widget unless the -// text has actually changed. +// and cleared using 'ClearDirty'. Assuming that the GameMode +// is maintaining a text widget of some sort, the GameMode +// can transfer the contents of this buffer into the text +// widget only when the dirty bit is set. +// +// Note that the GameMode is not obligated to use this class. +// If the GameMode wants to use some other framework to +// implement the virtual console, that's perfectly fine. // //////////////////////////////////////////////////////////////