From 1b0d96934f244b431828f19c15fc2650e5ed5df5 Mon Sep 17 00:00:00 2001 From: jyelon Date: Mon, 1 Jun 2026 20:52:48 -0400 Subject: [PATCH] Readme updates --- Plugins/UEWingman/README.md | 274 +++++++++++++++++++++++------------- 1 file changed, 180 insertions(+), 94 deletions(-) diff --git a/Plugins/UEWingman/README.md b/Plugins/UEWingman/README.md index a5cd740e..9fc846d0 100644 --- a/Plugins/UEWingman/README.md +++ b/Plugins/UEWingman/README.md @@ -1,12 +1,14 @@ # UE Wingman -UE Wingman is a tool that allows an AI to control the unreal -editor. When you're using it, it feels like the AI is right -there looking at the editor with you. You'll be able to -watch as it creates graph nodes and wires them together, -you'll see it add components to your blueprints, you'll see -it design widget hierarchies for you, and you'll see it write -shaders for your materials. +ue-wingman is a command-line program that allows an AI +to use the unreal editor. The AI can use this command +to do things like: + +* Create blueprints +* Wire up graph nodes +* Edit parameters in the details panel +* Manipulate actor components +* Build widget hierarchies The tool is not complete, not by a long shot. There are tons of Unreal Editor functions that the AI just can't @@ -14,60 +16,152 @@ access yet. Even so, I think it's useful: it has pretty comprehensive support to allow the AI agent to help create blueprints, widget blueprints, and materials. -## How Does it Work? - -This tool adds a command line interpreter plugin to the Unreal -Editor. You can type commands, and the plugin in the editor -will execute them. +## Example Commands +The AI can print out a graph in a text form, so it can read +it. Here's the dump of a simple event graph that doesn't +have much in it yet: ``` - $ ue-wingman Graph_Dump /Game/Testing/BP_Test,graph:EventGraph +$ ue-wingman.py Graph_Dump /Game/Testing/BP_Test,graph:EventGraph - node K2Node_Event_0: Event BeginPlay - output-pins OutputDelegate +node K2Node_Event_0: Event BeginPlay + output-pins OutputDelegate - node K2Node_Event_2: Event Tick - output-pins OutputDelegate, DeltaSeconds +node K2Node_Event_2: Event Tick + output-pins OutputDelegate, DeltaSeconds ``` -The ue-wingman command has tons of subcommands: Graph_Dump, -GraphNode_Add, GraphPin_Connect, BlueprintComponent_Add, -Widget_Add, and so forth. Using these commands, it's -possible to examine and modify blueprints, widgets, and -materials. +Here, the AI asks: what kinds of graph nodes can I add to the +event graph, that contain the words "Hit Result Under Cursor": -But, of course, these commands aren't really intended for humans. -They're intended for an AI agent. +``` +$ ue-wingman.py GraphNode_SearchTypes /game/testing/bp_test,graph:EventGraph 50 "Hit Result under Cursor" -## Why Choose this Particular Unreal AI Plugin? +=== Hit Result under Cursor === -There are a *lot* of Unreal Engine AI plugins out there. Some of -them are, shall we say, not carefully engineered. I'm a -reasonably skilled software engineer and I've designed -this plugin to be robust and capable of sustained development. +PlayerController|Game|Player|GetHitResultUnderCursorForObjects +PlayerController|Game|Player|GetHitResultUnderCursorByChannel +``` -This plugin is also designed to be as broadly general as -possible. I've seen plugins that claim "can create 22 different -kinds of graph nodes!" This makes me ask: why not just -provide the *entire catalog* of all possible graph nodes? -I've seen plugins claim "you can edit 15 different material -expression properties!" Why not provide access to *all* -editable material expression properties? I've tried to make -every tool in this plugin as capable as possible, with as few -limits as possible. +Next, the AI adds a node to the event graph: -Some of the MCPs out there expose the entire Unreal API to -the AI agent. That is not entirely safe. AI agents will -very often carelessly call API functions they shouldn't, -crashing your editor or corrupting your assets. All the -commands supported by this MCP are error-checked. It is -intended that they won't let the AI do something that would -crash your editor. +``` +$ ue-wingman.py GraphNode_Add /game/testing/bp_test,graph:EventGraph "PlayerController|Game|Player|GetHitResultUnderCursorByChannel" 100 100 +Spawned: PlayerController|Game|Player|GetHitResultUnderCursorByChannel +node K2Node_CallFunction_0: Get Hit Result Under Cursor by Channel + pos 96, 96 + input-pin PlayerController self = + input-pin ETraceTypeQuery TraceChannel = TraceTypeQuery1 + input-pin bool bTraceComplex = TRUE + output-pins HitResult HitResult, bool ReturnValue +``` -This MCP is very extensible. Adding a new command requires -a relatively small amount of code. I'm hoping some others -in the community will eventually start contributing new -commands. +The AI changes the value of 'bTraceComplex' on the hit-result node: + +``` +$ ue-wingman.py GraphNode_SetDefault /game/testing/bp_test,graph:EventGraph,node:K2Node_CallFunction_0 bTraceComplex False +Done. +``` + +The AI can view the details panel for almost any object. +Here's a partial dump from an enhanced input action: + +``` +$ ue-wingman.py Details_Dump /game/luprex/InputActions/IA_Menu + +Action: + editable bool bTriggerWhenPaused = False + editable bool bReserveAllMappings = False + editable EInputActionValueType ValueType = Boolean + editable EInputActionAccumulationBehavior AccumulationBehavior = TakeHighestAbsoluteValue + editable Array Triggers = +... +``` + +## List of Current Commands, Excluding Asset Creation Commands + +``` +ActorComponent_Add Blueprint Class Component Parent +ActorComponent_Remove Component +ActorComponent_Reparent Component Parent + +Asset_Backup Asset +Asset_ContentBrowse Path +Asset_Delete Asset Force +Asset_FindReferences Asset +Asset_Rename Asset NewPath +Asset_Restore Asset +Asset_Search Query Type Limit + +Blueprint_AddInterface Blueprint Interface +Blueprint_Compile Blueprint +Blueprint_Dump Blueprint +Blueprint_RemoveInterface Blueprint Interface PreserveFunctions +Blueprint_Reparent Blueprint Parent + +BlueprintGraph_Add Blueprint Graph GraphType [Variables...] +BlueprintGraph_Remove Graph + +BlueprintOverride_Add Blueprint Function +BlueprintOverride_ShowMenu Blueprint + +Create_MaterialInstance Path ParentMaterial + +Details_Dump Object +Details_Get Object Property +Details_Set Object Property Value + +Documentation_Command Command +Documentation_Commands +Documentation_CreateAssets Query Verbose +Documentation_Manual +Documentation_Section Section + +Editor_ListOpenAssets +Editor_OpenAsset Asset + +EventDispatcher_Add Blueprint Dispatcher [Variables...] +EventDispatcher_Remove Blueprint Dispatcher + +Graph_Dump Graph Details + +GraphNode_Add Graph Type PosX PosY +GraphNode_ChooseMenu Node Item +GraphNode_Dump Node Details +GraphNode_GetComment Node +GraphNode_Remove Node +GraphNode_Rename Node Name +GraphNode_SearchTypes Graph MaxResults [Queries...] +GraphNode_SetComment Node Comment +GraphNode_SetDefault Graph Node Name Value +GraphNode_SetPosition Graph Node X Y +GraphNode_ShowMenu Node + +GraphPin_Connect Graph [SourcePin_TargetPin...] +GraphPin_Disconnect Graph [Pins...] + +Material_Compile Material +Material_DumpParameters Material + +MaterialInstance_ClearParameter MaterialInstance Parameter +MaterialInstance_DumpParameters MaterialInstance +MaterialInstance_SetParameter MaterialInstance Parameter Value + +SysInfo_Factories +SysInfo_PackageContents Package + +TypeName_Search Query Limit + +Variables_Add Object [Variables...] +Variables_Dump Object +Variables_Modify Object [Variables...] +Variables_Remove Object [Variables...] + +Widget_Add Blueprint Type Name Parent IsVariable +Widget_Remove Widget +Widget_Reparent Widget Parent +Widget_SearchTypes MaxResults [Queries...] +``` ## Installation @@ -75,68 +169,60 @@ There are two parts to UE Wingman: * The Unreal Plugin, which does 99% of the work. -* The python program "ue-wingman.py" +* The command-line program "ue-wingman.py" -The python program is actually less than 100 lines of code: -all it does is package up its command line arguments, send -them to the plugin, and let the plugin do the work. Then it -prints the output. +The command-line program is actually less than 100 lines of +python. All it does is package up its command line +arguments, send them to the plugin, and let the plugin do +the work. Then it prints the output. To install it, +just copy it into your PATH. I like to remove the ".py" +extension, but that's optional. -If you build Unreal from source, the best way to install the -plugin is to drop the entire UEWingman source folder into -your Plugins folder. Then do a build. Restart the editor, and -go into your plugins configuration. Enable the UE Wingman -plugin. You're done. +Currently, the only way to install the plugin is to build +it from source. I just don't have to the time to make +a binary distribution. Building from source is simple: +just drop the entire plugin folder into one of two places: -If you don't build from source, then unfortunately, you're -out of luck. Precompiled plugins must be built for every -different OS, for every different engine version. I just -don't have the means to do that right now. +* Option 1: UnrealEngine/Engine/Plugins/UEWingman +* Option 2: YourGame/Plugins/UEWingman -After installing the plugin, you need to install the two -python programs. They are both short and simple: all they -do is establish a network connection to the plugin, and then -send the command you typed. They require python 3.6 or later, -and no other dependencies. +Then, do a build using unreal build tool. Restart the +editor, and go into your plugins configuration. Enable the +UE Wingman plugin. You're done. -To install the human version, ue-wingman.py, just drop it into -a folder on your PATH. +## The Built-in Documentation -## The "User Manual" - -You might be interested in seeing the "user manual" for the -plugin. To get that, you type this: +The tool has a built-in user manual, which you can access +by typing this: ``` $ ue-wingman.py Documentation_Manual ``` -Of course, you're not the intended user: your AI agent is. +Alternately, you can just type ue-wingman with no arguments. +This manual isn't really intended for humans, it's meant +for the AI agent, so it's a little dense. + You should put a note into your agent's system prompt to -let it know about the ue-wingman.py command, and to let +let it know about the ue-wingman command, and to let it know that it can type ue-wingman.py Documentation_Manual. -This in turn will tell your agent about this command: + +The manual mentions several other commands, including +these, which emit documentation for the built-in subcommands: ``` $ ue-wingman.py Documentation_Commands +$ ue-wingman.py Documentation_Command ``` Using these commands, you can learn more about what this plugin can do. -## Fun things to Try - -I really started enjoying this plugin when I asked my agent -to make me a "cool looking material, something psychedelic -and weird." It made a neat kaleidoscope thing. I then -asked it to make me an animated rendering of the mandelbrot -set. It's fun to watch it do things like that. - ## History and Credits -When I myself needed an MCP for unreal development, I did a -survey of the plugins out there. I ended up choosing a -plugin called "Blueprint MCP" by David Gundry: +When I myself needed a tool like this, I did a survey of the +plugins out there. I ended up choosing a plugin called +"Blueprint MCP" by David Gundry: @@ -146,16 +232,16 @@ pretty much the whole thing. So this whole project is actually a fork of Blueprint MCP. There's very little of the original code remaining. However, you will find snippets here and there. -Even though I ended up rewriting most of the code, it really -was useful to have a functioning starting point. It meant I -could improve one thing at a time, without having to try to -get everything working all at once. So I'm quite grateful -to David Gundry and his work. +Even though I ended up rewriting essentially all of the +code, it really was useful to have a functioning starting +point. It meant I could improve one thing at a time, +without having to try to get everything working all at once. +So I'm quite grateful to David Gundry and his work. ## Software License UE Wingman is licensed under the MIT license, a copy of which -is enclosed. Its predecessor (of which it is a fork) was also +is enclosed. Its predecessor Blueprint MCP was also under the MIT license, so everything works out.