talezero/integration
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Readme updates

Browse Source
This commit is contained in:
jyelon
2026-06-01 20:52:48 -04:00
parent cc34e36cd2
commit 1b0d96934f
1 changed files with 180 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

274
Plugins/UEWingman/README.md
View File

@@ -1,12 +1,14 @@
# UE Wingman # UE Wingman
UE Wingman is a tool that allows an AI to control the unreal ue-wingman is a command-line program that allows an AI
editor. When you're using it, it feels like the AI is right to use the unreal editor. The AI can use this command
there looking at the editor with you. You'll be able to to do things like:
watch as it creates graph nodes and wires them together,
you'll see it add components to your blueprints, you'll see * Create blueprints
it design widget hierarchies for you, and you'll see it write * Wire up graph nodes
shaders for your materials. * Edit parameters in the details panel
* Manipulate actor components
* Build widget hierarchies
The tool is not complete, not by a long shot. There are The tool is not complete, not by a long shot. There are
tons of Unreal Editor functions that the AI just can't tons of Unreal Editor functions that the AI just can't
@@ -14,60 +16,152 @@ access yet. Even so, I think it's useful: it has pretty
comprehensive support to allow the AI agent to help create comprehensive support to allow the AI agent to help create
blueprints, widget blueprints, and materials. blueprints, widget blueprints, and materials.
## How Does it Work? ## Example Commands
This tool adds a command line interpreter plugin to the Unreal
Editor. You can type commands, and the plugin in the editor
will execute them.
The AI can print out a graph in a text form, so it can read
it. Here's the dump of a simple event graph that doesn't
have much in it yet:
``` ```
$ ue-wingman Graph_Dump /Game/Testing/BP_Test,graph:EventGraph $ ue-wingman.py Graph_Dump /Game/Testing/BP_Test,graph:EventGraph
node K2Node_Event_0: Event BeginPlay node K2Node_Event_0: Event BeginPlay
output-pins OutputDelegate output-pins OutputDelegate
node K2Node_Event_2: Event Tick node K2Node_Event_2: Event Tick
output-pins OutputDelegate, DeltaSeconds output-pins OutputDelegate, DeltaSeconds
``` ```
The ue-wingman command has tons of subcommands: Graph_Dump, Here, the AI asks: what kinds of graph nodes can I add to the
GraphNode_Add, GraphPin_Connect, BlueprintComponent_Add, event graph, that contain the words "Hit Result Under Cursor":
Widget_Add, and so forth. Using these commands, it's
possible to examine and modify blueprints, widgets, and
materials.
But, of course, these commands aren't really intended for humans. ```
They're intended for an AI agent. $ ue-wingman.py GraphNode_SearchTypes /game/testing/bp_test,graph:EventGraph 50 "Hit Result under Cursor"
## Why Choose this Particular Unreal AI Plugin? === Hit Result under Cursor ===
There are a *lot* of Unreal Engine AI plugins out there. Some of PlayerController|Game|Player|GetHitResultUnderCursorForObjects
them are, shall we say, not carefully engineered. I'm a PlayerController|Game|Player|GetHitResultUnderCursorByChannel
reasonably skilled software engineer and I've designed ```
this plugin to be robust and capable of sustained development.
This plugin is also designed to be as broadly general as Next, the AI adds a node to the event graph:
possible. I've seen plugins that claim "can create 22 different
kinds of graph nodes!" This makes me ask: why not just
provide the *entire catalog* of all possible graph nodes?
I've seen plugins claim "you can edit 15 different material
expression properties!" Why not provide access to *all*
editable material expression properties? I've tried to make
every tool in this plugin as capable as possible, with as few
limits as possible.
Some of the MCPs out there expose the entire Unreal API to ```
the AI agent. That is not entirely safe. AI agents will $ ue-wingman.py GraphNode_Add /game/testing/bp_test,graph:EventGraph "PlayerController|Game|Player|GetHitResultUnderCursorByChannel" 100 100
very often carelessly call API functions they shouldn't, Spawned: PlayerController|Game|Player|GetHitResultUnderCursorByChannel
crashing your editor or corrupting your assets. All the node K2Node_CallFunction_0: Get Hit Result Under Cursor by Channel
commands supported by this MCP are error-checked. It is pos 96, 96
intended that they won't let the AI do something that would input-pin PlayerController self = <default>
crash your editor. input-pin ETraceTypeQuery TraceChannel = TraceTypeQuery1
input-pin bool bTraceComplex = TRUE
output-pins HitResult HitResult, bool ReturnValue
```
This MCP is very extensible. Adding a new command requires The AI changes the value of 'bTraceComplex' on the hit-result node:
a relatively small amount of code. I'm hoping some others
in the community will eventually start contributing new ```
commands. $ ue-wingman.py GraphNode_SetDefault /game/testing/bp_test,graph:EventGraph,node:K2Node_CallFunction_0 bTraceComplex False
Done.
```
The AI can view the details panel for almost any object.
Here's a partial dump from an enhanced input action:
```
$ ue-wingman.py Details_Dump /game/luprex/InputActions/IA_Menu
Action:
editable bool bTriggerWhenPaused = False
editable bool bReserveAllMappings = False
editable EInputActionValueType ValueType = Boolean
editable EInputActionAccumulationBehavior AccumulationBehavior = TakeHighestAbsoluteValue
editable Array<InputTrigger> Triggers =
...
```
## List of Current Commands, Excluding Asset Creation Commands
```
ActorComponent_Add Blueprint Class Component Parent
ActorComponent_Remove Component
ActorComponent_Reparent Component Parent
Asset_Backup Asset
Asset_ContentBrowse Path
Asset_Delete Asset Force
Asset_FindReferences Asset
Asset_Rename Asset NewPath
Asset_Restore Asset
Asset_Search Query Type Limit
Blueprint_AddInterface Blueprint Interface
Blueprint_Compile Blueprint
Blueprint_Dump Blueprint
Blueprint_RemoveInterface Blueprint Interface PreserveFunctions
Blueprint_Reparent Blueprint Parent
BlueprintGraph_Add Blueprint Graph GraphType [Variables...]
BlueprintGraph_Remove Graph
BlueprintOverride_Add Blueprint Function
BlueprintOverride_ShowMenu Blueprint
Create_MaterialInstance Path ParentMaterial
Details_Dump Object
Details_Get Object Property
Details_Set Object Property Value
Documentation_Command Command
Documentation_Commands
Documentation_CreateAssets Query Verbose
Documentation_Manual
Documentation_Section Section
Editor_ListOpenAssets
Editor_OpenAsset Asset
EventDispatcher_Add Blueprint Dispatcher [Variables...]
EventDispatcher_Remove Blueprint Dispatcher
Graph_Dump Graph Details
GraphNode_Add Graph Type PosX PosY
GraphNode_ChooseMenu Node Item
GraphNode_Dump Node Details
GraphNode_GetComment Node
GraphNode_Remove Node
GraphNode_Rename Node Name
GraphNode_SearchTypes Graph MaxResults [Queries...]
GraphNode_SetComment Node Comment
GraphNode_SetDefault Graph Node Name Value
GraphNode_SetPosition Graph Node X Y
GraphNode_ShowMenu Node
GraphPin_Connect Graph [SourcePin_TargetPin...]
GraphPin_Disconnect Graph [Pins...]
Material_Compile Material
Material_DumpParameters Material
MaterialInstance_ClearParameter MaterialInstance Parameter
MaterialInstance_DumpParameters MaterialInstance
MaterialInstance_SetParameter MaterialInstance Parameter Value
SysInfo_Factories
SysInfo_PackageContents Package
TypeName_Search Query Limit
Variables_Add Object [Variables...]
Variables_Dump Object
Variables_Modify Object [Variables...]
Variables_Remove Object [Variables...]
Widget_Add Blueprint Type Name Parent IsVariable
Widget_Remove Widget
Widget_Reparent Widget Parent
Widget_SearchTypes MaxResults [Queries...]
```
## Installation ## Installation
@@ -75,68 +169,60 @@ There are two parts to UE Wingman:
* The Unreal Plugin, which does 99% of the work. * The Unreal Plugin, which does 99% of the work.
* The python program "ue-wingman.py" * The command-line program "ue-wingman.py"
The python program is actually less than 100 lines of code: The command-line program is actually less than 100 lines of
all it does is package up its command line arguments, send python. All it does is package up its command line
them to the plugin, and let the plugin do the work. Then it arguments, send them to the plugin, and let the plugin do
prints the output. the work. Then it prints the output. To install it,
just copy it into your PATH. I like to remove the ".py"
extension, but that's optional.
If you build Unreal from source, the best way to install the Currently, the only way to install the plugin is to build
plugin is to drop the entire UEWingman source folder into it from source. I just don't have to the time to make
your Plugins folder. Then do a build. Restart the editor, and a binary distribution. Building from source is simple:
go into your plugins configuration. Enable the UE Wingman just drop the entire plugin folder into one of two places:
plugin. You're done.
If you don't build from source, then unfortunately, you're * Option 1: UnrealEngine/Engine/Plugins/UEWingman
out of luck. Precompiled plugins must be built for every * Option 2: YourGame/Plugins/UEWingman
different OS, for every different engine version. I just
don't have the means to do that right now.
After installing the plugin, you need to install the two Then, do a build using unreal build tool. Restart the
python programs. They are both short and simple: all they editor, and go into your plugins configuration. Enable the
do is establish a network connection to the plugin, and then UE Wingman plugin. You're done.
send the command you typed. They require python 3.6 or later,
and no other dependencies.
To install the human version, ue-wingman.py, just drop it into ## The Built-in Documentation
a folder on your PATH.
## The "User Manual" The tool has a built-in user manual, which you can access
by typing this:
You might be interested in seeing the "user manual" for the
plugin. To get that, you type this:
``` ```
$ ue-wingman.py Documentation_Manual $ ue-wingman.py Documentation_Manual
``` ```
Of course, you're not the intended user: your AI agent is. Alternately, you can just type ue-wingman with no arguments.
This manual isn't really intended for humans, it's meant
for the AI agent, so it's a little dense.
You should put a note into your agent's system prompt to You should put a note into your agent's system prompt to
let it know about the ue-wingman.py command, and to let let it know about the ue-wingman command, and to let
it know that it can type ue-wingman.py Documentation_Manual. it know that it can type ue-wingman.py Documentation_Manual.
This in turn will tell your agent about this command:
The manual mentions several other commands, including
these, which emit documentation for the built-in subcommands:
``` ```
$ ue-wingman.py Documentation_Commands $ ue-wingman.py Documentation_Commands
$ ue-wingman.py Documentation_Command
``` ```
Using these commands, you can learn more about what this Using these commands, you can learn more about what this
plugin can do. plugin can do.
## Fun things to Try
I really started enjoying this plugin when I asked my agent
to make me a "cool looking material, something psychedelic
and weird." It made a neat kaleidoscope thing. I then
asked it to make me an animated rendering of the mandelbrot
set. It's fun to watch it do things like that.
## History and Credits ## History and Credits
When I myself needed an MCP for unreal development, I did a When I myself needed a tool like this, I did a survey of the
survey of the plugins out there. I ended up choosing a plugins out there. I ended up choosing a plugin called
plugin called "Blueprint MCP" by David Gundry: "Blueprint MCP" by David Gundry:
<https://github.com/mirno-ehf/ue5-mcp> <https://github.com/mirno-ehf/ue5-mcp>
@@ -146,16 +232,16 @@ pretty much the whole thing. So this whole project is actually
a fork of Blueprint MCP. There's very little of the original a fork of Blueprint MCP. There's very little of the original
code remaining. However, you will find snippets here and there. code remaining. However, you will find snippets here and there.
Even though I ended up rewriting most of the code, it really Even though I ended up rewriting essentially all of the
was useful to have a functioning starting point. It meant I code, it really was useful to have a functioning starting
could improve one thing at a time, without having to try to point. It meant I could improve one thing at a time,
get everything working all at once. So I'm quite grateful without having to try to get everything working all at once.
to David Gundry and his work. So I'm quite grateful to David Gundry and his work.
## Software License ## Software License
UE Wingman is licensed under the MIT license, a copy of which UE Wingman is licensed under the MIT license, a copy of which
is enclosed. Its predecessor (of which it is a fork) was also is enclosed. Its predecessor Blueprint MCP was also
under the MIT license, so everything works out. under the MIT license, so everything works out.