integration/Plugins/UEWingman/README.md

# UE Wingman

ue-wingman is a command-line program that allows an AI
to use the unreal editor.  The AI can use this command
to do things like:

* Create blueprints
* Wire up graph nodes
* Edit parameters in the details panel
* Manipulate actor components
* Build widget hierarchies

The tool is not complete, not by a long shot.  There are
tons of Unreal Editor functions that the AI just can't
access yet. Even so, I think it's useful: it has pretty
comprehensive support to allow the AI agent to help create
blueprints, widget blueprints, and materials.

## Example Commands

The AI can print out a graph in a text form, so it can read
it.  Here's the dump of a simple event graph that doesn't
have much in it yet:
```
$ ue-wingman.py Graph_Dump /Game/Testing/BP_Test,graph:EventGraph
 
node K2Node_Event_0: Event BeginPlay
    output-pins OutputDelegate
 
node K2Node_Event_2: Event Tick
   output-pins OutputDelegate, DeltaSeconds
```

Here, the AI asks: what kinds of graph nodes can I add to the
event graph, that contain the words "Hit Result Under Cursor":

```
$ ue-wingman.py GraphNode_SearchTypes /game/testing/bp_test,graph:EventGraph 50 "Hit Result under Cursor"

=== Hit Result under Cursor ===

PlayerController|Game|Player|GetHitResultUnderCursorForObjects
PlayerController|Game|Player|GetHitResultUnderCursorByChannel
```

Next, the AI adds a node to the event graph:

```
$ ue-wingman.py GraphNode_Add /game/testing/bp_test,graph:EventGraph "PlayerController|Game|Player|GetHitResultUnderCursorByChannel" 100 100
Spawned: PlayerController|Game|Player|GetHitResultUnderCursorByChannel
node K2Node_CallFunction_0: Get Hit Result Under Cursor by Channel
    pos 96, 96
    input-pin PlayerController self = <default>
    input-pin ETraceTypeQuery TraceChannel = TraceTypeQuery1
    input-pin bool bTraceComplex = TRUE
    output-pins HitResult HitResult, bool ReturnValue
```

The AI changes the value of 'bTraceComplex' on the hit-result node:

```
$ ue-wingman.py GraphNode_SetDefault /game/testing/bp_test,graph:EventGraph,node:K2Node_CallFunction_0 bTraceComplex False
Done.
```

The AI can view the details panel for almost any object.
Here's a partial dump from an enhanced input action:

```
$ ue-wingman.py Details_Dump /game/luprex/InputActions/IA_Menu

Action:
    editable bool bTriggerWhenPaused = False
    editable bool bReserveAllMappings = False
    editable EInputActionValueType ValueType = Boolean
    editable EInputActionAccumulationBehavior AccumulationBehavior = TakeHighestAbsoluteValue
    editable Array<InputTrigger> Triggers = 
...
```

## List of Current Commands, Excluding Asset Creation Commands

```
ActorComponent_Add Blueprint Class Component Parent
ActorComponent_Remove Component
ActorComponent_Reparent Component Parent

Asset_Backup Asset
Asset_ContentBrowse Path
Asset_Delete Asset Force
Asset_FindReferences Asset
Asset_Rename Asset NewPath
Asset_Restore Asset
Asset_Search Query Type Limit

Blueprint_AddInterface Blueprint Interface
Blueprint_Compile Blueprint
Blueprint_Dump Blueprint
Blueprint_RemoveInterface Blueprint Interface PreserveFunctions
Blueprint_Reparent Blueprint Parent

BlueprintGraph_Add Blueprint Graph GraphType [Variables...]
BlueprintGraph_Remove Graph

BlueprintOverride_Add Blueprint Function
BlueprintOverride_ShowMenu Blueprint

Create_MaterialInstance Path ParentMaterial

Details_Dump Object
Details_Get Object Property
Details_Set Object Property Value

Documentation_Command Command
Documentation_Commands
Documentation_CreateAssets Query Verbose
Documentation_Manual
Documentation_Section Section

Editor_ListOpenAssets
Editor_OpenAsset Asset

EventDispatcher_Add Blueprint Dispatcher [Variables...]
EventDispatcher_Remove Blueprint Dispatcher

Graph_Dump Graph Details

GraphNode_Add Graph Type PosX PosY
GraphNode_ChooseMenu Node Item
GraphNode_Dump Node Details
GraphNode_GetComment Node
GraphNode_Remove Node
GraphNode_Rename Node Name
GraphNode_SearchTypes Graph MaxResults [Queries...]
GraphNode_SetComment Node Comment
GraphNode_SetDefault Graph Node Name Value
GraphNode_SetPosition Graph Node X Y
GraphNode_ShowMenu Node

GraphPin_Connect Graph [SourcePin_TargetPin...]
GraphPin_Disconnect Graph [Pins...]

Material_Compile Material
Material_DumpParameters Material

MaterialInstance_ClearParameter MaterialInstance Parameter
MaterialInstance_DumpParameters MaterialInstance
MaterialInstance_SetParameter MaterialInstance Parameter Value

SysInfo_Factories
SysInfo_PackageContents Package

TypeName_Search Query Limit

Variables_Add Object [Variables...]
Variables_Dump Object
Variables_Modify Object [Variables...]
Variables_Remove Object [Variables...]

Widget_Add Blueprint Type Name Parent IsVariable
Widget_Remove Widget
Widget_Reparent Widget Parent
Widget_SearchTypes MaxResults [Queries...]
```

## Installation

There are two parts to UE Wingman:

* The Unreal Plugin, which does 99% of the work.

* The command-line program "ue-wingman.py"

The command-line program is actually less than 100 lines of
python.  All it does is package up its command line
arguments, send them to the plugin, and let the plugin do
the work.  Then it prints the output.  To install it,
just copy it into your PATH.  I like to remove the ".py"
extension, but that's optional.

Currently, the only way to install the plugin is to build
it from source.  I just don't have to the time to make
a binary distribution.  Building from source is simple:
just drop the entire plugin folder into one of two places:

* Option 1: UnrealEngine/Engine/Plugins/UEWingman
* Option 2: YourGame/Plugins/UEWingman

Then, do a build using unreal build tool.  Restart the
editor, and go into your plugins configuration.  Enable the
UE Wingman plugin.  You're done.

## The Built-in Documentation

The tool has a built-in user manual, which you can access
by typing this:

```
$ ue-wingman.py Documentation_Manual
```

Alternately, you can just type ue-wingman with no arguments.
This manual isn't really intended for humans, it's meant
for the AI agent, so it's a little dense.

You should put a note into your agent's system prompt to
let it know about the ue-wingman command, and to let
it know that it can type ue-wingman.py Documentation_Manual.

The manual mentions several other commands, including
these, which emit documentation for the built-in subcommands:

```
$ ue-wingman.py Documentation_Commands
$ ue-wingman.py Documentation_Command
```

Using these commands, you can learn more about what this
plugin can do.

## History and Credits

When I myself needed a tool like this, I did a survey of the
plugins out there.  I ended up choosing a plugin called
"Blueprint MCP" by David Gundry:

   <https://github.com/mirno-ehf/ue5-mcp>

It was not bad, but it had some limitations, and I started
doing work to improve it.  Incrementally, I ended rewriting
pretty much the whole thing.  So this whole project is actually
a fork of Blueprint MCP.  There's very little of the original
code remaining.  However, you will find snippets here and there.

Even though I ended up rewriting essentially all of the
code, it really was useful to have a functioning starting
point.  It meant I could improve one thing at a time,
without having to try to get everything working all at once.
So I'm quite grateful to David Gundry and his work.

## Software License

UE Wingman is licensed under the MIT license, a copy of which
is enclosed.  Its predecessor Blueprint MCP was also
under the MIT license, so everything works out.