# UE Wingman ue-wingman is a command-line program that allows an AI to use the unreal editor. The AI can use this command to do things like: * Create blueprints * Wire up graph nodes * Edit parameters in the details panel * Manipulate actor components * Build widget hierarchies The tool is not complete, not by a long shot. There are tons of Unreal Editor functions that the AI just can't access yet. Even so, I think it's useful: it has pretty comprehensive support to allow the AI agent to help create blueprints, widget blueprints, and materials. ## Example Commands The AI can print out a graph in a text form, so it can read it. Here's the dump of a simple event graph that doesn't have much in it yet: ``` $ ue-wingman.py Graph_Dump /Game/Testing/BP_Test,graph:EventGraph node K2Node_Event_0: Event BeginPlay output-pins OutputDelegate node K2Node_Event_2: Event Tick output-pins OutputDelegate, DeltaSeconds ``` Here, the AI asks: what kinds of graph nodes can I add to the event graph, that contain the words "Hit Result Under Cursor": ``` $ ue-wingman.py GraphNode_SearchTypes /game/testing/bp_test,graph:EventGraph 50 "Hit Result under Cursor" === Hit Result under Cursor === PlayerController|Game|Player|GetHitResultUnderCursorForObjects PlayerController|Game|Player|GetHitResultUnderCursorByChannel ``` Next, the AI adds a node to the event graph: ``` $ ue-wingman.py GraphNode_Add /game/testing/bp_test,graph:EventGraph "PlayerController|Game|Player|GetHitResultUnderCursorByChannel" 100 100 Spawned: PlayerController|Game|Player|GetHitResultUnderCursorByChannel node K2Node_CallFunction_0: Get Hit Result Under Cursor by Channel pos 96, 96 input-pin PlayerController self = input-pin ETraceTypeQuery TraceChannel = TraceTypeQuery1 input-pin bool bTraceComplex = TRUE output-pins HitResult HitResult, bool ReturnValue ``` The AI changes the value of 'bTraceComplex' on the hit-result node: ``` $ ue-wingman.py GraphNode_SetDefault /game/testing/bp_test,graph:EventGraph,node:K2Node_CallFunction_0 bTraceComplex False Done. ``` The AI can view the details panel for almost any object. Here's a partial dump from an enhanced input action: ``` $ ue-wingman.py Details_Dump /game/luprex/InputActions/IA_Menu Action: editable bool bTriggerWhenPaused = False editable bool bReserveAllMappings = False editable EInputActionValueType ValueType = Boolean editable EInputActionAccumulationBehavior AccumulationBehavior = TakeHighestAbsoluteValue editable Array Triggers = ... ``` ## List of Current Commands, Excluding Asset Creation Commands ``` ActorComponent_Add Blueprint Class Component Parent ActorComponent_Remove Component ActorComponent_Reparent Component Parent Asset_Backup Asset Asset_ContentBrowse Path Asset_Delete Asset Force Asset_FindReferences Asset Asset_Rename Asset NewPath Asset_Restore Asset Asset_Search Query Type Limit Blueprint_AddInterface Blueprint Interface Blueprint_Compile Blueprint Blueprint_Dump Blueprint Blueprint_RemoveInterface Blueprint Interface PreserveFunctions Blueprint_Reparent Blueprint Parent BlueprintGraph_Add Blueprint Graph GraphType [Variables...] BlueprintGraph_Remove Graph BlueprintOverride_Add Blueprint Function BlueprintOverride_ShowMenu Blueprint Create_MaterialInstance Path ParentMaterial Details_Dump Object Details_Get Object Property Details_Set Object Property Value Documentation_Command Command Documentation_Commands Documentation_CreateAssets Query Verbose Documentation_Manual Documentation_Section Section Editor_ListOpenAssets Editor_OpenAsset Asset EventDispatcher_Add Blueprint Dispatcher [Variables...] EventDispatcher_Remove Blueprint Dispatcher Graph_Dump Graph Details GraphNode_Add Graph Type PosX PosY GraphNode_ChooseMenu Node Item GraphNode_Dump Node Details GraphNode_GetComment Node GraphNode_Remove Node GraphNode_Rename Node Name GraphNode_SearchTypes Graph MaxResults [Queries...] GraphNode_SetComment Node Comment GraphNode_SetDefault Graph Node Name Value GraphNode_SetPosition Graph Node X Y GraphNode_ShowMenu Node GraphPin_Connect Graph [SourcePin_TargetPin...] GraphPin_Disconnect Graph [Pins...] Material_Compile Material Material_DumpParameters Material MaterialInstance_ClearParameter MaterialInstance Parameter MaterialInstance_DumpParameters MaterialInstance MaterialInstance_SetParameter MaterialInstance Parameter Value SysInfo_Factories SysInfo_PackageContents Package TypeName_Search Query Limit Variables_Add Object [Variables...] Variables_Dump Object Variables_Modify Object [Variables...] Variables_Remove Object [Variables...] Widget_Add Blueprint Type Name Parent IsVariable Widget_Remove Widget Widget_Reparent Widget Parent Widget_SearchTypes MaxResults [Queries...] ``` ## Installation There are two parts to UE Wingman: * The Unreal Plugin, which does 99% of the work. * The command-line program "ue-wingman.py" The command-line program is actually less than 100 lines of python. All it does is package up its command line arguments, send them to the plugin, and let the plugin do the work. Then it prints the output. To install it, just copy it into your PATH. I like to remove the ".py" extension, but that's optional. Currently, the only way to install the plugin is to build it from source. I just don't have to the time to make a binary distribution. Building from source is simple: just drop the entire plugin folder into one of two places: * Option 1: UnrealEngine/Engine/Plugins/UEWingman * Option 2: YourGame/Plugins/UEWingman Then, do a build using unreal build tool. Restart the editor, and go into your plugins configuration. Enable the UE Wingman plugin. You're done. ## The Built-in Documentation The tool has a built-in user manual, which you can access by typing this: ``` $ ue-wingman.py Documentation_Manual ``` Alternately, you can just type ue-wingman with no arguments. This manual isn't really intended for humans, it's meant for the AI agent, so it's a little dense. You should put a note into your agent's system prompt to let it know about the ue-wingman command, and to let it know that it can type ue-wingman.py Documentation_Manual. The manual mentions several other commands, including these, which emit documentation for the built-in subcommands: ``` $ ue-wingman.py Documentation_Commands $ ue-wingman.py Documentation_Command ``` Using these commands, you can learn more about what this plugin can do. ## History and Credits When I myself needed a tool like this, I did a survey of the plugins out there. I ended up choosing a plugin called "Blueprint MCP" by David Gundry: It was not bad, but it had some limitations, and I started doing work to improve it. Incrementally, I ended rewriting pretty much the whole thing. So this whole project is actually a fork of Blueprint MCP. There's very little of the original code remaining. However, you will find snippets here and there. Even though I ended up rewriting essentially all of the code, it really was useful to have a functioning starting point. It meant I could improve one thing at a time, without having to try to get everything working all at once. So I'm quite grateful to David Gundry and his work. ## Software License UE Wingman is licensed under the MIT license, a copy of which is enclosed. Its predecessor Blueprint MCP was also under the MIT license, so everything works out.