162 lines
5.7 KiB
Markdown
162 lines
5.7 KiB
Markdown
# UE Wingman
|
|
|
|
UE Wingman is a tool that allows an AI to control the unreal
|
|
editor. When you're using it, it feels like the AI is right
|
|
there looking at the editor with you. You'll be able to
|
|
watch as it creates graph nodes and wires them together,
|
|
you'll see it add components to your blueprints, you'll see
|
|
it design widget hierarchies for you, and you'll see it write
|
|
shaders for your materials.
|
|
|
|
The tool is not complete, not by a long shot. There are
|
|
tons of Unreal Editor functions that the AI just can't
|
|
access yet. Even so, I think it's useful: it has pretty
|
|
comprehensive support to allow the AI agent to help create
|
|
blueprints, widget blueprints, and materials.
|
|
|
|
## How Does it Work?
|
|
|
|
This tool adds a command line interpreter plugin to the Unreal
|
|
Editor. You can type commands, and the plugin in the editor
|
|
will execute them.
|
|
|
|
```
|
|
$ ue-wingman Graph_Dump /Game/Testing/BP_Test,graph:EventGraph
|
|
|
|
node K2Node_Event_0: Event BeginPlay
|
|
output-pins OutputDelegate
|
|
|
|
node K2Node_Event_2: Event Tick
|
|
output-pins OutputDelegate, DeltaSeconds
|
|
```
|
|
|
|
The ue-wingman command has tons of subcommands: Graph_Dump,
|
|
GraphNode_Add, GraphPin_Connect, BlueprintComponent_Add,
|
|
Widget_Add, and so forth. Using these commands, it's
|
|
possible to examine and modify blueprints, widgets, and
|
|
materials.
|
|
|
|
But, of course, these commands aren't really intended for humans.
|
|
They're intended for an AI agent.
|
|
|
|
## Why Choose this Particular Unreal AI Plugin?
|
|
|
|
There are a *lot* of Unreal Engine AI plugins out there. Some of
|
|
them are, shall we say, not carefully engineered. I'm a
|
|
reasonably skilled software engineer and I've designed
|
|
this plugin to be robust and capable of sustained development.
|
|
|
|
This plugin is also designed to be as broadly general as
|
|
possible. I've seen plugins that claim "can create 22 different
|
|
kinds of graph nodes!" This makes me ask: why not just
|
|
provide the *entire catalog* of all possible graph nodes?
|
|
I've seen plugins claim "you can edit 15 different material
|
|
expression properties!" Why not provide access to *all*
|
|
editable material expression properties? I've tried to make
|
|
every tool in this plugin as capable as possible, with as few
|
|
limits as possible.
|
|
|
|
Some of the MCPs out there expose the entire Unreal API to
|
|
the AI agent. That is not entirely safe. AI agents will
|
|
very often carelessly call API functions they shouldn't,
|
|
crashing your editor or corrupting your assets. All the
|
|
commands supported by this MCP are error-checked. It is
|
|
intended that they won't let the AI do something that would
|
|
crash your editor.
|
|
|
|
This MCP is very extensible. Adding a new command requires
|
|
a relatively small amount of code. I'm hoping some others
|
|
in the community will eventually start contributing new
|
|
commands.
|
|
|
|
## Installation
|
|
|
|
There are two parts to UE Wingman:
|
|
|
|
* The Unreal Plugin, which does 99% of the work.
|
|
|
|
* The python program "ue-wingman.py"
|
|
|
|
The python program is actually less than 100 lines of code:
|
|
all it does is package up its command line arguments, send
|
|
them to the plugin, and let the plugin do the work. Then it
|
|
prints the output.
|
|
|
|
If you build Unreal from source, the best way to install the
|
|
plugin is to drop the entire UEWingman source folder into
|
|
your Plugins folder. Then do a build. Restart the editor, and
|
|
go into your plugins configuration. Enable the UE Wingman
|
|
plugin. You're done.
|
|
|
|
If you don't build from source, then unfortunately, you're
|
|
out of luck. Precompiled plugins must be built for every
|
|
different OS, for every different engine version. I just
|
|
don't have the means to do that right now.
|
|
|
|
After installing the plugin, you need to install the two
|
|
python programs. They are both short and simple: all they
|
|
do is establish a network connection to the plugin, and then
|
|
send the command you typed. They require python 3.6 or later,
|
|
and no other dependencies.
|
|
|
|
To install the human version, ue-wingman.py, just drop it into
|
|
a folder on your PATH.
|
|
|
|
## The "User Manual"
|
|
|
|
You might be interested in seeing the "user manual" for the
|
|
plugin. To get that, you type this:
|
|
|
|
```
|
|
$ ue-wingman.py Documentation_Manual
|
|
```
|
|
|
|
Of course, you're not the intended user: your AI agent is.
|
|
You should put a note into your agent's system prompt to
|
|
let it know about the ue-wingman.py command, and to let
|
|
it know that it can type ue-wingman.py Documentation_Manual.
|
|
This in turn will tell your agent about this command:
|
|
|
|
```
|
|
$ ue-wingman.py Documentation_Commands
|
|
```
|
|
|
|
Using these commands, you can learn more about what this
|
|
plugin can do.
|
|
|
|
## Fun things to Try
|
|
|
|
I really started enjoying this plugin when I asked my agent
|
|
to make me a "cool looking material, something psychedelic
|
|
and weird." It made a neat kaleidoscope thing. I then
|
|
asked it to make me an animated rendering of the mandelbrot
|
|
set. It's fun to watch it do things like that.
|
|
|
|
## History and Credits
|
|
|
|
When I myself needed an MCP for unreal development, I did a
|
|
survey of the plugins out there. I ended up choosing a
|
|
plugin called "Blueprint MCP" by David Gundry:
|
|
|
|
<https://github.com/mirno-ehf/ue5-mcp>
|
|
|
|
It was not bad, but it had some limitations, and I started
|
|
doing work to improve it. Incrementally, I ended rewriting
|
|
pretty much the whole thing. So this whole project is actually
|
|
a fork of Blueprint MCP. There's very little of the original
|
|
code remaining. However, you will find snippets here and there.
|
|
|
|
Even though I ended up rewriting most of the code, it really
|
|
was useful to have a functioning starting point. It meant I
|
|
could improve one thing at a time, without having to try to
|
|
get everything working all at once. So I'm quite grateful
|
|
to David Gundry and his work.
|
|
|
|
## Software License
|
|
|
|
UE Wingman is licensed under the MIT license, a copy of which
|
|
is enclosed. Its predecessor (of which it is a fork) was also
|
|
under the MIT license, so everything works out.
|
|
|
|
|