Two kinds of AI supported by Gua
It changes code and uses Debug information, logs, and screenshots to verify the running result and fix problems.
It does not change code. It participates as a player using only Player-approved UI, World information, and operations.
1. Give AI coding agents a way to verify running games
The coding agent writes the game; Gua provides the runtime UI layer it uses to check the result. Through gui-mcp, the agent can access the Semantic UI Tree, actions, waits, logs, screenshots, recordings, visual comparisons, and test execution.
The AI development loop
- Implement the feature
The AI coding agent changes Godot GDScript/C# or Unity C# and the game UI.
- Launch the game
The Godot GDScript add-on collects Controls. The Unity UPM package automatically collects runtime UI Toolkit, uGUI, and TextMeshPro controls. Both publish through the Gua WebSocket bridge.
- Inspect the Semantic UI Tree
The agent calls
get_ui_treeand identifies a control from its ID, role, text, and state without depending on engine-specific object APIs. - Operate and wait for the result
It sends click, value, selection, scroll, or key actions and waits for the expected UI state.
- Verify from evidence and fix
On failure, it inspects the tree, engine logs, screenshot, or visual diff, changes the implementation, and runs the test again.
2. Let an AI agent player play the release build
In a release build, the AI is a participant in the game rather than a test operator with developer access. The game fixes the Player profile and exposure policy that select visible UI, World Objects, and permitted UI operations. Semantic Game Actions and Raw Input are separate permissions and remain unavailable until explicitly enabled.
The World Object Tree describes opted-in objectives, enemies, doors, checkpoints, and other game concepts without dumping scene nodes or internal state.
Semantic UI Actions operate menus. Semantic Game Actions such as jump or move, and Raw Input only when needed, drive gameplay.
Private objects, internal values, and Debug logs are excluded, and screenshots are denied by default. The AI cannot elevate itself to Debug.
Held input is isolated by owner and returns to neutral after lease expiry, disconnect, unregister, or engine shutdown.
Development AI versus AI player
| AI development and verification | AI player in a release build | |
|---|---|---|
| Goal | Verify implementation and fix problems | Participate and play as a player |
| Observation profile | Developer-managed Debug | Host-fixed Player |
| Observation | UI, logs, screenshots, and diagnostics | Approved UI and World information; no Debug logs |
| Operations | Semantic operations needed for testing and diagnosis | Approved UI operations plus separately authorized gameplay input |
| Connection | Primarily gui-mcp for native games | Player-configured gui-mcp, or gua-webmcp for web games |
Engine integration
| Engine | Runtime UI source | How Gua starts |
|---|---|---|
| Godot 4.7 | Standard Control tree | The GDScript add-on attaches an automatic adapter and starts the bridge. |
| Unity 6 | UI Toolkit, uGUI, and TextMeshPro runtime UI | The UPM package starts automatically before scene load and runs collection after each frame. |
| Both | Semantic UI Tree v2 | gui-mcp uses the same locators, actions, waits, logs, screenshots, Visual, and Recording tools. |
How it maps to Playwright on the web
| Web UI automation | Game UI automation with Gua |
|---|---|
| Browser | Running game |
| DOM | Semantic UI Tree |
| locator | Semantic locator by ID, role, text, and state |
| click / fill / keyboard | Semantic actions such as click / set value / press key |
| expect / wait | GuaAssertions / wait_for_node |
| screenshot / visual test | Gua screenshot / visual baseline comparison |
Frequently asked questions
Can Gua test Godot UI?
Yes. The recommended Godot 4.7 GDScript add-on reflects standard Controls and supports semantic locators, actions, waits, assertions, screenshots, and visual comparison.
Can Gua test Unity UI?
Yes. The Unity 6 UPM package automatically reflects runtime UI Toolkit, uGUI, and TextMeshPro controls on supported Windows x64 Mono builds. The AI uses the same semantic MCP tools as it does with Godot.
Can an AI play a release build?
Yes. Native games can use Player-configured gui-mcp, and web games can use gua-webmcp. The AI observes approved UI and World information and plays with approved UI operations and gameplay input. Development Debug information and Player permissions remain separate.
Is Gua a Godot or Unity Editor MCP?
No; the roles are complementary. Editor tools operate the development environment, while Gua observes, operates, and verifies the runtime UI of a running Godot or Unity game.
Does Gua locate buttons with image recognition?
No. Primary targeting uses IDs, roles, text, and state published by the game. Screenshots remain available as evidence for rendered output and visual comparison.
Does Gua replace the game engine or AI?
No. Gua is the automation layer connecting a game runtime to tests, Inspector, development AI, and AI agent players.
Review the Godot 4.7 integration and test features.
Unity UI Testing →Review the Unity 6 runtime integration and external tests.
MCP Server →Connect an AI to a native game.
Browser-native WebMCP →Connect an AI player to a released web game.
AI agent exposure policy →Restrict the information and UI operations exposed to Player.
Game input →Configure the operations available to an AI player.