UI Automation

Control and automate your Tauri application's UI. Webview tools work across Linux, Windows, and macOS. Native dialog automation is Windows-only.

Multi-Window Support

All webview tools support targeting specific windows in multi-window applications. Use the optional windowId parameter to specify which window to interact with. If not specified, tools default to the "main" window.

Discovering Windows

Use manage_window with action: "list" to discover all available windows:

javascript

{
  "tool": "manage_window",
  "action": "list"
}

Response:

json

{
  "windows": [
    {
      "label": "main",
      "title": "My App",
      "url": "http://localhost:1420/",
      "focused": true,
      "visible": true,
      "isMain": true
    },
    {
      "label": "settings",
      "title": "Settings",
      "url": "http://localhost:1420/settings",
      "focused": false,
      "visible": true,
      "isMain": false
    }
  ],
  "defaultWindow": "main",
  "totalCount": 2
}

Getting Window Info

Use action: "info" to get detailed information about a specific window:

javascript

{
  "tool": "manage_window",
  "action": "info",
  "windowId": "main"
}

Response:

json

{
  "width": 800,
  "height": 600,
  "x": 100,
  "y": 100,
  "title": "My App",
  "focused": true,
  "visible": true
}

Resizing Windows

Use action: "resize" to resize a window to specific dimensions:

javascript

{
  "tool": "manage_window",
  "action": "resize",
  "width": 1024,
  "height": 768
}

Response:

json

{
  "success": true,
  "windowLabel": "main",
  "width": 1024,
  "height": 768,
  "logical": true
}

By default, dimensions are in logical pixels (respects display scaling). Set logical: false for physical pixels. The resize will fail if the window has fixed size constraints or is not resizable.

Targeting a Specific Window

Add windowId to any webview tool to target a specific window:

javascript

// Execute JavaScript in the settings window
{
  "tool": "webview_execute_js",
  "script": "document.title",
  "windowId": "settings"
}

// Take a screenshot of the main window (explicit)
{
  "tool": "webview_screenshot",
  "windowId": "main"
}

driver_session

Manage UI automation session lifecycle. Initializes console log capture and prepares the webview for automation. Supports remote device connections via the host parameter.

Parameters

Name	Type	Required	Description
`action`	string	Yes	Action to perform: 'start', 'stop', or 'status'
`host`	string	No	Host address to connect to (e.g., '192.168.1.100'). Falls back to `MCP_BRIDGE_HOST` or `TAURI_DEV_HOST` env vars
`port`	number	No	Port to connect to (default: 9223)

Actions

start - Start a new session, connecting to the Tauri app
stop - Stop the current session and disconnect
status - Check current connection status without changing state. Returns the app's identifier (bundle ID) which can be used to determine if the session is connected to the correct app

Connection Strategy

When starting a session, the tool uses the following connection strategy:

Try localhost first - Most reliable for simulators, emulators, and desktop apps
Fall back to configured host - If localhost fails and a remote host is configured
Auto-discover - Scan port range on localhost for running apps
Graceful fallback - Return success message even if no app found (allows IPC-only mode)

Example

javascript

// Start an automation session (default - localhost)
{
  "tool": "driver_session",
  "action": "start"
}

// Connect to a real iOS device on the network
{
  "tool": "driver_session",
  "action": "start",
  "host": "192.168.1.100"
}

// Connect to a specific port
{
  "tool": "driver_session",
  "action": "start",
  "port": 9225
}

// Check connection status
{
  "tool": "driver_session",
  "action": "status"
}

Response

Start/Stop:

Session started with app: My App (localhost:9223)

Status:

json

{
  "connected": true,
  "app": "My App",
  "identifier": "com.example.my-app",
  "host": "localhost",
  "port": 9223
}

The identifier field contains the app's bundle ID (e.g., com.example.my-app). Use this to verify you're connected to the correct application before reusing an existing session.

Note: The identifier field may be null if the Tauri app uses an older version of the MCP Bridge plugin that doesn't provide app identification. In this case, you cannot verify the app identity and should start a new session if uncertain.

Environment Variables

MCP_BRIDGE_HOST - Default host when host parameter not provided
TAURI_DEV_HOST - Fallback host (same as Tauri CLI uses for mobile dev)
MCP_BRIDGE_PORT - Default port when port parameter not provided

Remote Device Setup

For real iOS/Android devices on the network:

Ensure your development machine and device are on the same network
The Tauri plugin binds to 0.0.0.0 by default, allowing remote connections
Use the device's IP address as the host parameter

Android alternative: Use adb reverse tcp:9223 tcp:9223 to forward the port, then connect to localhost.

Note: No external driver process required.

Native Windows Dialogs

native_dialog_snapshot and native_dialog_interact automate native message, confirmation, single- or multi-file Open, folder selection, and Save dialogs. These windows are outside the webview DOM, so webview_interact, webview_keyboard, and webview_screenshot cannot inspect or control them.

The tools require an active driver_session and an interactive Windows desktop. Discovery is restricted to visible dialogs in the connected Tauri process whose bounded owner chain leads back to the targeted Tauri window. This includes nested prompts such as a Save overwrite confirmation without exposing desktop-wide automation or dialogs from other applications. Windows toast notifications are not dialogs and are not supported.

native_dialog_snapshot

Returns a bounded semantic snapshot containing control types, automation IDs, names, semantic roles, supported UI Automation patterns, owner depth, parent-dialog references, and opaque elementRef values. File-dialog snapshots include actionable navigation controls, the current-location control, navigation-tree items, and selectable file-system entries. Use windowId and appIdentifier with the same targeting behavior as webview tools. timeoutMs accepts 100–10000 milliseconds.

javascript

{
  "tool": "native_dialog_snapshot",
  "windowId": "main",
  "minOwnerDepth": 1,
  "timeoutMs": 2000
}

minOwnerDepth defaults to 1, where the Tauri window directly owns the dialog. Set it to 2 to wait for a prompt owned by another app-owned dialog, such as an overwrite confirmation. Nested dialogs are returned before their parents.

Controls advertise one or more supported actions:

invoke — InvokePattern, typically buttons such as accept, cancel, yes, or no
setValue — ValuePattern, used for a complete absolute filename/path
setPaths — ValuePattern, used for up to 100 complete absolute existing file paths in a multi-select Open dialog
select — SelectionItemPattern, where the dialog exposes a selectable control

Semantic roles are derived from automation IDs and control patterns rather than localized button text. Names and dialog text are returned to the caller for inspection but are not written to bridge logs.

native_dialog_interact

Apply one advertised action to an elementRef from the latest snapshot:

javascript

{
  "tool": "native_dialog_interact",
  "action": "invoke",
  "elementRef": "native_..."
}

For Open or Save dialogs, set the filename field to a complete absolute path, then invoke the accept button:

javascript

{
  "tool": "native_dialog_interact",
  "action": "setValue",
  "elementRef": "native_...",
  "value": "C:\\tmp\\fixture.txt"
}

For a multi-file Open dialog, use the filename field's advertised setPaths action and then invoke accept:

javascript

{
  "tool": "native_dialog_interact",
  "action": "setPaths",
  "elementRef": "native_...",
  "paths": [
    "C:\\tmp\\first.txt",
    "C:\\tmp\\second.txt"
  ]
}

For a folder picker, use setValue with a complete absolute directory path and invoke accept. To respond to a Save overwrite prompt, invoke Save, take a new snapshot with minOwnerDepth: 2, and invoke the confirmation's advertised affirmative action. Back, forward, up, address/current-location, navigation-tree, and file-system entry controls are included when Windows exposes their corresponding UI Automation patterns.

Element references are session-bound, expire after 30 seconds, and are otherwise ephemeral. A new snapshot replaces the previous reference set; invoking or selecting a control invalidates references for that dialog, while setValue and setPaths preserve them so the previously discovered accept control can be invoked. If the dialog changes or closes, the tool returns a stale-reference error and the caller must take another snapshot.

Supported scope includes single- and multi-file Open, Save, folder selection, overwrite confirmation, message acknowledgement, ask/confirm responses, absolute-path entry, cancellation, nested app-owned dialogs, and navigation controls that expose the patterns above. Arbitrary custom controls without UI Automation patterns, permission prompts outside the app's ownership chain, notification/toast automation, image recognition, and coordinate-based native control remain out of scope.

webview_find_element

Find UI elements using CSS, XPath, or text selectors.

Parameters

Name	Type	Required	Description
`selector`	string	Yes	Element selector
`strategy`	string	No	Selector strategy: 'css', 'xpath', 'text' (default: 'css')
`windowId`	string	No	Window label to target (defaults to 'main')

Example

javascript

// Find a button by CSS selector
{
  "tool": "webview_find_element",
  "selector": "#submit-button",
  "strategy": "css"
}

// Find by text content
{
  "tool": "webview_find_element",
  "selector": "Submit",
  "strategy": "text"
}

Response

Returns element information including tag name, text content, and attributes.

read_logs

Read logs from various sources: webview console logs, Android logcat, iOS simulator logs, or desktop system logs.

Parameters

Name	Type	Required	Description
`source`	string	Yes	Log source: 'console', 'android', 'ios', 'system'
`lines`	number	No	Number of log lines to retrieve (default: 50)
`filter`	string	No	Regex or keyword to filter logs
`since`	string	No	ISO timestamp to filter logs since
`windowId`	string	No	Window label for console logs (defaults to 'main')

Sources

console - JavaScript console logs from the webview (requires active session)
android - Android logcat output
ios - iOS simulator logs
system - Desktop system logs (macOS/Linux)

Example

javascript

// Get webview console logs
{
  "tool": "read_logs",
  "source": "console"
}

// Get console logs matching a pattern
{
  "tool": "read_logs",
  "source": "console",
  "filter": "error|warning"
}

// Read Android logcat
{
  "tool": "read_logs",
  "source": "android",
  "filter": "com.myapp",
  "lines": 100
}

// Read system logs
{
  "tool": "read_logs",
  "source": "system",
  "lines": 50
}

Response

Returns log entries from the specified source with timestamps and log levels.

webview_dom_snapshot

Get a structured DOM snapshot of a Tauri app's webview for AI consumption.

Parameters

Name	Type	Required	Description
`type`	`'accessibility'` \| `'structure'`	Yes	Snapshot type
`selector`	string	No	CSS selector to scope the snapshot
`windowId`	string	No	Window label to target
`appIdentifier`	string \| number	No	App identifier

Snapshot Types

accessibility - Uses aria-api for comprehensive, spec-compliant accessibility computation:

WAI-ARIA 1.3 role computation
Accessible names and descriptions
ARIA states (disabled, expanded, checked, etc.)
Best for understanding UI semantics and finding interactive elements

structure - DOM structure tree with:

Element tag names
Element IDs (if present)
CSS classes (if present)
data-testid attributes (if present)
Best for understanding page layout and debugging CSS selectors

Accessibility Snapshot Format

yaml

- heading "Page Title" [level=1] [ref=e0]:
- navigation [ref=e1]:
  - list [ref=e2]:
    - listitem [ref=e3]:
      - link "Home" [ref=e4]
- main [ref=e5]:
  - button "Submit" [disabled] [ref=e6]
  - textbox "Enter name" [ref=e7]

Structure Snapshot Format

yaml

- body [ref=e0]:
  - div#app.container [ref=e1]:
    - header.header [ref=e2]:
      - nav.nav-menu [ref=e3]
    - main.content [ref=e4]:
      - form#login-form [ref=e5] [data-testid=login]:
        - input#username [ref=e6]
        - button.btn.btn-primary [ref=e7]

Element References

Each element includes a ref attribute (e.g., [ref=e0]) that can be used with other webview tools to target that specific element. Simply pass the ref ID as the selector parameter:

javascript

// First, get a snapshot to see available refs
{ "tool": "webview_dom_snapshot", "type": "accessibility" }

// Then use a ref to interact with an element
{ "tool": "webview_interact", "action": "click", "selector": "ref=e7" }

// Or type into an input
{ "tool": "webview_keyboard", "action": "type", "selector": "ref=e6", "text": "hello" }

Refs work with: webview_interact, webview_keyboard, webview_get_styles, webview_find_element, and webview_wait_for.

WARNING

Refs are regenerated each time you call webview_dom_snapshot. If the DOM changes significantly, run a new snapshot to get updated refs.

Example

javascript

// Accessibility snapshot of entire page
{
  "tool": "webview_dom_snapshot",
  "type": "accessibility"
}

// Structure snapshot of a specific component
{
  "tool": "webview_dom_snapshot",
  "type": "structure",
  "selector": ".login-form"
}

Scoped Snapshots

Use the selector parameter to snapshot a subtree. If the selector matches multiple elements, each match is returned as a separate labeled snapshot.

Response

Returns a YAML-formatted tree with:

Accessibility type: Element roles, accessible names, ARIA states, ref IDs
Structure type: Tag names, IDs, CSS classes, data-testid attributes, ref IDs
Metadata footer with generation timestamp and element count

webview_select_element

Activate a visual element picker overlay in the Tauri app. The user sees a blue highlight following their cursor and can click to select an element. The tool returns rich metadata about the selected element plus a cropped screenshot of it.

This is useful when you want the user to point out a specific UI element so you can discuss it, debug it, or make changes to it.

Slash Command

You can also use the /select slash command for a guided element selection workflow. See Prompts for details.

Parameters

Name	Type	Required	Description
`timeout`	number	No	Timeout in ms for user to pick an element (5000–120000, default: 60000)
`windowId`	string	No	Window label to target (defaults to 'main')
`appIdentifier`	string \| number	No	App identifier for multi-app setups

Example

javascript

// Activate the element picker with default timeout
{
  "tool": "webview_select_element"
}

// Activate with a longer timeout
{
  "tool": "webview_select_element",
  "timeout": 120000
}

Response

Returns two content items:

Text — Formatted element metadata including:
- Tag name, ID, classes
- CSS selector and XPath
- Bounding rect (position and dimensions)
- HTML attributes
- Text content (truncated to 200 characters)
- Computed CSS styles
- Parent chain (tag, id, classes, dimensions for each ancestor)
Image — A cropped PNG screenshot of just the selected element.

If the user presses Escape or clicks the X button, the tool returns a cancellation message.

How It Works

A translucent blue overlay appears in the Tauri app
As the user moves their cursor, elements are highlighted with a blue outline
The user clicks an element to select it
The overlay is removed and metadata + screenshot are returned
The user can cancel by pressing Escape or clicking the close button

webview_get_pointed_element

Retrieve element metadata for an element the user previously pointed at via Alt+Shift+Click in the Tauri app. This is the passive counterpart to webview_select_element — the user can Alt+Shift+Click elements at any time while using the app, and then later the agent can retrieve the metadata.

Parameters

Name	Type	Required	Description
`windowId`	string	No	Window label to target (defaults to 'main')
`appIdentifier`	string \| number	No	App identifier for multi-app setups

Example

javascript

// Get the element the user Alt+Shift+Clicked
{
  "tool": "webview_get_pointed_element"
}

Response

If the user has Alt+Shift+Clicked an element, returns the same content as webview_select_element:

Text — Formatted element metadata (tag, id, classes, selector, styles, parent chain, etc.)
Image — A cropped PNG screenshot of the element.

If no element has been pointed, returns an instruction message telling the user to Alt+Shift+Click an element first.

When to Use Which

webview_select_element — The agent initiates the picker. Best when the AI wants to ask the user to select something.
webview_get_pointed_element — The user initiates the selection. Best when the user says "look at this element I pointed at".

UI Automation ​

Multi-Window Support ​

Discovering Windows ​

Getting Window Info ​

Resizing Windows ​

Targeting a Specific Window ​

driver_session ​

Parameters ​

Actions ​

Connection Strategy ​

Example ​

Response ​

Environment Variables ​

Remote Device Setup ​

Native Windows Dialogs ​

native_dialog_snapshot ​

native_dialog_interact ​

webview_find_element ​

Parameters ​

Example ​

Response ​

read_logs ​

Parameters ​

Sources ​

Example ​

Response ​

webview_dom_snapshot ​

Parameters ​

Snapshot Types ​

Accessibility Snapshot Format ​

Structure Snapshot Format ​

Element References ​

Example ​

Scoped Snapshots ​

Response ​

webview_select_element ​

Parameters ​

Example ​

Response ​

How It Works ​

webview_get_pointed_element ​

Parameters ​

Example ​

Response ​

UI Automation

Multi-Window Support

Discovering Windows

Getting Window Info

Resizing Windows

Targeting a Specific Window

driver_session

Parameters

Actions

Connection Strategy

Example

Response

Environment Variables

Remote Device Setup

Native Windows Dialogs

native_dialog_snapshot

native_dialog_interact

webview_find_element

Parameters

Example

Response

read_logs

Parameters

Sources

Example

Response

webview_dom_snapshot

Parameters

Snapshot Types

Accessibility Snapshot Format

Structure Snapshot Format

Element References

Example

Scoped Snapshots

Response

webview_select_element

Parameters

Example

Response

How It Works

webview_get_pointed_element

Parameters

Example

Response