Media understanding

OpenClaw can summarize inbound media (image/audio/video) before the reply pipeline runs, so command parsing and routing work off short text instead of raw bytes. Understanding auto-detects local tools or provider keys, or you can configure explicit models. Original media is always delivered to the model as usual; when understanding fails or is disabled, the reply flow continues unchanged. Vendor plugins register capability metadata (which provider supports which media type, default model, priority). OpenClaw core owns the shared tools.media config, fallback order, and reply-pipeline integration.

How it works

Collect attachments

Collect inbound attachments (MediaPaths, MediaUrls, MediaTypes).

Select per capability

For each enabled capability (image/audio/video), select attachments per the attachments policy (default: first attachment only).

Choose a model

Pick the first eligible model entry (size + capability + auth available).

Fall back on failure

If a model errors, times out, or the media exceeds maxBytes, try the next entry.

Apply on success

Body becomes an [Image], [Audio], or [Video] block. Audio also sets {{Transcript}}; command parsing uses caption text when present, otherwise the transcript. Captions are preserved as User text: inside the block.

Config

tools.media holds a shared model list plus per-capability overrides:

{
  tools: {
    media: {
      concurrency: 2, // max concurrent capability runs (default)
      models: [
        /* shared list, gate with capabilities */
      ],
      image: {
        /* optional overrides */
      },
      audio: {
        /* optional overrides */
        echoTranscript: true,
        echoFormat: '📝 "{transcript}"',
      },
      video: {
        /* optional overrides */
      },
    },
  },
}

Per-capability (image/audio/video) keys:

Key	Type	Default	Notes
`enabled`	`boolean`	auto (`false` disables)	Set `false` to turn off auto-detect for this capability
`models`	array	none	Preferred before the shared `tools.media.models` list
`prompt`	`string`	`"Describe the {media}."` (+ maxChars guidance)	Image/video only by default
`maxChars`	`number`	`500` (image/video), unset (audio)	Output is trimmed if the model returns more
`maxBytes`	`number`	image `10485760`, audio `20971520`, video `52428800`	Oversized media skips to the next model
`timeoutSeconds`	`number`	`60` (image/audio), `120` (video)
`language`	`string`	unset	Audio transcription hint
`baseUrl`/`headers`/`providerOptions`/`request`	-	-	Provider request overrides; see Tools and custom providers
`attachments`	object	`{ mode: "first", maxAttachments: 1 }`	See Attachment policy
`scope`	object	unset	Gate by channel/chatType/keyPrefix
`echoTranscript`	`boolean`	`false`	Audio only: echo the transcript back to the chat before agent processing
`echoFormat`	`string`	`'📝 "{transcript}"'`	Audio only: `{transcript}` placeholder

Deepgram-specific options go under providerOptions.deepgram (the top-level deepgram: { detectLanguage, punctuate, smartFormat } field is deprecated but still read).

Model entries

Each models[] entry is a provider entry (default) or a CLI entry:

Provider entry
CLI entry

{
  type: "provider", // default if omitted
  provider: "openai",
  model: "gpt-5.5",
  prompt: "Describe the image in <= 500 chars.",
  maxChars: 500,
  maxBytes: 10485760,
  timeoutSeconds: 60,
  capabilities: ["image"], // optional, for multi-modal shared entries
  profile: "vision-profile",
  preferredProfile: "vision-fallback",
}

{
  type: "cli",
  command: "gemini",
  args: [
    "-m",
    "gemini-3-flash",
    "--allowed-tools",
    "read_file",
    "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
  ],
  maxChars: 500,
  maxBytes: 52428800,
  timeoutSeconds: 120,
  capabilities: ["video", "image"],
}

CLI templates can also use {{MediaDir}} (directory containing the media file), {{OutputDir}} (scratch dir created for this run), and {{OutputBase}} (scratch file base path, no extension).

Provider credentials

Provider media understanding uses the same auth resolution as normal model calls: auth profiles, environment variables, then models.providers.<providerId>.apiKey. tools.media.*.models[] entries do not accept an inline apiKey field.

{
  models: {
    providers: {
      openai: { apiKey: "<OPENAI_API_KEY>" },
      moonshot: { apiKey: "<MOONSHOT_API_KEY>" },
    },
  },
}

See Tools and custom providers for profiles, env vars, and custom base URLs.

Rules and behavior

Media exceeding maxBytes skips that model and tries the next one.
Audio files under 1024 bytes are treated as empty/corrupt and skipped before transcription; the agent gets a deterministic placeholder transcript instead.
If the active primary image model already supports vision natively, OpenClaw skips the [Image] summary block and passes the original image into the model directly. MiniMax is an exception: minimax, minimax-cn, minimax-portal, and minimax-portal-cn always route image understanding through the plugin-owned MiniMax-VL-01 media provider, even if legacy MiniMax M2.x chat metadata claims image input (only MiniMax-M3 and later are treated as natively vision-capable).
If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded media://inbound/* refs so image/PDF tools or a configured image model can still inspect them instead of losing the attachment.
Explicit openclaw infer image describe --file <path> --model <provider/model> (alias: openclaw capability image describe) runs that image-capable provider/model directly, including Ollama refs such as ollama/qwen2.5vl:7b when a matching image-capable model is configured under models.providers.ollama.models[].
If <capability>.enabled is not false but no models are configured, OpenClaw tries the active reply model when its provider supports the capability.

Auto-detect (default)

When tools.media.<capability>.enabled is not false and no models are configured, OpenClaw tries these in order and stops at the first working option:

Configured image model (image only)

agents.defaults.imageModel primary/fallback refs, unless the active reply model already supports vision natively. Prefer provider/model refs; bare refs are qualified from configured image-capable provider model entries only when the match is unique.

Active reply model

The active reply model, when its provider supports the capability.

Provider auth (audio only, before local CLIs)

Configured models.providers.* entries that support audio are tried before local CLIs. Bundled provider priority order (ties break alphabetically by provider id): Groq/OpenAI → xAI → Deepgram → OpenRouter → Google/SenseAudio → Deepinfra/ElevenLabs → Mistral.

Local CLIs (audio only)

First installed local binary, in this order:

sherpa-onnx-offline (requires SHERPA_ONNX_MODEL_DIR with tokens.txt/encoder.onnx/decoder.onnx/joiner.onnx)
whisper-cli (whisper-cpp; uses WHISPER_CPP_MODEL or a bundled tiny model)
whisper (Python CLI; defaults to the turbo model, downloads automatically)

Provider auth (image/video)

Configured models.providers.* entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin.Bundled provider priority order (ties break alphabetically by provider id):

Image: Anthropic/OpenAI → Google → MiniMax → Deepinfra → MiniMax Portal → Z.AI
Video: Google → Qwen → Moonshot

Antigravity CLI (image/video only)

First installed agy or antigravity binary (override with OPENCLAW_ANTIGRAVITY_CLI), sandboxed against the media’s directory.

To disable auto-detection for a capability:

{
  tools: {
    media: {
      audio: {
        enabled: false,
      },
    },
  },
}

Binary detection is best-effort across macOS/Linux/Windows; ensure the CLI is on PATH (~ is expanded), or set an explicit CLI model entry with a full command path.

Proxy support (audio/video provider calls)

Provider-based audio and video understanding honors standard outbound proxy environment variables, including NO_PROXY/no_proxy bypass rules: HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, https_proxy, http_proxy, all_proxy. Lowercase vars take precedence over uppercase. If none are set, media understanding uses direct egress; if the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch. Image understanding does not go through this proxy path.

Capabilities

Set capabilities on a models[] entry to restrict it to specific media types. For shared lists, OpenClaw infers defaults per bundled provider:

Provider	Capabilities
`openai`, `anthropic`, `minimax`	image
`minimax-portal`	image
`moonshot`	image + video
`openrouter`	image + audio
`google` (Gemini API)	image + audio + video
`qwen`	image + video
`deepinfra`	image + audio
`mistral`	audio
`zai`	image
`groq`, `xai`, `deepgram`, `senseaudio`	audio
Any `models.providers.<id>.models[]` catalog with an image-capable model	image

For CLI entries, set capabilities explicitly to avoid surprising matches; if omitted, the entry is eligible for every capability list it appears in.

Provider support matrix

Capability	Providers	Notes
Image	Anthropic, Codex app-server, Deepinfra, Google, MiniMax, MiniMax Portal, Moonshot, OpenAI, OpenAI Codex OAuth, OpenRouter, Qwen, Z.AI, config providers	Vendor plugins register image support; `openai/` can use API-key or Codex OAuth routing; `codex/` uses a bounded Codex app-server turn; image-capable config providers auto-register.
Audio	Deepgram, Deepinfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter, SenseAudio, xAI	Provider transcription (Whisper/Groq/xAI/Deepgram/OpenRouter STT/Gemini/SenseAudio/Scribe/Voxtral).
Video	Google, Moonshot, Qwen	Provider video understanding via vendor plugins; Qwen video understanding uses the standard DashScope endpoints.

MiniMax note: minimax, minimax-cn, minimax-portal, and minimax-portal-cn image understanding always comes from the plugin-owned MiniMax-VL-01 media provider, even if legacy MiniMax M2.x chat metadata claims image input.

Model selection guidance

Prefer the strongest current-generation model for each media capability when quality and safety matter.
For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
CLI fallbacks (whisper-cli, whisper, gemini) help when provider APIs are unavailable.
parakeet-mlx: with --output-dir, OpenClaw reads <output-dir>/<media-basename>.txt when the output format is txt or unspecified; other formats fall back to stdout.

Attachment policy

Per-capability attachments controls which attachments are processed:

mode

"first" | "all"

default:"first"

Process only the first selected attachment, or all of them.

maxAttachments

number

default:"1"

Cap the number processed.

prefer

"first" | "last" | "path" | "url"

Selection preference among candidate attachments.

When mode: "all", outputs are labeled [Image 1/2], [Audio 2/2], etc.

File-attachment extraction

Extracted file text is wrapped as untrusted external content before it’s appended to the media prompt, using boundary markers like <<<EXTERNAL_UNTRUSTED_CONTENT id="...">>> / <<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>> plus a Source: External metadata line.
This path intentionally omits the long SECURITY NOTICE: banner to keep the media prompt short; the boundary markers and metadata still apply.
A file with no extractable text gets [No extractable text].
If a PDF falls back to rendered page images, OpenClaw forwards those images to vision-capable reply models and keeps the placeholder [PDF content rendered to images] in the file block.

Config examples

Shared models + overrides
Audio + video only
Image only
Multi-modal single entry

{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-5.5", capabilities: ["image"] },
        {
          provider: "google",
          model: "gemini-3-flash-preview",
          capabilities: ["image", "audio", "video"],
        },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["image", "video"],
        },
      ],
      audio: {
        attachments: { mode: "all", maxAttachments: 2 },
      },
      video: {
        maxChars: 500,
      },
    },
  },
}

{
  tools: {
    media: {
      audio: {
        enabled: true,
        models: [
          { provider: "openai", model: "gpt-4o-mini-transcribe" },
          {
            type: "cli",
            command: "whisper",
            args: ["--model", "base", "{{MediaPath}}"],
          },
        ],
      },
      video: {
        enabled: true,
        maxChars: 500,
        models: [
          { provider: "google", model: "gemini-3-flash-preview" },
          {
            type: "cli",
            command: "gemini",
            args: [
              "-m",
              "gemini-3-flash",
              "--allowed-tools",
              "read_file",
              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
            ],
          },
        ],
      },
    },
  },
}

{
  tools: {
    media: {
      image: {
        enabled: true,
        maxBytes: 10485760,
        maxChars: 500,
        models: [
          { provider: "openai", model: "gpt-5.5" },
          { provider: "anthropic", model: "claude-opus-4-8" },
          {
            type: "cli",
            command: "gemini",
            args: [
              "-m",
              "gemini-3-flash",
              "--allowed-tools",
              "read_file",
              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
            ],
          },
        ],
      },
    },
  },
}

{
  tools: {
    media: {
      image: {
        models: [
          {
            provider: "google",
            model: "gemini-3.1-pro-preview",
            capabilities: ["image", "video", "audio"],
          },
        ],
      },
      audio: {
        models: [
          {
            provider: "google",
            model: "gemini-3.1-pro-preview",
            capabilities: ["image", "video", "audio"],
          },
        ],
      },
      video: {
        models: [
          {
            provider: "google",
            model: "gemini-3.1-pro-preview",
            capabilities: ["image", "video", "audio"],
          },
        ],
      },
    },
  },
}

Status output

When media understanding runs, /status includes a per-capability summary line:

📎 Media: image ok (openai/gpt-5.5) · audio skipped (maxBytes)

Notes

Understanding is best-effort. Errors do not block replies.
Attachments are still passed to models even when understanding is disabled.
Use scope to limit where understanding runs (for example, only DMs).

​How it works

​Config

​Model entries

​Provider credentials

​Rules and behavior

​Auto-detect (default)

​Proxy support (audio/video provider calls)

​Capabilities

​Provider support matrix

​Model selection guidance

​Attachment policy

​File-attachment extraction

​Config examples

​Status output

​Notes

​Related

How it works

Config

Model entries

Provider credentials

Rules and behavior

Auto-detect (default)

Proxy support (audio/video provider calls)

Capabilities

Provider support matrix

Model selection guidance

Attachment policy

File-attachment extraction

Config examples

Status output

Notes

Related