Audio and voice notes

​

What it does

1. Locates the first audio attachment (local path or URL) and downloads it if needed.
2. Enforces maxBytes before sending to each model entry.
3. Runs the first eligible model entry in order (provider or CLI); if an entry fails or skips (size/timeout), the next entry is tried.
4. On success, replaces Body with an [Audio] block and sets {{Transcript}}.

​

Auto-detection (default)

1. Active reply model, when its provider supports audio understanding.
2. Configured provider auth — any models.providers.* entry with auth available for a provider that supports audio transcription. This is checked before local CLIs, so a configured API key always wins over a local binary on PATH. Provider priority when multiple are configured: Groq, OpenAI, xAI, Deepgram, Google, SenseAudio, ElevenLabs, Mistral.
3. Local CLIs (only if no provider auth resolved), checked in this order:
   • sherpa-onnx-offline (requires SHERPA_ONNX_MODEL_DIR with tokens.txt, encoder.onnx, decoder.onnx, and joiner.onnx)
   • whisper-cli (from whisper-cpp; uses WHISPER_CPP_MODEL or a bundled tiny model)
   • whisper (Python CLI; downloads models automatically)

​

Config examples

​

Provider + CLI fallback (OpenAI + Whisper CLI)

{
  tools: {
    media: {
      audio: {
        enabled: true,
        maxBytes: 20971520,
        models: [
          { provider: "openai", model: "gpt-4o-transcribe" },
          {
            type: "cli",
            command: "whisper",
            args: ["--model", "base", "{{MediaPath}}"],
            timeoutSeconds: 45,
          },
        ],
      },
    },
  },
}

​

Provider-only with scope gating

{
  tools: {
    media: {
      audio: {
        enabled: true,
        scope: {
          default: "allow",
          rules: [{ action: "deny", match: { chatType: "group" } }],
        },
        models: [{ provider: "openai", model: "gpt-4o-transcribe" }],
      },
    },
  },
}

​

Provider-only (Deepgram)

{
  tools: {
    media: {
      audio: {
        enabled: true,
        models: [{ provider: "deepgram", model: "nova-3" }],
      },
    },
  },
}

​

Provider-only (Mistral Voxtral)

{
  tools: {
    media: {
      audio: {
        enabled: true,
        models: [{ provider: "mistral", model: "voxtral-mini-latest" }],
      },
    },
  },
}

​

Provider-only (SenseAudio)

{
  tools: {
    media: {
      audio: {
        enabled: true,
        models: [{ provider: "senseaudio", model: "senseaudio-asr-pro-1.5-260319" }],
      },
    },
  },
}

​

Echo transcript to chat (opt-in)

{
  tools: {
    media: {
      audio: {
        enabled: true,
        echoTranscript: true, // default is false
        echoFormat: '📝 "{transcript}"', // optional, supports {transcript}
        models: [{ provider: "openai", model: "gpt-4o-transcribe" }],
      },
    },
  },
}

​

Notes and limits

• Provider auth follows the standard model auth order (auth profiles, env vars, models.providers.*.apiKey).
• Groq setup details: Groq.
• Deepgram picks up DEEPGRAM_API_KEY when provider: "deepgram" is used. Setup details: Deepgram.
• Mistral setup details: Mistral.
• SenseAudio picks up SENSEAUDIO_API_KEY when provider: "senseaudio" is used. Setup details: SenseAudio.
• Audio providers can override baseUrl, headers, and providerOptions via tools.media.audio.
• Default size cap is 20MB (tools.media.audio.maxBytes). Oversize audio is skipped for that model and the next entry is tried.
• Audio files below 1024 bytes are skipped before provider/CLI transcription.
• Default maxChars for audio is unset (full transcript). Set tools.media.audio.maxChars or a per-entry maxChars to trim output.
• OpenAI auto-detect default is gpt-4o-transcribe; set model: "gpt-4o-mini-transcribe" for a cheaper/faster option.
• Use tools.media.audio.attachments to process multiple voice notes (mode: "all" plus maxAttachments, default 1).
• Transcript is available to templates as {{Transcript}}.
• tools.media.audio.echoTranscript is off by default; enable it to send a transcript confirmation back to the originating chat before agent processing.
• tools.media.audio.echoFormat customizes the echo text (placeholder: {transcript}; default 📝 "{transcript}").
• CLI stdout is capped at 5MB; keep CLI output concise.
• CLI args should use {{MediaPath}} for the local audio file path. Run openclaw doctor --fix to migrate deprecated {input} placeholders from older audio.transcription.command configs (retired key: audio.transcription, replaced by tools.media.audio.models).

​

Proxy environment support

• HTTPS_PROXY / https_proxy
• HTTP_PROXY / http_proxy
• ALL_PROXY / all_proxy

​

Mention detection in groups

1. If a voice message has no text body and the group requires mentions, OpenClaw performs a preflight transcription of the first audio attachment.
2. The transcript is checked for mention patterns (for example @BotName, emoji triggers).
3. If a mention is found, the message proceeds through the full reply pipeline.

• Set channels.telegram.groups.<chatId>.disableAudioPreflight: true to skip preflight transcript mention checks for that group.
• Set channels.telegram.groups.<chatId>.topics.<threadId>.disableAudioPreflight to override per-topic (true to skip, false to force-enable).
• Default is false (preflight enabled when mention-gated conditions match).

​

Gotchas

• Scope rules use first-match-wins; chatType is normalized to direct, group, or channel.
• Ensure your CLI exits 0 and prints plain text; JSON output needs to be massaged via jq -r .text.
• For parakeet-mlx, if you pass --output-dir, OpenClaw reads <output-dir>/<media-basename>.txt when --output-format is txt (or omitted); non-txt output formats fall back to stdout parsing.
• Keep timeouts reasonable (timeoutSeconds, default 60s) to avoid blocking the reply queue.
• Preflight transcription only processes the first audio attachment for mention detection. Additional audio attachments are processed during the main media-understanding phase.

​

Related

• Media understanding
• Talk mode
• Voice wake