Web fetch

​

Quick start

await web_fetch({ url: "https://example.com/article" });

​

Tool parameters

​

Result

• Request metadata: url, finalUrl, status, extractMode, and extractor
• Optional response metadata: contentType, title, and warning (omitted when absent)
• Wrapped content metadata: externalContent, truncated, length, rawLength, fetchedAt, tookMs, and text
• Optional cached: true on a cache hit
• Optional spill: { path, chars, truncated? } when truncated content was written to a private temporary file; truncated is present only when that file contains partial source content

​

How it works

​

Progress updates

Fetching page content...

​

Config

{
  tools: {
    web: {
      fetch: {
        enabled: true, // default: true
        provider: "firecrawl", // optional; omit for auto-detect
        maxChars: 20000, // default output chars; capped by maxCharsCap
        maxCharsCap: 20000, // hard cap for maxChars param
        maxResponseBytes: 750000, // max download size before truncation (32000-10000000)
        timeoutSeconds: 30,
        cacheTtlMinutes: 15,
        maxRedirects: 3,
        useTrustedEnvProxy: false, // let a trusted HTTP(S) env proxy resolve DNS
        readability: true, // use Readability extraction
        userAgent: "Mozilla/5.0 ...", // override User-Agent
        headers: {
          // optional; every value is treated as sensitive
          "X-Routing-Target": "staging",
        },
        ssrfPolicy: {
          dangerouslyAllowPrivateNetwork: false, // broad private-network opt-in; keep false by default
          allowedHostnames: ["internal.example"], // narrow exact host exception
          allowRfc2544BenchmarkRange: true, // opt-in for trusted fake-IP proxies using 198.18.0.0/15
          allowIpv6UniqueLocalRange: true, // opt-in for trusted fake-IP proxies using fc00::/7
        },
      },
    },
  },
}

​

Firecrawl fallback

{
  tools: {
    web: {
      fetch: {
        provider: "firecrawl", // optional; omit for auto-detect from available credentials
      },
    },
  },
  plugins: {
    entries: {
      firecrawl: {
        enabled: true,
        config: {
          webFetch: {
            // apiKey: "fc-...", // optional; omit for keyless starter access
            baseUrl: "https://api.firecrawl.dev",
            onlyMainContent: true,
            maxAgeMs: 172800000, // cache duration (2 days)
            timeoutSeconds: 60,
          },
        },
      },
    },
  },
}

• tools.web.fetch.provider selects the fetch fallback provider explicitly.
• If provider is omitted, OpenClaw auto-detects the first ready web-fetch provider from configured credentials. Non-sandboxed web_fetch can use installed plugins that declare contracts.webFetchProviders and register a matching provider at runtime. The official Firecrawl plugin provides this fallback today.
• Sandboxed web_fetch calls allow bundled providers plus installed providers whose official npm or ClawHub provenance is verified. Today that permits the official Firecrawl plugin; third-party external fetch plugins stay excluded.
• If Readability is disabled, web_fetch skips straight to the selected provider fallback. If no provider is available, it fails closed.

​

Custom request headers

{
  tools: {
    web: {
      fetch: {
        headers: {
          "X-Routing-Target": "${WEB_FETCH_ROUTING_TARGET}",
        },
      },
    },
  },
}

• Values are plain strings and support ${VAR} environment substitution like any other config string. Structured SecretRef values are not accepted.
• Headers apply only to the direct web_fetch request. Provider fallbacks such as Firecrawl call their own API and never receive these headers.
• Entries are validated when the request is built, not at config load, so one bad entry is dropped while the rest still apply. Config load stays permissive on purpose: a fail-closed validation error over a single header-name typo would disable the whole surface. Every dropped entry is logged by name.
• Dropped names:
  • Accept, Accept-Language, and User-Agent belong to the fetch and readability contract. Use tools.web.fetch.userAgent for the user agent.
  • Framing and hop-by-hop names such as Content-Length, Transfer-Encoding, Connection, and Upgrade, which a request either rejects outright or ignores.
  • Names that are not valid HTTP tokens, such as "X Routing Target".
• Dropped values: bytes a request cannot carry (CR, LF, NUL, or any character above U+00FF). Missing environment variables are reported by config loading; the global $${VAR} escape remains available when the literal ${VAR} text is intentional.
• Two entries whose names differ only in case collapse to the later entry, so a request never carries a comma-joined value the receiving gateway cannot parse. The dropped name is logged without either value. If the later entry is unusable, neither value is sent.
• Rejection happens before the cache key is computed, so the key always matches the bytes actually sent: changing a header that is really sent partitions the fetch cache, while adding one that gets dropped does not.
• When a redirect crosses origins, the guarded-fetch safe allowlist is applied. Routing headers outside that list are dropped; standard safe headers such as Cache-Control, Content-Type, and Range are preserved.

​

Trusted env proxy

​

Limits and safety

• maxChars is clamped to tools.web.fetch.maxCharsCap (default 20000)
• Response body is capped at maxResponseBytes (default 750000, clamped to 32000-10000000) before parsing; oversized responses are truncated with a warning
• Private/internal hostnames are blocked
• tools.web.fetch.ssrfPolicy.allowedHostnames allows exact trusted hosts while leaving other private/internal targets blocked
• tools.web.fetch.ssrfPolicy.dangerouslyAllowPrivateNetwork broadly permits private-network targets; enable it only when model-selected URLs are trusted in this deployment
• tools.web.fetch.ssrfPolicy.allowRfc2544BenchmarkRange and tools.web.fetch.ssrfPolicy.allowIpv6UniqueLocalRange are narrow opt-ins for trusted fake-IP proxy stacks; leave them unset unless your proxy owns those synthetic ranges and enforces its own destination policy
• Redirects are checked and limited by maxRedirects (default 3)
• tools.web.fetch.headers values are redacted from exposed config and debug captures, sent to the initial fetched host, and retained on redirects only when the existing guarded-fetch policy allows them
• useTrustedEnvProxy is an explicit opt-in and should only be enabled for operator-controlled proxies that still enforce outbound policy after DNS resolution
• web_fetch is best-effort — some sites need the Web Browser

​

Tool profiles

{
  tools: {
    allow: ["web_fetch"],
    // or: allow: ["group:web"]  (includes web_fetch, web_search, and x_search)
  },
}

​

Related

• Web Search — search the web with multiple providers
• Web Browser — full browser automation for JS-heavy sites
• Firecrawl — Firecrawl search and scrape tools