Skip to main content

​
Agent default

Agent sessions run trusted development tests, changed gates, typecheck/lint, and builds locally by default, broadening only when the touched contract requires it. Never execute untrusted repository tooling locally. Use Crabbox when the environment is part of the proof: clean-machine, install/package, Docker, E2E, live, desktop, or cross-platform work, or when the operator explicitly requests remote proof. Do not use Crabbox merely as generic compute offload. The configured Testbox workflow hydrates credentials, so untrusted contributor or fork code must use secretless fork CI or sanitized direct AWS Crabbox instead. Do not pre-warm for anticipated work. Acquire the backend lazily when the first environment-sensitive command is ready, reuse the returned tbx_... id for later remote commands, sync the current checkout on every run, and stop it before handoff. After the first successful reuse, the wrapper records the lease’s base, dependency, and Testbox workflow fingerprint under .crabbox/testbox-leases/. Source-only edits keep reusing the warmed box. A changed merge base, lockfile, package-manager input, wrapper, or Testbox workflow fails closed and requires a fresh lease. Every run still syncs the current checkout. OPENCLAW_TESTBOX_ALLOW_STALE=1 is only for intentional diagnostics, not release proof. Local test commands below are the normal trusted development path. Keep proof proportional to the touched contract. For untrusted proof, lazily warm with --provider aws. Every run must set CRABBOX_ENV_ALLOW=CI, pass --provider aws --no-hydrate, and use a fresh temporary remote HOME before installing dependencies or running tests. Use a newly warmed lease dedicated to that untrusted source; never reuse a trusted or previously hydrated lease. Launch an installed trusted Crabbox binary from a clean trusted main checkout and fetch only the remote PR with --fresh-pr; never execute the untrusted checkout’s wrapper or config locally. Unset CRABBOX_AWS_INSTANCE_PROFILE and fail closed unless resolved aws.instanceProfile is empty. Before any install/test, use trusted absolute-path tools to require an IMDSv2 token, prove the IAM credentials endpoint returns 404, and verify remote git rev-parse HEAD equals the full reviewed PR head SHA. Bind the lease to that SHA and stop/rewarm when the head changes. Upload trusted scripts/crabbox-untrusted-bootstrap.sh from clean main alongside --fresh-pr; it installs pinned Node/pnpm, verifies the SHA and package-manager pin, isolates HOME, installs dependencies, then executes the requested test. If the broker cannot prove no role or no remote PR exists, use secretless fork CI. Do not use hydrate-github, --no-sync, or a credential-hydrated Testbox workflow. Unset all CRABBOX_TAILSCALE* overrides, force --network public --tailscale=false, clear exit-node/LAN flags, and require crabbox inspect to report public networking with no Tailscale state before uploading any script.

​
Routine local order

  1. pnpm test:changed for changed-scope Vitest proof.
  2. pnpm test <path-or-filter> for one file, directory, or explicit target.
  3. pnpm test only when you intentionally need the full local Vitest suite.
Codex and other linked/sparse worktrees can run local tests and checks. When the dependency install is ready, use the normal commands above. If pnpm would reconcile a shared install, use the direct Node harnesses to bypass that package-manager preflight:
  • Bounded focused proof with ready dependencies: node scripts/run-vitest.mjs <path-or-filter>.
  • Changed typecheck/lint/guard proof: node scripts/check-changed.mjs.
For remote-environment proof, invoke node scripts/crabbox-wrapper.mjs directly. Avoid local pnpm crabbox:run in linked worktrees because pnpm may reconcile dependencies before the remote wrapper starts.

​
Core commands

Test wrapper runs end with a short [test] passed|failed|skipped ... in ... summary; Vitest’s own duration line stays the per-shard detail.

​
Shared test state and process helpers

  • src/test-utils/openclaw-test-state.ts: use from Vitest when a test needs an isolated HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, config fixture, workspace, agent dir, or auth-profile store.
  • pnpm test:env-mutations:report: non-blocking report of tests/harnesses that mutate HOME, OPENCLAW_STATE_DIR, OPENCLAW_CONFIG_PATH, OPENCLAW_WORKSPACE_DIR, or related env keys directly. Use it to find migration candidates for the shared test-state helper.
  • test/helpers/openclaw-test-instance.ts: process-level E2E tests needing a running Gateway, CLI env, log capture, and cleanup in one place.
  • Docker/Bash E2E lanes that source scripts/lib/docker-e2e-image.sh can pass docker_e2e_test_state_shell_b64 <label> <scenario> into the container and decode it with scripts/lib/openclaw-e2e-instance.sh; multi-home scripts can pass docker_e2e_test_state_function_b64 and call openclaw_test_state_create <label> <scenario> in each flow. node --import tsx scripts/lib/openclaw-test-state.mts -- create --label <name> --scenario <name> --env-file <path> --json writes a sourceable host env file (the -- before create keeps newer Node runtimes from treating --env-file as a Node flag). Lanes that launch a Gateway can source scripts/lib/openclaw-e2e-instance.sh for entrypoint resolution, mock OpenAI startup, foreground/background launch, readiness probes, state env export, log dumps, and process cleanup.

​
Control UI, TUI, and extension lanes

  • Control UI mocked E2E: pnpm test:ui:e2e runs the Vitest + Playwright lane that starts the Vite Control UI and drives a real Chromium page against a mocked Gateway WebSocket. Tests live in ui/src/**/*.e2e.test.ts; shared mocks/controls live in ui/src/test-helpers/control-ui-e2e.ts. pnpm test:e2e includes this lane. Use Testbox/Crabbox only when clean Linux/browser parity is part of the proof. In a linked worktree, node scripts/run-vitest.mjs run --config test/vitest/vitest.ui-e2e.config.ts --configLoader runner ui/src/e2e/chat-flow.messaging.e2e.test.ts avoids pnpm dependency reconciliation for a targeted local run.
  • TUI PTY tests: node scripts/run-vitest.mjs run --config test/vitest/vitest.tui-pty.config.ts runs the fast fake-backend PTY lane. OPENCLAW_TUI_PTY_INCLUDE_LOCAL=1 or pnpm tui:pty:test:watch --mode local runs the slower tui --local smoke, which mocks only the external model endpoint. CI also sets OPENCLAW_TUI_PTY_USE_BUILT_CLI=1 after building dist/; use that flag only when exact-head built artifacts already exist. Assert stable visible text or fixture calls, not raw ANSI snapshots.
  • pnpm test:extensions and pnpm test extensions run all extension/plugin shards. Heavy channel plugins, the browser plugin, and OpenAI run as dedicated shards; other plugin groups stay batched. pnpm test extensions/<id> runs one bundled plugin lane.
  • Source files with sibling tests map to that sibling before falling back to wider directory globs. Helper edits under src/channels/plugins/contracts/test-helpers, src/plugin-sdk/test-helpers, and src/plugins/contracts use a local import graph to run importing tests instead of broad-running every shard when the dependency path is precise.
  • Contract directory targets fan out to their contract lanes: pnpm test src/channels/plugins/contracts runs the four channel contract configs and pnpm test src/plugins/contracts runs the plugin contracts config, since the generic channels/plugins projects exclude contracts/**.
  • auto-reply splits into three dedicated configs (core, top-level, reply) so the reply harness does not dominate the lighter top-level status/token/helper tests.
  • Selected plugin-sdk and commands test files route through dedicated light lanes that keep only test/setup.ts, leaving runtime-heavy cases on their existing lanes.
  • Base Vitest config defaults to pool: "threads" and isolate: false, with the shared non-isolated runner enabled across repo configs.
  • pnpm test:channels runs vitest.channels.config.ts.

​
Gateway and E2E

  • Gateway tests are included in the untargeted pnpm test full suite; run them alone with pnpm test:gateway.
  • pnpm test:e2e: repo E2E aggregate = pnpm test:e2e:gateway && pnpm test:ui:e2e.
  • pnpm test:e2e:gateway: gateway end-to-end smoke tests (multi-instance WS/HTTP/node pairing). Defaults to threads + isolate: false with one worker in vitest.e2e.config.ts; opt into parallelism with OPENCLAW_E2E_WORKERS=<n> (capped at 16), and enable verbose logs with OPENCLAW_E2E_VERBOSE=1.
  • pnpm test:live: provider live tests (Claude/Minimax/DeepSeek/z.ai/etc, gated by *.live.test.ts). Requires API keys and LIVE=1 (or OPENCLAW_LIVE_TEST=1) to unskip; verbose output with OPENCLAW_LIVE_TEST_QUIET=0.

​
Full Docker suite (pnpm test:docker:all)

Builds the shared live-test image, packs OpenClaw once as an npm tarball, builds/reuses a bare Node/Git runner image plus a functional image that installs that tarball into /app, then runs Docker smoke lanes through a weighted scheduler. scripts/package-openclaw-for-docker.mjs is the stable local/CI package packer entrypoint and validates the tarball plus dist/postinstall-inventory.json before Docker consumes it.
  • Bare image (OPENCLAW_DOCKER_E2E_BARE_IMAGE): installer/update/plugin-dependency lanes; mounts the prebuilt tarball instead of copied repo sources.
  • Functional image (OPENCLAW_DOCKER_E2E_FUNCTIONAL_IMAGE): normal built-app functionality lanes.
  • Lane definitions: scripts/lib/docker-e2e-scenarios.mts. Planner: scripts/lib/docker-e2e-plan.mts. Executor: scripts/test-docker-all.mjs.
  • node scripts/test-docker-all.mjs --plan-json emits the scheduler-owned CI plan (lanes, image kinds, package/live-image needs, state scenarios, credential checks) without building or running Docker.
Scheduling knobs (env vars, defaults in parentheses): Env var pattern for resource caps is OPENCLAW_DOCKER_ALL_<RESOURCE>_LIMIT (resource name uppercased, non-alphanumerics collapsed to _). Other behavior: the runner preflights Docker by default, cleans stale OpenClaw E2E containers, shares provider CLI tool caches between compatible lanes, and stops scheduling new pooled lanes after the first failure unless OPENCLAW_DOCKER_ALL_FAIL_FAST=0 is set. If one lane exceeds the effective weight/resource cap on a low-parallelism host, it can still start from an empty pool and run alone until it releases capacity. Per-lane logs, summary.json, failures.json, and phase timings write under .artifacts/docker-tests/<run-id>/; use pnpm test:docker:timings <summary.json> to inspect slow lanes and pnpm test:docker:rerun <run-id|summary.json|failures.json> to print cheap targeted rerun commands.

​
Notable Docker lanes

​
Sandbox compatibility lanes

​
Local PR gate

For local PR land/gate checks, run:
  • pnpm check:changed
  • pnpm check
  • pnpm check:test-types
  • pnpm build
  • pnpm test
  • pnpm check:docs
If pnpm test flakes on a loaded host, rerun once before treating it as a regression, then isolate with pnpm test <path/to/test>. For memory-constrained hosts:
  • OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test
  • OPENCLAW_VITEST_FS_MODULE_CACHE_PATH=/tmp/openclaw-vitest-cache pnpm test:changed

​
Test performance tooling

  • pnpm test:perf:imports: enables Vitest import-duration + import-breakdown reporting, while still using scoped lane routing for explicit file/directory targets. pnpm test:perf:imports:changed scopes the same profiling to files changed since origin/main.
  • pnpm test:perf:changed:bench -- --ref <git-ref> benchmarks the routed changed-mode path against the native root-project run for the same committed git diff; pnpm test:perf:changed:bench -- --worktree benchmarks the current worktree change set without committing first.
  • pnpm test:perf:profile:main writes a CPU profile for the Vitest main thread (.artifacts/vitest-main-profile); pnpm test:perf:profile:runner writes CPU + heap profiles for the unit runner (.artifacts/vitest-runner-profile).
  • pnpm test:perf:groups --full-suite --allow-failures --output .artifacts/test-perf/baseline-before.json: runs every full-suite Vitest leaf config serially and writes grouped duration data plus per-config JSON/log artifacts. Full-suite reports isolate files by default so retained module graphs and GC pauses from earlier files are not charged to later assertions; pass -- --no-isolate only when intentionally profiling shared-worker accumulation. pnpm test:perf:groups:compare .artifacts/test-perf/baseline-before.json .artifacts/test-perf/after-agent.json compares grouped reports after a performance-focused change.
  • Full, extension, and include-pattern shard runs update local timing data in .artifacts/vitest-shard-timings.json; later whole-config runs use those timings to balance slow and fast shards. Include-pattern CI shards append the shard name to the timing key, which keeps filtered shard timings visible without replacing whole-config timing data. Set OPENCLAW_TEST_PROJECTS_TIMINGS=0 to ignore the local timing artifact.

​
Benchmarks

Optional env: MINIMAX_API_KEY, MINIMAX_BASE_URL, MINIMAX_MODEL, ANTHROPIC_API_KEY. Default prompt: “Reply with a single word: ok. No punctuation or extra text.”
Presets:
  • startup: --version, --help, health, health --json, status --json, status
  • real: health, status, status --json, sessions, sessions --json, tasks --json, tasks list --json, tasks audit --json, agents list --json, gateway status, gateway status --json, gateway health --json, config get gateway.port
  • all: both presets combined
Output includes sampleCount, avg, p50, p95, min/max, exit-code/signal distribution, and max RSS per command. --cpu-prof-dir / --heap-prof-dir write V8 profiles per run.Saved output: pnpm test:startup:bench:smoke writes .artifacts/cli-startup-bench-smoke.json; pnpm test:startup:bench:save writes .artifacts/cli-startup-bench-all.json (runs=5 warmup=1). Checked-in fixture: test/fixtures/cli-startup-bench.json, refreshed by pnpm test:startup:bench:update, compared by pnpm test:startup:bench:check.
Defaults to the built CLI entry at dist/entry.js; run pnpm build first. Pass --entry scripts/run-node.mjs to measure the source runner instead, and keep those results separate from built-entry baselines.
Case ids: default, skipChannels (channel startup skipped), oneInternalHook, allInternalHooks, fiftyPlugins (50 manifest plugins), fiftyStartupLazyPlugins (50 startup-lazy manifest plugins).Output includes first process output, /healthz, /readyz, HTTP listen log time, Gateway ready log time, CPU time, CPU core ratio, max RSS, heap, startup trace metrics, event-loop delay, and plugin lookup-table detail metrics. The script sets OPENCLAW_GATEWAY_STARTUP_TRACE=1 in the child Gateway environment./healthz is liveness (HTTP server can answer). /readyz is usable readiness (startup plugin sidecars, channels, and ready-critical post-attach work have settled). Startup hooks dispatch asynchronously and are not part of the readiness guarantee. Ready log time is the Gateway’s internal timestamp, useful for process-side attribution but not a substitute for the external /readyz probe.Use JSON output or --output when comparing changes. Use --cpu-prof-dir only after trace output points at import, compile, or CPU-bound work that phase timings alone cannot explain.
macOS and Linux only (uses SIGUSR1 for in-process restarts; fails immediately on Windows). Same built-entry default and --entry scripts/run-node.mjs override as gateway startup above.
Case ids: skipChannels, skipChannelsAcpxProbe (ACPX startup probe on), skipChannelsNoAcpxProbe (probe off), default, fiftyPlugins.Output includes next /healthz, next /readyz, downtime, restart ready timing, CPU, RSS, startup trace metrics for the replacement process, and restart trace metrics for signal handling, active-work drain, close phases, next start, ready timing, and memory snapshots. The script sets OPENCLAW_GATEWAY_STARTUP_TRACE=1 and OPENCLAW_GATEWAY_RESTART_TRACE=1.Use this benchmark when a change touches restart signaling, close handlers, startup-after-restart, sidecar shutdown, service handoff, or readiness after restart. Start with skipChannels to isolate Gateway mechanics from channel startup; use default or plugin-heavy cases only after the narrow case explains the restart path. Trace metrics are attribution hints, not verdicts — judge a restart change from multiple samples, the matching owner span, /healthz//readyz behavior, and the user-visible restart contract.

​
Onboarding E2E (Docker)

Optional; only needed for containerized onboarding smoke tests. Full cold-start flow in a clean Linux container:
Drives the interactive wizard via a pseudo-tty, verifies config/workspace/session state, then starts the gateway and runs openclaw health.

​
QR import smoke (Docker)

Ensures the maintained QR runtime helper loads under the supported Docker Node runtimes (Node 24 default, Node 22 compatible):