Writing and adding tests

Test Temp Directories

Use the shared helpers in test/helpers/temp-dir.ts for test-owned temporary directories so ownership is explicit and cleanup stays in the test lifecycle:

import { afterEach } from "vitest";
import { useAutoCleanupTempDirTracker } from "../helpers/temp-dir.js";

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("openclaw-example-");
  // use workspace
});

useAutoCleanupTempDirTracker(afterEach) intentionally exposes no manual cleanup method - Vitest owns cleanup after each test. Older lower-level helpers (makeTempDir, cleanupTempDirs, createTempDirTracker) still exist for tests that have not migrated; avoid new usage of them and avoid new bare fs.mkdtemp* calls unless a test is explicitly verifying raw temp-dir behavior. When a bare temp dir is genuinely needed, add an auditable allow comment with a reason:

// openclaw-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

node scripts/report-test-temp-creations.mjs reports new bare temp-dir creation and new manual shared-helper usage in added diff lines, without blocking existing cleanup styles. It follows the same test-path classification as scripts/changed-lanes.mjs and skips the shared helper implementation itself. check:changed runs this report for changed test paths as a warning-only CI signal (GitHub warning annotations, not failures).

Agent reliability evals (skills)

We already have a few CI-safe tests that behave like “agent reliability evals”:

Agent admission, run-ID responses, and abort requests through the real Gateway with a mock OpenAI provider (src/gateway/gateway.test.ts).
End-to-end wizard flows that validate session wiring and config effects (src/gateway/gateway.test.ts).

What’s still missing for skills (see Skills):

Decisioning: when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
Compliance: does the agent read SKILL.md before use and follow required steps/args?
Workflow contracts: multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.

Future evals should stay deterministic first:

A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.

Adding regressions (guidance)

When you fix a provider/model issue discovered in live:

Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation)
If it’s inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars
Prefer targeting the smallest layer that catches the bug:
- provider request conversion/replay bug -> direct models test
- gateway session/history/tool pipeline bug -> gateway live smoke or CI-safe gateway mock test
SecretRef traversal guardrail:
- src/secrets/exec-secret-ref-id-parity.test.ts derives one sampled target per SecretRef class from registry metadata (listSecretTargetRegistryEntries()), then asserts traversal-segment exec ids are rejected.
- If you add a new includeInPlan SecretRef target family in src/secrets/target-registry-data.ts, update classifyTargetClass in that test. The test intentionally fails on unclassified target ids so new classes cannot be skipped silently.

Start here

FAQ

Testing

Diagnostics

Community and meta

Writing and adding tests

Test Temp Directories

Agent reliability evals (skills)

Adding regressions (guidance)

​Test Temp Directories

​Agent reliability evals (skills)

​Adding regressions (guidance)

Test Temp Directories

Agent reliability evals (skills)

Adding regressions (guidance)