Skip to main content
The Matrix QA lane runs the bundled @openclaw/matrix plugin against a disposable Tuwunel homeserver in Docker, with temporary driver, SUT, and observer accounts plus seeded rooms. It is the live transport-real coverage for Matrix. Maintainer-only tooling. Packaged OpenClaw releases omit qa-lab, so openclaw qa only runs from a source checkout, which loads the bundled runner directly with no plugin install step. For broader QA framework context, see QA overview.

​
Quick start

pnpm openclaw qa matrix --profile fast --fail-fast
Plain pnpm openclaw qa matrix runs --profile all and does not stop on first failure. Shard the full inventory across parallel jobs with --profile transport|media|e2ee-smoke|e2ee-deep|e2ee-cli.

​
What the lane does

  1. Provisions a disposable Tuwunel homeserver in Docker (default image ghcr.io/matrix-construct/tuwunel:v1.5.1, server name matrix-qa.test, port 28008) behind a bounded redacting request/response recorder.
  2. Registers three temporary users: driver (sends inbound traffic), sut (the OpenClaw Matrix account under test), observer (third-party traffic capture).
  3. Seeds rooms required by the selected scenarios (main, threading, media, restart, secondary, allowlist, E2EE, verification DM, etc.).
  4. Runs the substrate-neutral matrix-qa-v1 protocol probe against the recorded Tuwunel boundary. Unit tests prove the probe contract with the Matrix protocol fixture; the canonical QA transport adapter host in #99707 owns real Crabline target wiring.
  5. Starts a child OpenClaw gateway with the real Matrix plugin scoped to the SUT account.
  6. Runs scenarios in sequence, observing events through the driver/observer Matrix clients and deriving route/state expectations from the recorded traffic.
  7. Tears down the homeserver, writes report and evidence artifacts, then exits.

​
CLI

pnpm openclaw qa matrix [options]

​
Common flags

FlagDefaultDescription
--profile <profile>allScenario profile. See Profiles.
--fail-fastoffStop after the first failed check or scenario.
--scenario <id>-Run only this scenario. Repeatable. See Scenarios.
--output-dir <path><repo>/.artifacts/qa-e2e/matrix-<timestamp>Where reports, summary, route/state inventory, observed events, and the output log are written. Relative paths resolve against --repo-root.
--repo-root <path>process.cwd()Repository root when invoking from a neutral working directory.
--sut-account <id>sutMatrix account id inside the QA gateway config.

​
Provider flags

The lane uses a real Matrix transport but the model provider is configurable:
FlagDefaultDescription
--provider-mode <mode>live-frontiermock-openai for deterministic mock dispatch or live-frontier for live frontier providers. The legacy alias live-openai still works.
--model <ref>provider defaultPrimary provider/model ref.
--alt-model <ref>provider defaultAlternate provider/model ref where scenarios switch mid-run.
--fastoffEnable provider fast mode where supported.
Matrix QA does not accept --credential-source or --credential-role. The lane provisions disposable users locally; there is no shared credential pool to lease against.

​
Profiles

ProfileUse it for
all (default)Full catalog. Slow but exhaustive.
fastRelease-gate subset that exercises the imperative live transport contract: mention gating, allowlist block, reply shape, restart resume, reaction observation, exec approval metadata delivery, and E2EE basic reply.
transportTransport-level threading, DM, room, autojoin, mention/allowlist, approval, and reaction scenarios.
mediaImage, audio, video, PDF, EPUB attachment coverage.
e2ee-smokeMinimum E2EE coverage: basic encrypted reply, thread follow-up, bootstrap success.
e2ee-deepExhaustive E2EE state-loss, backup, key, and recovery scenarios.
e2ee-cliopenclaw matrix encryption setup and verify * CLI scenarios driven through the QA harness.
The exact mapping lives in extensions/qa-matrix/src/runners/contract/scenario-catalog.ts.

​
Scenarios

The shared Matrix adapter exposes these canonical YAML scenarios through openclaw qa suite --channel-driver live --channel matrix:
  • channel-chat-baseline
  • thread-follow-up
  • thread-isolation
  • thread-reply-override
  • dm-shared-session
  • dm-per-room-session
subagent-thread-spawn remains available through explicit --scenario subagent-thread-spawn selection, but is not part of the default shared Matrix set until live child-completion proof is stable. The remaining imperative scenario id list is the MatrixQaScenarioId union in extensions/qa-matrix/src/runners/contract/scenario-catalog.ts. Categories:
  • threading: matrix-thread-root-preservation, matrix-thread-nested-reply-shape
  • top-level / DM / room: matrix-top-level-reply-shape, matrix-room-*, matrix-dm-*
  • streaming and tool progress: matrix-room-partial-streaming-preview, matrix-room-quiet-streaming-preview, matrix-room-tool-progress-*, matrix-room-block-streaming
  • media: matrix-media-type-coverage, matrix-room-image-understanding-attachment, matrix-attachment-only-ignored, matrix-unsupported-media-safe
  • routing: matrix-room-autojoin-invite, matrix-secondary-room-*
  • reactions: matrix-reaction-*
  • approvals: matrix-approval-* (exec/plugin metadata, chunked fallback, deny reactions, threads, and target: "both" routing)
  • restart and replay: matrix-restart-*, matrix-stale-sync-replay-dedupe, matrix-room-membership-loss, matrix-homeserver-restart-resume, matrix-initial-catchup-then-incremental
  • mention gating, bot-to-bot, and allowlists: matrix-mention-*, matrix-allowbots-*, matrix-allowlist-*, matrix-multi-actor-ordering, matrix-inbound-edit-*, matrix-mxid-prefixed-command-block, matrix-observer-allowlist-override
  • E2EE: matrix-e2ee-* (basic reply, thread follow-up, bootstrap, recovery key lifecycle, state-loss variants, server backup behavior, device hygiene, SAS / QR / DM verification, restart, artifact redaction)
  • E2EE CLI: matrix-e2ee-cli-* (encryption setup, idempotent setup, bootstrap failure, recovery-key lifecycle, multi-account, gateway-reply round-trip, self-verification)
Pass --scenario <id> (repeatable) to run a hand-picked set; combine with --profile all to ignore profile gating.

​
Environment variables

VariableDefaultEffect
OPENCLAW_QA_MATRIX_TIMEOUT_MS1800000 (30 min)Hard upper bound on the entire run.
OPENCLAW_QA_MATRIX_CANARY_TIMEOUT_MS45000Bound for the initial canary reply. Release CI raises this on shared runners so a slow first gateway turn does not fail before scenario coverage starts.
OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS8000Quiet window for negative no-reply assertions. Clamped to <= the run timeout.
OPENCLAW_QA_MATRIX_CLEANUP_TIMEOUT_MS90000Bound for Docker teardown. Failure surfaces include the recovery docker compose ... down --remove-orphans command.
OPENCLAW_QA_MATRIX_TUWUNEL_IMAGEghcr.io/matrix-construct/tuwunel:v1.5.1Override the homeserver image when validating against a different Tuwunel version.
OPENCLAW_QA_MATRIX_PROGRESSon0 silences [matrix-qa] ... progress lines on stderr. 1 forces them on.
OPENCLAW_QA_MATRIX_CAPTURE_CONTENTredacted1 keeps message body and formatted_body in matrix-qa-observed-events.json. Default redacts to keep CI artifacts safe.
OPENCLAW_QA_MATRIX_DISABLE_FORCE_EXIToff1 skips the deterministic process.exit after artifact write. The default forces exit because matrix-js-sdk’s native crypto handles can keep the event loop alive past artifact completion.
OPENCLAW_RUN_NODE_OUTPUT_LOGunsetWhen set by an outer launcher (e.g. scripts/run-node.mjs), Matrix QA reuses that log path instead of starting its own tee.

​
Output artifacts

Written to --output-dir (default <repo>/.artifacts/qa-e2e/matrix-<timestamp> so successive runs do not overwrite each other):
  • matrix-qa-report.md: Markdown protocol report (what passed, failed, was skipped, and why).
  • matrix-qa-summary.json: Structured summary suitable for CI parsing and dashboards.
  • matrix-qa-route-state-manifest.json: Dynamic matrix-qa-v1 inventory keyed by scenario id. It records redacted route/body shapes, request ordering, observed retries, errors, sync-token continuity, and device/key/media/backup state families observed during that run. This is executable evidence, not a checked-in baseline.
  • matrix-qa-observed-events.json: Observed Matrix events from the driver and observer clients. Bodies are redacted unless OPENCLAW_QA_MATRIX_CAPTURE_CONTENT=1; approval metadata is summarized with selected safe fields and a truncated command preview.
  • matrix-qa-output.log: Combined stdout/stderr from the run. If OPENCLAW_RUN_NODE_OUTPUT_LOG is set, the outer launcher’s log is reused instead.

​
Triage tips

  • Run hangs near the end: matrix-js-sdk native crypto handles can outlive the harness. The default forces a clean process.exit after artifact write; if you set OPENCLAW_QA_MATRIX_DISABLE_FORCE_EXIT=1, expect the process to linger.
  • Cleanup error: look for the printed recovery command (a docker compose ... down --remove-orphans invocation) and run it manually to release the homeserver port.
  • Flaky negative-assertion windows in CI: lower OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS (default 8 s) when CI is fast; raise it on slow shared runners.
  • Need redacted bodies for a bug report: rerun with OPENCLAW_QA_MATRIX_CAPTURE_CONTENT=1 and attach matrix-qa-observed-events.json. Treat the resulting artifact as sensitive.
  • Different Tuwunel version: point OPENCLAW_QA_MATRIX_TUWUNEL_IMAGE at the version under test. The lane checks in only the pinned default image.

​
Live transport contract

Matrix is one of three live transport lanes (Matrix, Telegram, Discord) that share a single contract checklist defined in QA overview: Live transport coverage. qa-channel remains the broad synthetic suite and is intentionally not part of that matrix.
  • QA overview: overall QA stack and live transport contract
  • QA Channel: synthetic channel adapter for repo-backed scenarios
  • Testing: running tests and adding QA coverage
  • Matrix: the channel plugin under test