Full Release Validation is the release product-validation umbrella. Most work
happens in child workflows so a failed box can be rerun without restarting the
whole release. Run release preparation before freezing the Code SHA; it
refreshes Control UI locale output when the background bot has not landed it
yet, then enforces the same strict zero-fallback check used by release CI.
Freeze the product-complete pre-changelog commit and its target context as the
Code SHA/ref, and select one trusted workflow commit and context as the
Tooling SHA/ref, then run:
main.
provider also accepts anthropic or minimax for cross-OS onboarding and the
end-to-end agent turn. Regular release/* targets accept only the branch’s final
package version or a matching beta prerelease. Tideclaw alpha validation uses
its exact alpha tag and matching alpha branch. The helper maps beta releases and
exact alpha tags to the beta profile and final versions to stable. Pass
alternate workflow inputs with -f key=value; use -f release_profile=full
only for the broad advisory sweep.
fail_fast defaults to false, so dispatched child workflows finish and expose
independent failures together. In that mode, the parent makes no child
cancellation calls. Pass -f fail_fast=true only when the shorter
first-failure path is preferable; Release Decision then cancels only the exact
still-active child that owns the blocking failure.
After dispatch, the parent writes one immutable
full-release-execution-plan-<run-id> artifact and preserves the same bytes in
an exact run-ID Actions cache. It records selected and
required coverage, gate results, reuse identity, the original parent attempt,
and every exact child run ID, attempt, title, workflow ref, and Tooling SHA.
Decision, Drain, manifest generation, evidence verification, and the final
verifier consume the artifact for their current attempt. Collector retries
restore the immutable cached copy, validate it, and upload the artifact again
for the retry; they never rebuild the plan or redispatch tests. A missing or
evicted cache fails closed, so start a new validation instead of retrying that
stale parent.
Release Decision also repeats canonical reuse-chain validation before a reused
run can pass. The sealed target SHA, evidence SHA, policy, changed-path set,
selected run, root run, source manifest, trusted tooling identity, and child
tuple must all still match.
On a parent retry, final verification selects the newest available Release
Decision and Diagnostic Drain artifacts independently. Both must bind the same
immutable plan and exact child tuple; their source attempts remain recorded in
the artifacts and may differ when only one collector needed a retry.
The helper creates a temporary release-ci/* ref pinned to the Tooling SHA,
passes the Validation SHA as both the candidate ref and expected_sha, and
deletes the temporary ref after successful validation and strict evidence
verification. If Release Decision reports a blocker while Diagnostic Drain is
still collecting failures, the helper exits nonzero immediately and keeps both
temporary refs for reruns and diagnosis. The Validation SHA equals the Code
SHA for product validation or the Release SHA for changelog-only validation; it
is not a third release identity. The workflow rejects malformed or mismatched
expected SHAs before child dispatch. Every child must report the same Tooling
SHA. Pass
-f reuse_evidence=false to force a fresh run. Regular release-branch runs
require --workflow-sha with the recorded full SHA, which must remain reachable
from current origin/main. The helper rejects a pinned Tooling SHA that does
not declare the current release-isolation contract or the expected_sha
dispatch input; it never silently substitutes newer tooling. The workflow never
creates or updates repository refs itself.
The main-lineage requirement above applies to the initial validation tooling
selection. Once release publication binds that Tooling SHA to an exact protected
lightweight release-publish/<12sha>-<provenance-run> tag, the live tag-to-SHA
mapping remains authoritative even when main advances. The suffix records
tag-creation provenance, not the current parent run id. Publication must re-read
that exact tag and revalidate the exact parent run tuple immediately before each
core or plugin npm publish or dist-tag mutation. A missing, moved, annotated, or
wrong-SHA tag, parent mismatch, or disallowed parent state fails closed. Other
privileged writers require their dependent enforcement changes before the
protected-tag publication route is globally complete.
Extended-stable exception
Extended-stable publish requires a run whose workflow and target are both the canonical branch:pnpm ci:full-release or release-ci/*. Publish binds the run’s
branch, head/target SHA, manifest workflowRef, ID, and attempt to the canonical
branch and release commit.
Backport product failures; make the smallest behavior-preserving repair for
frozen-target tooling; retry provider, approval, or runner failures without a
source change. Any branch change needs a complete new run. Do not omit required
package, installer, update, channel, or live behavior because the target is old.
For a regular release, when the Code SHA is green, generate and commit only
CHANGELOG.md. This new commit is the Release SHA. Run the same helper for
the Release SHA. Product evidence is reused only when GitHub proves the Release
SHA descends from the Code SHA and the complete changed path set is exactly
CHANGELOG.md; npm preflight and package/install acceptance still run on the
Release SHA.
The conceptual phases map to current inputs:
beta-publish:release_profile=beta,run_release_soak=falsepostpublish-confidence: exact published package plusrun_release_soak=trueor explicit focused groupsstable-publish:release_profile=stable
all excludes broad live/E2E soak and QA-live lanes. Stable and
full always run the soak. Stable publication rejects a validation manifest
without soak and blocking product-performance evidence.
Package Acceptance normally builds the candidate tarball from the resolved
ref, including full-SHA runs dispatched with pnpm ci:full-release. After a
beta publish, pass release_package_spec=openclaw@YYYY.M.PATCH-beta.N to reuse
the shipped npm package across release checks, Package Acceptance, cross-OS,
release-path Docker, and package Telegram. Use package_acceptance_package_spec
only when Package Acceptance should intentionally prove a different package.
The Codex plugin live package lane follows the same state: published
release_package_spec values derive codex_plugin_spec=npm:@openclaw/codex@<version>;
SHA/artifact runs pack extensions/codex from the selected ref; and operators
can set codex_plugin_spec directly for npm:, npm-pack:, or git: plugin
sources. The lane grants the explicit Codex CLI install approval required by
that plugin, then runs Codex CLI preflight and same-session OpenAI agent turns.
Its final zero-retry, medium-thinking turn sends visible progress with omitted
Codex final, reads randomized workspace inputs, writes their exact artifact,
and sends explicit completion. This catches the v2026.7.1 regression where an
ordinary progress send terminated the turn.
Use -f skip_package_telegram_e2e=true only when the release owner explicitly
defers the Package Acceptance Telegram E2E to a follow-up beta. The input is
rejected for stable and full, recorded in validation evidence, and does not disable the focused
rerun_group=npm-telegram workflow.
Top-level stages
Forrerun_group=all, a Check for reusable validation evidence job runs
first. It looks for the newest prior green full validation with the same release
profile, effective soak setting, and validation inputs. Exact-target reruns use
exact-target-full-validation-v1. A descendant whose complete delta is exactly
CHANGELOG.md uses changelog-only-release-v1; every product lane is skipped
and the verifier independently rechecks the GitHub commit comparison, immutable
parent artifact, child runs, and dispatch logs. Any other target change requires
a fresh Code SHA validation. Pass reuse_evidence=false to force a fresh full
run. Evidence reuse runs only from main or a canonical SHA-pinned
release-ci/* ref whose workflow commit remains on trusted main lineage;
other workflow refs run the selected lanes fresh.
Fresh package-facing validation prepares one immutable tarball plus one Docker
image artifact before dispatching Plugin Prerelease and OpenClaw Release Checks.
Both children verify the same package SHA, artifact IDs, service digests,
producer run attempt, and Docker archive digest before use. The package-independent
bare Docker layer uses a content-addressed GHCR cache; candidate-specific images
remain immutable GitHub artifacts. Focused runs with an explicit published
package spec keep the existing package path instead.
Also for rerun_group=all, a Verify Docker runtime image assets job builds
the runtime-assets Docker target with
OPENCLAW_EXTENSIONS=diagnostics-otel,codex. It runs in parallel with the
other stages and is enforced by the umbrella verifier; lanes no longer wait for
it before dispatching. A narrower rerun_group skips this preflight.
The five child-dispatch jobs own dispatch and exact identity capture only. They
emit the child run ID, run attempt, and URL, then finish. Release Decision owns
the blocking answer; Diagnostic Drain owns complete terminal evidence. The
immutable execution plan owns child identity across collector attempts. The
decision state is one of
qualifying, blocked_diagnostics_running, passed,
blocked_complete, orchestration_error, or cancelled_with_children.
Persistent GitHub API failures are orchestration errors. A child whose workflow
path, display title, ref, Tooling SHA, run ID, or attempt changes is a distinct
provenance mismatch.
blocked_diagnostics_running is safe for immediate diagnosis but not for a
retry until Diagnostic Drain is terminal. orchestration_error authorizes
collector recovery against the same exact child identities, never test
redispatch. blocked_complete means diagnostics are complete; it does not
claim a drain is still running.
The umbrella always dispatches product performance in artifact-only mode.
OpenClaw Performance permits report publication only for scheduled runs or a
manual dispatch that explicitly sets publish_reports=true. The artifact-only
guard must complete successfully, proving the publisher job stayed skipped.
Fresh and reused evidence records
controls.performanceReportPublication=artifact-only; the verifier and reuse
selector reject evidence without the matching normalized performance-child
proof.
The verifier uploads the canonical manifest as
full-release-validation-<run-id>-<run-attempt>. Evidence tooling validates
its artifact ID, digest, producer run, and attempt before downloading that exact
artifact ID. It caps the downloaded ZIP, verifies its bytes against the REST
sha256: digest, and streams the only allowed bounded manifest entry without
extracting the archive. A stable-name alias remains temporarily for older
publish consumers. The verifier always prefers the attempt-qualified artifact;
as a transition, it accepts the stable name only for an attempt-1 manifest v2
producer. It rejects that legacy name for later attempts and manifest v3.
Concurrency is keyed by Validation SHA, Tooling SHA, and rerun group and does
not cancel an older run. Parent cancellation or timeout leaves adopted
identity-checked children running and records cancelled_with_children when
the state collector can complete its cancellation handoff. Cancel an exact
child explicitly when it is no longer useful. Do not run a second foreground
watcher when the SHA-pinned helper already owns the parent; use
release-ci-summary --watch only after the helper has returned or when the
parent was dispatched separately.
Release checks stages
OpenClaw Release Checks is the largest child workflow. It resolves the target
once and validates the umbrella’s shared package artifact when available. A
direct or focused dispatch prepares its own release-package-under-test
artifact when package or Docker-facing stages need it.
Docker release-path chunks
The Docker release-path stage runs these chunks whenlive_suite_filter is
empty:
Use targeted
docker_lanes=<lane[,lane]> on the reusable live/E2E workflow when
only one Docker lane failed. The release artifacts include per-lane rerun
commands with package artifact and image reuse inputs when available.
Release profiles
release_profile mostly controls live/provider breadth inside release checks.
It does not remove normal full CI, Plugin Prerelease, install smoke, package
acceptance, or QA parity. Stable and full profiles always run exhaustive
repo/live E2E, Docker release-path, and QA-live soak coverage. The beta profile
adds those lanes only with run_release_soak=true, an explicit qa-live
controller retry, or the direct child’s manual qa aggregate. Package
Acceptance supplies the canonical package Telegram E2E for every candidate, so
the umbrella does not duplicate that live poller.
Full-only additions
These suites are skipped bystable and included by full:
stable includes native-live-src-gateway-profiles-anthropic-smoke and
native-live-src-gateway-profiles-opencode-go-smoke; full uses the broader
Anthropic and OpenCode Go model shards instead. Focused reruns can still use the
aggregate native-live-src-gateway-profiles-anthropic or
native-live-src-gateway-profiles-opencode-go handles.
Focused reruns
Usererun_group to avoid repeating unrelated release boxes:
Use
live_suite_filter with rerun_group=live-e2e when one live suite failed.
The former release-checks aggregate retry handle is invalid. It silently
expanded to every release-check lane, including package and Docker setup. Pick
one concrete group after classifying the failed surface.
The umbrella/controller also rejects qa; direct OpenClaw Release Checks
dispatches may use it only as a deliberate manual aggregate of qa-parity and
qa-live. Live, QA-live, and cross-OS filters must match their owning group.
Mismatches fail before scheduling and never widen to an unfiltered run.
Valid filter ids are defined in the reusable live/E2E workflow, including
docker-live-models, live-gateway-docker,
live-gateway-anthropic-docker, live-gateway-google-docker,
live-gateway-minimax-docker, live-gateway-advisory-docker,
live-cli-backend-docker, live-cli-cache-docker, live-acp-bind-docker, and
live-codex-harness-docker.
For a focused QA transport rerun, set rerun_group=qa-live and use the
canonical selector qa-live-matrix, qa-live-buzz, qa-live-telegram,
qa-live-discord, qa-live-whatsapp, or qa-live-slack.
The live-gateway-advisory-docker handle is an aggregate rerun handle for its
three provider shards, so it still fans out to all advisory Docker gateway jobs.
Use cross_os_suite_filter with rerun_group=cross-os when one cross-OS lane
failed. The filter accepts an OS id, a suite id, or an OS/suite pair, for
example windows/packaged-upgrade, windows, or packaged-fresh. Cross-OS
summaries include per-phase timings for packaged upgrade lanes, and long-running
commands print heartbeat lines so a stuck update is visible before the job
timeout.
QA release-check failures block normal release validation only for selected
Matrix, Telegram, and QA runtime tool coverage lanes. QA parity, runtime
parity, and the gated Discord, WhatsApp, and Slack live lanes are advisory and
publish status artifacts without blocking the release verifier. Tideclaw
alpha runs may still treat non-package-safety release-check lanes as advisory. With
release_profile=beta, the Run repo/live E2E validation live-provider suites
are advisory: third-party model deployments change underneath a release, so
beta surfaces their failures as warnings while stable and full profiles keep
them blocking. When
live_suite_filter explicitly requests a gated QA live lane such as Discord,
WhatsApp, or Slack, the matching OPENCLAW_RELEASE_QA_*_LIVE_CI_ENABLED repo
variable must be enabled; otherwise input capture fails instead of silently skipping the lane.
Use controller groups qa-parity or qa-live for fresh QA evidence. A direct
manual OpenClaw Release Checks dispatch may use qa to aggregate both.
Evidence to keep
Keep theFull Release Validation summary as the release-level index. It links
child run ids and includes slowest-job tables. Classify failures as product,
harness/tooling/provenance, infrastructure/credential, or wrapper. Only a
confirmed product failure changes the Code SHA. Use one diagnosis, one fix when
needed, and one narrow retry, then reassess; do not automatically rerun all.
Narrow evidence is not publish authorization by itself.
For a regular release, record both Code SHA and Release SHA, the reuse policy
and changed-path set, the green Code SHA parent run, and the lightweight Release
SHA parent run. For extended-stable, record the canonical branch, exact release
SHA, fresh parent run id and attempt, workflow ref, every child run, and any
frozen-target compatibility repair or intentional omission.
Useful artifacts:
release-package-under-testfromOpenClaw Release Checks- Docker release-path artifacts under
.artifacts/docker-tests/ - Package Acceptance
package-under-testand Docker acceptance artifacts - Cross-OS release-check artifacts for each OS and suite
- QA parity, runtime parity, and selected Matrix, Buzz, Telegram, Discord, WhatsApp, or Slack artifacts
Workflow files
.github/workflows/full-release-validation.yml.github/workflows/openclaw-release-checks.yml.github/workflows/openclaw-live-and-e2e-checks-reusable.yml.github/workflows/plugin-prerelease.yml.github/workflows/install-smoke.yml.github/workflows/install-smoke-reusable.yml.github/workflows/openclaw-cross-os-release-checks-reusable.yml.github/workflows/package-acceptance.yml.github/workflows/openclaw-performance.yml.github/workflows/npm-telegram-beta-e2e.yml