Tests, Snapshots, and Runtime Invariants

Conclusion: correctness is not proved by one green run, but by evidence layers that cannot replace one another

This is not a weak-to-strong pipeline whose final aggregate score supersedes the earlier tiers. The repository's own testing policy states that line coverage proves execution, not that the shipped feature works; snapshots, real entry paths, platform-specific checks, and runtime invariants therefore own different counterexample spaces.TINV-POLICY

1. The evidence stack: each layer answers a different question

2. Per-file 100%: large files cannot subsidize small ones

The V8 coverage include surface is packages/*/*/src/**/*.{ts,tsx}, not the whole repository. Examples and vendor code lie outside that include glob; within it, type-only files, self-executing bins and workers, and an explicit set of GUI or composition debt are excluded.TINV-COVERAGE-SCOPEAfter file exclusions and source inline ignores take effect, statements, branches, functions, and lines must each reach 100% for every included file that remains measured.TINV-COVERAGE-BAR

included production file
        │
        ├── statements 100%
        ├── branches   100%
        ├── functions  100%
        └── lines      100%

aggregate average cannot compensate for one file

This matters in a composition-heavy repository: a new small companion or adapter cannot disappear behind strong coverage in another large file. But the precise reading is “100% per file within the measured set after file exclusions and inline ignores,” not “100% of all production paths in the repository,” and certainly not “100% of semantics.”

3. Exemption does not delete a test: instrumentation and correctness become parallel gates

Compiler-analysis and real-subprocess fixtures may be excluded from the instrumented run, but the same coverage aggregate runs those suites beside it under plain Vitest. The admission rule says that every coverage-measured file they execute must already be fully covered by other suites.TINV-COVERAGE-EXEMPTA mechanical test further requires each CLI filter and exclude glob to select the same non-empty file set and forbids overlapping roster entries.TINV-COVERAGE-EXEMPT-TEST

Platform conditions also change the effective include surface: Windows excludes suites that require a POSIX shell, every non-Windows host excludes ACL source that executes only on Windows, and hosts without real pwsh exempt the related source in step with suites that self-skip.TINV-PLATFORM-COVERAGEA coverage report therefore has to be interpreted with its platform and tool availability; one machine's 100% cannot be generalized to execution of every platform path.

4. Keyless snapshots replace the nondeterministic model boundary with committed scripts

replay is the default mode: it does not load .env, starts real subprocess paths from committed model responses, and compares assembled requests, normalized protocol or transcript output, and durable logs. Only record reads credentials and calls the real API; refresh remains keyless and uses existing scripts to rewrite current expected outputs. Snapshot files run concurrently only in replay when the configured bound permits it; record and refresh files remain serial to avoid quota contention or concurrent corruption of goldens.TINV-SNAPSHOT-MODE Inside the ACP suite, scenario tests likewise use a concurrent suite only in replay.TINV-SNAPSHOT-SCENARIO-MODE

The CI snapshot gate depends on a build and sets DSH_EXAMPLE_MODE=lib: example and package snapshots start built artifacts under plain Node, while script snapshots run their real source entry paths.TINV-SNAPSHOT-GATEThis preserves independent evidence between “source tests pass” and “the published shape loads.”

5. Replay derives calls from the durable log and compares both wire and log

deriveReplayScript() splits model calls at terminating finish chunks in assistant/chunk events; if turn or step changes before the previous call terminates, it fails loudly. A compaction summary explicitly marked as a local LLM call reconstructs blocks, usage, and a terminal chunk at its log position. Throws and hangs cannot be losslessly derived from an ordinary log alone and require an override.TINV-REPLAY-DERIVE

recorded session JSONL
        │
        ├── assistant/chunk + finish ──► positional replay entries
        ├── marked compaction summary ─► reconstructed local call
        └── throw / hang gap ──────────► explicit override required
                                             │
real assembled subprocess ◄─────────────────┘
        │
        ├── normalized stdout vs committed wire snapshot
        └── normalized harvested JSONL vs committed session fixtures

For each scenario not skipped by the current mode, platform, or PowerShell availability, the ACP suite runs the real assembled entry, first rejects acceptance of UNKNOWN_TOOL as successful behavior, then normalizes session IDs, cwd, and other volatile values. It finally performs per-file stdout matching and one-to-one normalized comparison of harvested parent and child logs with fixtures; mode explicitly controls record and refresh writes.TINV-SNAPSHOT-COMPAREThe scenario directories also have closure guards: orphan directories, missing input/stdout/session files, incorrect override presence, or wrong header-sidecar ownership fail.TINV-SNAPSHOT-FIXTURES

6. The runtime invariant registry centralizes attribution and lifecycle, while owners retain rules

InvariantRegistry provides global enablement plus package allowlists and blocklists. Even when a filter disables an installer, its package name remains reserved, preventing two companions from silently claiming one owner. An enabled installer runs in a dedicated child fiber and receives fail() bound to its package name; a failure throws an error with stable INVARIANT code and packageName. Startup failure disposes the child and releases the reservation, while the normal disposer completes child teardown before release.TINV-REGISTRY

The registry imports no product packages and does not centralize a giant rule table. The repository convention asks each package's ./invariant companion to check event or mutable-data relationships that package owns, while type existence, method presence, and fixed pure-function results remain type, load, or unit-test concerns. The gate can verify the registered npm owner and the presence of a no-runtime-rule marker; whether a rule is semantically owned by that package and whether an explanation is adequate still require review.

7. “Every package has a companion” is proved separately by a source gate, test topology, and artifact consumption

The source gate scans every packages/*/*/package.json, checking the ./invariant export, published files, peer and development dependencies, and TypeScript references. It then parses each companion AST, requiring registration of exactly its own npm name, named name/inject/apply exports, no default export, and actual use of the failure reporter by a non-empty installer. An empty installer must contain the No runtime invariant: marker; the gate does not judge the explanation's specificity or quality.TINV-COMPANION-GATE

An ordinary Vitest root first mounts an enabled registry, then selects only the current package's companion from the test path, and uses a readiness dependency to prevent the target plugin from starting early.TINV-TEST-HOST A dedicated topology test mounts every companion, passes each through real Loader unwrapExports, and verifies that every package name is reserved.TINV-INVARIANT-TOPOLOGY-TEST

8. Concrete invariants check relationships instead of restating implementation

The Session companion keeps a lastSeq/openTurn/openStep/nextTurn/nextStep/pendingCalls trace for each Session. Candidate core-execution events must preserve monotonic sequence numbers and their required Turn/Step enclosure. An append-mode tool result must cite a prior call in the same Step unless it is the synthetic TOOL_NOT_STARTED result; ending the Step clears any unmatched pending calls. Relationships for merge-extensible events remain with their owners rather than being guessed by Session.TINV-SESSION-RULES

The mutation timing is more important: internal/dispatch performs pure validation and stages a transition, while the trace commits only after session/event is actually published. If a later pre-commit listener vetoes, the staged transition is abandoned and diagnostic state never advances ahead of the real log; on companion reload, the baseline is rebuilt from existing durable events.TINV-SESSION-COMMIT

The Agent-loop companion checks loop-built requests at the front of llm/stream: request and messages must be frozen, the session ID must identify a live Session, messages must equal the immediate durable-log derivation, and model/system/temperature/maxTokens/stop/tools must equal the folded request header.TINV-LOOP-REQUESTThis is the distinctive value of a runtime invariant: each object can be locally valid, while only comparison of the outgoing request with committed log facts exposes composition desynchronization.

9. Configuration closure proves two reachability graphs before Loader startup

verify-runtime-closure performs BFS from workspace dependencies in the executable deploy manifest, expanding ordinary and optional dependencies, and requires every encountered non-optional workspace peer to be supplied explicitly by the root runtime manifest. Failure output retains the full chain from runtime to missing peer.TINV-RUNTIME-CLOSURE

verify-cordis-config scans all Loader YAML, recursively validates entries and patches, and resolves plugin specifiers from examples, apps, and bundles back to the appropriate manifest dependencies. For local packages it also requires source launch to resolve through tsconfig.base.json paths to .ts/.tsx, preventing a development machine with stale lib/ from masking a missing mapping in a clean checkout. Directory-picker backends selected by runtime strings are added to the closure explicitly.TINV-CONFIG-CLOSURE

The configuration AST has a semantic boundary too: only fields the Loader really interpolates, config and entry-level disabled, may be dynamic. A !!js value in id/name/group/inject/intercept/isolate becomes truthy data rather than an expression, so the gate rejects it and pre-parses disabled syntax.TINV-CONFIG-METADATA

10. Generated documents are recomputable source projections, not hand-maintained copies

doc-sync is not one Markdown linter. It composes code-fence typechecking, Cordis/client/tool/config/persistence catalogs, document graphs, scoped events, links, source ownership, type equivalence, translation pairing, site projection, and a documentation-site production build.TINV-DOC-SYNC

The config catalog rebuilds from package entries, config types, JSDoc, and static Schemastery schemas. Every package must classify, type references must be collision-free, and every enumerable schema path must be locatable in the declared type.TINV-CONFIG-CATALOGIt first folds keys from intersected schemas recursively, then rejects missing members; --check compares the computed result byte-for-byte with the committed document.TINV-CONFIG-CATALOG-CHECK

The Cordis catalog scans Context merges and Events members, requires every service or event scope to map to a page, rejects stale exemptions, requires generated-region markers on both language pages, and compares complete outputs with committed files.TINV-CORDIS-CATALOGverify-type-equiv separately partitions byte-identical paired-language copies as derivatives, then requires one-to-one correspondence between every primary ts type-equiv or ts public-api block and its manifest entry and compares that primary block with source structure and JSDoc.TINV-TYPE-EQUIV

11. The platform matrix assigns different evidence responsibilities instead of copying one command

The Linux pull-request workflow separates Node 24 static, coverage, and consumer jobs.TINV-CI-LINUXThe consumer gate graph owns the build, package-consumption checks, assembled snapshots, a browser snapshot, and built-bin smoke.TINV-CI-CONSUMERSA separate matrix runs compatibility contracts on Node 22.19 and 26 so the primary version's green result cannot hide support-boundary regressions.TINV-NODE-MATRIX

Windows has two distinct signals. The pull-request windows workflow job delegates to a runner that provisions checksum-verified Windows Node, checks that it reports win32 x64, and runs only workspace build and documentation-site production build surfaces.TINV-WINDOWSTINV-WINE-RUNNER windows-native runs the complete inventory on a real Windows kernel, but reports independently and is absent from all-checks-passed.needs. The final all checks passed job uses if: always() to aggregate named jobs and treats failed, cancelled, and skipped as failure.TINV-REQUIRED-VERDICT

12. The exact meaning of green and the remaining blind spots

The essential engineering discipline is not to add a larger “total test” label, but to name the needed counterexample during planning: local branch, real Loader composition, durable transcript, cross-event relationship, deployment closure, generated contract, or specific kernel. Only then does a new test land in the tier that would actually turn red for the real regression.

13. Verification scope for this chapter

All repository facts are pinned to commit 47f943859bef60e4160492346772ded9b24f765a. This work did not run the full build, full coverage, full snapshot suite, browser suite, real-API tests, Windows or Wine checks, performance tests, or stress tests.

The first focused Vitest run covered the coverage-exempt roster, exhaustive companion topology, registry lifecycle, Session invariants, LLM replay, snapshot harness and normalizer, and Cordis config helpers. All 256 tests in 9 files passed. Representative assertions cover all companions registering through Loader, registry attribution and atomic rollback, a Session veto not advancing the trace, and positional replay failure semantics for throw, hang, exhaustion, and underrun.TINV-INVARIANT-TOPOLOGY-TESTTINV-REGISTRY-TESTTINV-SESSION-TESTTINV-REPLAY-TEST

pnpm exec vitest run \
  scripts/coverage-exempt.spec.ts \
  scripts/test-invariants.spec.ts \
  packages/runtime-diagnostics/invariants/tests/service.spec.ts \
  packages/core/session/tests/invariant.spec.ts \
  packages/test-support/llm-replay/tests/llm-replay.spec.ts \
  packages/test-support/acp-snapshot/tests/harness.spec.ts \
  packages/test-support/acp-snapshot/tests/normalize.spec.ts \
  scripts/cordis-config-files.spec.ts \
  scripts/verify-cordis-config.spec.ts

Test Files  9 passed (9)
Tests      256 passed (256)
Duration   22.17s

The second run used the replay and source defaults with both environment variables unset; the reproduction command below pins them explicitly. It ran the snapshot config read-only for canonical session-fixture layout and the translation-prompt snapshot: 2 files and 2 tests passed. The layout guard scans all session-format JSONL and rejects a noncanonical packed representation.TINV-FIXTURE-LAYOUT-TEST

DSH_SNAPSHOT=replay DSH_EXAMPLE_MODE=src \
pnpm exec vitest run --config vitest.snapshot.config.ts \
  scripts/session-fixture-layout.snapshot.ts \
  scripts/translation-prompt.snapshot.ts

Test Files  2 passed (2)
Tests      2 passed (2)
Duration   1.98s

The third run likewise used the replay and source defaults with both variables unset; the reproduction command pins them explicitly. It selected only the ACP handshake scenario. Its real source-mode assembled process passed: 1 test ran and passed, while 85 tests in the same file were skipped by the -t handshake filter. This is not a full ACP snapshot result.

DSH_SNAPSHOT=replay DSH_EXAMPLE_MODE=src \
pnpm exec vitest run --config vitest.snapshot.config.ts \
  examples/acp-agent/tests/acp.snapshot.ts -t handshake

Test Files  1 passed (1)
Tests      1 passed | 85 skipped (86)
Duration   14.70s

Finally, six read-only structural gates all completed with exit code 0. verify-package-invariants reported 219 hand-owned companions; verify-runtime-closure reported 109 workspace packages; verify-cordis-config reported 120 config files; the config catalog was current; all 93 generated Cordis files or regions were current; and verify-type-equiv reported 384 blocks matching source and JSDoc plus 384 paired derivatives. The focused Cordis metadata tests also cover the allowed disabled expression and three rejection paths.TINV-CONFIG-TEST

pnpm run verify-package-invariants
pnpm run verify-runtime-closure
pnpm run verify-cordis-config
pnpm run verify-config-catalog
pnpm run verify-cordis-catalog
pnpm run verify-type-equiv