Projections, Checkpoints, and Cold Start
Which disposable acceleration state exists beyond the complete event log
Conclusion: the event log owns truth; checkpoints and projections own timing and cost
DeepSeek Harness does not persist a second conversational truth beside the Session log. It keeps the append-only SessionEvent stream and SessionHeader authoritative, then layers three different mechanisms over them: semantic flush barriers before irreversible work, in-memory projection cells for fast current views, and durable projection checkpoints that shorten some folds. A browser receives yet another derived layer—watermarked baselines and non-durable push frames—and converges them back toward the log.
The architecture is strongest when those layers remain distinct. A semantic checkpoint proves that earlier facts reached storage; a projection checkpoint merely remembers a fold; a watermark says how far a value has observed; a baseline is a transport cut; and a client row is disposable convergence state. Treating any one of them as the log itself creates exactly the recovery bugs the design is trying to avoid.
1. Six names for six different contracts
| Layer | Representation | Authority | What it proves |
|---|---|---|---|
| Durable fact | SessionHeader + append-only SessionEvent | Source of truth | Which metadata and committed events the backend actually stores |
| Semantic checkpoint | Awaited sessions.flush(session) | Durability barrier, not data | All participating flush listeners settled for the admitted prefix |
| Live projection cell | {state, observedSeq} per unit and live Session | Disposable derived state | The last event passed through that unit's fold |
| Projection checkpoint | {ver, seq, val} per key | Durable cache only | A versioned fold shortcut bound to one log lifecycle |
| Host baseline | {asOfSeq, values} | Derived wire snapshot | One consistent cut suitable for seeding a client |
| Client row | key -> {value, seq} | Disposable convergence state | The newest accepted host value for that key |
The persisted unit is the existing Session event, not a parallel persisted-message model. Non-replayable metadata remains in the Header. Projection checkpoint types explicitly say that a row is never authoritative and must be discarded when its version or claimed log endpoint is unusable.ARCH-LOGSESSION-FORMAT-HEADERPROJ-AUTHORITY-LAYERS
2. A semantic checkpoint is an awaited barrier, not a stored checkpoint object
append facts to the live Session
↓
sessions.flush(session)
├─ persistence listener drains write-behind
├─ any other scoped durability listener settles
└─ first failure is thrown only after all listeners settle
↓
construct model stream or enter top-level tool body
SessionStore.flush() resolves the exact live Session's scoped listeners, invokes all of them, waits with Promise.allSettled, and then throws the first rejection if any. Its return value only says whether a durability listener participated. There is no durable “checkpoint record” produced by this API.
The standard policy places this barrier before the downstream model stream is constructed, before a top-level tool body can run, and at every agent/pre-step boundary. Model and tool boundaries fail closed; nested tool dispatches reuse the already-durable outer call.LOG-CHECKPOINT-POLICY
3. Write-behind separates hot-path admission from durable quiescence
Clone on admission
Every committed event enters a persistence-owned queue through structuredClone.
Batch on a fixed deadline
The first pending event arms one bounded timer; the hot append path does not wait for disk.
Retain failures
A failed batch is prepended back in original order, and automatic retry pauses.
Join one barrier
Concurrent explicit flush callers share the same drain to a quiescent queue.
An explicit flush cancels the batching timer, waits for overlapping work, retries retained events, and drains until no active or pending write remains. Per-session-ID serialization prevents durable appends for one log from interleaving.LOG-WRITE-BEHIND
Persistence and checkpoint scheduling are separate plugins. Loading a backend without the semantic policy is valid, but a crash may then lose events still inside the batching window or an outstanding write. The shipped base composition mounts both the JSONL persistence provider and the checkpoint policy explicitly.PROJ-SHIPPED-COMPOSITION
4. Projection units are synchronous folds over every committed event
ProjectionDefinition<K, S> = {
key,
schema, // validates the wire-facing view
init(): S,
apply(S, event): S, // pure and synchronous
view(S): Value,
stateVersion
}
The registry subscribes once to session/event. Every event passes every registered unit's apply; an uninterested unit must return the same state reference. Only a changed reference causes schema validation and a change notification. Missing cells lazily fold the full in-memory log, while observedSeq advances even when the state reference does not.PROJ-REGISTRY-DRIVE
The synchronous requirement is load-bearing. A carrier can copy a Session's event snapshot and immediately call snapshot() in the same JavaScript turn, obtaining values whose shared asOfSeq is exactly session.seq - 1. An async unit would tear that cut.
5. Thirteen production keys share one framework but encode different lifecycles
| Key / version | Fold semantics | Production source |
|---|---|---|
todos v2 | Latest whole todo/write; cleared by the next turn/start | packages/todo/tool-todo/src/index.ts:122-147 |
plan v1 | command/run(plan) records intent; plan/mode commits active state and clears pending | packages/plan/plan-mode/src/index.ts:235-265 |
goal v4 | Last-wins durable goal/change projection | packages/goal/goal/src/index.ts:201-212 |
title v1 | Latest session/title | packages/session/session-title/src/index.ts:304-316 |
sessionStats v1 | Whole-log turn, Step, model, tool, TTFT, and decode totals | packages/session/session-stats/src/projection.ts:1-21,88-182 |
tokenUsage v1 | Usage samples replace the prior sample for the same Turn and Step | packages/llm/token-meter/src/usage-projection.ts:97-140 |
contextPressure v4 | Provider prompt sample plus signed Surface movement estimates the next request | packages/llm/token-meter/src/usage-projection.ts:142-205 |
contextBreakdown v2 | Latest request envelope plus message-surface shadow pricing | packages/llm/token-meter/src/breakdown-projection.ts:31-69 |
subagentTiming v2 | Completed and currently active post-descriptor Turn duration | packages/subagent/subagent/src/projection.ts:36-85 |
subagent v2 | Last valid durable mode/label descriptor, including its seq | packages/subagent/subagent/src/projection.ts:131-155 |
permissions v1 | Three whole-value knob events, viewed against composition defaults | packages/interaction/permission-presets/src/index.ts:227-251 |
sessionListMetadata v1 | Blank-to-nonblank state and latest human-prompt time for listing | packages/host/apiproxy/src/api-proxy.ts:1289-1300 |
imageLimits v1 | Per-boot attachment limits; constant state, live-service view, baseline only | packages/host/apiproxy/src/api-proxy.ts:1302-1323 |
This is a repository-wide inventory of production registrations at the pinned commit, not a promise that all thirteen keys are active in every deployment. Registration is composition-dependent, and the process-wide table also means a key mounted by one preset can appear in snapshots for Sessions whose own preset never produces a meaningful value.
6. A watermark advances on observation; a push occurs only on change
event seq N enters every unit
├─ apply returns same reference
│ observedSeq = N
│ no session/projection frame
└─ apply returns new reference
observedSeq = N
schema.parse(view(state))
emit { key, value, seq: N }
This distinction explains why imageLimits never emits a push yet still appears in a tail baseline, and why a quiet value in a full snapshot can legitimately be stamped at the log's current asOfSeq. The watermark means “observed through,” not “last changed at.”
The Host translates registry changes into non-persistent session/projection mux frames. The history tail—and only the tail—carries a full baseline. An attached Session copies events and snapshots projections with no await between them; a detached Session folds its inspected events.PROJ-HOST-CARRIERLOG-HISTORY-CUT
7. The durable projection cache is a versioned shortcut, never a read authority
| Guard | Why it exists | Failure behavior |
|---|---|---|
stateVersion | State shape or fold semantics may change | Discard that key's row and refold |
createdAt + cwd | A Session ID names a slot, not a unique lifecycle forever | Discard the whole unrelated record |
| Anchored floor | A cached row may claim events beyond a shrunken or repaired stored log | Detect overreach and reread from seq 0 |
| Lossless JSON boundary | Internal unit state must be durably serializable | Fail that cache write; never mutate the Session log |
The session_projcache domain is version 3. Each Session record binds one complete key-to-row checkpoint to a log identity. Domain-version incompatibility can discard the cache medium because the event log remains sufficient for reconstruction.PROJ-CACHE-SCHEMA
restoreFloor() chooses one event below the lowest usable watermark. restore() accepts a row only when its version matches and its seq lies within the supplied suffix's provable bounds. An unsafe partial restore throws so the caller can reread from zero instead of blending incompatible state.PROJ-REGISTRY-RESTORE
8. The implemented cold-cache ladder is not wired into shipped cold History
implemented SessionProjectionCache.coldSnapshot(id)
cache rows → restoreFloor → persistence.readFrom(tail)
→ registry.restore → fail-soft refreshed write-back
shipped ordinary cold History
persistence.inspect(full logical log)
→ registry.restore({}, all events, 0)
shipped Session list
cachedSnapshot(header) only, zero log I/O
coldSnapshot() validates log identity, tail-folds from an anchored floor, falls back to one full read after identity or endpoint failure, and writes refreshed rows back without failing the read.PROJ-CACHE-LADDER
At the pinned commit, production search finds the coldSnapshot method definition and generated API catalog, but no shipped caller. session.list uses only cachedSnapshot; ordinary and subagent cold History already load an inspected full log and fold from zero; subagent listing uses a cached row when conclusive and otherwise performs a full inspection. The cache is therefore a shipped list/metadata accelerator, not an active cold-History tail-replay accelerator.PROJ-CACHE-INTEGRATION
9. Inspect, prepare, and resume form a cold-start protocol, not three aliases
| Operation | Logical recovery | Durable mutation | Ownership / publication |
|---|---|---|---|
inspect(id) | Validates and adds deterministic closers in memory | No repair commit | No Agent; may retain an exact prepared graph for reuse |
readFrom(id, seq) | None; returns valid physical stored suffix | None | No preparation cache or publication |
prepare(id) | Builds the balanced exact Session | Commits repair if needed, then reloads | Exclusively reserves the unpublished Session |
Agent resume | Consumes the preparation | Persists later live suffixes normally | Runs setup, publishes exact Session + Agent, or rolls back |
Cold inspections can present a balanced logical turn while leaving a torn or interrupted physical tail untouched. Preparation uses phases loading, ready, committing, and reserved; same-ID inspectors share a load, while only the exact reserved Session may publish.LOG-PERSISTENCE-READ-FACESLOG-PREPARED-COMMIT
The Agent factory fuses caller, owner-fiber, and factory cancellation around persistence.prepare. Setup runs before publication; failure disposes the prepared Agent and releases the reservation. SessionStore.enter is the final same-ID collision boundary.PROJ-COLD-RESUME
10. Browser cold open is deliberately not Agent resume
Restore selection
The browser remembers its current Session address and selects it after the list arrives.
Open History
Session.open() requests the tail page; it does not request an Agent.
Inspect cold storage
The Host uses an attached Session when available, otherwise persistence inspection.
Resume only on demand
Prompt, rename, model, command, and similar Agent operations enter the shared resolver.
A cold Session must be listed by persistence and carry a project cwd to be Web-servable. History can still degrade to generic tool cards when the recorded preset no longer supplies a standing presenter scope. A real resume is stricter: it must pass format, continuity, ownership, preset composition, setup, and final publication checks.
The shared resolver reuses a live Agent, deduplicates same-ID resumes, inspects ownership and composition, then calls agents.resume. Not-found and subagent ownership receive typed errors; every other genuine resume failure remains an error. A fresh Session is created only on explicit create/adopt paths after persistence confirms the ID is absent.PROJ-HOST-COLD-OPENPROJ-COLD-RESUME
11. Client convergence combines a full baseline, sparse pushes, and generation repair
history tail baseline { asOfSeq: N, values: all registered keys }
↓ seed
resident ProjectionValueStore
↑ apply only if frame.seq is greater
session/projection { key, value, seq }
new mux generation:
session/subscribed { lastSeq: D }
→ remove client rows with seq > D
→ refetch list and resync opened History windows
The manager owns one projection store per Session even before a Session object exists. Both baselines and push frames use strict higher-seq-wins. A full History baseline also clears omitted keys at or below its cut; a partial session.list block applies only carried keys and deliberately clears nothing.PROJ-CLIENT-STORE
session/subscribed.lastSeq removes rows that claim state beyond the Host's durable endpoint, allowing lower-seq truth reconstructed after a crash to land. History installation then drains buffered live events, drops overlap, and refetches the tail when it detects a seq hole.PROJ-CLIENT-MANAGERLOG-CLIENT-STITCH
The mux and host feeds are independent streams with no shared total-order claim. events.mux({since}) also ignores since in v1. The supported convergence story is reopen the streams, refresh list state, refetch History, and repair each domain—not resume one globally linear stream offset.
12. Rebuild and invalidation operate at four different scopes
| Change | Invalidated state | How truth returns |
|---|---|---|
| Another event appends | Live cell advances eagerly | Changed units push; full baseline later stamps every value through the new cut |
| Unit state or fold semantics change | Persisted row with old stateVersion | Discard row and fold from a provable log prefix |
| Session ID is recreated or storage root changes | Whole checkpoint record with mismatched identity | Discard all rows for that unrelated lifecycle |
| Last registrant unloads | Registration and every live WeakMap cell for the key | A later snapshot omits the key; re-registration lazily folds the in-memory log |
| Projection-cache domain format changes | Entire derived medium | Drop it; the Session log remains sufficient |
No live registry API hydrates its WeakMap cells from a persisted checkpoint. The cache's detached restore() path returns a snapshot and refreshed checkpoint, but a resumed live Session still lazily folds its in-memory event array on first registry touch. There is also no cache eviction or retention surface; records accumulate until out-of-band maintenance removes them.
13. Four documentation claims have drifted from current production control flow
| Documented claim | Current implementation | Evidence |
|---|---|---|
| Duplicate projection keys throw | Same-version registrations share one ref-counted unit; only version mismatch throws | session-projection/README.md:11 vs src/index.ts:194-220 |
Titles do not join session.list | List rows carry projection blocks; cold cached titles seed the manager before open | host/apiproxy/README.md:33 vs api-proxy.ts:1725-1764 |
| History resumes an unattached Session and has no persistence-only read path | Cold History uses inspect() and explicitly does not publish an Agent | client/connection/README.md:23-26 vs api/sessions.ts:264-283 |
skill.list resolves a cold Session from its Header | The handler first requires ctx.sessions.get(sessionId); unattached Sessions return not found | host/apiproxy/README.md:59 vs api-proxy.ts:3205-3245 |
14. Source-level risk: detach can briefly let the projection cache lead the log
Session detach
1. remove Session from SessionStore
2. emit contained, non-awaited session/disposed observers
├─ persistence starts asynchronous retirement flush
└─ projection cache snapshots and writes
live-entry check now fails → no sessions.flush()
The cache documentation says the log always leads the cache. That is enforced for an ordinary live write, but not strictly at detach: the Session has already left the store, so the cache skips its explicit flush and relies on the persistence listener's separate retirement drain. The cache source itself acknowledges possible residual overreach. A process crash between the two durable writes can therefore leave a checkpoint whose seq is ahead of the stored log.PROJ-CACHE-DETACH-WINDOW
coldSnapshot() could detect that overreach through its anchored floor, but no shipped caller uses it. The zero-I/O list read does not validate the stored log endpoint. A phantom list projection can consequently reach the client; a later lower-seq cold-History baseline loses under higher-seq-wins until a true resume emits session/subscribed and truncates the row, or the log advances beyond it.
15. Source-level risk: equal-seq replacement cannot cross a Host generation
The client rejects every update whose seq is lower than or equal to the resident row, while generation truncation removes only rows strictly beyond the new Host's durable lastSeq. If a Host restarts with the same log endpoint but a different value computed at that same endpoint, the old row survives and the new equal-seq baseline cannot replace it.PROJ-CLIENT-EQUAL-SEQ
imageLimits makes the scenario concrete. Its state is always null, its view reads the current boot's attachment configuration, and it intentionally never pushes. A browser that remains alive across a Host restart can retain old limits at seq N; subscribed keeps the row because it is not beyond N, and the fresh baseline at N loses as equal. The value converges only after another event advances the baseline, a full client reset, or a protocol that carries a generation/version dimension.
The same edge applies to a unit's view semantics changing at an unchanged log watermark, and to unload/reload of the same key. stateVersion protects the Host's persisted cache but is absent from the client wire row, so it cannot resolve this client-generation ambiguity.PROJ-IMAGE-LIMITS
16. Source-level risk: one optional unit can make ordinary History unavailable
A detached History baseline calls registry.restore({}, events, 0), which folds every registered unit. If any unit's apply, view, or schema rejects the log, ordinary session.history returns internal instead of serving the transcript without projections. Session listing explicitly contains projection failure per row, and subagent History also degrades without the block; ordinary History does not.PROJ-HISTORY-FAILURE
This is not merely corrupt state inside the failing domain: the registry's process-wide all-unit fold lets an unrelated projection block the base transcript read. Whether fail-loud is desired needs an explicit product decision. Current carrier behavior is inconsistent.
Unregistering the final unit deletes the registration and cells but emits no invalidation frame. Already connected clients retain that key until a later full tail baseline omits it. If the key is reloaded at the same log seq, the strict equal-seq rule can retain the previous value even after that baseline arrives.PROJ-REGISTRY-REGISTRATIONPROJ-CLIENT-STORE
17. Design assessment and verification status
| Choice | Benefit | Cost |
|---|---|---|
| One authoritative event log | Every derived layer can be discarded and rebuilt | Cold full folds remain expensive when acceleration is not connected |
| Synchronous, whole-value projection units | Atomic cuts and simple client replacement semantics | Every event touches every unit; large values ride every tail baseline |
| Versioned persisted fold shortcuts | No cache migrations are required for semantic changes | Version discipline is manual, and cache retention is unbounded |
| Inspect separately from resume | Cold browsing does not activate agents or mutate durable recovery state | Read and execution paths expose different logical/physical moments |
| Higher-seq-wins on the client | Stale baselines and replayed frames cannot regress current state | Seq alone cannot distinguish an equal-watermark Host generation |
This chapter traced the fixed upstream commit through Session flush dispatch, checkpoint policy, persistence write-behind, projection registry and all production registrations, projection-cache schema and read/write paths, Host History/list carriers, cold inspection and exact preparation, Agent resume, and browser projection/gap-repair stores. It also reviewed relevant contract tests in source. Dependencies were not installed and no tests, crash injection, or live reconnect experiment ran locally; every risk section above is therefore labeled as static deduction rather than reproduced behavior.
My Learning Notes
Autosaved only in this browser. Nothing is uploaded or committed. Export Markdown whenever you want to keep a copy.