Retries, Errors, Cancellation, and Recovery
Which layer owns the decision when failures cross different boundaries
Conclusion: failure recovery is a stack of commit protocols, not one universal retry
DeepSeek Harness splits recovery across five boundaries: the adapter normalizes provider faults into recordable facts; the Agent decides whether to retry only at the model-request failure seam; tool timeout and cancellation wait for started work to converge; checkpoints make intent durable before external side effects; and repair fixes only log structure it can prove, without guessing what the outside world did while the process was crashing.
The defining property is that it does not collapse different uncertainties into one error. A request may be replayable while a started write tool can only be marked outcome-unknown; a timeout can signal cancellation but cannot make an uncooperative Promise stopped; an in-memory append can mean admitted while only flush/fsync places it in the durable prefix. Those commit points reveal the real recovery model more clearly than a list of error codes.
1. Failure-ownership map: each layer recovers only what it can prove
| Layer | Facts it owns | Permitted recovery | Explicitly not responsible for |
|---|---|---|---|
| Provider adapter | HTTP, SSE, idle, caller abort | Produce a stable failure snapshot and close transport | Choosing business retry count |
| Request recovery | Current Turn/Step, provider policy, failure history | Wait, then resend the same model request | Swallowing middleware, consumer, or cleanup bugs |
| Tool runtime | Which calls started and whether abort was observed | Stop replenishment, drain started calls, complete ordered results | Hard-killing an uncooperative in-process tool |
| Persistence/checkpoint | Which events are durable | Flush before side effects; roll back failed writes and retain batches | Proving an external side-effect outcome |
| Crash repair | The final complete parseable prefix | Truncate torn tails; synthesize tool/step/turn closers | Inventing unpersisted tool results |
Recovery power comes from ownership boundaries, not the number of try/catch blocks. Guessing across a boundary turns a diagnosable failure into duplicated side effects, fabricated success, or an unreplayable log.
2. The adapter demotes only adapter-owned failures into terminal chunks
For a direct, unprepared ctx.llm.stream() call, the final adapter boundary owns adapter selection, asynchronous exact-model resolution, dispatch, iterator construction, and iteration; failures in those steps become one terminal finish(error|aborted). The Agent path first calls prepareCall() in buildRequest(): except for its NO_ADAPTER fallback, selection or configuration-resolution failures throw before the stream boundary, close the Turn as an error, and never enter agent/request-error.FAILURE-AGENT-PREPARE-BOUNDARY After preparation succeeds, adapter dispatch and iterator failures still become terminal chunks. Middleware, downstream consumer, and iterator-cleanup errors remain thrown; early consumer stop awaits iterator.return().FAILURE-ADAPTER-BOUNDARY
normalizeLlmFailure() trusts a carried snapshot only when the Error exposes matching own data descriptors for failure and code. It then reads and validates the snapshot fields inside a try, but those inner fields need not themselves be own properties. Message/code must be non-empty, while status, Retry-After, and requestId must satisfy type and range checks. A standalone third-party SDK code without a matching failure snapshot does not enter the Harness taxonomy directly; it degrades to UNKNOWN. The snapshot is copied and frozen rather than placing a live Error object in the event stream.FAILURE-SNAPSHOT
The narrow boundary makes provider faults recoverable through one protocol while preserving stack and visibility for plugin and consumer defects. Extension authors cannot assume that every stream throw will be retried automatically.
3. The DeepSeek adapter classifies HTTP advice, transport breaks, and idle separately
Non-2xx responses map 401/403, 429, 400, and 5xx into stable codes such as AUTH, RATE_LIMIT, INVALID_REQUEST, and SERVER. The adapter also extracts Retry-After in seconds or HTTP-date form and checks two possible request-id headers.FAILURE-DEEPSEEK-HTTP
Stable snapshot for one stream call
connection + credential + user id
│
├─ caller abort ───────────────→ ABORTED
├─ no progress on active read ─→ TIMEOUT
├─ other fetch/SSE failure ────→ TRANSPORT
└─ HTTP non-2xx ───────────────→ AUTH / RATE_LIMIT / SERVER / ...Each stream call resolves one connection-and-credential snapshot, so an in-flight request cannot mix in the next configuration generation. The idle watchdog counts only while one iterator demand is outstanding. On exit, the adapter aborts its consumer controller and awaits return() for an unexhausted iterator, avoiding “the upper layer recovered while the lower transport kept running.”FAILURE-DEEPSEEK-STREAM
4. agent/request-error is a narrow seam: only terminal request failure enters it
A Step opens once, with an internal while (true) rebuilding the request and assembler for each attempt. Only an assembler finish(error|aborted) invokes agent/request-error. The loop continues only when a listener returns { kind: "retry" }; otherwise it throws a structured LlmError. It writes the sole assistant/message and executes tools only after success.FAILURE-REQUEST-LOOP
Regression tests prove that request-middleware failure is not offered to the recovery listener; two consecutive provider failures followed by success remain Turn 1 / Step 1; cancellation wins even alongside a retry action; and a recovery listener failure closes the Turn instead of prompting another attempt.FAILURE-REQUEST-TEST
5. Retry policy belongs to the provider route; the executor only carries out a resolved decision
| Mode | Eligibility | Budget | Downstream composition |
|---|---|---|---|
normal | Only configured codes; defaults include EMPTY_RESPONSE, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT | Default: at most 2 after the first request; 500ms initial, 10s cap, 0.1 jitter | Delegate on mismatch, exhaustion, or over-cap provider delay |
always | Every model-request failure | Unbounded until success, cancellation, or disposal | Ask downstream first; accept its retry, log its failure, otherwise use local policy |
The resolver rejects unknown keys, invalid bounds, and empty or duplicate code lists, then produces a frozen route policy. The execution plugin has no retry-policy configuration of its own, preventing global executor settings from silently rewriting one provider.FAILURE-RETRY-POLICY
At execution time it finds prior retries with the same turn/step/provider/policyKey before selecting a delay. A valid provider Retry-After at or below the cap is used verbatim; an over-cap value delegates in normal mode and falls back to local backoff in always mode. llm/retry is appended before waiting, while llm/retry-started is appended only after the wait. Cancellation and plugin disposal can stop that wait, and disposal drains captured recovery work.FAILURE-RETRY-EXECUTOR
6. Retry retains raw chunks from failed attempts without committing a message or tool side effect
Unit tests leave failed partial text/tool-call chunks in the Step's event history while proving that they are not sources for an assistant/message, generate no tool/call, and execute the dangerous tool zero times. Only the successful attempt commits the Step's sole assistant message. The tests also pin retry scheduling recorded before the wait, exponential backoff, and error closure after exhaustion.FAILURE-RETRY-SEMANTICS-TEST
Real HTTP/SSE tests additionally cover a refused connection whose endpoint starts during backoff, stream disconnect, partial disconnect, empty completion, and a stalled body. The resent body is identical and stays in the same Step. One important counterexample is a validly ending partial EOF: it is classified STREAM_CLOSED, which is outside the default retry-code set, so it is not blindly retried merely because it looks network-like.FAILURE-WIRE-RECOVERY-TEST
Retaining raw chunks preserves forensic evidence while the derived transcript exposes only complete messages. Retry can still add latency, provider quota use, or billing. The Harness prevents duplicated tool side effects here, but it cannot undo a model request the provider already accepted.
7. Documentation drift: the current implementation retries within the same Turn and Step
packages/llm/llm-retry/README.md still says that every retry opens a freshly numbered Turn and that the loop closes the failed Turn before opening a retry Turn. That conflicts with the current Step-local loop and with unit and wire tests proving every attempt remains Turn 1 / Step 1.FAILURE-RETRY-DOC-DRIFT
At pinned commit 47f943859bef60e4160492346772ded9b24f765a, source and executable tests are authoritative: retry events keep the same coordinates, the Step writes one start/end pair, and only one successful assistant message is written. The README should be treated as stale semantics, not evidence of two implementation modes.
8. Tool timeout is a cooperative deadline, not a Promise race or hard kill
The timeout policy wraps only tools that declare timeoutMs. It fuses the caller signal with a local deadline, temporarily substitutes exec.signal during dispatch, and restores the original afterward. Most importantly, it awaits the downstream tool, replacing the final result with TOOL_TIMEOUT only when its own timeout code won first. It never abandons the still-running Promise.FAILURE-TOOL-TIMEOUT
The underlying deadline documentation explicitly says the signal only notifies and callers must stop their own work. AbortSignal.any preserves the first reason, while code-scoped timeoutOf() separates this layer's timeout from an outer cancellation. The idle watchdog likewise arms a timer only during an outstanding demand.FAILURE-TIMEOUT-SIGNAL
Tests cover three races: a cooperative tool returning after abort, a tool throwing an abort error, and caller abort arriving before or after timeout. When timeout wins, the Promise remains pending until the tool's cleanup gate is released. When caller abort wins, the ordinary abort remains and is not mislabeled as timeout.FAILURE-TIMEOUT-TEST
9. Parallel-tool cancellation: stop replenishment, drain starts, complete the shape for skipped calls
After abort, the parallel pool stops starting new calls but waits for every already-dispatched call to converge, committing results and additional context in model order. It then writes an ordered synthetic aborted-before-dispatch call/result for each model call that never started. A scheduler failure also allSettleds in-flight dispatches but does not fabricate successful recovery results.FAILURE-TOOL-DRAIN
A test that cancels from an assistant/message observer proves that the dangerous tool body never executes, yet the log contains a matching tool/call and TOOL_ABORTED_BEFORE_DISPATCH result; the next model request can replay that provider-valid pair.FAILURE-CANCEL-TOOL-TEST
The goal is not “make cancel return fastest.” It is to leave no anonymous background work after cancellation and retain a result for every assistant tool request. Quiescence is part of recovery correctness.
10. Agent cancellation is a state transition; cancel returning does not mean activity ended
By default, cancel() clears unclaimed Inbox work and aborts the current phase; keepInbox preserves only unclaimed work. Waking input arriving in the abort-to-idle window is reclassified to the next Turn and sets a wake latch. The old driver reaches idle after turn/end(aborted) and then starts it. The driver contains already reported errors, and consecutive Turns receive fresh AbortControllers.FAILURE-AGENT-CANCEL
Tests verify that post-abort wakes are not lost, default cancel clears queue and latch together, removing a latched message suppresses empty replay, slow convergence still accepts a latch, and a disposal cause never latches new work. Another test pins the default behavior of dropping the queued tail on mid-turn cancellation.FAILURE-CANCEL-TEST
11. Disposal is a shared quiescence point: cancel, drain, then unwind resources in reverse
The Agent lifecycle fuses caller, owner, and factory cancellation into one setup signal. Its memoized disposer runs once: issue a disposed-cause cancel, await whenIdle(), dispose the private scope, and only then detach from agent/session registries and release ownership bookkeeping. Every concurrent caller awaits the same Promise.FAILURE-AGENT-DISPOSAL
Lifecycle tests verify that turn/end precedes unregistration, concurrent owner unload and handle disposal share one quiescence, the old ID remains occupied until asynchronous scope cleanup completes, and a replacement with the same ID is permitted only afterward.FAILURE-DISPOSAL-TEST
This teardown can be slower than deleting a map entry and returning immediately. It prevents a new instance from overlapping old same-named resources and prevents tests from passing while production still has a timer, listener, tool, or transport alive.
12. Write-behind explicitly separates “event admitted” from “event durable”
Each live Session owns a pending queue, fixed batching deadline, active write, and shared flush barrier. Enqueue clones the event and returns immediately. Flush cancels the timer, awaits overlapping work, then drains until the queue is quiescent. On write failure, the batch is prepended in original order and automatic work pauses; a background failure is reported, while an explicit barrier rejects.FAILURE-WRITE-BEHIND
| Moment | What can be claimed | What cannot be claimed |
|---|---|---|
| Session append / enqueue returns | The live event entered persistence-owned memory | The disk has fsynced it |
| Background batch succeeds | The backend confirmed that stable prefix durable | Events that arrived later are also written |
| Flush resolves | The queue observed by the barrier drained to quiescence | External tool side effects completed |
| Flush rejects | The failed batch is retained and retryable | The caller may safely dispatch the side effect |
The coordinator uses reverse teardown to close event admission before the final flush, wait every per-session chain, and close the backend last. Session retirement likewise flushes before releasing exact-lifecycle state, and a close error cannot mask an earlier drain error.FAILURE-PERSISTENCE-WIRING
13. Semantic checkpoints put recoverable intent before irreversible side effects
model path: turn/start → claim → step/start → user/message → flush → adapter dispatch
tool path: assistant/message → tool/call → flush → tool body
next Step: previous response/result batch → pre-step flush → new requestThe Agent records its Turn/Step, input messages, and request header/context before constructing the session-tagged request and entering the stream.FAILURE-CHECKPOINT-AGENT-ORDERFAILURE-CHECKPOINT-REQUEST-PREFIX The llm/stream wrapper delays constructing the downstream stream until that request prefix is flushed. The tool path appends tool/call before scheduler dispatch, and dispatch must traverse the tools/execute waterfall before reaching the body.FAILURE-CHECKPOINT-TOOL-INTENTFAILURE-CHECKPOINT-TOOL-WRAPPER The checkpoint policy flushes a top-level call in that waterfall; an abort during the checkpoint returns the canonical aborted-before-dispatch result. Pre-step also flushes the preceding Step's response/result batch. Every checkpoint failure is fail-closed: neither adapter nor tool body runs.FAILURE-CHECKPOINT-POLICY
Unit tests prove that flush completes before the adapter or tool body runs, flush rejection prevents the side effect, cancellation during a checkpoint produces the structured not-dispatched result, and nested tools reuse the outer checkpoint. Those tests call llm.stream/tools.execute directly; they do not independently prove the upstream event-recording order.FAILURE-CHECKPOINT-TEST
A checkpoint establishes a happens-before relation: before a side effect occurs, recovery can at least see its intent. It is not an exactly-once promise for a provider or tool. The process can still crash after the side effect and before result fsync—the unknown-outcome window in the next section.
14. Hard-crash recovery protects the byte prefix first, then repairs event semantics
A separate-child E2E sends SIGKILL at model-dispatch and tool-side-effect boundaries. The child waits forever after the adapter receives the request or the tool writes its external-effect marker; the parent hard-kills it after observing that marker. After recovery, the first scenario retains the complete model-request prefix. The second proves tool/call was durable before the external marker and recovers a started tool without a result as TOOL_OUTCOME_UNKNOWN. Both 2/2 cases passed in this review.FAILURE-CRASH-FIXTUREFAILURE-CRASH-E2E
The raw JSONL scanner commits only newline-terminated, successfully decoded records with contiguous sequence numbers. The Zstd scanner lists only structurally complete frames and marks a final frame interrupted by EOF as torn.FAILURE-JSONL-RAW-SCANFAILURE-JSONL-ZSTD-SCAN The loader turns those complete boundaries into a stable prefix. For a torn tail, it preserves complete records and produces an opaque truncation marker; repair truncates first, then appends recovered events and synthetic closers.FAILURE-JSONL-PREFIX
Initial materialization uses a synced temporary file and a non-overwriting publish: POSIX publishes through link() and fsyncs the parent directory, while Windows uses non-replacing MoveFileExW(..., MOVEFILE_WRITE_THROUGH). Later append writes and fsyncs; a write/sync failure closes the handle, truncates to the old length, and fsyncs again so retry cannot duplicate a sequence. Repair truncation is fsynced too.FAILURE-JSONL-ATOMICITYFAILURE-JSONL-WIN32
JSONL regression tests cover a torn partial line, preservation of complete open events, synthetic step/turn closers, unchanged committed bytes, and a retry without sequence gaps after fsync failure rollback.FAILURE-JSONL-TEST
15. Crash repair fixes transcript truth, not external side-effect truth
| Last provable pre-crash shape | Synthetic result | Safe continuation |
|---|---|---|
Assistant requested a tool, but no tool/call | TOOL_NOT_STARTED | Retry if still needed |
tool/call exists, but no durable result | TOOL_OUTCOME_UNKNOWN | Retry only read-only/idempotent work; otherwise verify external state or ask the user |
| A Step is open | Append step/end; then close its open Turn with turn/end(interrupted) | Resume from a provider-valid transcript |
| A Turn is open, but no Step is open | Append only turn/end(interrupted) | Resume from the Turn boundary |
| The log is balanced | Generate no events | Leave existing facts unchanged |
The repairer reuses the last real timestamp and consecutive sequence numbers, synthesizes results for dangling tool calls first, then closes Step and Turn. “Started” is determined solely by a durable tool/call. Its model-facing recovery text explicitly forbids blind retry of possible side effects.FAILURE-CRASH-REPAIR
A backend-agnostic shared contract defines recovery for interrupted Turns, not-started tools, and unknown-outcome tools. The in-memory, JSONL, and SQLite backend suites each instantiate that contract, so those semantics are not bound only to JSONL.FAILURE-CRASH-CONTRACTFAILURE-CRASH-CONTRACT-MEMORYFAILURE-CRASH-CONTRACT-JSONLFAILURE-CRASH-CONTRACT-SQLITE
UNKNOWN may feel less seamless than automatic recovery, but it preserves the information boundary honestly. Automation should come from tool idempotency keys, query endpoints, or compensating actions; a log repairer must not invent reality it could not observe.
16. Misconfiguration is not one failure class: boot, hot update, and missing credentials use three strategies
| Scenario | Strategy | Reason |
|---|---|---|
| Invalid initial-composition structure/bounds | Fail loud before registration | No previous safe generation exists |
| Invalid live-settings candidate | Retain the complete last-good snapshot and log | Prevent cross-generation mixtures such as old endpoint plus new key |
| Missing credential | Keep route/catalog discoverable; each request returns MISSING_CREDENTIAL | The next request recovers after a key arrives, without restart |
| Top-level config/user-patch hot-update failure | Reject candidate, retain running tree; accept a later valid edit | Transactional replacement avoids half-applied state |
DeepSeek configuration has both schema and explicit-resolver validation. The first options() runs before adapter publication; a failing live snapshot returns the whole lastGood generation.FAILURE-CONFIG-RESOLUTION Credentials resolve lazily from that request's same connection snapshot. Retry policy is the registration-captured exception, so it updates through synchronous replace to avoid publishing an empty-route window.FAILURE-CREDENTIAL-ROUTE
Dynamic-configuration tests prove that the next request sees both a new endpoint and key; a keyless first request fails then succeeds after key insertion; policy replacement never makes the provider disappear; and an invalid generation containing a new URL leaks none of it before a later valid generation recovers.FAILURE-DYNAMIC-CONFIG-TEST
The application watcher explicitly fails loud for a present-but-invalid patch and hands a new patch set to one entry update; transactional loader semantics preserve the previous running tree.FAILURE-HMR-WATCH Tests cover last-good behavior after parse/validation failure and the next valid refresh.FAILURE-HMR-CONFIG-TEST Another suite covers the full user-patch sequence add → apply failure → parse failure → recovery → removal.FAILURE-HMR-USER-PATCH-TEST
17. Public-contract comparison with Codex: resuming interaction and constraining authority do not prove side-effect outcomes
| Dimension | DeepSeek Harness (chapter source) | Codex (public documentation only) |
|---|---|---|
| Recovery unit | Durable event-log prefix plus Turn/Step/tool closers | codex resume / /resume continues a saved chat; transcript and recorded working directory resume while files come from the current working tree |
| Execution safety | Checkpoints, cooperative abort, scope quiescence | The sandbox sets technical boundaries; approval policy decides when to stop and ask the user |
| Unknown side effects | Explicit TOOL_OUTCOME_UNKNOWN with idempotency-aware verification | The cited public pages do not specify a dangling-tool-outcome crash-repair protocol |
| Comparable conclusion | Application recovery protocol is verifiable from source and tests | Publicly confirmable contracts cover interaction continuation and authority boundaries; private internals are not inferred |
Codex documentation describes continuing a saved session with the resume command. Projects and chats further states that the transcript and recorded working directory resume while files come from the current working tree. Agent approvals & security and Sandboxing describe sandbox and approval as complementary controls (retrieved 2026-08-13).
18. Guarantees, tradeoffs, and gaps left to higher layers
| This layer can guarantee | When | Still owned above it |
|---|---|---|
| Transient model failures can retry boundedly or continuously | Failure enters adapter terminal protocol, policy permits, no cancel/disposal | Cost budget, global backoff, business SLA |
| Failed partial output cannot trigger a tool | Only a complete successful attempt becomes an assistant message | Provider-side cost and request deduplication |
| Tool timeout has a stable classification | The tool declares a timeout and honors its signal | Hard isolation for uncooperative code |
| In-process quiescence follows cancellation/disposal | Adapters, tools, and listeners honor their async contracts | Side effects already accepted by external services |
| A crash log becomes a provider-valid transcript | At least one complete durable prefix exists | Verification, idempotency, or compensation for unknown outcomes |
| Failed live config does not contaminate last-good generation | Updates use the managed transactional seam | Health monitoring for silently expired external secrets/endpoints |
The reusable design is not one error code but three rules: establish who owns the failure facts; record the recovery decision at the same coordinates as the failure; persist intent before irreversible effects and preserve unknown when the outcome cannot be proven.
19. Verification scope for this chapter
Every repository claim targets pinned upstream commit 47f943859bef60e4160492346772ded9b24f765a. This review ran 13 focused source-level test files with 407 cases, then separately ran the hard-crash file under the E2E configuration with 2 cases. All 409 tests across 14 files passed.
13 focused source-level files → 407 passed
1 hard-crash E2E file → 2 passed
total → 409 passedCoverage spans the provider adapter, request recovery, retry and real HTTP/SSE transport, timeout/cancel/disposal, checkpoint/JSONL repair, dynamic DeepSeek configuration, and application-config hot reload. These tests establish the boundary behavior described here; load, performance, cross-machine power loss, and live third-party API availability are not presented as verified.
My Learning Notes
Autosaved only in this browser. Nothing is uploaded or committed. Export Markdown whenever you want to keep a copy.