Observability, Telemetry, and Trajectories
The division of labor among event logs, projections, redaction, OTel, and UI traces
Conclusion: observability is four cooperating planes, not one universal trace stream
DeepSeek Harness does not make OpenTelemetry, the browser, or operational logging the source of truth. Its durable append-only Session log owns interaction facts; a best-effort telemetry seam hands selected copies to an OTel Logs pipeline; host and browser code builds paged projections for people; and local loggers retain operational failures that should neither enter the model transcript nor escape to untrusted callers. Session-query “trace” is a fifth view over the first plane: deterministic event relationships and session lineage, not distributed span tracing.
This separation is the chapter's central result. It makes replay and inspection possible even when export is disabled, keeps exporter failure out of the agent loop, and supports privacy gates at the outbound-copy boundary. It also means there is no single lossless observability feed: the ledger is semantically complete but local and sensitive; telemetry is externally useful, but capture handoff maintains only a high-water cursor with no durable acknowledgement or outbox, while the downstream SDK may retry, so the end-to-end path is neither lossless nor exactly-once; the browser sees a loaded window plus host projections; and local diagnostics can contain stacks that are absent from every other surface.
1. Signal-ownership map
| Plane | Primary unit | Authority | Loss / visibility boundary |
|---|---|---|---|
| Canonical Session ledger | Contiguous raw SessionEvent | Replay, model-visible history, lifecycle outcomes | Complete payloads may remain local and include secrets |
| Session telemetry + OTel | Ledger copy or operational record | External reporting and collector-side analysis | Best effort, redaction-dependent, capture-side high-water handoff; SDK may retry |
| Host/browser projections | History page, projection value, trajectory record | Human inspection and whole-log summaries | Raw history is windowed; derived values may degrade independently |
| Local operational logger | Warning / full diagnostic | Plugin, persistence, query, and projection health | Not replayable session truth and not necessarily exported |
| Session-query trace | Lineage or event-relationship result | Exact relationships over canonical logs | Optional model tools add workspace authorization and sanitization |
An operator must correlate planes by identity and sequence instead of asking one stream to answer every question. “What happened to the conversation?” begins at the ledger; “did export fail?” begins in local diagnostics; “what left the machine?” begins at sharing mode, redaction, and the OTel queue.
2. The canonical ledger is the interaction truth—and the largest privacy surface
The Session event map is explicitly append-only, lossless JSON with contiguous sequence numbers. It retains turn and step boundaries, raw stream chunks, assembled messages, raw tool arguments, tool results and private presentation metadata, todos, request headers, context markers, and plugin-extended events. Durable turn/end reasons distinguish completed, aborted, blocked, structured error, max-token, and crash-repaired interrupted outcomes.OBS-SESSION-LOG
A request/header is not a small correlation header. It can carry the effective model configuration, the complete rendered system prompt, and the full tool catalog with schemas; the Agent appends an initial, resume, or changed snapshot before dispatch whenever the effective header needs recording.OBS-REQUEST-HEADER
Logging enough to reproduce model-visible state provides unusually strong replay and forensic properties, but it makes the local log a high-value sensitive asset. “Telemetry disabled” means no OTel export; it does not mean messages, prompts, tool payloads, or local paths are absent from the canonical log.
3. Capture converts ledger facts into a small outbound contract
canonical Session event
│ structuredClone(event.data)
▼
SessionTelemetryRecord
channel: ledger | ops
time + severity
minimal identity attributes
complete structured body
│ synchronous redaction waterfall
▼
backend.emit() ── enqueue only ──→ exporter-owned queueLedger records preserve the event timestamp and complete cloned event.data, while attributes duplicate only session/event identity and optional lineage facts. Operational records have no event sequence and therefore cannot masquerade as ledger rows. The sink contract requires emit() to be a synchronous non-blocking enqueue; batching, retries, queue bounds, and loss after handoff belong to the backend SDK.OBS-TELEMETRY-CONTRACT
Capture applies one fixed volume projection: only the first assistant/chunk for each (turn, step) is handed off as the stream-start signal. The assembled assistant/message later carries byte-complete content. All other event types pass through as full bodies.OBS-TELEMETRY-PROJECTION
This is event mirroring with one deliberate high-frequency reduction, not a conventional span model. A collector can reconstruct ordered ledger records from (session.id, event.seq), but it does not receive one chunk record per token delta.
4. Redaction is synchronous, composable, fail-closed per record—and absent by default
session-telemetry/record is a synchronous waterfall on the outbound copy. Rules compose around next(); a rule may replace data beneath it, and a throwing rule withholds that one record while capture containment protects the agent loop. Live mode evaluates rules at append time; feedback-gated mode applies the currently mounted rules at feedback time to the canonical suffix after the handoff cursor—starting at firstLiveSeq when no cursor exists—through that feedback event. The canonical ledger is never rewritten.OBS-TELEMETRY-REDACTION
Tests pin four separate properties: no listener passes a secret unchanged; one rule scrubs both ledger and operational records; multiple listeners stack; and a throwing rule suppresses only its candidate. They also prove that the canonical log still contains the original secret after the exported copy is scrubbed.OBS-TEST-REDACTION
5. The handoff cursor means “offered to the backend,” not “delivered to the collector”
A module-scope WeakMap<Session, seq> survives telemetry fiber reloads while the Session object remains live. On adoption, the coordinator replays after that cursor; without a cursor it begins at firstLiveSeq, so a resumed process does not backfill the previous process's failed exports. The cursor advances only after backend.emit() returns, and its own documentation says it marks handed-off rather than delivered.OBS-TELEMETRY-HANDOFF
emit() acknowledges only an enqueue. A later queue overflow, exporter retry exhaustion, or process exit can lose a cursor-advanced record. More subtly, if record N throws but N+1 succeeds, the high-water cursor advances past N; hot reload will not retry the hole. This is a capture-side at-most-once high-water posture; downstream SDK retry is a separate layer, so it establishes neither end-to-end at-most-once nor exactly-once behavior.
Regression tests preserve the cursor across coordinator reload, avoid replaying already handed-off events, and show a rejected middle event does not prevent later events from being offered. They also verify first-chunk-only projection and non-blocking flush hints.OBS-TEST-HANDOFF
6. Three sharing modes place consent at different commit points
| Mode | Capture moment | What is released | Operational records |
|---|---|---|---|
FULL | Every live append | Each projected and redacted event immediately | agent-error and shutdown |
FEEDBACK_ONLY | Canonical feedback/record | Canonical suffix after the handoff cursor—or from firstLiveSeq without one—through that exact event | None from on-demand capture |
DISABLED | Never | Nothing; no SDK pipeline is constructed | None; local feedback warning only |
The backend validates mode before transport setup. FEEDBACK_ONLY accepts consent only when the emitted object is the exact event already committed at session.events[event.seq]; a look-alike bus value is ignored. It then reads and redacts the canonical suffix after the handoff cursor—or from firstLiveSeq when none exists—through the feedback sequence. DISABLED creates neither coordinator nor provider/processor/exporter state.OBS-TELEMETRY-MODESOBS-TEST-OTEL-MODES
The /feedback producer records one log-only event without starting model work. Its acknowledgement distinguishes full, feedback-gated, disabled, and unconfigured sharing, and notes only that the entry was logged—the append is not synchronously flushed to disk.OBS-FEEDBACK-DISCLOSURE
Feedback-gated sharing does not mean “send only my feedback text.” It may release all eligible current-lifecycle content—from firstLiveSeq when no cursor exists, or after the cursor—that survives the current redaction policy, but it does not resend resume/fork constructor seed in that process. Later events remain local until another feedback record.
7. Shipped default-off; the CLI gives the known telemetry row a final export opt-out
The base bundle mounts the OTel plugin but resolves an absent mode to DISABLED. Explicit FULL or FEEDBACK_ONLY uses an OTLP/HTTP logs endpoint, gzip, a 10-second batch cadence, queue and batch sizes of 2,048, a one-second exporter timeout, a 1.5-second processor export timeout, and a three-second outer shutdown deadline. The bundle mounts no redaction listener.OBS-BASE-DEFAULTOBS-TEST-OTEL-COMPOSITION
Within the apps/cli profile composition, a non-empty DSH_TELEMETRY_DISABLED value—including 0 or false—adds a disabling patch for the fixed row id session-telemetry-otel after the normal composition layers. If the composition has no row with that id, the launcher emits no patch; the opt-out is naturally satisfied only if telemetry is not mounted under another row id or backend.OBS-HARD-OPTOUT
The layered policy is privacy-biased: explicit positive mode selection is needed to upload, while an imprecise but non-empty kill switch resolves to off. Effective configuration inspection still matters, because a custom profile may choose a different backend, endpoint, queue, or redaction rule.
8. “OTel” currently means structured logs, not spans or meters
The implementation constructs LoggerProvider → BatchLogRecordProcessor → OTLPLogExporter, creates separate ledger and ops instrumentation scopes, maps record time to both OTel timestamps, maps three severities to INFO/WARN/ERROR, and forwards body and attributes. Its resource contains product name/version and a best-effort-persisted anonymous user.id.OBS-OTEL-PIPELINE
That user id is a random UUID stored under the Harness home. After persistence succeeds and converges, later processes reuse the same value. A narrow create-to-write race on concurrent first launch can still produce two process IDs for that run, while an unwritable home guarantees stability only within the current process. Deleting the file resets it on the next launch. It is not derived from hostname, IP address, or Git metadata.OBS-ANON-ID
At the pinned commit, the telemetry package's OTel dependencies are the generic API, Logs API/SDK, resources, the OTLP exporter base, and the OTLP log exporter, with no trace or metrics SDK; the implementation constructs only the Logs pipeline described above.OBS-OTEL-DEPENDENCIES The wire test sends OTLP/HTTP JSON through the real OTel SDK pipeline to a local scripted node:http mock collector, then verifies service.name/user.id, ledger/ops scopes, a source timestamp, severity, session.cwd/event attributes, and the shutdown marker.OBS-TEST-OTEL-WIRE
Do not infer “distributed traces are supported” from the package name or the word OTel. That claim must be re-audited if trace or metrics SDK dependencies, provider construction, span context propagation, or new exporters appear.
9. Batching and shutdown improve the tail but do not create delivery guarantees
The backend intentionally implements no per-turn flush(); normal export cadence belongs to the batch processor. On disposal, live sessions first enqueue a shutdown ops marker, then the coordinator awaits backend shutdown. The backend races provider.shutdown() against its own deadline because an SDK transport force-flush can outlive the processor's export timeout. Expiry rejects the wait but cannot cancel the underlying transport, and coordinator containment allows application teardown to continue.OBS-OTEL-SHUTDOWN
| Moment | What can be claimed | What cannot be claimed |
|---|---|---|
backend.emit() returns | The SDK accepted a synchronous enqueue attempt | The collector stored the record |
| Scheduled batch export succeeds | That SDK batch completed its export callback | All earlier process records are gap-free |
| Provider shutdown resolves | The SDK reports quiescence | The remote retention/query system is healthy |
| Outer deadline expires | Teardown is no longer blocked | Pending records survived process exit |
10. Failure semantics are richer in the ledger than in the exported ops channel
Capture marks only three cases as error severity: tool results whose own block has isError=true, turn/end with reason error, and live agent-error. Other core and plugin events default to info; an aborted turn, a max-token stop, and a clean shutdown therefore are not ERROR records merely because an operator may care about them.OBS-TELEMETRY-SEVERITY
The durable ledger retains structured turn-end reasons and tool-result error identity. Separately, throwError() emits the live agent/error with turn and step before rethrowing; the driver then contains an already-reported failure. Telemetry normalizes that arbitrary error to name and message only—no stack—and puts coordinates in ops attributes.OBS-DURABLE-FAILURE
Severity is a conservative capture default, not a full incident policy. Collector alerts that care about abort frequency, max-token saturation, missing closers, retries, or plugin-defined outcomes must inspect event type/body rather than filter only severity=ERROR.
11. Live UI errors and local logs expose different failure detail
The Host API contract defines host/agent-error as the only outlet for a live failure with no turn position. Its frame shape contains only session id and message, with no turn/step coordinates. It belongs to the host stream rather than the durable Session event log, so a disconnected browser can miss it.OBS-HOST-FAILURE
Local warnings cover failures intentionally kept out of the ledger: telemetry capture exceptions and backend shutdown failure;OBS-OPS-LOGS background persistence writes whose events remain buffered;OBS-PERSISTENCE-DIAGNOSTIC and projection computation that degrades session.list and subagent.history instead of blocking them.OBS-PROJECTION-DEGRADEOBS-SESSION-LIST-PROJECTIONOBS-SUBAGENT-HISTORY-PROJECTION Ordinary session.history projection is not inside those two fail-soft catches.OBS-SESSION-HISTORY-PROJECTION Session-query best-effort logs a complete printable stack/cause chain, uses a fixed placeholder for unprintable values, then translates the failure for a model caller.OBS-QUERY-DIAGNOSTIC
This avoids contaminating the transcript with infrastructure noise and avoids exposing stacks over product APIs. It also means postmortems need both durable logs and process logs. A clean transcript does not prove that telemetry, projection, indexing, or persistence background work was healthy.
12. Session statistics are a deterministic whole-log read model
The sessionStats projection reports distinct turns with a closed step, closed steps, model wall time, matched tool wall time, summed TTFT and its sample count, and decode duration/tokens. Its contract explicitly says values fold over the complete durable log independently of how much history a client has loaded.OBS-STATS-CONTRACT
The pure fold opens timing at step/start, captures the first non-empty token delta, closes model/decode timing at assistant/message, pairs tool calls/results by callId, counts at step/end, clears unresolved calls at turn/end, and clamps negative clock differences to zero. It guards provider usage and inherited object keys rather than trusting event payloads blindly.OBS-STATS-FOLD
The browser prefers the host projection and falls back to the visible-window fold only while the sessionStats projection value is unavailable. It displays turn/step counts, model/tool durations, average TTFT, decode tokens per second, cache hit ratio, and token totals; token billing comes from a separate projection.OBS-STATS-UI
13. Stats measure recorded event intervals, not provider billing or OTel metrics
| Figure | Recorded definition | Important exclusion / distortion |
|---|---|---|
| Turns / steps | Distinct turns containing step/end; all closed steps | Rejected or empty turns are not counted |
| Model time | step/start → assistant/message | Includes in-step retry waiting; cancelled partial steps are untimed |
| TTFT | step/start → first non-empty token delta | First token survives an in-step retry |
| Decode rate | Provider output tokens over first-token → message duration | Absent/invalid usage contributes no token-rate sample |
| Tool time | Each matched call/result interval | Parallel intervals are summed; unresolved calls are excluded |
Projection tests cover closed, rejected, cancelled, max-token, retry, tool-pair, invalid-usage, clock-skew, late-mount, and reload cases.OBS-TEST-STATS A paged-history browser test shows that the tail page already carries whole-log totals and loading older events does not change them.OBS-TEST-STATS-PAGING
These numbers are reproducible product telemetry, but they are not OTel instruments, exclusive CPU time, network-only latency, or invoice truth. Parallel tool durations can exceed wall-clock elapsed time, and retry waits remain part of the model interval by construction.
14. Raw history remains contiguous while the browser pages and repairs gaps
The history API pages on append-origin message boundaries so it never cuts a message from its chunk/tool events. Each entry includes the raw event plus an optional host-computed tool view. Only a tail page carries the in-flight partial and projection baseline, and reading history never resumes or publishes an Agent.OBS-HISTORY-API
The client prepends only a page whose final sequence is exactly adjacent to its current base; a discontinuous older page is logged to the console and dropped fail-soft. Live duplicates are removed by sequence. A forward gap buffers incoming events and repulls the tail page, then stitches the buffer into one contiguous raw window. Transport exceptions thrown by loadOlder or gap repair also go to the console, while an RPC error result silently preserves the current window.OBS-HISTORY-CLIENT
Live mux frames carry raw session/event values. Projection updates are explicitly transient and recomputed from replay, while allowlisted remote events may be forwarded verbatim with no projection or redaction at that path.OBS-RAW-FRAMES
Browser continuity is a consistency property of the loaded slice, not proof that the browser holds the complete log. Whole-log answers should come from host projections or exact query services; trajectory/search counts over the UI window must be labeled as window-scoped.
15. Trajectory is a browser event ledger, not the OTel trace pipeline
The trajectory package advertises a turn-aware, virtualized browser ledger with timeline, search, folding, local inspector, and older-page loading. Selection, request totals, and navigation cover the currently loaded window; in-flight durations remain blank rather than fabricated. It contributes no service and nothing to a model request.OBS-TRAJECTORY-CONTRACT
Its builder sorts definition contributions by raw sequence, assembles finalized records, requests, call schemas, partial assistant output, running calls, compaction boundaries, and turn errors. This is a target-specific read model over the shared Session window; it neither changes the canonical log nor reads the chat view as its authority.OBS-TRAJECTORY-BUILDER
Trajectory optimizes local forensic comprehension—chronology, prompt changes, tool causality, timing—not remote correlation. Calling it a “trace view” is reasonable in a product sense, but it has no trace/span ids, parent span context, sampling policy, or exporter semantics.
16. Prompt inspection is powerful—and makes browser access part of the privacy model
Trajectory consumes each request/header to reconstruct model configuration, full system text, and tool schemas, then calculates initial/system/tools change records against the prior prompt snapshot. The inspector can therefore explain exactly which prompt or tool catalog a request used.OBS-TRAJECTORY-PROMPT
The same completeness that makes trajectory useful for prompt drift analysis makes “safe observability UI” a product security question, not just a rendering concern.
17. Session-query “trace” means deterministic lineage and event relationships
The unified query service makes exact reads, filters, surfaces, and traces backend-independent; only full-text search is delegated to an indexing backend. A complete read uses a live-preferred source, replay-validates the raw log, and returns detached clones.OBS-QUERY-SERVICE
traceEvent identifies the target's current/shadowed/log-only surface class, positional replacement chain, replaced events, cited source events, and later events directly derived from the target. traceSession walks parent ancestry and sorted descendants, reports an unresolved parent explicitly, and rejects cycles instead of inventing a complete tree.OBS-QUERY-TRACE
A single-target exact load (such as readSession, readEvent, or traceEvent) reads a known live target directly from memory without consulting persistence, so backend failure cannot hide current history from those reads; corpus-wide traceSession, list, and filter operations do not have that guarantee. A persisted exact target is listed and then inspected; if it remains detached, its loaded and listed headers are checked for compatibility. If it becomes live during inspection, the detached compatibility branch is bypassed and the live snapshot wins.OBS-QUERY-CORPUS
No distributed tracing claim follows from these APIs. Their inputs are session ids and event sequence numbers, and their outputs are log-derived relationships. They do not start spans or propagate trace context.
18. Model-facing history access is opt-in, workspace-scoped, and re-authorized after observation
An optional plugin registers five tools: session search, event search, session lineage trace, event relationship trace, and exact event read. Shipped host compositions do not mount it by default. Exact reads can return the full unabridged target JSON, and a generic spill policy—not this package—must bound oversized inline results.OBS-QUERY-TOOLS
The caller identity comes only from the executing Agent. Self-read is allowed; cross-session reads require exact cwd equality, and a caller without a cwd can inspect only itself. Direct targets are authorized before the service call and checked again against the observed header afterward, limiting a source-change/TOCTOU escape. Unauthorized descendant subtrees become null boundary markers with no hidden ids.OBS-QUERY-AUTH
Current-session event search stops immediately before the active step/start, so the invoking output and tool call cannot search themselves.OBS-QUERY-CURRENT-BOUNDARY Service failures best-effort log full diagnostics locally, using a fixed placeholder for unprintable values, then cross one sanitizer that exposes stable model-safe codes/messages; caller cancellation and precise local authorization failures are preserved.OBS-QUERY-SANITIZE
Exact-string cwd equality is conservative but not a security principal equivalent to canonical filesystem identity: symlink-equivalent workspaces do not share authority, while every session intentionally created with the same string does. Deployment policy still decides whether the five powerful read tools should be visible.
19. Privacy boundary matrix
| Data | Canonical log / browser | FULL export without custom rules | FEEDBACK_ONLY (from firstLiveSeq without a cursor; otherwise after the cursor) | Structurally absent |
|---|---|---|---|---|
| User/assistant content | Full | Full captured copy | Eligible suffix through feedback | No |
| Tool arguments/results, command output, file content | Full when logged | Full captured copy | Eligible suffix through feedback | No |
| System prompt and tool schemas | Full request-header snapshots | Full captured copy | Eligible suffix through feedback | No |
| Todo, compaction, hook summary, feedback | Full event data | Full captured copy | Eligible suffix through feedback | No |
| Local cwd and lineage ids | Header/event context | Attributes/resource correlation | Released records | No |
| Adapter API credentials | Not Session events | Absent from this path | Absent from this path | Yes, by current structure |
| Error stack | Turn retains flattened/structured failure facts, not a full stack | Ledger retains those turn-failure facts; ops has name/message only; neither carries a full stack | An eligible suffix can include turn-failure facts; no ops capture | A full stack may remain only in local logs |
The backend README enumerates the complete categories that leave the machine in uploading modes and explicitly says no redaction rule ships with the seam. API keys are absent because adapter credentials are constructor inputs, not Session events—not because arbitrary secrets in prompts, files, commands, or tool results are detected automatically.OBS-PRIVACY-BOUNDARY
Once persisted and converged, the anonymous UUID is pseudonymous correlation, not unlinkability. It avoids deriving identity from machine/network metadata, but that stable value still permits aggregation across runs sharing one Harness home.
20. Public-contract comparison with Codex: both default OTel log export off, but their observable products differ
| Dimension | DeepSeek Harness at the pinned source | Codex public documentation only |
|---|---|---|
| Default | Session telemetry mode DISABLED | OTel log export is disabled by default; [otel].exporter must be set explicitly to otlp-http or otlp-grpc; anonymous usage/health metrics are a separate channel enabled by default |
| Log model | Near-mirror of canonical session events plus two ops records | Purpose-built events for conversations, API/SSE/WebSocket activity, prompts, tool decisions, and tool results |
| Prompt privacy | No built-in redactor; deployment supplies waterfall rules | User prompt content redacted unless log_user_prompt=true |
| Signals | OTLP Logs pipeline only in reviewed implementation | Docs list log, metrics, and trace exporter configuration; counters/histograms cover API, stream, WebSocket, and tool calls |
| Shutdown | SDK shutdown raced against a Harness-owned outer deadline | Docs say exporters batch asynchronously and flush on shutdown |
| Product trace | Separate log-derived session/event relationship queries | No equivalence inferred from public OTel configuration alone |
Codex's Observability and telemetry documentation describes opt-in OTel log export, redacted user prompts by default, representative structured events, metrics, batching, and shutdown flushing. Its configuration reference exposes distinct log, metrics, and trace exporter settings (retrieved 2026-08-13).
21. Drift watchlist and open operational questions
| Watch item | Why the conclusion could change | What to re-verify |
|---|---|---|
| OTel signal expansion | Trace/meter SDKs may be added | Providers, exporters, span context, metric instruments, resource attributes |
| Default sharing posture | Bundle or launcher composition may change | Effective mode, final opt-out ordering, endpoints, headers |
| Redaction baseline | A built-in or bundle-mounted rule may appear | Rule order, fail behavior, ledger/browser scope, test fixtures |
| Delivery semantics | An acknowledged queue or persisted outbox may replace handoff cursor | Ack point, retry identity, gap recovery, shutdown durability |
| Stats semantics | New events or timing anchors may alter totals | Retry/cancel/parallel-tool math and paging stability |
| History authorization | Workspace identity may move beyond raw cwd equality | Canonicalization, tenant/project identity, TOCTOU checks |
Source tests can prove mapping, containment, mode gates, and local batching behavior. They do not prove a production collector's authentication, TLS policy, retention, deletion, access control, alert rules, queue saturation behavior under sustained outage, or whether operators actually correlate ledger gaps with process logs. Those remain deployment evidence, not repository facts.
22. Verification scope for this chapter
Every repository claim targets pinned upstream commit 47f943859bef60e4160492346772ded9b24f765a. Focused verification covers telemetry capture/redaction/handoff, OTel wire mapping and mode composition, whole-log statistics, session-query tracing and model-tool authorization, browser trajectory assembly, and paged-history behavior.
14 focused unit/source files → 361 passed
1 real Loader E2E file → 3 passed
total → 364 passed across 15 files
structural parity → 23 sections / 45 evidence records, exact orderThe built-web paged-history E2E was inspected as evidence but not executed: this isolated task forbids a build and the pinned checkout has no web dist artifact. The 364 executed tests establish repository behavior at the pinned revision; they do not substitute for a live collector integration audit, privacy review of a real deployment's redaction rules, long-outage queue testing, or retention/access-control verification.
My Learning Notes
Autosaved only in this browser. Nothing is uploaded or committed. Export Markdown whenever you want to keep a copy.