Gaps, Limitations, and Open Questions
Semantics and evidence that remain incomplete during Developer Preview
Conclusion: there is no single completeness score, only a ledger of questions in different evidence states
DeepSeek Harness deliberately maintains a limitations ledger in its public repository, but the existence of that ledger does not make every limitation the same kind of claim. This chapter uses five strict categories: confirmed gaps directly established by source and public contracts; weak guarantees where a capability exists but delivery, isolation, or consistency ends at a stated boundary; documentation/code drift where public prose and implementation disagree in the same commit; platform differences where operating systems or runners provide different mechanisms and strength; and runtime experiment questions that static inspection cannot decide without a real network, process, crash, or concurrency fixture.
The classification prevents two common mistakes. An explicitly excluded capability is not automatically a defect, and the absence of a runtime experiment is not evidence that the system already fails. Every repository fact is pinned to commit 47f943859bef60e4160492346772ded9b24f765a.
1. The five evidence states are not interchangeable
| State | Criterion used here | Supported conclusion | Unsupported conclusion |
|---|---|---|---|
| Confirmed gap | Implementation, type contract, or public Known Limitations explicitly says absent, stubbed, reserved-but-unimplemented, or returns an unsafe value on a named path | The capability is absent or unsafe to rely on at this commit | A production incident has occurred |
| Weak guarantee | A capability exists, but acknowledgement, atomicity, isolation, consistency, completion, or recovery covers only a limited boundary | The caller owns the remaining policy | The implementation is worthless |
| Documentation/code drift | Public contract and implementation in the same commit directly disagree about a value or coverage surface | At least one side must change; consumers cannot trust both | All hand-written documentation is stale |
| Platform difference | Linux, macOS, Windows, or CI runners explicitly provide different mechanisms, strength, or verdict weight | Every conclusion must name platform and runner | The weaker platform is necessarily buggy |
| Runtime experiment question | The answer depends on DNS, a kernel, an SDK, a filesystem, crash timing, concurrent processes, or live-model behavior | A falsifiable experiment can be specified | Source alone proves pass or failure |
The repository gate scans every packages/*/*/package.json, requires a sibling README with the one canonical heading and at least one top-level bullet, and in this run reported 219 package READMEs checked, one whitelist entry, and full conformance.OPEN-LIMITATIONS-GATE
2. Confirmed gaps: two input boundaries can still carry a sensitive target or secret across
Settings wire redaction: redactSecrets recurses only through object, dict, and array. Its default branch explicitly says that a secret reachable only through a union, intersection, or transform is returned verbatim with no record of the miss.OPEN-SETTINGS-REDACTION-CODEThe public README adds a second path: schema.toJSON() carries a secret field's default to the client, and neither case currently fails closed.OPEN-SETTINGS-REDACTION-DOC
HTTP fetch SSRF: the provider bounds URL shape, protocol, size, timeout, content type, and same-origin redirects, but it explicitly has no private, loopback, link-local, or multicast destination block, no resolve-then-validate step, and no per-hop revalidation. The public contract calls it an SSRF primitive in a deployment that can reach sensitive internal targets and says not to enable it there.OPEN-WEB-SSRF
These are not merely opportunities for more defense in depth. The current wire and network admission paths cannot cover named schema shapes or destination classes. Existing schemas, cookie-free requests, same-origin redirects, and byte caps do not substitute for a fail-closed secret proof or destination policy.
3. Confirmed gap versus scope boundary: file confinement is not hostile-code isolation
The sandbox seam's entire policy vocabulary covers file effects. Network, process, syscall, device, and credential restrictions are outside its contract. It is also same-world confinement; denial classification depends on a stderr dialect, and runner diagnostics are in-band, so a malicious child can forge a diagnostic and cause false attribution, though not bypass the underlying confinement by doing so.OPEN-SANDBOX-SCOPE
The current workflow engine pays one worker thread per run. It prevents a synchronous script from blocking the host event loop and enables forced termination, but its documentation explicitly says node:vm is API shaping: an escaped script can recover host-process authority, and genuinely untrusted scripts need a different process or container engine.OPEN-WORKFLOW-ISOLATION
“Sandbox does not restrict networking” is an explicit scope of the implemented seam, not evidence that its file policy fails. “The current workflow engine cannot host hostile scripts” is a confirmed gap relative to an untrusted-code deployment. Together they show why the words sandbox, worker, and vm do not imply a general security boundary.
4. Confirmed gaps: long-running work, resource budgets, and lifecycle still have empty seats
Workflow supports foreground collection only and journals neither scripts, child progress, nor intermediate values, so a process restart cannot resume a run. The interface also has no cross-child token-budget vocabulary, and the holder rather than a service registry tracks each live run.OPEN-WORKFLOW-DURABILITYRalph accepts a worker's self-declared completion with no independent evaluator, while round count is the only aggregate effort cap; token, price, and elapsed-time budgets remain deferred.OPEN-RALPH-VERIFICATION
Attachment objects are immutable and may be shared by resumed and forked sessions, but retention and reference-aware garbage collection are not implemented; the local backend publicly says it retains objects indefinitely.OPEN-ATTACHMENT-RETENTIONOPEN-ATTACHMENT-LOCAL-RETENTIONThe generic atomic-write primitive provides visible same-directory rename atomicity but does not fsync the file or directory, so it is not crash durability; a lock orphaned by owner death also requires operator recovery after proving that no writer remains.OPEN-ATOMIC-DURABILITY
These gaps do not erase the current value of foreground workflows, immutable attachments, or atomic replacement. They bound the product semantics that cannot be promised: process-resumable orchestration, an aggregate model budget, automatic object reclamation, and crash-durable commit for generic atomic-write callers.
5. Weak guarantee: recovery preserves unknown outcomes instead of creating exactly-once execution
Checkpoint policy persists execution intent before a top-level tool body. If a crash lands after the call is durable but before its result is durable, recovery writes TOOL_OUTCOME_UNKNOWN. The contract explicitly says it cannot prove whether the external effect completed and therefore does not retry automatically; a side-effecting provider should use exec.callId as an idempotency key when supported.OPEN-CHECKPOINT-UNKNOWN
The guarantee is “recover the intent and do not disguise uncertainty as success,” not exactly-once effects in an arbitrary external system. This is not a missing universal algorithm: without provider-side idempotency, querying, or compensation, an in-process runtime cannot infer the real-world outcome from a durable call alone.
6. Weak guarantee: the telemetry cursor is a handoff watermark, not a delivery acknowledgement
Telemetry disclosure claims only that a record was handed nonblockingly to a backend; batching, retry, and loss policy belong to its SDK. A module-scope cursor records the highest handed-off sequence rather than a delivered sequence, and resume does not backfill records that a previous process failed to deliver.OPEN-TELEMETRY-HANDOFF
The seam also ships no redaction rule. Without a listener, records leave the process as captured, and data in a backend queue may be lost at crash time.OPEN-TELEMETRY-LIMITSThe OTel backend leaves authentication, TLS, throttling, and live-collector behavior to the upstream SDK, and its shutdown deadline cannot cancel transport.OPEN-OTEL-BOUNDARY
“Full” means every eligible event enters the handoff path, not that it was delivered. “Feedback-only” means an eligible prefix is replayed at feedback time, not that a capture-time snapshot exists. End-to-end delivery, deduplication, privacy, and retention require separate proof in the deployment and collector.
7. Confirmed gaps: reserved seats in the public API and client lifecycle
The session.list request accepts an optional cursor, but v1 returns every persisted session and the cursor is only an unimplemented reserved seat. events.mux.since is likewise ignored; reconnect means reopening the stream and refetching history.OPEN-API-RESERVED-SEATSOPEN-EVENT-RESUME-SEAT
The client runtime's public limitations also say that loader.unload is a not-implemented stub, with no general chain from fiber disposal through registration and style removal.OPEN-CLIENT-UNLOAD
These are pinpointed missing capabilities, not evidence that reconnect or effect cleanup is wholly absent. Reconnect already has a reopen-plus-refetch path, and many concrete contributions clean up through fiber effects. The exact gaps are wire-level cursor resume, scalability of the unpaginated list, and a general module-unload chain.
8. Documentation/code drift: host.describe.version is not the value its contract specifies
The public Host API comment defines version as the package version of apps/cli.OPEN-HOST-VERSION-CONTRACTThe implementation in the same commit carries a TODO and returns the constant 0.0.1, while that commit's CLI package version is 0.1.0-rc.5.OPEN-HOST-VERSION-IMPLOPEN-CLI-VERSION
This is not another name for missing protocol-version negotiation; the contract specifically describes a host application version. Until corrected, consumers cannot treat the field as the real application version or infer compatibility from it.
A smaller public documentation omission appears in the sandbox seam's implementation summary: the parent README's implementation parenthetical lists only Linux and macOS, while the same page's error text and the provider README include the Windows ACL runner. The provider's existence is unaffected; the summary coverage needs correction.OPEN-SANDBOX-DOC-OMISSIONOPEN-SANDBOX-PLATFORM-LADDER
9. Platform differences: one “sandbox” or “process cleanup” label does not imply one strength
| Surface | Linux | macOS | Windows | Correct interpretation |
|---|---|---|---|---|
| Local file confinement | bwrap first, then Landlock; older ABIs may report partial | Seatbelt, depending on deprecated sandbox-exec | ACL restricted token, explicitly partial because of Everyone grants and NTFS hard links | enforcement is a backend- and ABI-specific fact, not a property of the mode name alone |
| Terminal process observation | Tree/session inspection; a child calling setsid can leave the observable set | ps snapshots; a child reparented before the first snapshot can be lost | No terminal inspector; ordinary tree kill is best-effort taskkill | Process-tree termination is not continuous OS containment |
The provider exposes rather than hides these differences: Windows ACL and some Landlock ABIs report partial, while Seatbelt's future viability depends on Apple retaining its policy engine.OPEN-SANDBOX-PLATFORM-LIMITSThe local subprocess provider likewise lists Windows best-effort behavior, the Linux/macOS inspector boundary, and daemonized-descendant escape.OPEN-PROCESS-PLATFORM-LIMITS
A Linux bwrap result cannot be extrapolated to Windows ACL, and Windows being partial does not mean it provides no constraint. A deployment claim must name the mode, backend, kernel or ABI, reported enforcement, and effects actually tested.
10. Verification gaps: deterministic replay has a named unexplored state space
LLM replay binds live sessions to recorded parent and child scripts in first-call order, which assumes sequential delegation. Concurrent siblings would bind nondeterministically, and stronger keying is deferred. A pure pre-chunk throw, cancel or hang, and an unmarked external summarizer call also need an explicit sidecar.OPEN-REPLAY-CONCURRENCY
For each eligible recorded scenario, the ACP snapshot suite runs the real assembled entry, normalizes volatile values, and separately compares stdout plus harvested parent and child logs with fixtures.TINV-SNAPSHOT-COMPAREIts session harvest supports raw JSONL only; compressed JSONL and SQLite compositions have no harvest path.OPEN-SNAPSHOT-BACKEND-GAPA green result is therefore evidence for the recorded scenario and harvested channels it actually compared, not concurrent-child interleavings or the same path through every persistence backend.
The missing capability is counterexample generation and fixture collection, not a proven production-semantics failure. Stronger replay identity and backend-neutral harvesting must land before that state space can enter deterministic regression.
11. CI aggregate boundary: a signal can exist outside one workflow's aggregate
The pull-request windows job runs Windows Node under Wine on hosted Linux and is named in all-checks-passed.needs. Its invoked runner verifies the Windows Node archive and win32 x64 identity, then runs only the workspace build and production-site build surfaces.OPEN-CI-WINDOWS-SIGNALSTINV-WINE-RUNNERA real-Windows-kernel job runs the complete inventory but is explicitly absent from that needs list.OPEN-CI-VERDICTThe hosted serial reference jobs for Linux and macOS are currently if: false; self-hosted serial lanes run only on master pushes.OPEN-CI-SERIAL-LANES
Real-API E2E is a separate workflow. Fork and Dependabot pull requests skip the entire job under the secret model, and GitHub reports that job-level skip as a successful check.OPEN-REAL-API-CIThe master-push real-kernel sandbox matrix covers bwrap, two Landlock architectures, and Seatbelt, and its command asserts that both platform files ran so a self-skip cannot appear green.OPEN-SANDBOX-CINeither workflow is part of the all-checks-passed dependency graph in ci.yml.
This does not mean “CI has no tests”; the repository has independent observational and reference signals. Static workflow files establish triggers, jobs, and the scope of the all-checks-passed aggregate. This study did not verify the repository's live ruleset or required-status configuration, so it does not claim which checks can actually block a merge.
12. Cross-process consistency and resource bounds: local serialization is not a global guarantee
Message feedback's per-Session mutation queue serializes one service process only. Storage-domain has no cross-process conditional write, so several Host processes writing one storage root have no compare-and-swap or lost-update guarantee. It also has no durable Session deletion cascade and no item-count or aggregate-row byte cap.OPEN-FEEDBACK-CONSISTENCYThe settings seam likewise serializes by namespace only in-process; cross-process convergence belongs to the provider, and the local file provider resolves same-namespace conflicts last-write-wins.OPEN-SETTINGS-CROSS-PROCESS
For resources, the browser /api bridge buffers each full request body in memory, with a default resident bound of 160 MiB. The native-command helper buffers both stdout and stderr without bounds and is currently justified only by callers expected to produce tiny output.OPEN-API-BODY-BOUNDOPEN-NATIVE-OUTPUT-BOUND
A single-process queue and today's small-output callers are valid local premises, not multi-writer consistency or general resource isolation. Deployments sharing a storage root, broadening the command surface, or accepting concurrent image requests near the limit must validate memory and conflict behavior separately.
13. Runtime experiments: every question needs a falsifiable verdict
| Priority | Question | Minimum fixture | Falsifiable verdict | Evidence state |
|---|---|---|---|---|
| P0 | Can any wire-exposed secret schema leak? | Place a canary secret in union, intersection, transform, default, and error-text paths and read it through the real settings describe carrier | Unsupported schemas fail before publication; no response, error, or serialized schema contains the canary | Current gap confirmed; fix efficacy needs experiment |
| P0 | Can HTTP fetch cross DNS rebinding or redirects into an internal address? | Dual-answer DNS, TTL switches, IPv4/IPv6 and mapped forms, per-hop redirects, and a local metadata-canary server | The actual destination is validated after resolution and at every hop; any non-public address is rejected before socket connect | Current gap confirmed; defense completeness needs experiment |
| P0 | Can a hostile workflow script escape and read host authority? | Run CPU, memory, filesystem, network, and credential canaries in a candidate separate-process or container engine | Only explicit capabilities are reachable, OS boundaries reject escape, and cancel or kill leaves no surviving child | Current engine non-boundary confirmed; replacement needs experiment |
| P1 | How does a tool effect converge across crash windows? | Kill the process at intent flush, external commit, and result append, with and without provider support for callId idempotency | The log always recovers; without idempotency it reports only unknown, while with idempotency a retry creates no second effect | Weak guarantee; provider-dependent |
| P1 | What telemetry is actually lost, duplicated, or disclosed under collector failure? | Use a real OTLP collector to inject TLS/auth errors, 429, disconnect, queue saturation, hung shutdown, and crash. Run a current-baseline arm with no redaction listener, then a candidate-policy arm that explicitly mounts a redaction rule; place distinct privacy canaries in both. | The baseline arm quantifies disclosure, loss, and duplication without pretending to pass privacy. The candidate arm passes only if its canary never leaves, receiver dedupe works, and measured loss and shutdown behavior match the declared target policy. | SDK and deployment owned; static inspection cannot decide |
| P1 | Do two processes writing one feedback or settings root lose updates? | Before the run, declare either a strong-consistency target or a last-write-wins characterization target. Then make two independent Host processes perform barrier-synchronized updates plus kill, restart, and watcher replacement. | For the strong target, every conflict is detectable and no update is silently lost. For characterization, repeated schedules produce and document the actual winner and lost-update window; that result remains explicitly non-atomic and is not counted as a strong-consistency pass. | Local guarantee known; deployment behavior needs experiment |
| P1 | What are the real platform boundaries of sandboxing and process cleanup? | Run one file, network, process, hard-link, reparent, and setsid corpus under bwrap, each Landlock ABI, Seatbelt, and Windows ACL | Reported enforcement matches observed effects, failure is closed, and survivors and false attribution have platform-specific results | Platform differences known; kernel behavior needs experiment |
| P2 | Can concurrent-sibling replay and non-raw backends regress deterministically? | Release two child first model calls simultaneously and harvest from raw, compressed, and SQLite backends | Script identity does not depend on scheduling and all three backends yield the same normalized logical log | Test-capability gap confirmed |
| P2 | Do resource limits fit the deployment under concurrency? | Send concurrent API bodies near 160 MiB and make the native helper emit controlled large stdout and stderr | Resident memory, rejection point, cleanup, and operator diagnostics remain inside a predeclared budget | Static model gives one-request or unbounded behavior; capacity needs experiment |
14. Priority: close authority-expanding seams before scale and convenience work
| Priority | Work | Why | Completion condition |
|---|---|---|---|
| P0 | Fail-closed settings wire description, HTTP destination policy, and hostile-workflow-engine selection | Failure can carry a secret, internal network target, or host-process authority across a surface users may treat as a boundary | Code fix, the canary or escape experiment above, and converged public contract |
| P1 | Telemetry redaction and outbox policy, cross-process CAS, real-platform blocking signals, and workflow journal and budget | These determine privacy, recovery, consistency, and long-running operational promises | Declare target semantics first, then prove them with fault injection or a native lane |
| P2 | API cursor and since, client unload, attachment GC, snapshot-backend parity, and resource tuning | These chiefly affect scale, lifecycle completeness, and test coverage and should not displace authority seams | Compatibility path, safe migration or GC, and regression proof are complete |
| Immediate documentation fix | host.describe.version and the sandbox platform summary | There is a direct contradiction or omission with low correction cost and consumer impact | Code and public prose agree in one commit |
15. Verification scope for this chapter
- Used only source, package READMEs, and GitHub workflows from the public DeepSeek Harness repository at the pinned commit; neither prose nor evidence contains another project's name, path, or identifier.
- Rechecked every cited live path and line range; confirmed gaps, weak guarantees, documentation/code drift, platform differences, and experiment questions carry distinct labels.
- Ran only the structural documentation gate
pnpm run verify-package-readme-limitations: 219 package READMEs, one whitelist entry, exit code 0. - Did not run the full build, unit, coverage, snapshot, browser, real-API, collector, crash-injection, multiprocess, native Windows or macOS, or kernel-sandbox experiments; no related claim is presented as runtime proof.
- The Chinese and English pages have identical section, table, claim and caveat, and evidence-reference structure and first-use order.
My Learning Notes
Autosaved only in this browser. Nothing is uploaded or committed. Export Markdown whenever you want to keep a copy.