shard
Star
Fork
Watch
shard ARCHITECTURE.md
69 kB · 1197 lines
Markdown
at main

Shard — Architectural Foundation #

A coding agent harness built around three ideas: an agnostic core that depends on nothing optional, a deterministic conversation compiler for compaction, and a client/server split where a daemon supervises many isolated agent processes and any number of clients (web, orchestrator, SDK) attach over a shared protocol.

This is a living reference. Mark it up as contracts stabilize.

1. Guiding Principles #

  1. The core is a closed dependency layer. It depends on nothing optional. Optional capabilities depend on the core, never the reverse. Arrows point inward only.
  2. Everything is a seam. Any behavior a user might want to override sits behind an interface — tools, permissions, modes, compaction, extraction, providers, code intelligence, transports. No if (codeIndex) branches in the turn loop; capability presence is resolved at registration time, not call time.
  3. Deterministic by default; enrichment optional. Default compaction is a pure function over the normalized conversation. CodeIndex/LSP enrich but are never required.
  4. Data outlives behavior. The session JSONL is append-only and self-describing. Tools, extractors, and prompts can change across versions; old sessions still replay and compact.
  5. Capabilities compose, they don't special-case. MCP servers, extensions, CodeIndex, and built-ins all register through one ToolRegistry and pass through one PermissionGate.
  6. Lineage is never deleted. Compaction reindexes, it never destroys. The agent can always recall prior work — this is what makes deterministic compaction safe.
  7. One tool interface, every origin. Built-in, extension, MCP, and CodeIndex-backed tools are all Tool. One registry, one gate, one render path, one permission model.
  8. Client/server split. The agent loop lives in a server process; clients are thin views + controllers over a transport-agnostic protocol. shardd is to agents what tmux is to sessions.
  9. Contracts are wire-safe. Every L1 type is serializable data — no functions, no class instances. This discipline is what makes multi-client, detach/reattach, and replay work.

2. Deployment Topology #

graph TD
    subgraph Clients[Clients - thin view and controller]
        WEB[Web client single agent]
        ORCH[Orchestrator web client - all agents]
        SDK[SDK - in-process, no socket]
    end
    subgraph Daemon[shardd - supervisor daemon]
        REG[Agent registry list spawn kill attach]
        PROXY[Protocol proxy multiplex agents over socket]
        STATIC[Serve web UI static assets]
        AUTH[Local token auth]
    end
    subgraph Agents[Agent processes - one per session, isolated]
        A1[Agent session 1 owns turn loop tools compactor]
        A2[Agent session 2]
        A3[Subagent child - headless durable Agent owned by daemon]
    end
    WEB -->|WebSocket| Daemon
    ORCH -->|HTTP| Daemon
    SDK -->|in-process direct, no socket| A1
    REG --> A1
    REG --> A2
    JOB[JobCatalog + Coordinator - durable background jobs/runs]
    COORD -->|owns jobs/runs, drives children| A3
    A1 -.subagent_launch over unix socket.-> JOB
    PROXY --> A1
    PROXY --> A2

Three process roles, one protocol:

Role What it is Owns
shardd (daemon/supervisor) The one thing you run. Thin. Agent lifecycle, protocol proxy, durable JobCatalog/SubagentJobCoordinator, web serving, auth. Never runs agent logic.
Agent process One per session. The turn loop, tools, compactor, session JSONL — everything in L2.
Client Web, orchestrator, or SDK. Rendering, input, permission prompts, UI dialogs.

Why child processes for agents (not in-process): isolation and crash containment. One agent's bad tool call, OOM, or native segfault must not take down the orchestrator or other agents. This also makes subagents a first-class citizen for free — a subagent is a headless agent process the daemon owns on behalf of a parent, with its own durable Agent, session, gate, grants, and parent-scoped alias (§9).

3. Layered Architecture #

graph TD
    subgraph L4[L4 - Integration surfaces]
        CLI[CLI entrypoint - subcommand dispatch]
        WEB[Web client - SvelteKit plus shadcn-svelte]
        ORCH[Orchestrator - aggregation views over registry API]
        SDK[SDK - in-process embedding, transport is a no-op subscriber]
        DAEMON[shardd supervisor - lifecycle, proxy, auth, static serving]
    end
    subgraph L3[L3 - First-party optional capabilities]
        CODE[CodeIndex] ; LSP[LSP] ; MCP[MCP] ; SUB[Subagent tool]
    end
    subgraph L2[L2 - Core harness lives in the agent process]
        SESSION[AgentSession] ; TREG[ToolRegistry] ; GATE[PermissionGate] ; MODE[ModeController] ; COMP[Compactor] ; PROV[Provider] ; SKILLS[Agent Skills + typed loaders] ; BUS[EventBus]
    end
    subgraph L1[L1 - Contracts - wire-safe interfaces]
        C[Tool PermissionGate Mode Compactor Extractor Provider CodeIntel ExtensionApi EventBus Event RenderDescriptor Decision]
    end
    subgraph L0[L0 - Foundation types]
        T[Message NormalizedBlock ToolResult SessionEntry - all serializable]
    end
    CLI --> C
    WEB --> C
    ORCH --> C
    SDK --> SESSION
    DAEMON --> C
    CODE --> C ; LSP --> C ; MCP --> C ; SUB --> C
    SESSION --> TREG ; SESSION --> GATE ; SESSION --> MODE ; SESSION --> COMP ; SESSION --> PROV ; SESSION --> SKILLS ; SESSION --> BUS
    C --> T
    COMP -. optional enrichment via ctx.codeIndex .-> CODE

The dependency rule: Layer N may depend only on N−1 and below. L2 never imports L3. L3 implements L1 contracts and registers itself into L2 via the ExtensionApi. The single dashed edge (Compactor → CodeIndex) is the optional enrichment path, resolved through the L1 CodeIntelligenceProvider interface — the core holds the interface, not the impl.

4. Core Contracts (the seams) #

Each contract below is a registration point. The harness ships default implementations; extensions can replace any of them. Every contract is wire-safe (serializable data only).

4.1 Tool — a capability the model can invoke #

Self-describing, self-executing, optionally self-rendering via a transport-neutral descriptor.

  • name, description, inputSchema (typebox → JSON-schema-for-model + TS type + validation from one source)
  • isReadOnly: boolean
  • execute(input, ctx): Promise<ToolResult>
  • optional render hooks returning RenderDescriptors (see 4.5)

Registered via registerTool. Every origin (built-in, extension, MCP, CodeIndex-backed) produces the same shape.

4.2 ToolRegistry #

Name→tool resolution, mode-aware filtering, dedupe. The single source of "what tools exist this turn."

4.3 PermissionGate — async, over-the-wire #

Decides allow / allow_always / deny / ask for a (tool, input) pair before execution.

type Decision =
  | { kind: "allow" }
  | { kind: "allow_always"; rule: Rule }   // persists an allowance
  | { kind: "deny"; reason: string }
  | { kind: "ask"; prompt: string; id: string };  // emit permission_request, await response

Layered evaluation, first match wins: deny-rules → allow_always grants → mode policy → default ask. Grants persist to disk per-agent (see §13). See §6 for the multi-client resolution model.

4.4 Mode — a policy bundle, not a code path #

Filters the toolset, contributes a system-prompt section, casts a permission vote, hooks turn-end (e.g. plan-step extraction). plan/build are default impls; review/debug are future modes, not code paths. The loop never branches on mode === 'plan'.

Plan → build workflow: in plan mode the model is instructed to end its turn with a ## Implementation Plan section; the mode's pure onTurnEnd extractor returns a checkpoint intent carrying a PlanArtifact (ordered steps), which the loop persists generically as a lineage checkpoint (artifactKind plan). Switching to build mode (/mode build) reuses the latest valid plan on the active branch (## Active Plan prompt section, resolved by parentId ancestry — an off-branch plan never leaks) with no mode === 'build' branch. "No plan means no prompt change." The plan survives compaction (append-only lineage).

4.5 RenderDescriptor — transport-neutral rendering #

A tool returns structured data each client renders its own way:

type RenderDescriptor =
  | { kind: "text"; text: string }
  | { kind: "kv"; pairs: { k: string; v: string }[] }
  | { kind: "diff"; path: string; hunks: Hunk[] }
  | { kind: "tree"; root: TreeNode }
  | { kind: "progress"; label: string; current: number; total?: number }
  | { kind: "status"; label: string; tone: "info" | "warn" | "error" };
  • Web renders descriptors to Svelte components.
  • The core never imports a rendering library.
  • Third-party tool extensions render correctly in every client without per-client adapters — they emit descriptors, clients ship renderers for the closed descriptor set. New kinds are additive and versioned.

4.6 Compactor + Extractor — deterministic, self-contained #

Decides when to compact, prepares the boundary, produces a deterministic summary. The internal seam is the Extractor:

Aspect Shape
Extractor (blocks: NormalizedBlock[], ctx: ExtractorContext) => string[]
ExtractorContext { codeIndex?: CodeIntelligenceProvider; previousSummary?: string; compactionIndex: number }

ctx.codeIndex is optional. Default extractors operate on blocks alone via regex (ported from pi-vcc). If a provider is registered, the typeCatalog/symbolChanges extractors prefer it for real signatures. This single field is the entirety of the CodeIndex coupling.

4.7 CodeIntelligenceProvider (optional) #

outline / symbol / signatures / invalidate. Powers the code_outline and code_symbol tools and (optionally) enriches extractors. Source bodies remain the existing bounded read tool's responsibility. Tree-sitter WASM is the first-party impl; LSP is a richer impl layered on top. Remove the module and the core still compacts.

4.8 Provider #

Turns (messages, tools, systemPrompt) into an assistant message with tool_calls. Abstracts OpenAI/Anthropic/local. Router picks by config.

4.9 Specialized typed resource loaders + Agent Skills #

Discovery is NOT a single generic ResourceLoader flattening everything into one ResourceRecord. Each runtime format owns a specialized, typed loader with its own format, trust gate, precedence, diagnostics, and bounded output:

Loader Kind Output type Notes
Agent-Skills loader (core/agent-skills) SKILL.md packages SkillRegistrySnapshot Standards-compatible packages; merged by parsed front-matter name; see below.
ContextFileLoader (core/resources) SHARD.md / AGENTS.md ContextFileSnapshot Layered outer→inner; project files require trust.
AgentDefinitionLoader agents/*.md AgentDefinitionSnapshot Parsed-name merge; project requires trust. Discovered/exposed; model-facing subagent launch is future scope.
PermissionRuleLoader rules/*.json PermissionRuleSnapshot Ordered first-match; untrusted project allow rules dropped (deny survives).
ExtensionLoader (core/extensions) executable .ts/.js installed via ExtensionHost Code, not metadata — project sources require affirmative trust before import.

Precedence is fixed per loader (typically bundled → user → project; the rule loader is most-specific-first for first-match). Roots are injected (never hard-coded) so the same loader serves bundled, user-config, and project layouts. The shared ResourceFilesystem seam (normalizePath/resolveContained/readFileText) backs the read-only loaders so tests run hermetically; ExtensionLoader has its own independent fs + importer seams.

Agent Skills lifecycle #

An Agent-Skill is a directory containing a SKILL.md whose YAML front-matter declares a non-empty name (kebab-case) and description. Optional standard metadata (license, compatibility, version, author, keywords, allowed-tools, metadata) is retained in the typed registry for inspection; the startup catalog remains limited to name, description, and location. Metadata is never trusted for behavior.

Locations (bundled → user → project precedence; project wins on parsed-name collisions, first-match for rules):

  • User: ~/.shard/skills and ~/.agents/skills.
  • Project: .shard/skills and .agents/skills under cwd and each ancestor up to the git root (highest precedence).

Progressive disclosure: discovery produces a metadata-only catalog (<available_skills> — name: description (location) per skill) composed into the base system prompt by the canonical bootstrap. The model never sees skill bodies up front; it calls the built-in load_skill({ name }) tool (registered through ExtensionApi iff ≥1 skill is live — absence behavior). On a successful load load_skill:

  1. Resolves the name ONLY through the live SkillRegistry (names, never paths — no traversal vector); re-reads the winning SKILL.md from disk on every call (no cache), reparses + revalidates the current front-matter + identity.
  2. Returns the COMPLETE body (front-matter stripped, never truncated), the absolute skill directory, and a sorted list of at most 256 scripts//references//assets/ resource paths, up to eight directory levels (names only — bodies are never read). The model uses the returned directory as the base when resolving relative references from the body.
  3. Emits a skill-activation checkpoint intent; the turn loop persists it generically (linked to the just-appended tool_result).

Activation durability: while the tool_result carrying the body is still in the live window, the body is NOT re-injected into the system prompt (no duplication — the model already sees it). After compaction drops that row, the activation is rehydrated into the prompt from the checkpoint (## Activated Skills section, same <skill> envelope) — no silent loss, stable attribution. Active-branch authority is the parentId chain, so an off-branch activation never leaks. recall recovers the body across compaction boundaries (the JSONL is append-only). Depth/count/byte bounds and full-body output consistency are enforced; symlink loops are deduped by canonical realpath (broken links skipped). The complete rendered load result is capped at 128 KiB; an oversized skill is an explicit error with NO checkpoint and is never silently truncated. Rehydrated activated-skill prompt content has a 160 KiB aggregate cap: newest activations are included first and bodies are never partially truncated.

4.10 EventBus — serializable event stream #

Events are the wire format. Every Event type is plain data (no functions, no Error objects — serialize messages + codes). The in-process SDK uses an in-process subscriber; networked clients use the same event stream over a socket. Transport is just another EventSubscriber.

4.11 ExtensionApi — the single registration surface #

The only integration surface extensions receive:

  • on(event, handler)
  • registerTool(tool), registerMode(mode), registerCompactor(c), registerExtractor(e)
  • registerCodeIntelligence(p), registerProvider(p), registerSkillLoader(fn)
  • registerMcpTransport(name, factory)
  • ui — select/confirm/input/editor/notify (wire dialogs, see §6)

One seam, everything pluggable. No other integration path exists. First-party optional capabilities (CodeIndex, LSP, MCP, skills, subagents) wire themselves in as bundled extensions; if you don't load them, they don't exist.

5. The Turn Loop #

graph TD
    A[Assemble context: resources, mode, system prompt] --> B[turn_start event]
    B --> C[Call Provider with messages plus tools plus system prompt]
    C --> D[Receive assistant message text plus tool_calls]
    D --> E[Append assistant message to JSONL]
    E --> F{tool_calls present?}
    F -->|yes| G[For each tool_call in order]
    G --> H[tool_call event]
    H --> I[PermissionGate check]
    I -->|deny| J[Record denial as tool_result]
    I -->|ask| K[Emit permission_request to all clients, await first response]
    I -->|allow| L[Tool execute input ctx]
    L --> M[Append tool_result to JSONL, tool_result event]
    M --> N{file mutated?}
    N -->|yes| O[Notify CodeIndex to invalidate if registered]
    N -->|no| P[Next tool_call]
    O --> P
    J --> P
    K --> P
    P -->|all calls done| P2{Context at 100% of window?}
    P2 -->|yes| S2[Run compaction pipeline mid-turn, rebuild system prompt]
    P2 -->|no| C
    S2 --> C
    F -->|no text-only| Q[turn_end event]
    Q --> R{Compactor shouldCompact? above reserve threshold}
    R -->|yes| S[Run compaction pipeline, write compaction entry, set live window]
    R -->|no| T[Done: await next user turn]
    S --> T

Two invariants:

  • Mode filtering happens before the call (toolset narrowing) and the gate is defense-in-depth at execution. Neither knows about the other; both can independently block.
  • Compaction runs at boundaries, never inside a provider call or tool execution. The default tier fires between turns — after turn_end — when the context passes the reserve threshold (~80% of the window): it can wait until the agent goes idle. The emergency tier fires mid-turn, at the loop boundary after a call reports the context at 100% of the window: the next provider call would overflow, so compaction cannot wait for the turn to end. By that boundary every tool_call in the window has its result, so the cut still lands on a safe boundary (a completed tool cycle); the system prompt is rebuilt so the turn's remaining provider calls carry the handoff summary.

6. Multi-Client Model — Broadcast, First-Response-Wins #

Decision (locked): any attached client can answer permission prompts and UI dialogs. This is flexible at the cost of races, which we accept and define crisply.

Semantics #

  • A permission_request or ui_request is broadcast to all attached clients. Any of them may respond.
  • First response wins. The daemon proxy dedupes: once a response is accepted for a request id, late responses are discarded silently (not errored) — they are simply no longer relevant.
  • When a request is resolved, the agent emits a permission_resolved / ui_resolved event to all clients so they can dismiss their pending dialogs. No client is left hanging.
  • Grants are per-agent, not per-client. An allow_always from any client persists and applies regardless of who is attached later.
  • No primary/promotion protocol. This is the simplicity win of the flexible model — no "who is primary" state to track or hand off.
  • Headless policy still applies when zero clients are attached (see §6.2).

Conflict resolution #

If two clients disagree (one allows, one denies), whoever responds first decides. This is the accepted race. It is rare in practice (usually one human at one client) and the alternative — coordination/quorum — costs more than it saves for a local-first tool.

6.1 Wire flow #

sequenceDiagram
    participant A as Agent process
    participant D as shardd proxy
    participant C1 as Client 1
    participant C2 as Client 2 Web
    A->>D: permission_request id req-7
    D->>C1: broadcast permission_request req-7
    D->>C2: broadcast permission_request req-7
    C2->>D: permission_response req-7 allow_always
    D->>A: forward first response, drop late
    A->>D: permission_resolved req-7 allow_always
    D->>C1: broadcast permission_resolved req-7
    D->>C2: broadcast permission_resolved req-7
    Note over C1,C2: both dismiss their dialogs

6.2 Headless policy #

When zero clients are attached (subagents, detached agents, SDK-with-no-UI), an unanswered ask cannot block forever. Apply a per-agent headlessPolicy:

  • Default: deny for write/mutating tools, allow for read-only tools.
  • Configurable per-agent: deny | allow_if_rule_matches | queue (queue until a client attaches — useful for long-running detached agents a human will rejoin).
  • Timeout safety: an unanswered ask after N seconds (configurable, default ~60s) falls back to headlessPolicy. Prevents a dead agent if a client disconnects mid-prompt.
  • Client swap (tmux-style detach/reattach): if a client disconnects during an ask, the request stays open; a reattaching client receives pending permission_requests.

6.3 Roles, hierarchy, and the no-responder fallback (subagents) #

Managed subagents add three additive rules on top of the broadcast + first-response-wins model above. They never weaken §6.1–§6.2 for unleased/root agents.

  • Attachment roles. agent.attach takes a role of interactive (default) or observer. Both receive the event stream; only interactive connections answer permission/UI requests. Observers (e.g. a dashboard watching a run) never count as responders. A connection may promote/demote itself on re-attach.
  • Hierarchical routing. A delegated child's permission/UI request goes to the child's own interactive clients first; if the child has none, it is routed up the durable parent chain to the nearest interactive ancestor (retaining the child agent_id so a response routes back to the child). Parent Runtime may be stopped while its human client stays attached; hierarchy routing uses durable parent links + interactive attachments, not parent process liveness. Pending delegated controls survive client detach/reattach via ancestor sync.
  • No-responder fallback. Only when there is no interactive responder anywhere in the hierarchy does the proxy synthesize a bottom-case answer so no request hangs: permission_request → permission_response{source:'headless', decision:'deny'} (the gate maps it to a terminal deny — §6.2 parity for root/no-parent agents); ui_request → ui_response{value:null} (the bridge emits ui_resolved{value:null}). An internal SubagentJobCoordinator event subscription and observer sockets never count as responders.

7. The Protocol — one definition, multiple transports #

Define the protocol once as a set of messages; carry it over whatever transport fits the client. The messages are identical regardless of transport.

Transport Used by Why
stdio JSONL daemon ↔ child Runtime (incl. headless subagent children), headless agent, SDK subprocess Simplest; no port allocation; the private daemon↔Runtime channel
Unix socket Bundled subagent tools ↔ daemon (short-lived job clients) No network exposure; fast; auto-discovered at ~/.shard/daemon.sock
WebSocket Web client, orchestrator Bidirectional, low-latency; one persistent connection carries commands + events

7.1 Framing rule #

Streamed transports (stdio, unix socket): JSONL with LF-only record delimiter. Do not use Node readline (it splits on U+2028/U+2029, valid inside JSON strings). Split on \n only. Pin this in the protocol spec — it's the kind of detail that bites in production.

WebSocket transport: one JSON message per WebSocket text frame. WS is already message-framed, so no newline delimiting is needed — the frame boundary is the record boundary. Commands and events share the same connection and the same message envelope shape (see §7.2). Keepalive via WS ping/pong; clients should reconnect with backoff on drop and re-subscribe to the agent stream(s) they were watching.

7.2 Per-agent protocol (one client ↔ one agent) #

Commands (client → agent): prompt, steer (queue mid-turn), subscribe, permission_response, ui_response, abort, session_op (compact, fork, list), set_mode.

Events (agent → client, streamed): message_delta (text/tool_call deltas), tool_call, tool_result, permission_request, permission_resolved, ui_request, ui_resolved, turn_start/turn_end, compaction, usage (tokens/cost/context), status.

7.3 Orchestrator protocol (client ↔ daemon registry) #

Commands: agent.list, agent.create (config: cwd, model, mode, skills, permissions; optional opaque session_id adopts a legacy session), agent.kill, agent.attach (returns a per-agent stream handle the daemon proxies), agent.status, daemon.shutdown.

Events: agent.spawned, agent.exited, agent.status_changed (idle/streaming/waiting-for-permission/error). AgentInfo.runtime_state is required and independently reports the Runtime lifecycle (stopped, starting, running, or failed).

The web orchestrator calls agent.list to populate the dashboard, then agent.attach to get a live event stream for the one it wants to watch — without losing the registry view.

8. Process Model & Lifecycle #

sequenceDiagram
    participant U as User
    participant W as Web client
    participant D as shardd daemon
    participant A as Agent process
    U->>W: shard web
    W->>D: connect WebSocket, daemon boots if down
    D-->>W: daemon ready, prints URL
    U->>W: open URL, enroll, create agent
    W->>D: agent.create cwd model mode
    D->>A: spawn shard agent --id --socket
    A-->>D: agent.ready
    D-->>W: agent.attach stream handle
    W->>A: prompt via proxy
    A-->>W: streamed events, permission_request, ui_request
    W->>A: permission_response, ui_response
    A-->>D: turn_end, usage
    U->>W: close browser tab
    Note over A: agent keeps running, headlessPolicy governs prompts
    U->>W: reopen URL, attach to same agent
    W->>D: agent.list, agent.attach
    D-->>W: live stream for A

Daemon responsibilities (the daemon owns identity; Runtime processes are replaceable):

  1. Lifecycle — start/stop Runtime generations, fence writers, validate hello/sync, and coalesce concurrent starts. PIDs are diagnostic only and are never durable identity.
  2. Proxy — multiplex per-agent protocol over the socket; route commands/events and sync. Attach reservations buffer live events until authoritative sync is sent; buffered events are then drained exactly once before atomic promotion to the attached set.
  3. Registry — agent.* plus durable manifests under ~/.shard/agents/<id>/manifest.json.
  4. Session recovery — retain append-only JSONL and restore it under a writer lock after Runtime exit or daemon restart.
  5. Static serving/auth — web assets and local token metadata.
  6. Durable subagent jobs — JobCatalog persists job/run records under ~/.shard/jobs/<job_id>/, and SubagentJobCoordinator leases active runs, schedules single/parallel/chain, applies {previous}, runs wait/status/steer/relaunch/stop, and reconciles interrupted runs on restart (§9). Ownership is the durable parent_agent_id; an internal Broker event subscription + inbound lease guard keep result routing + direct-child controls correct.

A daemon restart reloads manifests and lazily restores Agents on attach. Runtime exit and agent.kill retain the manifest and transcript; a later attach creates a new generation. Legacy adoption resolves an opaque session id and canonical cwd daemon-side, commits one binding before startup, and never copies or rewrites transcript JSONL.

9. Subagents (daemon-managed background jobs) #

A subagent is an agent process the daemon spawns headless on behalf of a parent agent. Rather than the parent owning the child's lifecycle, the daemon owns durable Jobs and Runs: the parent submits a job, the daemon validates + commits durable records and aliases before acknowledging, then materializes/drives children asynchronously. The parent never blocks on child startup, model output, or cleanup.

9.1 Tools (six, separate — separate capabilities) #

The parent agent reaches subagents through six bundled tools (L3, src/subagent/**). Each is an independent allowlist capability — no tool implies another.

Tool Mutating? Returns after
subagent_launch yes the daemon acknowledges the durable job (preallocated IDs + aliases); never waits for child startup/output/cleanup. A terminal child report is pushed into the active parent as a steer or wakes an idle parent as a fresh turn. Validates the whole request before side effects.
subagent_wait no (read-only) any/all handles reach a terminal state, or timeout elapses. Timeout stops waiting, never work.
subagent_steer yes the strict steer is drained into the active turn_id (injected) or conflicts. Accepted only while the run is actively executing.
subagent_relaunch yes a new job_id/run_id is committed on the same durable Agent/session/config/grants/alias. Activе duplicates conflict.
subagent_status no (read-only) authoritative Agent/job/run snapshots. Omitting targets lists the parent's latest 50 directly-owned jobs (newest first) — the handle/alias recovery path after a parent Runtime restart.
subagent_stop yes cancellation is accepted (queued → cancelled; active → abort→graceful→kill escalation).

subagent_launch additionally requires a trusted target definition + launch-policy eligibility. Management tools are available to daemon-managed agents subject to the effective tool allowlist even if the current definition snapshot is empty.

9.2 Machine IDs vs friendly aliases #

  • Opaque, typed, authoritative IDs: daemon-generated agent_id, job_id, run_id are the wire/storage identity. Never derive identity from display strings. Prefix families (agent-, job-, run-) make them distinguishable at a glance.
  • Durable friendly aliases: every accepted initial run gets a reserved alias plus a preallocated agent_id before launch acknowledgement, copied into the child manifest on materialization. Launch returns recognizable definition_name, alias, and display_name (definition_name · alias, e.g. reviewer · Juniper) immediately.
  • Scope + uniqueness: the alias scope is the durable parent's direct children plus accepted-but-not-materialized reservations, unique under Unicode NFKC + trim + lowercase. Empty/control/path-separator/machine-ID-like/oversize values are rejected. Different parents may each have a Juniper.
  • Pool precedence (pinned at acceptance): agent-definition front-matter names > XDG user config subagents.names > a stable bundled friendly pool. The resolved pool is pinned into the accepted allocation context so later config edits never rename existing Agents.
  • Allocation: the first configured base alias with the fewest existing durable sibling assignments (pool order breaks ties); first use is Juniper, then Juniper 2, Juniper 3… An alias is never reused while that durable sibling Agent exists, regardless of Runtime/run state.
  • Resolution order (all tools): exact typed IDs first, then parent-scoped aliases case-insensitively. Resolution never crosses ownership boundaries. The display string reviewer · Juniper is accepted as a convenience only by splitting/validating both components; bare alias is canonical. Ambiguity/corrupt duplicates fail loudly.

9.3 Job/Run model + background orchestration #

  • Job = daemon-private durable record (job_id, parent_agent_id, strategy single|parallel|chain, ordered run IDs, state, timestamps). Run = one delegated prompt/turn (run_id, job/index/alias, optional child agent_id, state, deadline, latest text, usage, error). Run states: queued|starting|active|completed|failed|timed_out| cancelled|interrupted. Job records live under the daemon home (<home>/jobs/<job_id>/…), are not session entries, and never appear in AgentInfo. Task/result text is local private data — it never enters logs, errors, or list broadcasts.
  • Atomic acceptance: IDs + the full job record are allocated/committed before the launch acknowledgement. Invalid requests create no record/Agent/Runtime. Materialization is idempotent: crash recovery either creates the pinned manifest from the reservation or marks the unstarted run interrupted; it never reallocates/renames.
  • One Agent, one queued/active run globally; the coordinator leases it while active.
  • Single/parallel create/start eligible children asynchronously. Chain starts only the current step; on terminal success it applies exact non-recursive replaceAll('{previous}', previousText) and schedules the next. First-step {previous} is invalid. Parallel is ordered all-settled (one failure does not cancel siblings; the job is failed if any run fails/times out/cancels, but every result is retained). Chain is fail-fast and records stopped_at.
  • Relaunch is not clone: same agent_id, session history, grants, system prompt, capability config, launch policy, and alias; new job/run IDs and task. Ownership is direct — a parent tool manages only jobs it launched and children whose manifest carries its parent_agent_id.
  • Daemon SubagentJobCoordinator consumes tagged Broker events through an internal subscription (not a socket attachment). It binds each run to the exact turn_id observed at turn_start, captures message_complete only while that turn is current, and completes only on the matching turn_end/terminal error. Stale events are ignored. State is persisted before scheduling successors/cleanup, so stdout pump ordering + turn correlation prevent chain/relaunch misattribution. Coordinator observers never count as interactive clients.
  • Termination: on terminal success/failure, only the Runtime is stopped after persisting the run result; the Agent/session/job are retained. Definition timeout > 0 is a per-run whole-lifecycle deadline (schedule/start through terminal cleanup); omitted/0 means no deadline.
  • Completion reporting: after a terminal transition is persisted, the coordinator pushes a structured child report into the owning parent's model conversation. An active parent is steered at its next provider boundary; an idle/stopped parent is restored and started with a fresh turn. subagent_wait/subagent_status remain synchronization and recovery tools, not prerequisites for normal completion delivery.
  • Restart: terminal records survive daemon and parent restarts; parent Runtime restart does not affect execution. On daemon process restart, records are loaded but formerly nonterminal runs are marked interrupted and dependent chain work fails/cancels rather than replaying a possibly side-effecting prompt; the parent may explicitly relaunch the durable Agent. Automatic exactly-once crash replay is a non-goal for v1.

9.4 Strict steering, leases, and control-channel loss #

  • Active-only steer uses a managed child handshake, not check-then-send. The coordinator sends a strict steer carrying the leased run_id/turn_id + a correlation ID. AgentSession atomically accepts it only for that in-flight turn, resolves injected only when it is actually drained into that turn at the next provider boundary, and resolves conflict + removes it if the turn terminates first. It can never remain in the idle steer queue or leak into a relaunch.
  • Lease guard: the Broker consults a SubagentLeaseGuard before forwarding commands to a leased child. Direct prompt conflicts (would misattribute results); permission/UI responses pass (resolve the child gate); direct human steer/abort are delegated through the coordinator's strict active-turn / cancellation path and recorded against the run. Direct prompt is therefore blocked while leased. Unleased agents are unaffected (parity).
  • abort(): a parent subagent_steer/direct human steer and a direct human abort both traverse the handshake and are recorded against the run. Steers are lineage-true user messages, create no runs, do not reset the run deadline, and do not alter result routing. Parent completion/abort after subagent_launch acknowledges does not affect jobs; parent abort while inside subagent_wait closes/removes only that wait registration. subagent_stop is the explicit background cancellation mechanism.
  • ToolContext cancellation: the turn loop owns one turn-level AbortController spanning provider + tool execution, exposed via ToolContext.signal (additive — distinct from the provider-call controller so the carefully-tuned steer/abort drain ordering is unchanged). It cancels socket calls/wait registrations but does not cancel an already-accepted background job. Terminal reporting is daemon-driven and independent of the launching tool call. If a launch response is lost after commit, subagent_status recent-job discovery is the recovery path.
  • Control-channel loss: daemon control-channel loss closes child stdin; children abort, release writer locks, and exit within a bounded test deadline (an explicit exit is added if event-loop keepalives prevent natural termination).

9.5 Transport + layering #

  • Daemon ↔ child Runtime: private stdio JSONL.
  • Bundled subagent tools ↔ daemon: short-lived Unix-socket JSONL clients. subagent_wait keeps only its wait request/socket open; job execution itself is daemon-owned.
  • src/subagent/** is L3: tool definitions + thin daemon job client on L1 protocol/contracts. It must not import core/daemon/cli/transports. The durable JobCatalog/Coordinator live in src/daemon/** (L4) and consume Broker/Orchestrator services directly — there is no long-lived L3 runner inside the parent Runtime. Core gets only a generic injected bootstrap-contribution seam; L4 injects L3 extensions, and ExtensionHost remains the sole registration surface. Boundary lint + tests enforce src/subagent/**.

This is the pi subagent design (child subprocess via stdio), generalized to the daemon model and extended with durable ownership, recognizable aliases, and explicit lifecycle controls. The win: subagents get isolation, their own permission gates, their own compaction — they're full agents, not nested loops — while the parent keeps its own transcript free of child progress (wait/status return structured results; delegated permission UI stays control-plane only).

10. Data Model #

Session = append-only JSONL, one entry per line. Each entry has stable id, parentId (lineage chain), timestamp, type, details.

Entry type Purpose
message user/assistant, with content blocks (text/tool_call/tool_result nested by id)
tool_result paired to its call by id; carries isError, exit codes
compaction summary, messageRange: [firstId, lastId], tokensBefore/After, compactor name, index
checkpoint (optional) mode transitions, metadata snapshots

Lineage = the same JSONL. Compaction partitions it into addressable spans (compaction:0, compaction:1, …). The live context the model sees = [latest compaction summary] + [entries after firstKeptEntryId]. Nothing is ever deleted; recall walks the full file.

11. Compaction Pipeline (deterministic, self-contained) #

Three pure functions — fast (pi-vcc: 30–470ms on sessions with thousands of messages), unit-testable, deterministic:

  1. normalize(messages) → NormalizedBlock[] — flatten the stream; pair tool_call ↔ tool_result via a look-ahead index built once and shared across extractors.
  2. buildSections(blocks, ctx) → SectionData — run the extractors.
  3. formatSummary(sections, previousSummary) → string — render to the section doc.
type NormalizedBlock =
  | { kind: "user"; text: string }
  | { kind: "assistant"; text: string }
  | { kind: "tool_call"; name: string; args: Record<string, unknown> }
  | { kind: "tool_result"; name: string; text: string; isError: boolean }
  | { kind: "bash"; command: string; output: string; exitCode: number | undefined }
  | { kind: "thinking"; text: string };

Section order (stable → volatile, for prompt-cache survival) #

# Section Source Stability
1 Session Goal first user turns stable
2 User Preferences explicit pref statements accumulates
3 Files And Changes tool_call args append-only
4 Commits bash git output append-only
5 Type Catalog regex over read results or CodeIndex if registered structural
6 Outstanding Context error-classifier over recent tail volatile
7 Earlier Turns one-liner per turn grows downward

Then ---, then a Brief Transcript of recent turns kept verbatim.

Worth stealing verbatim from pi-vcc #

  • outstandingContext — classify the last ~25 blocks into priority-tagged lines ([ERROR] [tsc] …, [ERROR] [bash:exit 1] …, [WARN] [tests] …, [INFO] [no matches] …). Surfaces the agent's current failure state into the summary.
  • Resolution detection — if a [tsc] error's file was edited after the error in the tail, retag [RESOLVED]. Stops re-fixing fixed issues.
  • Handoff preamble + ...recall: breadcrumbs — fixed preamble on every summary; breadcrumbs let a recall hit trace to the originating turn.

12. Lineage + recall #

graph LR
    A[Full session JSONL never deleted] --> B[compaction 0 covers msg ids 1-200]
    A --> C[compaction 1 covers msg ids 201-450]
    A --> D[Live window: latest summary plus entries after 450]
    E[recall tool] --> A
    E --> F[scope: all or latest or compaction N]
    E --> G[query plus recent plus expand by index]

recall searches across all compaction boundaries, returns a paginated index-addressable view, and expand drills into specific entries without re-fetching everything. This is the feature that makes deterministic compaction viable long-term — the agent isn't blind to pre-compaction work, it just has to ask.

// recall tool shape
{
  scope: "all" | "latest" | "compaction:<n>",
  query: string,            // keyword/regex over normalized blocks
  recent?: number,         // default 25 entries from the chosen scope
  expand?: number[],       // fetch full text of entry indices from the previous page
}

13. Config & Discovery #

User-level storage splits by data class: config (user-authored files a human edits — config.json, agent definitions, themes) lives under the XDG config dir; state (runtime data shard writes — sessions, manifests, grants, jobs, sockets, logs) lives under the ~/.shard state home. Config is never mixed into the state home: the two classes have different lifetimes, backup stories, and sharing semantics.

~/.config/shard/             # user CONFIG (XDG: $XDG_CONFIG_HOME/shard) — user-authored,
                             # human-edited files
  config.json                # providers/models/permissions/compaction + subagents.names
                             # (the global friendly-alias pool)
  agents/                    # user agent definitions (*.md) — overrides bundled, yields to
                             # project; the legacy ~/.shard/agents/*.md location is still
                             # read as a fallback (the config-dir copy wins collisions)
  themes/                    # web UI theme files (.json/.toml)

~/.shard/                    # user STATE (runtime data; SHARD_HOME override) — written by
                             # shard itself
  daemon.json                # { pid, socketPath, httpPort, token }
  daemon.sock                # unix socket (local clients)
  agents.json                # legacy registry snapshot (migration input only)
  agents/<id>/               # durable manifest.json + session JSONL, forks, and
                             # permission-grants.json (per-agent allow_always grants)
  jobs/<job_id>/             # durable daemon-owned subagent job/run records (manifest.json)
  sessions/                  # CLI direct-run session transcripts (+ forks/)
  usage.jsonl                # append-only token-usage log (aggregated into usage stats)
  rules/                     # user permission rule sets (deny/allow)
  skills/                    # Agent-Skills packages (SKILL.md)
  extensions/                # *.ts/*.js extension modules
  worktrees/                 # daemon-managed worktrees (managedWorkspaceRoot)
  security-v1.json           # browser-principal/security store (§14)

~/.agents/skills/            # native user Agent-Skills location (alongside ~/.shard/skills)

.shard/                      # project-level (overrides/extends user)
  skills/                     # project Agent-Skills (requires --trust-project)
  agents/                     # project agent definitions (requires trust)
  rules/                      # project permission rules (allow rules require trust)
  extensions/                 # project extension modules (code; requires trust)
  ...                         # same structure ...

.agents/skills/              # native project Agent-Skills location (alongside .shard/skills)
SHARD.md                      # project context file, layered into the system prompt
AGENTS.md                     # compatibility context file (ecosystem familiarity)

Discovery order + precedence #

Bundled defaults → user → project (project wins on parsed-name collisions). Each format owns its loader (§4.9); per-loader notes:

  • Agent Skills merge by parsed front-matter name (not filename/dir); cross-layer overrides are info, same-scope collisions are warning. Project skill dirs are scanned under cwd and each ancestor up to the git root (cwd wins).
  • Context files (SHARD.md then AGENTS.md) layer outer→inner (bundled → user → project least-specific-ancestor → cwd). AGENTS.md compatibility is on by default (opt out per-run).
  • Agent definitions merge by parsed name; the user layer reads the config-dir agents/ (authoritative) over the legacy state-home fallback (see the tree above); permission rules compose first-match (project cwd→ancestors → user → bundled — most-specific first).
  • Subagent friendly-alias pool resolves subagents.names in the XDG config file (~/.config/shard/config.json; a non-empty string list), overridden per-definition by agent-definition front-matter names, falling back to a stable bundled pool. Invalid entries are dropped; the resolved pool is pinned at job acceptance so config edits never rename existing Agents (§9.2). It contains no secrets — safe to print in a config show.

Per-run project trust (load-bearing) #

Project-layer instruction/code resources — SHARD.md/AGENTS.md, agent definitions, allow permission rules, Agent-Skills, and executable extensions — are filtered/skipped by default (with diagnostics, no code executed). Affirmative trust is per-run, never persisted or inferred:

  • shard agent (direct): --trust-project.
  • shard daemon --trust-project forwards --trust-project to every spawned agent child.
  • SDK: createShard({ resources: { projectTrusted: true } }). Absent → user-layer discovery with project trust off (safe default).

Bundled/user sources are trusted by being explicitly configured. See §14.6 for extension trust specifically.

14. Security #

The daemon is the only network-exposed surface; agent processes speak over a local unix socket or stdio and never bind a port. Hardening therefore concentrates on the daemon's HTTP/WebSocket listener and on what an authenticated client is allowed to do.

14.1 Network binding #

  • Default: loopback only. The daemon binds 127.0.0.1:<port> and the unix socket ~/.shard/daemon.sock. Never 0.0.0.0 unless --remote is explicitly passed.
  • Unix socket has no network exposure — local job clients use it; no port, no CORS, no token-over-the-wire concerns for the local path.
  • Remote mode is opt-in and shipped in saga vr7kgx.9 (three configurations: loopback default, direct non-loopback bind, and trusted reverse proxy). See §14.3.1 for the transport matrix, threat model, and trust anchor; WEB-DEPLOYMENT.md for recipes. The TLS / authz considerations in §14.5 still apply.

14.2 Origin allowlist + CORS #

O3 is locked: the daemon serves the web client, so the normal path is single-origin and needs no CORS. CORS config exists for the cases that aren't single-origin. Host and Origin checks are exact serialized-value comparisons; hostname suffixes, alternate loopback names, forwarded host values, and DNS-rebinding aliases are never normalized into an allowlist hit. The HTTP listener must validate the exact loopback Host before routing, and the WS upgrade must require the exact Origin.

  • Dev workflow — Vite dev server on localhost:5173 talking to the daemon on its port.
  • Separately-hosted client — if someone serves the SPA elsewhere later.
  • Browser preflight — the WS upgrade and any fetch to a cross-origin daemon trigger preflight; without an allowlist the browser blocks them.

Config in daemon.json:

{
  "allowed_origins": ["http://localhost:5173", "http://127.0.0.1:5173"]
}

Semantics:

  • Echo only allowlisted origins in Access-Control-Allow-Origin. Never use * with credentials — the WS handshake carries the token, so wildcard + credentials is unsafe.
  • Allow methods GET, POST, OPTIONS; allow headers Authorization, Content-Type.
  • WS upgrade gate: reject the WebSocket handshake (400) if the Origin header is absent or not in the allowlist. This is the primary CSRF defense for the socket — browsers always send Origin on WS upgrades, and a non-browser attacker without the token is rejected at §14.3 anyway.
  • Default allowlist = the daemon's own origin only, so the out-of-the-box config is safe; dev just requires adding the Vite origin.

14.3 Authentication #

Browser authentication uses an owner-only, versioned durable store under configurable SHARD_HOME (default ~/.shard). The store contains browser-principal IDs, credential hashes, and timestamps only; raw credentials, setup tokens, enrollment codes, and WS tickets are never persisted or logged. Writes use a restrictive directory/file mode and write-temp/fsync/atomic rename, with explicit errors on corruption or I/O failure.

  • Enrollment: live enrollment returns a short-lived display code and a separate high-entropy claim nonce. Codes are rate-limited and a successful claim consumes the pending enrollment. A manually pasted setup token is high entropy, short-lived, and one-time.
  • Session cookie: authenticated sessions use a __Host- HttpOnly, Secure, SameSite=Strict cookie; it is never placed in a URL. Credential comparisons are constant-time.
  • WS handshake: the daemon issues a short-lived, single-use ticket bound to principal, session, and exact Origin. Reject tickets that are expired, replayed, or bound to another principal/session/origin. The WS Origin gate remains mandatory.

14.3.1 Remote mode (saga vr7kgx.9) #

Remote mode lets a browser on another machine reach the web client. It is opt-in, selected by shard daemon flags and validated loudly at startup (the daemon never silently degrades into an unsafe remote configuration). The locked design lives in .saga/plans/vr7kgx.9.md; the normative protocol text is PROTOCOL §4.1.2 and the deployment recipes are in WEB-DEPLOYMENT.md.

Transport configurations (D1).

Config Bind Transport Trust anchor Cookie
Loopback (default) 127.0.0.1 plaintext, loopback-only none HttpOnly; SameSite=Strict (no Secure)
Direct non-loopback <ip> plaintext, network-facing none (forwarded headers ignored) non-Secure (under --allow-insecure-remote)
Trusted reverse proxy 127.0.0.1/<ip> TLS at the proxy; plaintext to the daemon configured upstream peer Secure + __Host- iff X-Forwarded-Proto: https

Threat-model summary (remote mode). Loopback's "acceptable only because the daemon is loopback-bound" premise (§4.3) no longer holds once a remote browser can reach the listener:

  • Network MITM / passive sniff — mitigated by TLS at the proxy (https path). Under --allow-insecure-remote the session cookie is knowingly interceptable on untrusted networks; the operator confines non-TLS remote to a trusted/VPN/SSH-forwarded path and the daemon logs a startup warning.
  • Credential replay / cross-origin use — mitigated by exact Origin allowlist, SameSite=Strict, Secure (on https), HttpOnly, and the ticket bound to exact serialized Origin.
  • DNS rebinding — already mitigated by exact Host/Origin match. Remote mode keeps exact-match; expectedHost is the configured publicHost, never loosely inferred.
  • Proxy-header spoofing (new risk the reverse-proxy path introduces) — mitigated by the trusted-proxy trust anchor below.
  • CSRF — mitigated by SameSite=Strict + exact Origin on every state-changing request.
  • Stolen cookies on a non-HTTPS transport — Secure cookies are never sent over plaintext; a misconfigured proxy cannot mint Secure cookies for a hijacked plaintext session because the daemon refuses to issue/honor a Secure-form cookie when the effective scheme is not https.

Trusted-proxy trust anchor (D2). --trust-proxy <upstream> is opt-in and disabled by default. When set, forwarded headers are honored only from the configured upstream peer (loopback = 127.0.0.1/::1, or an exact IP literal); from any other peer forwarded headers are ignored and the request is treated as direct. X-Forwarded-Proto (or the configured headerScheme) sets the effective scheme; X-Forwarded-Host (or headerHost) is cross-checked against the pinned publicHost and rejected on mismatch — it is never used to derive expectedHost. X-Forwarded-For is never consumed (informational only; the peer address is the trust anchor). Under a non-loopback direct bind (no proxy) forwarded headers are ignored entirely, so a network peer cannot influence the scheme/host.

Cookie / origin policy split by effective scheme (D3/D4).

  • publicHost and the origin allowlist are mandatory whenever --web-host is non-loopback or --trust-proxy is set; missing either is a loud startup failure. The implicit http://127.0.0.1:<port> origin is added to the allowlist only in pure loopback mode.
  • The session cookie is issued Secure + __Host- iff the issuing request's effective scheme is https; otherwise non-Secure. A Secure cookie arriving over a non-https effective scheme is rejected at /bootstrap, and enrollment refuses to set a Secure cookie over plaintext.
  • --allow-insecure-remote is the explicit two-step opt-in required for any non-https remote operation; setting --web-host to a non-loopback address alone does not imply consent.

hug1mc invariant amendment. The hug1mc "any future remote mode must require HTTPS and Secure cookies" clause is relaxed to permit the explicit, two-step opt-in non-TLS remote mode above. All other hug1mc invariants (per-browser enrollment, the Sec-WebSocket-Protocol ticket, exact Origin matching, frame-ancestors 'none', selected-agent-only interactive attachment) stand unchanged.

14.4 Authorization (v1: single trust level) #

v1 grants a single trust level: anyone with the token can create/kill agents, send prompts, and answer permission requests for any agent. This is acceptable only because the daemon is loopback-bound. The defense-in-depth that still applies:

  • The PermissionGate (§4.3) still governs every tool call — a compromised client can make the agent ask to do things, but writes still route through the gate, the headless policy (§6.2), and any allow_always grants the real user already configured.
  • Agent processes are sandboxed children — the token grants agent control, not OS escape; the agent is still bound to its cwd and permission rules.

Real multi-tenant authz (per-agent ACLs, per-user identity) is a future item gated on remote mode.

14.5 Transport security #

  • Local = plaintext-safe. Loopback traffic never leaves the machine; TLS adds overhead with no benefit locally. The unix-socket path isn't networked at all.
  • Remote (shipped, saga vr7kgx.9): the canonical path is a trusted reverse proxy that terminates TLS and forwards to the loopback daemon; the daemon does not terminate TLS itself in v1 (a direct daemon-TLS mode remains out of scope). An explicit --allow-insecure-remote opt-in permits a deliberate non-TLS remote path on trusted/VPN/SSH-forwarded networks — see §14.3.1 for the cookie-exposure risk and the hug1mc invariant amendment.

14.6 Extension & subagent trust #

  • Extensions run in-process with full agent privileges. Only load extensions from sources you trust (bundled, ~/.shard/extensions/, or verified packages). An untrusted extension can register tools that bypass the gate's intent — treat extension loading as code execution.
  • Project extensions require affirmative per-run trust (--trust-project / SDK projectTrusted). Without it, a project extension source is discovered but never imported (a trust diagnostic, no code runs). Bundled/user sources are trusted by being explicitly configured. A reload(id) re-checks project trust before re-importing (a reload is still code execution).
  • Canonical containment: every discovered source is realpath-resolved and its canonical path must be contained within its extension directory; the import uses the canonical path, so a symlink swapped after discovery cannot escape (no TOCTOU window). Broken/ cyclic links are deduped or rejected.
  • Failure isolation: a broken/untrusted/malformed/throwing source is diagnosed with its source identity and the remaining loads continue; earlier successful installs stand. A failing setup never silently breaks the registry.
  • Subagents inherit the daemon's trust boundary but get their own permission gates, grant stores, headless policies (§6.2–§6.3), and strict capability allowlists (§9). A subagent cannot exceed the permissions its delegated config grants. Agent definitions are trusted sources for child system prompts/models/modes/tools/skills/launch policy/names; --trust-project governs project-layer definitions as usual.

15. Extension Model #

An extension is a .ts/.js module default-exporting function(api: ExtensionApi). Loaded from ~/.shard/extensions/, project .shard/extensions/, and compatibility extensions/ directories in deterministic bundled → user → project order. The ExtensionApi is the only integration surface — there is no other way to hook the core.

  • Default-export API: export default (api) => { ... }. The setup may be sync or async (it may await between registrations); the host keeps the extension's identity pinned across awaited registrations. It may return a disposer (sync or async), run on host shutdown in reverse install order; disposer errors are swallowed + attributed so one buggy disposer can't starve the rest.
  • Source attribution + diagnostics: every registration made during an install is attributed to the installing extension's identity; duplicate/replacement registrations produce registered/replaced diagnostics (the host never silently drops a registration — same-key replaces, last wins).
  • ExtensionApi-only canonical bootstrap: the default startup path installs bundled tool/mode/compactor extensions, recall, load_skill, the resolved provider, and trusted dynamic extensions all through host.install — the runtime ToolRegistry/ ModeController/compactor/provider are built from the host snapshot only. The core never registers a capability directly into a registry; ExtensionApi is the exclusive seam.
  • Diagnostic surfaces: the host exposes registration diagnostics, installed-extension identities, capability resolvers (resolveTool/resolveProvider/resolveCompactor), and a plain snapshot() for bootstrap/session assembly.

Bundled tools, modes, compaction, recall, conditional Agent Skills loading, and the provider all enter through this same API. Future optional L3 capabilities such as CodeIndex, LSP, MCP, and subagents must use it as well. This keeps the core honest.

16. Repo Layout #

Single package, strict internal module boundaries, split to a monorepo later. Splitting into packages before contracts stabilize causes churn. Internal boundaries give the architectural guarantee now, with the option to physicalize later without rework.

shard/
├── src/
│   ├── types/              # L0: Message, NormalizedBlock, SessionEntry, ToolResult
│   ├── contracts/          # L1: Tool, PermissionGate, Mode, Compactor, Extractor,
│   │                       #     Provider, CodeIntelligence, ExtensionApi, EventBus,
│   │                       #     Event, RenderDescriptor, Decision
│   ├── protocol/           # L1: per-agent + orchestrator message sets, framing
│   ├── core/               # L2: AgentSession, ToolRegistry, built-in tools,
│   │                       #     default gate/modes/compactor, recall, load_skill,
│   │                       #     provider impls, agent-skills + typed resource loaders,
│   │                       #     ExtensionHost/Loader, event bus
│   ├── daemon/             # L4: shardd supervisor - lifecycle, proxy, registry, auth
│   ├── transports/         # L4: stdio, unix-socket, websocket transport impls
│   ├── codeindex/          # L3: tree-sitter CodeIntelligenceProvider + code.* tools
│   ├── lsp/                # L3: LSP bridge layered on codeindex
│   ├── mcp/                # L3: MCP client + Tool adapter
│   ├── subagent/           # L3: subagent tool + process spawning (future)
│   ├── web/                # L4: SvelteKit + shadcn-svelte web client
│   ├── orchestrator/       # L4: aggregation views over the registry API (part of web)
│   ├── sdk/                # L4: embedding API, in-process subscriber
│   └── cli/                # L4: entrypoint
└── tests/

Enforce the boundary with a lint rule (no-restricted-imports): core/ may not import from codeindex/, lsp/, mcp/, subagent/. Agent Skills + the typed resource loaders live in core/ (L2) — discovery is a core capability, not an optional L3 module. That rule is the architecture's enforcement. Additionally, contracts/ and types/ may not import from anywhere above L1.

17. Decisions #

# Decision Status Resolution
1 Runtime Locked Bun — fast startup (daemon + many agent subprocesses), native TS
2 Package structure Locked Single package + lint-enforced boundaries; split later
3 Schema lib Locked Typebox — JSON-schema + TS type + validation from one source
4 Client rendering Locked Clients render transport-neutral RenderDescriptors themselves; the core never imports a rendering library. New descriptor kinds are additive and versioned.
5 Provider abstraction Locked OpenAI-compat lingua franca + native Anthropic
6 Context file name Locked SHARD.md default; optionally also read AGENTS.md
7 Optional capabilities location Locked First-party extensions (CodeIndex, LSP, MCP, skills, subagents), not core
8 Agent process model Locked Child processes; daemon as supervisor
9 Transports Locked One protocol, three transports: stdio + unix socket + WebSocket
10 Web client stack Locked SvelteKit + shadcn-svelte
11 Multi-client permission policy Locked Broadcast to all clients; first-response-wins; late responses discarded
12 No-client / headless policy Locked Default deny writes / allow reads; configurable per-agent (deny / allow_if_rule_matches / queue); ~60s timeout fallback
13 Auth for web Locked (v1) Local token from daemon.json; revisit for remote later
14 Compaction strategy Locked Deterministic, LLM-free, pi-vcc style; LLM compactor a swappable alternative later
15 CodeIndex as core dependency Locked Not a dependency — optional enrichment via ExtractorContext.codeIndex
16 Web transport Locked WebSocket — bidirectional, one connection for commands + events; replaces the SSE+POST split
17 Web client deployment Locked Served by the daemon (single-origin); token flows in the WS handshake; configurable origin allowlist (§14.2) for dev/cross-origin

Still open (decide when reached in build order) #

# Decision Notes
O1 Subagent process transport Locked — two complementary transports: daemon↔child Runtime is private stdio JSONL (isolation + the pi design generalized); parent subagent tools↔daemon use short-lived Unix-socket JSONL clients (the daemon owns jobs/runs, §9.5). A subagent is never a peer the orchestrator drives directly mid-run; it is a daemon-owned headless child surfaced by management tools/status.
O4 Orchestrator agent discovery metadata what per-agent info the registry exposes (name, cwd, model, mode, status, usage)
O5 Session forking UX Locked (saga 3aliod) — fork-to-new-session is the single split primitive (session_op: fork); branch/rewind dropped. Web per-message fork UI: saga 3aliod.1
O6 Permission rule DSL exact shape of Rule predicates (regex for bash, glob for paths, etc.)

18. Build Order #

The daemon and protocol enter the critical path early, because everything downstream is a client over them. CodeIndex and LSP stay on the parallel enrichment track, never blocking the critical path.

  1. L0–L1: foundation types (all serializable) + contracts, including RenderDescriptor and wire-safe Event/Decision.
  2. Protocol spec: define the message set + framing rules. Written doc; the contract between every client and the daemon.
  3. Agent process + loop: JSONL, one provider, read/write/bash, EventBus emitting serializable events.
  4. In-process SDK (transport = no-op subscriber). First runnable end-to-end agent, no daemon yet.
  5. stdio transport — wrap the agent as a headless process speaking the protocol. Also the subagent substrate.
  6. shardd daemon: lifecycle + registry + unix-socket proxy.
  7. Web client over WebSocket (renders RenderDescriptors). First interactive client against the daemon.
  8. ToolRegistry + edit/grep/find/ls + PermissionGate (async, wire-safe) + build mode.
  9. Deterministic Compactor + compaction entries.
  10. recall tool + lineage. ← compaction now safe to enable.
  11. plan mode + skills loader + formalized ExtensionApi.
  12. Subagents (headless agent processes via the daemon).
  13. MCP registry + tool adapter.
  14. WebSocket transport + web client (single-agent view) — ship with the §14 security baseline: loopback binding, origin allowlist + CORS, token-on-handshake. Security is not a later pass; the web client is the first network-exposed surface.
  15. Orchestrator web view (registry API + multi-agent dashboard).
  16. (Parallel, any time after step 9) CodeIndex + code.* tools → enriches compactor's Type Catalog via ctx.codeIndex.
  17. LSP layered on CodeIndex.

Critical path: types → protocol → agent process → daemon → core tools/permissions/compaction → web → orchestrator.

19. Load-Bearing Decisions (the ones that must not erode) #

These six decisions hold the architecture up. If any erodes, revisit the foundation:

  1. The layer rule — arrows point inward only; L2 never imports L3.
  2. The extractor-ctx seam — ExtractorContext.codeIndex? is the entire CodeIndex coupling.
  3. Lineage-never-deleted — compaction reindexes; recall walks the full JSONL.
  4. Client/server split — agent loop in a server process; clients are thin over a protocol.
  5. Wire-safe L1 contracts — RenderDescriptor, async Decision, serializable Event.
  6. Broadcast + first-response-wins — any client can answer; first wins; late discarded; all clients get *_resolved.