A crawler that asks GitHub, ClickHouse, a model or a web page the same question twice should read the first answer back. raw-store keys a request by the SHA-256 of its normalised description (a JSON spec, in its RFC 8785 canonical form), keeps the answer's bytes in a content-addressed heap (cas-eio), and binds keys to answers in an SQLite index (index.db, in WAL mode, on mono's pure-OCaml sqlite). Each binding records when the answer was fetched and last confirmed, its validators (ETag, Last-Modified, a mark such as the newest updatedAt it covers) and whether it is sealed.

A freshness policy decides when a stored answer is still the answer: Sealed (always: past months, model replies, derivations), Snapshot d (fetched in snapshot d), Revalidate span (ask again with If-None-Match after span; a 304 keeps the digest), Probe mark (a cheap probe reported nothing past the stored mark) and Max_age span. Raw_store.resolve applies it and fetches at most once per key, across the fibers of a process and the processes sharing a store.

Installation #

$ opam install raw-store

If opam cannot find the package, it may not yet be released in the public opam-repository. Add the overlay repository, then install it:

$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
$ opam update
$ opam install raw-store

Usage #

Two spellings of one HTTP request are one key: the URL is normalised as RFC 3986 section 6.2.2 describes and its query parameters are sorted.

# let k0 = Raw_store_http.key ~meth:"get"
    "HTTPS://api.github.com/repos/ocaml/dune/pulls?state=all&per_page=100";;
val k0 : Raw_store.Key.t = <abstr>
# let k1 = Raw_store_http.key ~meth:"GET"
    "https://api.github.com/repos/ocaml/./dune/pulls?per_page=100&state=all";;
val k1 : Raw_store.Key.t = <abstr>
# Raw_store.Key.equal k0 k1;;
- : bool = true
# Raw_store.Key.canonical k0;;
- : string =
"{\"method\":\"GET\",\"src\":\"http\",\"url\":\"https://api.github.com/repos/ocaml/dune/pulls?per_page=100&state=all\"}"

resolve fetches on a miss and reads the stored answer while the policy says it is fresh. Here the fetch counts its calls; the second resolution in the same snapshot does not call it:

# let calls = ref 0;;
val calls : int ref = {contents = 0}
# let fetch _validators =
    incr calls;
    Raw_store.Body ({|[{"number":1}]|}, Raw_store.Validators.v ~etag:{|"abc"|} ());;
val fetch : 'a -> Raw_store.fetched = <fun>
# let pulls () =
    Eio_mock.Backend.run @@ fun () ->
    let store = Raw_store.in_memory ~clock:(Eio_mock.Clock.make ()) () in
    let policy = Raw_store.Policy.Snapshot "2026-10-08" in
    let answer () =
      match Raw_store.resolve store ~policy ~fetch k0 with
      | Ok (_, body) -> body
      | Error (`Gone status) -> Fmt.str "gone (%d)" status
    in
    let first = answer () in
    let second = answer () in
    (first, second);;
val pulls : unit -> string * string = <fun>
# pulls ();;
- : string * string = ("[{\"number\":1}]", "[{\"number\":1}]")
# !calls;;
- : int = 1

A store on disk is opened with Raw_store.v; several processes may open the same directory. A model reply is keyed on the resolved model and the prompt, and sealed once asked:

let reply ~sw ~clock ~dir ~ask prompt =
  let store = Raw_store.v ~sw ~clock dir in
  let key =
    Raw_store.Key.of_spec
      (Raw_store.Spec.llm ~provider:"anthropic"
         ~model:"claude-sonnet-4-5-20250929" ~prompt_version:"journal-3"
         ~system:"Summarise the week." prompt)
  in
  let fetch _ = Raw_store.Body (ask prompt, Raw_store.Validators.none) in
  match Raw_store.resolve store ~policy:Sealed ~fetch key with
  | Ok (_, reply) -> Some reply
  | Error (`Gone _) -> None

A paged answer is written page by page with Raw_store.Chain, so a crash resumes after the last committed page, and sealed when complete. Raw_store.gc removes the objects nothing names once they are older than a grace period; Raw_store.verify rehashes what the index names.

Keys of model calls #

Raw_store.Spec.llm keys a call on the model, the prompt and the context, and on nothing else: the resolved model identifier (the one the provider reports, not an alias such as sonnet), the system prompt and instructions with the caller's prompt version, every piece of context the model is shown (evidence, exemplars, a draft, earlier answers) as a named digest, and the call options that change the answer. A model upgrade or a change to the evidence asks again; an unchanged context reads the stored reply. Raw_store.Key.diff names the parts two keys differ in:

# let ask evidence =
    Raw_store.Key.of_spec
      (Raw_store.Spec.llm ~provider:"anthropic"
         ~model:"claude-sonnet-4-5-20250929" ~prompt_version:"journal-3"
         ~context:[ ("evidence", `Text evidence) ]
         "Summarise the week.");;
val ask : string -> Raw_store.Key.t = <fun>
# Raw_store.Key.diff (ask "3 PRs merged") (ask "4 PRs merged");;
- : string list = ["context.evidence"]

cas and cas-eio (this repository) store the objects; this package adds the index from requests to objects, freshness and collection.
builder-graph's Derivation keys a build step by its inputs; Raw_store.Spec.derivation makes such a key one more request key, so step outputs live in the same store.
requests has an RFC 9111 cache in memory; this store is persistent, shared between processes and keyed on a normalised request rather than a URL.
cache (this repository) is an in-memory TTL cache for values that are cheap to recompute.

Licence #

ISC. See LICENSE.md.

Clone this repository

raw-store #

Installation #

Usage #

Keys of model calls #

Related work #

Licence #