github-query #
A GitHub GraphQL and REST client whose every answer is kept in a cas-store.
A crawler that asks GitHub the same question twice should read the first
answer back, and several crawlers on one token should share its rate limit.
github-query keys each query by its canonical text, its variables (RFC 8785)
and the principal asking, and resolves it through Cas_store.resolve under a
freshness policy: Snapshot d asks once per snapshot, Sealed once ever,
Probe mark again only when a cheap probe reports activity past the stored
mark.
Every query sent also selects rateLimit { cost remaining resetAt }. The
stored answer has that member removed and is written in its RFC 8785 form, so
two fetches of unchanged content store one object; the rate data goes to a
budget kept in the store per token, which every client of that store and
token reads before sending, in this process or another. A 403 or 429 with
Retry-After or a spent X-RateLimit-Remaining pauses that budget until the
time it gives; 5xx statuses and timeouts are retried with exponential
backoff; a batch of aliased selections that GitHub refuses with Resource
limits for this query exceeded is asked again in halves, down to one
selection.
Installation #
$ opam install github-query
If opam cannot find the package, it may not yet be released in the public
opam-repository. Add the overlay repository, then install it:
$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
$ opam update
$ opam install github-query
Usage #
A client takes a store, a token and a transport. Github_query.requests
posts with a requests session; here a stub transport answers instead and
counts the requests it gets:
# let sent : string list ref = ref [];;
val sent : string list ref = {contents = []}
# let transport ~headers:_ body =
sent := body :: !sent;
Ok (Github_query.response ~status:200
{|{"data":{"rateLimit":{"cost":1,"remaining":4999,"resetAt":"2026-10-08T12:00:00Z"},
"repository":{"stargazerCount":1234}}}|});;
val transport : headers:'a -> string -> (Github_query.response, 'b) result =
<fun>
A template is the query text and a function from a value to its variables. Two layouts of one query are one key:
# let stars =
Github_query.Template.v
"query($owner: String!, $name: String!) {
repository(owner: $owner, name: $name) { stargazerCount }
}"
(fun (owner, name) ->
[ ("owner", Json.Value.string owner); ("name", Json.Value.string name) ]);;
val stars : (string * string) Github_query.Template.t = <abstr>
# Github_query.Template.text stars;;
- : string =
"query($owner:String!$name:String!){repository(owner:$owner name:$name){stargazerCount}}"
Asked twice in one snapshot, the query is sent once, and the answer comes back
without its rateLimit:
# let run () =
Eio_mock.Backend.run @@ fun () ->
let clock = Eio_mock.Clock.make () in
let store = Cas_store.in_memory ~clock () in
let gq = Github_query.v ~clock ~store ~token:"ghp_..." transport in
let policy = Cas_store.Policy.Snapshot "2026-10-08" in
let ask () =
match Github_query.query gq ~policy stars ("ocaml", "dune") with
| Ok answer -> Json.Value.to_string answer
| Error e -> Fmt.str "%a" Github_query.pp_error e
in
let first = ask () in
let second = ask () in
(first, second, Option.map fst (Github_query.budget gq));;
val run : unit -> string * string * int option = <fun>
# run ();;
- : string * string * int option =
("{\"data\":{\"repository\":{\"stargazerCount\":1234}}}",
"{\"data\":{\"repository\":{\"stargazerCount\":1234}}}", Some 4999)
# List.length !sent;;
- : int = 1
Github_query.batch asks one selection per item under aliases, a hundred to
a query by default. Each item is answered on its own: its alias's value,
Not_found when GitHub answers NOT_FOUND for it, or the errors whose path
starts at its alias, so a repository GitHub does not know is told from one it
refused to show. Github_query.Probe.repos uses it to ask, for many
repositories at once, their node id, current name, visibility, pushedAt and
the latest updatedAt of their pull requests and issues: a few points per
hundred repositories. A crawl then sends its costly queries under
Cas_store.Policy.Probe newest, so a repository whose newest activity is the
mark already stored is not asked again.
Github_query.nodes pages a GraphQL connection: given a template with an
$after variable and the path of the connection in the answer, it follows
pageInfo { hasNextPage endCursor } page by page, each page a query under
the caller's policy, and is every node or an error, never the nodes read
before a failed page. Github_query.Search.nodes reads a search past
GitHub's cap of 1,000 results by splitting its created: (or another
qualifier's) time range until each part is under the cap.
Github_query.Rest sends GET requests to the REST API for what GraphQL
does not expose, through the same store and budgets. Each page is stored under
its URL with the target of its Link: rel="next" header, so
Github_query.Rest.pages follows a paged answer from the store as it would
from GitHub. Budgets are kept per token and per rate-limit resource (graphql,
core, search), as GitHub counts them; a stale page with an ETag is asked
again with If-None-Match, and the 304 that answers it costs no budget.
Github_query.mutate sends a mutation through no store: every call is sent,
and it is sent again only after GitHub refused it for its rate limit, never
after a 5xx or a timeout, when it may have been applied.
Github_query.Project reads a GitHub Project's fields and its items' field
values, and sets or clears an item's fields in one mutation, by field and
option name, unarchiving an archived item first and archiving it again last.
Related work #
cas-storekeeps the answers, the budget and the claims that make one fetch per key across processes.requestsis the HTTP client behindGithub_query.requestsandGithub_query.Rest.requests; its own retry logic should be off, since this client retries with GitHub's rate limit headers in view.github-unixcovers GitHub's REST API with typed Lwt bindings and keeps no answers; GitHub's owngh api graphqlsends one query with no store and no shared budget.caretakersyncs a GitHub Project with local files over Lwt;Github_query.Projecttakes its field-value mutations and its archived-item round trip.
Licence #
ISC. See LICENSE.md.