owi-cli

Command line interface for accessing the Open Web Index shards and slices.

Fork

owi-cli docs source http.md

17 kB · 457 lines

Markdown

at main

mgrani · fix(http,core): return the descriptive record, and fix two accessors that hid it · 2mo ago

4dec34a3

HTTP API Preview #

OWILIX includes an optional FastAPI server for local automation and controlled service demos. It is a preview API in v5.5.0: the endpoint surface is useful, but the server should be treated as a local service unless an external access-control layer is placed in front of it.

There is no caller authentication. The dependency guarding 25 routes — require_upstream_session, formerly named require_auth — checks only whether the server holds a usable LEXIS credential. It never inspects the request. There are no securitySchemes in the OpenAPI document and no operation declares security.

Two consequences worth stating plainly:

A 401 means the server is logged out, not you are not allowed.
Anything that can reach the port can drive every route, including POST /remote/remove, POST /remote/push and POST /local/rm, under the server's single shared identity.

Do not expose owilix http serve on a public or shared network as-is. Run it on localhost, or behind a reverse proxy, tunnel or gateway that authenticates callers.

Install #

The normal CLI install remains lightweight. Install the optional dependencies when you need the HTTP server:

uv pip install "owilix[http]"

Docker builds install the http extra by default.

Run Locally #

owilix http serve

The command binds to 127.0.0.1:8080 by default. Keep that default for local use. If you bind to 0.0.0.0, protect the service with a reverse proxy, firewall, tunnel, or another access-control mechanism.

Useful checks:

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/auth/status

Authentication Flow #

The server can start an OAuth device login and store the resulting refresh token in the same OWILIX token location used by the CLI:

curl -X POST http://127.0.0.1:8080/auth/device/start
curl http://127.0.0.1:8080/auth/device/<state-id>

Remote and query endpoints require the server process to have a refresh token or PY4LEXIS_TOKEN. This is a process-credential check, not caller-level API authorization.

POST /auth/logout reverses a device login: it deletes the stored refresh token and discards any pending device-login attempts, so GET /auth/status reports authenticated: false afterwards.

curl -X POST http://127.0.0.1:8080/auth/logout

When the server was started with PY4LEXIS_TOKEN in its environment, logout cannot revoke that credential — the endpoint returns 409 with {"error": "env_token_auth_active"} and leaves the token in place. Restart the process without the variable instead.

Endpoint Surface #

Core endpoints:

GET /health
GET /auth/status
POST /auth/logout
POST /auth/device/start
GET /auth/device/{state_id}
GET /jobs
GET /jobs/{job_id}

Local endpoints:

GET /local/ls
POST /local/free and POST /local/rm
POST /local/insert
POST /local/export
POST /local/analyze
GET /local/quota
POST /local/quota
POST /local/cleanup

Remote endpoints:

GET /remote/ls (can take minutes on object-store remotes — see Listing cost)
POST /remote/search
POST /remote/pull
POST /remote/push
POST /remote/upload
POST /remote/remove
POST /remote/doctor (asynchronous: returns a job record)
POST /remote/doctor/status (synchronous: returns the status document inline; reports readable per repository, and fails one whose metadata cannot be read)
POST /remote/catalog
POST /remote/status
POST /remote/diff

Query endpoints:

POST /query/less
POST /query/sites
POST /query/analyze
POST /query/aggregate
POST /query/stats
POST /query/slice
POST /query/stream

GET /remote/ls and GET /local/ls accept ?include=metadata — see Full descriptive metadata.

Derived artifact endpoints:

GET /remote/catalog/summary — declared vs observed across a scope
GET /remote/datasets/{id}/summary — one dataset, with its verification record
GET /remote/collections/{c}/verify — last verification outcome per dataset

Workflow monitoring endpoints (Airflow):

GET /workflow/dags
GET /workflow/runs
GET /workflow/executions
GET /workflow/summary

These are synchronous — they are read-only queries that answer in seconds, and a monitoring caller wants the answer rather than a job handle. See Workflow monitoring.

Configuration and plugin endpoints:

GET /config
POST /config/get
POST /config/set
GET /plugin/list
POST /plugin/run

Long-running operations return a job record. Poll GET /jobs/{job_id} until the job reaches succeeded or failed. Job state is in-process and is lost when the server restarts.

Local Dataset Quota #

The HTTP server can report and enforce a local dataset cache budget based on the filesystem that backs the local repository root.

Configure the quota:

curl -X POST http://127.0.0.1:8080/local/quota \
  -H 'content-type: application/json' \
  -d '{"max_usage_percent": 40}'

Check status:

curl http://127.0.0.1:8080/local/quota

Dry-run cleanup:

curl -X POST http://127.0.0.1:8080/local/cleanup \
  -H 'content-type: application/json' \
  -d '{"dry_run": true}'

Actual cleanup removes oldest local dataset copies until the quota is met:

curl -X POST http://127.0.0.1:8080/local/cleanup \
  -H 'content-type: application/json' \
  -d '{"dry_run": false}'

You can also set OWILIX_HTTP_LOCAL_MAX_USAGE_PERCENT.

Optional S3 Facade #

Set OWILIX_HTTP_S3_ENABLED=true to mount the local repository through a local S3-compatible facade at /s3.

Credentials are required unless local test mode is enabled:

export OWILIX_HTTP_S3_ENABLED=true
export OWILIX_HTTP_S3_ACCESS_KEY=local-access-key
export OWILIX_HTTP_S3_SECRET_KEY=local-secret-key
owilix http serve

For local-only testing, OWILIX_HTTP_S3_NO_AUTH=true disables S3 facade credential checks.

Full descriptive metadata #

GET /remote/ls returns eleven flat keys per dataset. Those come from metadata.get(), which returns a display rendering, not the stored value:

field	flat key gives	actual value
`creators`	`"Lukas Gienapp+7"`	8 author objects, with ORCIDs and affiliations
`descriptions`	ellipsis-truncated	full text
`publisher`	`{"name": "CORAL Project"}`	same
`rightsList`	(absent)	licence with `rights` and `rightsUri`

The +7 means "and seven more" — a page cannot recover eight authors from it. For anything that needs the record rather than a table row, ask for it:

curl "http://127.0.0.1:8080/remote/ls?specifier=lexis/collectionName=corpora&include=metadata"

Shape. metadata carries the record as stored, with DataCite fields nested under a datacite key:

{
  "id": "…", "title": "German Commons", "collectionName": "corpora",
  "metadata": {
    "datacite": {
      "creators": [{"name": "Lukas Gienapp", "givenName": "Lukas",
                    "nameIdentifiers": [{"nameIdentifier": "0000-0001-…",
                                         "nameIdentifierScheme": "ORCID"}],
                    "affiliation": [{"name": "University of Kassel"}]}, …],
      "publisher": {"name": "CORAL Project"},
      "publicationYear": 2025,
      "rightsList": [{"rights": "Open Data Commons Attribution License v1.0",
                      "rightsUri": "https://opendatacommons.org/licenses/by/1-0/"}],
      "relatedIdentifiers": [{"relatedIdentifier": "https://huggingface.co/…",
                              "relatedIdentifierType": "URL"}]
    },
    "schema": "http://datacite.org/schema/kernel-4"
  }
}

Why opt-in. An unfiltered listing is ~1,900 records and most callers want the flat keys. include takes a comma-separated list; unknown values are ignored rather than rejected.

Absent fields are omitted, never null-filled. Mirrors carry several metadata shapes — one deployment has six distinct key sets between 33 and 43 keys, with relatedIdentifiers on only 65% of datasets. Nothing keys on a fixed key set or count, schema is read as the version marker where present and tolerated where absent, and a record whose metadata cannot be rendered reports metadata_error while the rest of the page still answers — one malformed sidecar does not fail the listing.

No re-pull or migration is required: old sidecars work as they are.

Derived Artifacts #

What summarize, verify and reindex write beside the data, readable without scraping console output or holding object-store credentials.

curl "http://127.0.0.1:8080/remote/catalog/summary?specifier=owi-up"
curl "http://127.0.0.1:8080/remote/datasets/<id>/summary?specifier=owi-up"

GET /remote/catalog/summary reports declared against observed for every dataset in a scope, plus two counts that must not be confused:

field	meaning
`materialised_count`	datasets measured and found to hold data
`unmeasured_count`	datasets never summarized — `materialised` is `null`

unmeasured_count is not 0 materialised. A dataset that has never been summarized has no observation at all, and reporting it as empty is the mistake that told visitors 586 collections were mirrored when 4 held data. observed is null for those, never {files: 0}.

GET /remote/datasets/{id}/summary adds the verification record when one exists:

{
  "id": "0027fdbc-…",
  "declared":  { "files": 567, "objects": 250000, "bytes": 14748809896 },
  "observed":  { "files": 0,   "bytes": 0 },
  "materialised": false,
  "stats_computed_at": "2026-08-07T18:26:48+00:00",
  "verification": {
    "tier": "checksum", "dataset_digest": "01001c23…",
    "checked": 12, "invalid": 1, "unreadable": 0, "ok": false,
    "computed_at": "2026-08-07T19:02:11+00:00"
  },
  "partial": false, "failed": []
}

verification: null means never verified, which is not the same as verified and clean. Branch on its presence before branching on ok.

Status codes carry the same care as everything else here: a dataset that is absent from a complete listing is 404; a dataset absent from a partial listing is 503 with partial: true, because we cannot say it is missing when we could not see everything. Both list endpoints carry partial / failed as described under Partial Listings.

Workflow Monitoring #

Four read-only endpoints over the LEXIS Airflow instance, sharing their filter logic with owi workflow so the two cannot drift.

An execution is one task instance: one task, in one DAG run, with a state, a duration and the operator that ran it. That is the unit worth counting.

curl "http://127.0.0.1:8080/workflow/summary?days=7"
curl "http://127.0.0.1:8080/workflow/executions?state=failed&days=30&limit=20"
curl "http://127.0.0.1:8080/workflow/executions?since=2026-08-01&until=2026-08-05"
curl "http://127.0.0.1:8080/workflow/runs?dag_id=my_dag&state=failed"

Filters (all optional, all accepting comma-separated values):

parameter	meaning
`dag_id`, `task_id`, `operator`	identity
`state`	`success`, `failed`, `skipped`, `queued`, `up_for_reschedule`
`pool`, `queue`, `hostname`	where it ran
`min_duration`	seconds, floor
`since`, `until`	`YYYY-MM-DD` or ISO-8601; naive values are read as UTC
`days`, `hours`	relative window; defaults to the last 7 days
`offset`, `limit`	paging (`executions` only)

Values are OR within a field and AND across fields, so ?state=failed,up_for_reschedule&dag_id=X means "either of those states, in DAG X". The response echoes back window and filters, so a caller does not have to remember what it asked for.

GET /workflow/summary groups the same filtered set by state, operator and DAG, with avg_duration_sec and p95_duration_sec. Those are null rather than 0 when nothing reported a duration — a task that never recorded one is not a task that took no time.

GET /workflow/runs carries the partial contract: if one DAG cannot be read the others are still returned, with partial: true and failed: [...].

Status codes: 400 for an unparseable date, 401 when there is no LEXIS credential, 502 when Airflow is unreachable. An empty result is 200 with total: 0 — distinguishable from all three.

Partial Listings #

GET /remote/ls and GET /local/ls say whether the answer is the whole answer:

{
  "count": 584, "total": 584, "offset": 0, "limit": null, "has_more": false,
  "datasets": [ /* ... */ ],

  "partial": false,        // always present; false when nothing was skipped
  "failed": []             // sources that contributed less than they hold
}

When something was skipped, partial is true, failed names the sources, and an incomplete array carries the reason:

{
  "total": 0, "has_more": false, "datasets": [],
  "partial": true,
  "failed": ["owi-up"],
  "incomplete": [
    {
      "source": "owi-up",
      "reason": "error",
      "message": "[Errno 22] Precondition Failed"
    }
  ]
}

reason is a small closed set — auth, unreachable, timeout, error, partial_read — because the reason is what a consumer should branch on: auth is operator-actionable and probably persistent, timeout is probably transient and worth retrying. Matching on message text is fragile and is how this class of failure goes unnoticed.

Why this exists. A listing that fails every read used to return total: 0 with HTTP 200, which is indistinguishable from a store that genuinely holds nothing — and consumers that retire anything absent from the latest listing would quietly retire the lot. partial is the bit that tells them not to.

Backward compatible. partial is false and failed is [] for a complete answer, incomplete is present only when there is something to report, and no existing field changed. A caller that ignores all three behaves exactly as before.

Listing Cost #

GET /remote/ls enumerates. For a LEXIS repository the catalogue is queried and the answer comes back quickly; for an object-store remote (s3a/s3) the store itself is walked, so the cost tracks the size of the store rather than the size of the answer — and it is slowest when it has least to say. Measured against a full OWI cluster:

specifier	time	result
`owi-up/collectionName=main` (s3a)	84.1 s	0 datasets
`lexis/collectionName=main`	7.1 s	584 datasets

Consequences for a synchronous caller:

Budget minutes, not seconds. A 30 s client timeout against an object-store remote will fail every time, and it fails in the way least likely to be diagnosed correctly — as a connection error rather than as a slow answer.
Cache the result if you are rendering it repeatedly (a dashboard panel, a health view). The answer changes far more slowly than the cost of asking.
Do not use ls as a liveness probe. Use POST /remote/doctor/status, which reports per-access-level existence with a handful of prefix checks and does not enumerate. It answers "is this remote reachable, and does it hold anything at each access level" — which is what a health light needs.

There is deliberately no cheap dataset count endpoint. A count over an object store requires listing the keys under the prefix; there is no cheaper primitive to expose. The measurement above is the evidence: that 84 s call returned zero datasets, so no per-dataset metadata was ever fetched and the time went entirely into walking the store. A "count without enumerating" would have to be served from a maintained index or manifest rather than from the store, which is a different feature and not something ls can be made to do.

Known Limits #

HTTP endpoint authorization is preview-level; use only on localhost or behind external access control.
Jobs are in-process only and disappear after restart.
Job cancellation and durable progress records are not implemented.
remote/ls pagination currently slices after fetching backend results — so a page request costs the same as the full listing (see Listing cost).
The S3 facade has been smoke-tested; broader compatibility with aws s3, boto3, and rclone remains follow-up work.