owi-cli
Command line interface for accessing the Open Web Index shards and slices.
Star
Fork
Watch
owi-cli docs changes.md
39 kB · 443 lines
Markdown
at main

Owilix Changes #

Human-curated changelog tracking significant changes across branches.

Format #

Each entry follows:

## [Branch] @ [commit-hash] - YYYY-MM-DD

### Summary
Brief description of what was accomplished.

### Changes
- List of specific changes

### Breaking Changes (if any)
- List of breaking changes

Unreleased #

main @ v5.10.9 - 2026-08-08 #

The descriptive record, and two accessors that lied about it #

Summary: Asks 1, 2 and 3a/3b from the openwebindex.eu dashboard. Each was reproduced against live data before being changed.

Changes:

  • ?include=metadata on /remote/ls and /local/ls, sourced from as_json_dict(). Verified against the real German Commons record: 8 creators with ORCIDs where get() said Lukas Gienapp+7, a 1,385-character description where get() truncated, and a licence where get() returned None. Swept as_json_dict() across all 1,864 LEXIS records: zero failures, across all six shape variants.
  • DatasetMetadata.__getitem__ now agrees with .get() for advertised-but-absent keys, so dict(md) works. Six keys raised before.
  • RightsListMetadata.short_repr() no longer assumes rightsIdentifier. This is why rightsList was missing from the flat projection: the KeyError was swallowed by .get().
  • Response models declared for the listing endpoints, with coercing validators so typing cannot turn one odd sidecar into a 500.
  • require_auth renamed to require_upstream_session, alias kept, wire error code kept, docs made explicit about the absence of caller authentication.

Two mistakes caught during the work, both worth recording:

  • Declaring the response model initially emitted metadata: null and metadata_error: null on every row — precisely the null-filling the report asked us to avoid. Fixed with response_model_exclude_unset=True, verified by diffing the default response.
  • Two workflow tests written yesterday pinned a fixture date and queried a relative days=7 window. They passed on the day written and failed the next morning, looking like a regression in unrelated code. Fixtures are now relative to now.

Breaking Changes: None. include is opt-in, the default listing is byte-identical, require_auth still resolves, and the 401 code is unchanged.

Not addressed (reported, deferred by the reporter): /health cannot fail and there is no readiness probe; /remote/ls pagination re-walks the backend per page with no cache.

main @ v5.10.8 - 2026-08-07 #

The last two plan items: --since, and reading verification back #

Summary: verify --since and GET /remote/collections/{c}/verify. All four gaps recorded in Part III are now closed, with one deliberate non-delivery documented.

Changes:

  • --since skips datasets verified more recently than the cutoff, making a weekly pass over a large mirror practical. Tier order is respected — a recorded checksum satisfies a readable request, not the reverse — and a record whose computed_at is missing or unparseable is treated as stale, so incomplete information re-verifies rather than skips.
  • GET /remote/collections/{c}/verify exposes the last outcome per dataset with ok / failing / unverified counted apart. The distinction is the point: a dashboard that reads "no record" as "failing" reports unverified data as broken.
  • GET /collections/{c}/index deliberately not built — the index is an internal optimisation over the sidecars, and the catalog summary already serves consumers. Recorded rather than left implied.

Verified on the fixture: --since 2026-08-06 skipped all 3 as fresh (0 verified); --since 2026-08-09 re-verified all 3; a malformed date is rejected with a validation error rather than silently ignored.

Breaking Changes: None — new flag and new endpoint.

Plan status: M0–M6 complete, all Part III gaps closed.

main @ v5.10.7 - 2026-08-07 #

ls --observed: the claim and the measurement, side by side #

Summary: M1's acceptance criterion, the last operationally meaningful gap in the summarize/integrity plan.

Changes:

  • observed_totals() in core/tasks/remote.py aggregates stats.json across a listing, returning measured, unmeasured, observed, declared and the ids of datasets holding no data — counts rather than one total, because summing across unmeasured datasets produces a number that looks like a measurement and is not.
  • owi remote ls --observed prints it beneath the normal table.

Two design choices worth recording:

  • Additional block, not a changed summary line. Mutating the default alters numbers every consumer reads, and the value here is the comparison: declared 570 files against measured 3 says something neither figure says alone.
  • Unmeasured is its own count. 0 files measured and 3 datasets never summarized are different statements, and collapsing them is precisely the 586-vs-4 error this closes.

Verified on the fixture both before and after summarize: unmeasured 3 / measured 0 first, then measured 3 files against declared 570, with the empty dataset named.

Breaking Changes: None — opt-in flag, default output unchanged.

Remaining from the plan: verify --since (F7's last cost control) and a per-collection verification endpoint (M5 shipped in a different shape). Neither is operationally urgent.

main @ v5.10.6 - 2026-08-07 #

Write-back, and the phantom dataset it would have created #

Summary: F5 delivered, plus documentation housekeeping recording which acceptance criteria are still unmet.

Changes:

  • summarize --write-back/--no-write-back, and --summary --write-back placing _collection_summary.json beside each collection.
  • The guard that had to come first: list() treats any *.json in a collection directory as a dataset sidecar, so the aggregate would have appeared in every listing as a phantom — the latest.json failure shape, self-inflicted. COLLECTION_ARTIFACTS excludes owilix's own collection-level artifacts from listings, the index, and the fingerprint. Verified on the fixture: 3 datasets before writing the aggregate, 3 after.
  • Part III of the plan now records shipped-vs-open per acceptance criterion, after "essentially complete" was claimed here once and was wrong.

Tested against the local file:// fixture only. --write-back writes into the repository, and the LEXIS credential in this environment is currently valid, so running it against a real specifier would have written to production storage.

Breaking Changes: None. --write-back defaults to true, which is the previous behaviour.

Still open: remote ls --observed (the number a catalogue reads is still declared-only), verify --since, and a per-collection verification endpoint.

main @ v5.10.5 - 2026-08-07 #

Derived artifacts over HTTP, and checksums that can actually be re-run #

Summary: M5 plus the incremental mode M4 unblocked. Two bugs found by running the code rather than reading it.

Changes:

  • GET /remote/catalog/summary and GET /remote/datasets/{id}/summary expose what M0–M4 write beside the data. Unmeasured is kept distinct from measured-as-empty, and verification: null distinct from verified-and-clean — the same 0-vs-NULL rule, now at the API boundary.
  • Incremental checksums: reuse a recorded digest when size and mtime both match, labelled cached and counted. --recheck-all and --only-missing bound a run.
  • Bug found while testing: checksums.json was verifying itself. Self-referential by construction — hashing it changes it — so it could never converge and would rewrite on every run. Derived artifacts are now excluded, which also stops a regenerated README changing dataset_digest.
  • Bug found while testing: fill_file_details fetched only size, so the incremental reuse and the iRODS-checksum preference — both written, both tested against injected data — could never fire against a real repository. Switched to fs.info(), same round trip, and the tests that passed for the wrong reason now exercise the real path.

Also corrected: the new HTTP tests initially passed a NameError off as "no stats", because _read_stats_json swallows exceptions and json was not imported in that test module. Worth noting as a pattern — a helper that catches broadly turns a test bug into a plausible-looking result.

Breaking Changes: None. New endpoints; verification behaviour is additive and --recheck-all restores the previous full-pass semantics.

Still open from the plan: nothing in M0–M6 except cost controls beyond --max-files/--max-datasets/--only-missing (F7's --since), and the CLI/HTTP surfaces for reindex scheduling.

main @ v5.10.4 - 2026-08-07 #

A collection index that refuses to be trusted when it should not be #

Summary: M4. _index.parquet per collection plus remote reindex, shipped together because an index without a rebuild path is a liability rather than a cache.

Changes:

  • New owilix/core/index.py: fingerprinting, build, read, and an IndexUnusable exception whose only correct handling is to walk — raising rather than returning None makes that impossible to forget at the call site.
  • FileBasedRepository.list() consults the index per collection and falls through to the existing walk on any failure.
  • remote reindex walks the sidecars once and writes the index; owi remote ls needs no flag.
  • Staleness is a fingerprint over each sidecar's name, size and mtime, recomputed from one ls. Chosen over a bare object count because it also catches a resized or re-uploaded sidecar at identical cost, since the listing already returns the details.

Verified end to end on the fixture: walk and index return identical results; adding a dataset without reindexing yields 4 not 3; a corrupted index yields 4; a deleted index yields 4. All four paths agree with the source.

Breaking Changes: None. No index means the previous behaviour exactly.

Second customer: M6 tier 3 currently re-reads every byte each pass because it has no way to know what changed. The index is what an incremental checksum mode would build on — that is now the obvious next step for it.

Still open from the plan: M5 (HTTP exposure of the derived artifacts), and incremental checksums on top of this index.

main @ v5.10.3 - 2026-08-07 #

Verification: presence, readability, checksums — and what a hash cannot tell you #

Summary: M6 tiers 2–3. owi remote verify with three cost tiers, format-aware corruption detection, and results stored beside the data.

Changes:

  • New owilix/core/tasks/verify.py and owi remote verify. Tiers presence / readable / checksum.
  • Format validation belongs in the checksum tier, per the user requirement: a truncated Parquet hashes perfectly, so hashing alone certifies corruption rather than detecting it. checksum and format_valid are independent facts; format_valid: null for uncheckable formats keeps "unknown" distinct from "invalid" — the same 0-vs-NULL rule as everywhere else in this plan.
  • Hashes stored alongside the file (checksums.json) so non-iRODS clients can verify independently; md5 kept beside sha256 for MD5-only tooling.
  • iRODS checksums read where available, labelled checksum_source so a reader can distinguish verified from asserted.
  • Multipart ETag -N detected and refused as MD5; a genuine single-part disagreement is recorded, never repaired.
  • dataset_digest over sorted (path, size, sha256), with unhashed files contributing ? so partial can never masquerade as complete.

Verified: on the local fixture, a deliberately truncated parquet produced a valid SHA-256 and format_valid: false with the parser's reason ("Parquet magic bytes not found in footer"), while the healthy dataset reported rows, row groups and columns. checksums.json written for all three datasets.

Breaking Changes: None — new command and a new derived artifact.

Still open from the plan: M4 (_index.parquet + remote reindex) and M5 (HTTP exposure). M6 tier 3 has no incremental mode yet — it re-reads everything each pass, which is what M4's index would fix.

main @ v5.10.2 - 2026-08-07 #

README gets a machine-readable head, without restoring the churn #

Summary: M3 of the summarize/integrity plan. YAML front matter on README.md, prose untouched.

Changes:

  • _readme_front_matter emits the index-row fields as YAML; split_front_matter parses them back and is tolerant of a malformed or absent head.
  • The interaction that mattered: computed_at now appears in the README too, so the M2 unchanged-write check would never have matched and every README would have been rewritten on every run — reinstating the exact churn that could invalidate a dircache ETag and, before v5.9.0, empty a listing. _stats_computed_at_only_diff now strips volatile keys from front matter as well as from stats.json, and compares the prose literally. Caught by running the fixture, not by reading the diff.
  • _yaml_safe coerces unrepresentable metadata values rather than raising.

Verified: front matter round-trips; a second --force pass writes nothing; the human section is byte-identical when only the timestamp moves.

Breaking Changes: None. Additive head; existing readers of the prose are unaffected.

Still open from the plan: M4 (_index.parquet + remote reindex — where the risk concentrates, and its own release), M5 (HTTP exposure), M6 (verify tiers, checksums, cost controls).

main @ v5.10.1 - 2026-08-07 #

Observed vs declared, provenance, and an end to write churn #

Summary: M0–M2 of the summarize/integrity plan, in the order the plan set. All additive.

Changes:

  • M0 provenance: stats.json records computed_at and owilix_version. It had none — the fields the plan credited it with are remote doctor output from a different command. Provenance was pulled ahead of the index because index rows that cannot say when they were computed reintroduce the ambiguity below at a larger scale.
  • M0 honest empties: parquet_files_found plus materialised separate measured, empty from nothing measured. _generate_stats_json returned the same literal for both.
  • M1 observed vs declared: declared from the sidecar, observed from the store. observed reports 0 rather than null when a listing succeeded and found nothing, because zero is a measurement; declared reports null for fields the sidecar lacks.
  • M2 content-addressed writes: _write_if_changed skips a write when bytes would not change. stats.json is compared ignoring computed_at alone — otherwise the timestamp would defeat the fix — and an unreadable existing object is written rather than assumed unchanged.

Verified on the same three-dataset fixture used to schedule the plan: the dataset declaring 567 files and 250k objects over an empty directory now records observed: {files: 0, bytes: 0}, materialised: false; a second --force pass writes nothing where it previously rewrote every object.

Breaking Changes: None. statistics[] keeps its shape and every new key is additive.

Still open from the plan: M3 (README front matter), M4 (_index.parquet + remote reindex, where the risk concentrates), M5 (HTTP exposure), M6 (verify tiers, checksums, cost controls).

main @ v5.10.0 - 2026-08-07 #

Workflow monitoring, and the reason it had never worked #

Summary: owi workflow (dags, runs, executions, summary) plus four HTTP endpoints, filterable by date and metadata, with JSON everywhere. The blocker turned out to be a one-line auth bug, not missing code.

Changes:

  • Root cause first: AirflowMonitor sent X-Api-Token, which the LEXIS gateway answers with 401 for a valid token. Probed all three variants against the live instance — Authorization: Bearer returns 200, X-Api-Token returns 401 — so the 963-line module in compat/ had never been able to work. That explains why it was unregistered and untested rather than merely neglected.
  • New shared layer owilix/core/tasks/workflow.py: window resolution, filtering, summarisation and the error taxonomy, called by both the CLI and the HTTP API so the filters cannot drift. The Airflow client itself is reused from compat/workflows.py rather than rewritten.
  • CLI owi workflow registered in the modern Typer app: dags, runs, executions, summary, each with --json.
  • Filters: --dag, --task, --operator, --state, --pool, --queue, --hostname, --min-duration, --since/--until, --days/--hours, --limit/--offset. OR within a field, AND across fields. Field names taken from real task instances (dag_id, task_id, operator, state, duration, pool, queue, hostname, …), not guessed.
  • HTTP: GET /workflow/{dags,runs,executions,summary} with the same filters as query parameters, synchronous because they answer in seconds. 400 bad date / 401 no credential / 502 Airflow unreachable; empty stays 200 with total: 0.
  • summarize_executions is reimplemented in the shared layer rather than reusing AirflowMonitor.summarize, because the latter re-fetches and cannot see a filter — a summary must describe exactly the set that was listed.
  • Durations report None rather than 0 when unmeasured, following the rule from the summarize/integrity plan.

Verified live: against the LEXIS Airflow with a fresh token — 9 DAGs, 435 executions in 7 days, 6 failed. The failures are heappe_keep and ref_token_keeper pairs recurring on 2026-08-03, 08-05 and 08-07, i.e. a token-keeper failing roughly every other day. Both CLI and HTTP paths exercised, including date-range filtering and the 400 on a malformed date.

Breaking Changes: None. New command group and new endpoints; nothing existing changed.

main @ v5.9.1 - 2026-08-07 #

Verified the summarize/integrity plan and gave it an execution schedule #

Summary: Docs only. Part II appended to docs/review/summarize_and_integrity_feature_plan_2026-08-07.md: verification against the code, three corrections, milestones M0–M6 with acceptance criteria, and standing rules. Part I left as submitted.

Changes:

  • Built a three-dataset collection against a FileBasedRepository (the class backing s3a remotes) including the 586-vs-4 shape — a sidecar declaring 567 files over an empty directory — and ran the real commands rather than reasoning about them.
  • Correction 1: stats.json carries no provenance. The plan credited it with owilix.version/py4lexis.version; those are remote doctor output. Provenance becomes a prerequisite (M0), not a preserved property.
  • Correction 2: the 0-vs-NULL rule the plan asks us to adopt is already violated — _generate_stats_json returns the same {"values": {}} for "measured, empty" and for "no parquet found". The rule now has a concrete first target.
  • Correction 3: host stats select a url column and silently skip when it is absent.
  • Measurements kept: ls reports 570 files / 250,200 objects where 3 / 200 exist; summarize fully describes a dataset holding nothing; --force rewrites byte-identical objects, confirming F4.1 is churn removal rather than an optimisation.
  • Sequencing: M0–M3 are small and safe (patch-sized); M4 (index + reindex) is flagged as where risk concentrates, because it puts a derived artifact in the read path — the one place this codebase has repeatedly turned a failure into a confident wrong answer — so it gets fallback-and-warn, a staleness check and its own release.

Breaking Changes: None — documentation only.

main @ v5.9.0 - 2026-08-07 #

One rewritten file no longer empties a mirror listing, successfully #

Summary: M1 from the backlog — the live production bug. GET /remote/ls?specifier=owi-up returned {"total":0,"has_more":false} with HTTP 200 against a bucket holding 591 datasets, while remote doctor called the same repository healthy. All five asks from docs/review/owilix_improvement_brief_2026-08-07.md, in the order given.

Changes:

  • Carry the outcome (owilix/core/listing.py): DatasetListing is a list subclass with complete and incomplete entries of {source, reason, message}; /remote/ls and /local/ls expose partial, failed and incomplete. reason is a closed set because the reason is what a consumer branches on. This also delivers the listing half of M2 — field naming follows the bug report's partial/failed with the M2 proposal's incomplete array alongside, so both documents' consumers are served.
  • Per-file tolerance: list() collects read errors, returns the survivors and reports the skipped count. On its own this turns 591 → 0 into 591 → 590 plus a warning.
  • Explicit size to fs.open: s3fs sets IfMatch only when size is None, from a dircache that never expires, so a long-lived server 412s forever on any object rewritten after it was first listed. list() already has the size from the listing it just did. Falls back for filesystems that reject size=.
  • status() reads one sidecar and reports readable, failing a repository whose metadata cannot be read. An empty repository stays healthy — readable is None, not False, when there is nothing to read.
  • Prune before loading: an exact collectionName= filter restricts the directories walked. Regex and negation forms still walk everything, and a pruned walk that finds nothing re-walks — collectionName is metadata, the directory is convention, and a confidently empty answer is the bug being fixed.

Verification: the reproduction was shrunk and run against both tags. At v5.8.0 it raises OSError [Errno 22] Precondition Failed, which the aggregate turns into an empty listing; at HEAD the same input returns the 5 survivors with complete=False. Suite 621 → 647.

Breaking Changes: None. No existing field or signature changed; DatasetListing is a list, partial defaults to false. Minor rather than patch because the response gained fields and doctor can now report unhealthy where it previously said connected — deployments that alert on doctor output may see a repository flip that was already broken.

Left open: M2's remote pull half (fail on partial discovery, --allow-partial) is a behaviour change for existing callers and was not folded into a fix release. M3 (observed vs declared) untouched.

main @ v5.8.0 - 2026-08-06 #

Auth failures named, listing cost documented, and the small stuff #

Summary: Closes F3, F4, F6 and a new F8 from the mirror-operability review. With v5.7.0 the review is fully released except one item knowingly left open.

Changes:

  • F3: New owilix/core/exceptions.py (OwilixError, AuthenticationError, looks_like_authentication_failure). LexisRepository.list re-raises auth-shaped failures instead of returning []; AggregatedRepository.list re-raises when an auth failure left nothing to return; remote_pull reports ErrorType.AUTH / ExitCode.AUTH_ERROR. Two findings worth keeping: the failure was swallowed at two layers, so fixing either alone would have changed nothing; and detection cannot go by exception type, because py4lexis reports a missing credential as Py4LexisAPIException rather than its own Py4LexisAuthException — the DDI API answers with an errorString it does not recognise. Phrase matching is kept specific, with a test that "Failed to tokenize response body" does not read as a login problem.
  • F4: Documented on /remote/ls and in remote ls --help with both measurements. The cheap existence check turned out to already exist — FileBasedRepository.status does one fs.exists per access level and never enumerates, and F7 in v5.7.0 is what stopped it raising for those backends — so the docs point at POST /remote/doctor/status rather than adding an endpoint. A cheap count is declined with the measurement as the reason: the 84 s call returned zero datasets, so load_metadata was never reached and all 84 s was listdir/ls; a shallow count would cost the same.
  • F6: ask_yes_no detects a non-interactive stdin and declines rather than raising from termios.tcgetattr. Declining rather than accepting is deliberate — default='y' is what Enter means for a human, not consent from a caller that was never asked. The two bare except: in core/tasks/remote.py are now except Exception with a debug log, so Ctrl-C is no longer swallowed. Needed a module logger in that file; it had none.
  • F8 (new, found while editing the docstring for F4): owi remote ls --help crashed with MarkupError, because the specifier syntax [/<filter>] in the docstring parses as a Rich closing tag and Typer renders docstrings as markup. Reproduced at v5.7.0, so it long predates this work. Nothing had ever invoked --help in a test; a sweep over all 57 commands now exists and fails at v5.7.0 on exactly remote ls.

Breaking Changes: None to the HTTP API shape. remote pull gains exit code 3 for an unusable credential, where it previously reported the v5.7.0 data error (4) or, before that, success. A non-interactive caller that omitted --yes previously crashed and now receives a declined answer — a behaviour change only in the sense that the previous behaviour was a traceback.

Left open, deliberately: a caller still cannot tell that a listing was partial when one repository failed and another answered. Fixing it means changing the list() contract and with it the /remote/ls response shape, so it is recorded under F3 rather than folded in.

main @ v5.7.0 - 2026-08-06 #

Discovery failures are visible, and doctor works on non-LEXIS backends #

Summary: Released F2 and F7 from the mirror-operability review, both found by operating 5.6.0 against a live ~51 TiB mirror rather than by reading the code. Minor rather than patch: both change what an existing call reports.

Changes:

  • F2 (1a11291): remote_pull returns success=False / ErrorType.DATA / ExitCode.DATA_ERROR when discovery matched no dataset, naming the specifier. Deliberately narrow — datasets discovered whose files are already present stays a success and stays exit 0, because conflating the two would make an idle nightly mirror look broken. The HTTP job layer maps success=False to a failed job (http_api/jobs.py:121), so API consumers inherit the distinction without an API shape change.
  • F7 (dbf2cca): progress_cb=None accepted and ignored on AbstractRepository.status and FileBasedRepository.status, forwarded by AggregatedRepository.status. remote doctor calls it unconditionally at core/tasks/remote.py:504; only LexisRepository had it, so every file-based backend answered with a TypeError instead of a status.
  • Reviewed both before release. Audited all four status() implementations and both call sites (cli/admin.py:56 passes nothing and is fine on the default); confirmed ddi.py and irods.py define no repository status(); confirmed the F2 job mapping actually exists rather than taking the commit message's word for it.
  • Added the regression test F7 shipped without (TestStatusProgressCbContract). Written against dynamically discovered backends rather than the three that existed, with a guard against vacuous passes, because F7 was a contract satisfied by one implementation and unenforced everywhere else. Verified to fail at dbf2cca~1 with the production error.
  • Fixed test pollution introduced by F2's own counter-case test: it requested tmp_path and never used it, so _write_pull_sidecars → core/sync.py resolved a bare MagicMock through __fspath__ and mkdir'd MagicMock/mock.local.create().path/<id>/ into the repo root on every run. Fixed rather than gitignored, and the absence is now asserted.

Breaking Changes: None to the HTTP API shape. remote pull's exit code and success flag change for the zero-discovered case — intended, and the reason for a minor bump. Automation that alerts on non-zero exit will now surface a permanently mis-scoped specifier instead of passing silently.

main @ v5.6.0 - 2026-08-04 #

Remote pull --push-to source selection, and HTTP logout / doctor status #

Summary: Fixed a self-cancelling query that made remote pull --push-to report "0 datasets found" and exit 0 for nearly every specifier — the reason the OWI mirror could not be filled. Landed the two HTTP endpoints the accompanying operability review called for, and recorded the review itself.

Changes:

  • remote_pull now passes spec.get("data_center") as the listing source instead of falling back to push_to_remote. The fallback combined with ignore_data_centers=[push_to_remote] into "list from X while excluding X", which is empty by construction. It hit every specifier whose data center is None — all, and every it4i:/lrz: zone alias, since the aliases reset the data center before setting a zone query — so only direct repository specifiers (lexis:…) ever worked. remote_ls was never affected because it already passed the data center alone, which is why ls and pull disagreed on identical input. ignore_data_centers is deliberately kept.
  • remote_doctor returns its results dict as CommandResult.object when as_json is set (previously always None), so the per-repository status, backend, DDI counts and per-zone breakdown are reachable without scraping rendered console output.
  • Added POST /auth/logout (DeviceAuthManager.logout() / clear_device_states()): deletes the refresh token, drops pending device logins, and returns 409 env_token_auth_active when PY4LEXIS_TOKEN is set, since the server cannot revoke a credential handed to it by its environment. _poll_until_complete now returns rather than raising KeyError when its state was cleared underneath it — reachable once states can be cleared.
  • Added POST /remote/doctor/status for a synchronous status document; POST /remote/doctor stays the async job form.
  • Recorded docs/review/remote_pull_and_mirror_operability_2026-08-04.md: findings reproduced against the live clusters rather than inferred. F1 (the --push-to defect) and F5 (doctor unreadable over HTTP) ship here. F2 remains open and is the most valuable of the rest — zero discovered datasets still takes the success path and is indistinguishable from "already up to date", which is what hid F1 for weeks. F3 (auth failure surfacing as a TypeError during error formatting), F4 (an 84 s remote ls against object-store remotes) and F6 (two bare except:, ask_yes_no assuming a TTY) are also open.
  • Verified during release that the review's F6 note about eval(files) does not apply to any build carrying the HTTP API: the bare form existed through v5.2.0, ast.literal_eval landed in v5.3.0 (fb24445), and the HTTP API first shipped in v5.5.0. Recorded as an upgrade note in CHANGELOG.md rather than a fix.
  • Bumped package metadata to v5.6.0 (minor, not patch: two new endpoints) and refreshed the Readme.md version badge, stale at 5.5.0 since that release.

Breaking Changes: None. Both endpoints are additive and no existing response shape changed.

main @ v5.5.2 - 2026-07-15 #

Query slice CIFF index fix #

Summary: Fixed query slice writing a header-only (empty) index.ciff.gz for the sliced dataset, and restored compatibility with ciff-toolkit 0.2.x.

Changes:

  • Filter each source dataset's CIFF index into its own temp file and merge the non-empty results per partition (previously every source was filtered onto the same target path, so the last source — often without matching documents — clobbered the index).
  • Warn on the console when a sliced index contains 0 documents or a partition has no matching source index.
  • Ported owilix/core/ciff.py to the ciff-toolkit 0.2.x CiffWriter API and bumped the dependency pin to >=0.2.2; drop_parallel returns (kept_docs, postings_lists).
  • Fixed UnboundLocalError when --search is combined with CIFF slicing; normalized parquet id values (incl. binary columns) before matching.
  • Added regression tests (clobbering, merging, partitioned layouts, binary ids, zero matches, keep=False) and a network-free end-to-end CLI test that asserts sliced index document counts.
  • Bumped package metadata for the v5.5.2 patch release.

Breaking Changes: None (requires ciff-toolkit>=0.2.2 at install time).

main @ v5.5.1 - 2026-06-02 #

Remote pull push-to payload routing fix #

Summary: Fixed LEXIS-backed remote pull --push-to <repo> so dataset payload files are streamed into the configured destination filesystem instead of being written under the process working directory.

Changes:

  • Made LexisRepository.get(..., filesystem=...) copy from the LEXIS filesystem into the provided destination fsspec filesystem.
  • Preserved existing local-download behavior when no destination filesystem is supplied.
  • Added regression coverage using in-memory fsspec filesystems.
  • Bumped package metadata for the v5.5.1 patch release.

Breaking Changes: None.

feature/http-server @ v5.5.0 - 2026-05-30 #

Optional HTTP API preview #

Summary: Added an optional FastAPI-based HTTP API for local OWILIX automation and controlled demos, released as a preview local service with explicit deployment limits.

Changes:

  • Added owilix http serve through the optional http extra.
  • Added health, device-login auth, jobs, local, remote, query, config, and plugin endpoints.
  • Added in-process background jobs for long-running operations and pollable /jobs/{job_id} status.
  • Added optional local S3-compatible facade mounted at /s3.
  • Added local dataset quota status, quota configuration, and dry-run-by-default cleanup endpoints.
  • Added Docker HTTP compose support with localhost-only published port by default.
  • Documented the HTTP API preview, local-only release posture, S3 facade, quota controls, and known limits.

Breaking Changes: None.

feature/remote-sync-sidecar @ 1b1ebafd9ad408abbc4d8784e62f27f74272a5b8 - 2026-04-30 #

Remote sync sidecar workflow #

Summary: Added a sidecar-based remote sync workflow for pull/status/push operations so users can pull a dataset, edit local files, inspect dirty state, and push changed files without requiring remote-side checksums.

Changes:

  • Added .owi-sync.json and .owi-files.json.gz sidecars capturing local and remote file baselines from remote pull
  • Added owi remote status with local dirty-state detection, optional remote refresh, details output, generated-file inclusion, and explicit not-found rows
  • Made owi remote push upload sidecar-detected local additions and modifications while excluding sync sidecars and editor swap files
  • Refreshed sidecar baselines after successful sidecar-aware push
  • Fixed dataset ID handling in local free and remote push
  • Fixed metadata update regressions during remote push
  • Documented the implemented workflow under docs/features/implemented/remote-sync-sidecar-inventory-workflow.md

Breaking Changes: None.

main - 2026-02-25 #

Remote upload S3-source support and release preparation #

Summary: Added direct S3 URI source support to owi remote upload, aligned repository upload plumbing for external filesystems, and prepared release metadata for v5.2.0.

Changes:

  • Extended remote_upload source handling to support local paths plus s3:///s3a:// URIs
  • Passed external source filesystems through aggregate/file/lexis repository put() methods
  • Added unit coverage for unsupported source schemes and S3-backed upload flows
  • Updated remote command documentation and examples
  • Updated installer dependency handling for python-http-irods-client to satisfy uv direct URL requirement during install
  • Pinned installer-side py4lexis==5.0.1 to avoid transitive URL dependency drift conflicts in uv
  • Bumped package version and finalized release changelog entries for v5.2.0

multifs - 2026-02-17 #

Multi-filesystem and CLI consistency updates #

Summary: Improved multi-filesystem behavior across CLI and manager flows, with corresponding tests and docs updates.

Changes:

  • Extended HTTP-to-iRODS fsspec integration in owilix/core/fsspec/http2irods.py
  • Updated CLI and context/output handling in owilix/cli/ modules
  • Updated manager/query flow integration in owilix/core/manager/manager.py and owilix/core/tasks/query.py
  • Expanded unit coverage in tests/owilix/core/fsspec/test_core_fsspec_unit.py
  • Updated dependency lock and configuration docs (uv.lock, pyproject.toml, docs/source/config.md)

py4lexis4 - 2026-01-02 #

Py4Lexis 5.x Migration Complete #

Summary: Full migration to py4lexis 5.x HTTP APIs, replacing python-irodsclient with HTTP-based iRODS access.

Changes:

  • Added Http2IrodsFileSystem fsspec backend for HTTP-based iRODS access
  • Created owilix/core/repository/ modular package (replaces monolithic repository.py)
  • Added OWILIXSession wrapper for unified session management
  • Implemented configuration V3 with migration support (owi config migrate)
  • Added new display formats: --format=short|wide|markdown for owi remote ls / owi local ls
  • Added config commands: config version, config migrate, config delete, config reset
  • Improved test infrastructure with progress indicators

Breaking Changes:

  • Deprecated repositories it4i, lrz removed - replaced by unified lexis
  • Config auto-migrated from V2 to V3 via owi config migrate

main - 2026-01-01 #

Documentation Alignment #

  • Added docs/source/contributing/ section with symlinks to dev docs
  • Updated index.rst toctree to include contributing section
  • Added AI agent onboarding section to docs/epics.md
  • Fixed external links in dev docs for Sphinx compatibility
  • Created branch documentation at docs/branch/feature-docs-alignment.md

History #

main @ initial - 2026-01-01 #

Summary #

Initial AI infrastructure setup for owilix.

Changes #

  • Added .agent/workflows/ with development, testing, documentation, and release workflows
  • Created docs/architecture.md with project overview
  • Created docs/CODING_GUIDELINES.md with coding standards
  • Created docs/epics.md for epic tracking
  • Created docs/branch/ for per-branch documentation
  • Created docs/external/py4lexis.md with best practices
  • Fixed pyproject.toml merge conflict from poetry→uv migration