owi-cli

Command line interface for accessing the Open Web Index shards and slices.

Fork

owi-cli docs architecture.md

10 kB · 286 lines

Markdown

at main

mgrani · feat(search): add remote index full-text search with --search option · 7mo ago

133fe6b8

Owilix Architecture #

Purpose #

owilix is a command-line tool for managing Open Web Index (OWI) data slices across distributed storage backends.

Goals #

Provide unified access to OWI datasets across multiple datacenters
Enable querying, slicing, and downloading of web index data
Support multiple storage backends (iRODS, S3, local filesystem)
Abstract complexity of distributed data access

Package Structure #

graph TB
    subgraph CLI ["CLI Layer (owilix/cli/)"]
        entry["__init__.py · cli_main()"]
        remote["remote.py"]
        query["query.py"]
        local["local.py"]
        config["config.py"]
        admin["admin.py"]
        common["_common/ · CLIContext · CommandExecutor · output · progress"]
    end

    subgraph MANAGER ["Manager Layer (owilix/core/manager/)"]
        mgr["manager.py · OWIlixManager"]
        session["session.py · OWILIXSession"]
        cfg["config.py · OWIlixConfig"]
    end

    subgraph REPO ["Repository Layer (owilix/core/repository/)"]
        agg["aggregate.py · AggregatedRepository"]
        lexis["lexis.py · LexisRepository"]
        ddi["ddi.py · OWILexisDatasetAPI"]
        file["file.py · LocalRepository / S3FileBasedRepository"]
        base["base.py · AbstractRepository"]
    end

    subgraph TASKS ["Task Layer (owilix/core/tasks/)"]
        tremote["remote.py · remote_pull / remote_ls / …"]
        tquery["query.py · query_less / query_sites / …"]
        tlocal["local.py"]
    end

    subgraph FS ["Filesystem / DB (owilix/core/)"]
        http2irods["fsspec/http2irods.py · Http2IrodsFileSystem"]
        duck["db/duckdb_executor.py · OWIDuckDBSelectExecutor"]
        presign["warc/presign_client.py · PresignClient"]
    end

    subgraph EXT ["External"]
        py4lexis["py4lexis · LexisSession"]
        irods["irods_http_client · IrodsHttpClient"]
        duckdb["duckdb"]
    end

    entry --> common
    entry --> remote & query & local & config & admin
    remote & query & local --> TASKS
    TASKS --> mgr
    mgr --> session & cfg & agg
    agg --> lexis & file
    lexis --> ddi & http2irods
    ddi --> py4lexis
    http2irods --> irods
    session --> py4lexis
    tquery --> duck
    duck --> duckdb
    lexis --> presign

Core Abstractions #

Dataset #

A Dataset (owilix/core/models/dataset.py) represents a single data slice identified by UUID, with:

Metadata (title, dates, access level, collection name, data center, zone)
Files (parquet, json, warc)
A repository back-reference for fetch operations

Repository #

A Repository (owilix/core/repository/base.py — AbstractRepository) provides storage access:

Type	Module	Backend	Use Case
`LocalRepository`	`file.py`	Local filesystem	Dev, staging, output
`S3FileBasedRepository`	`file.py`	S3-compatible	Cloud storage
`LexisRepository`	`lexis.py`	iRODS HTTP + LEXIS DDI API	Authenticated datacenter access
`AggregatedRepository`	`aggregate.py`	Multiple repos	Unified multi-datacenter view

Manager (OWIlixManager) #

owilix/core/manager/manager.py — central orchestrator:

Loads owilix.cfg via OWIlixConfig
Instantiates repository objects from config
Holds the authenticated OWILIXSession
Exposes remote_data (AggregatedRepository) and local (LocalRepository)
Parses dataset specifiers via parse_specifier()

Command → Repository → LEXIS Flow #

The path from a user command to a LEXIS API call:

sequenceDiagram
    participant User
    participant CLI as cli/__init__.py<br/>(Typer callback)
    participant Ctx as CLIContext<br/>(cli/_common/context.py)
    participant Mgr as OWIlixManager<br/>(core/manager/manager.py)
    participant Agg as AggregatedRepository<br/>(core/repository/aggregate.py)
    participant Repo as LexisRepository<br/>(core/repository/lexis.py)
    participant DDI as OWILexisDatasetAPI<br/>(core/repository/ddi.py)
    participant LEXIS as py4lexis<br/>LexisSession / HTTP

    User->>CLI: owi remote ls lrz:latest
    CLI->>Ctx: create_context() → ctx.obj
    Note over Ctx: CLIContext holds<br/>ctx.obj.owi = OWIlixManager
    CLI->>Mgr: parse_specifier("lrz:latest")
    Mgr-->>CLI: {data_center:"lrz", day:"latest"}
    CLI->>Agg: remote_data.list(datacenter="lrz", ...)
    Agg->>Repo: LexisRepository("lrz").list(...)
    Repo->>DDI: ddi_api.get_all_datasets(access, filters, ...)
    DDI->>LEXIS: HTTP POST /meta/search?start=0
    LEXIS-->>DDI: page of dataset metadata
    DDI->>LEXIS: HTTP POST /meta/search?start=N  [paginated]
    LEXIS-->>DDI: next page …
    DDI-->>Repo: list[Dataset]
    Repo-->>Agg: list[Dataset]
    Agg-->>CLI: deduplicated list[Dataset]
    CLI->>User: table output

Specifier Format #

Specifiers encode filters in a compact string:

<datacenter>:<date>#<days>/<key>=<value>

Examples:

lrz:latest — newest dataset from the lrz data center
it4i:2024-03-01 — datasets from that date at it4i
lrz:latest/language=en — filtered by metadata field
lrz:latest#30 — 30-day window

Parsed by OWIlixManager.parse_specifier() into a dict consumed by AggregatedRepository.list().

CLI Task Wiring #

Each CLI sub-command delegates to a task function and wraps it via CommandExecutor:

# in e.g. owilix/cli/remote.py
from owilix.cli._common.executor import execute_command
from owilix.core.tasks.remote import remote_pull

@remote_app.command("pull")
def pull(ctx: typer.Context, specifier: str, ...):
    result = execute_command(ctx.obj, remote_pull, ctx.obj.owi, specifier, ...)
    if not result.success:
        raise typer.Exit(code=result.exit_code)

execute_command times the call, catches and categorises exceptions, logs the result
Task functions (core/tasks/) are pure Python — they accept a manager and return a CommandResult
Never call LexisRepository or OWILexisDatasetAPI directly from CLI code

Multi-Datacenter Routing #

Configuration #

owilix.cfg declares one or more named remote repositories, each tied to a LEXIS project/instance:

repositories:
  config:
    lexis:
      repository: lexis
      options:
        project_id: proj-abc
    it4i:
      repository: lexis
      options:
        project_id: proj-def
    lrz:
      repository: lexis
      options:
        project_id: proj-ghi
    local:
      repository: file
      options:
        path: ~/.owi/{access}
  selected_remote: [lexis, it4i, lrz]
  selected_local: [local]

Dispatch and Deduplication #

AggregatedRepository (core/repository/aggregate.py) fans out list() calls:

Dispatches to all selected remote repos in parallel (ThreadPoolExecutor)
Collects results; groups Dataset objects by internalID
For datasets appearing in multiple repos, picks the one from the best-scoring repo (based on recent response-time stats stored in ~/.owi/logs/repository_stats.json)
Returns a flat, deduplicated list[Dataset], each with a repository back-reference

When a datacenter filter is given (from the specifier), only matching repos are queried.

Zone Routing within a Datacenter #

Each LexisRepository may span multiple iRODS zones. File operations are handled by ZoneRoutingFileSystem (core/fsspec/), which selects the correct Http2IrodsFileSystem instance per zone based on dataset metadata.

Authentication & Session #

Session Initialization #

OWIlixManager._get_lexis_session() tries authentication sources in order:

Credentials from owilix.cfg (username + password)
PY4LEXIS_TOKEN environment variable (access token)
Stored refresh token at ~/.owi/.tokens/refresh_token
URL-based login fallback (interactive or prompted)

The resulting LexisSession is wrapped as OWILIXSession (core/manager/session.py), which adds:

Lazy-loading of the iRODS wrapper (py4lexis.core.lexis_irods.iRODS)
get_irods_client() — returns IrodsHttpClient after proactively checking the token

Token Lifetime Risk #

Token refresh is triggered by calling a py4lexis internal method. Long-running operations (large downloads, queries over many files) can outlive the token's validity. There is currently no proactive periodic refresh during operations — see docs/backlog.md for the tracking item.

Environment Assumptions #

For development and testing:

Authentication: User is logged in with a valid refresh token or PY4LEXIS_TOKEN set
Network: Remote datacenters may be slow or unavailable; use --limit for testing
Data size: Full datasets can be large; --files "['**/*.parquet']" to restrict file types

Server Status #

The LEXIS/OWS server-side is currently in active development and outside the team's control.

This has direct implications for client development:

Server APIs may change, return unexpected status codes, or behave inconsistently across instances
Error responses may be malformed or lack standard HTTP semantics (e.g., 200 with an error body)
Timeouts and transient failures should be treated as normal operating conditions, not bugs in the client
The client must be defensive: validate all server responses, never assume a 2xx means success, and surface meaningful errors to the user even when the server provides none
Do not implement workarounds that depend on undocumented server behaviour; prefer explicit error handling and user-facing messages instead

Key Dependencies #

Package	Purpose
`py4lexis >= 5.0.1`	LEXIS datacenter API and session management
`irods_http_client`	Low-level iRODS HTTP API client
`fsspec`	Filesystem abstraction layer
`duckdb`	SQL queries on parquet files
`pydantic`	Configuration validation
`typer`	CLI framework (replaced legacy Click interface)
`rich`	Terminal output, tables, progress bars