owi-cli
Command line interface for accessing the Open Web Index shards and slices.
Star
Fork
Watch
owi-cli docs epics.md
6.5 kB · 157 lines
Markdown
at main

Owilix Epics #

Epic and backlog tracking for owilix development.

For AI Agents: This is your starting point. Check the In Progress section for current work, or the Backlog for planned work. Key resources:

  • docs/AI_DEVELOPMENT.md - AI development guide
  • docs/testing.md - Testing guide for AI agents
  • .agent/workflows/ - Workflows (invoke with /development, /testing, etc.)
  • docs/architecture.md - Project structure
  • docs/CODING_GUIDELINES.md - Coding standards
  • docs/branch/ - Per-branch documentation

Status Legend #

Status Meaning
🟢 Done Epic completed and merged
🟡 In Progress Epic has active branch
⚪ Planned Epic in backlog

Backlog #

Planned epics not yet started:

  • Workflow Execution & Statistics: Run workflows and report execution statistics through owilix — see the epic below and docs/backlog.md F3
  • Mirror Operability (M1–M3): Open items from the 2026-08 review campaign — a read failure must not become an empty success (live bug), partial listings must be able to say so, and datasets must be verifiable rather than merely listed. See docs/backlog.md.
  • Query Federation: Add support for federated queries across datacenters
  • Tag-based Specifiers: Allow tag-based query federation
  • Async Query Optimization: True async I/O implementation for query commands to improve performance (currently uses thread pools)
  • Dataset Cleaning: Tools for cleaning/validating datasets
  • Overall Clean-up: Code restructuring and cleanup
  • Security Hardening: Defer and track hardening tasks captured in docs/hardening.md

⚪ Workflow Execution & Statistics #

  • Branch: feature/workflows (not started)
  • Description: Drive workflow execution and report workflow statistics through owilix.
  • Detail: docs/backlog.md § Features → F3
  • Start here — this is not greenfield:
    • Execution partly exists: owi batch run <specifier> <taskfile.yaml> (owilix/cli/batch.py → compat.batch.BatchCommands), with workers, keep-going, resume and dry-run. Not exposed over HTTP.
    • Statistics exist but are stranded: owilix/compat/workflows.py is a 963-line Airflow 2.8 monitor (dags, dag_runs, summary) that is unregistered in the modern CLI, absent from the HTTP API, and untested. owi workflow is not a command.
  • Cycles (provisional — the first one decides the rest):
    • Cycle 1: Decide the scope of "workflow" (Airflow DAGs, owilix batch run task files, or both), and whether to port or delete compat/workflows.py
    • Cycle 2: Expose execution — batch run over HTTP, which needs durable jobs (F2's first item; jobs are in-process and lost on restart)
    • Cycle 3: Statistics surface, reusing what JobManager already records (operation, status, the three timestamps, logs)
  • Blocked-by, and worth taking seriously: M1–M3 in docs/backlog.md. Statistics computed over listings that can be silently empty, silently partial, or count a sidecar as a materialised dataset will be confidently wrong in exactly the way the 2026-08 campaign spent three releases fixing. M1 is a live bug and should land first.

⚪ Deployment & Release Prep #

  • Description: Prepare version 4.0.0 release and deployment infrastructure
  • Tasks:

In Progress #

(No active epics)

Completed #

� DuckDB Query Integration #

  • Branch: py4lexis4-duckdb (merged to py4lexis4)
  • Started: 2026-01-02
  • Completed: 2026-01-05
  • Description: Integrate DuckDB querying with new fsspec backend for direct iRODS queries
  • Details: docs/branch/py4lexis4-duckdb.md
  • Deliverables:
    • owilix/core/db/ package with OWIDuckDBSelectExecutor, OWIDuckDBAsyncExecutor
    • Query commands with Http2IrodsFileSystem integration
    • --async and --prefetch options for all query commands
    • Automatic session token refresh via callable

🟢 Py4Lexis 5.x Migration #

  • Branch: py4lexis4
  • Started: 2026-01-01
  • Completed: 2026-01-02
  • Description: Migrate to py4lexis 5.x with HTTP iRODS client
  • Details: docs/branch/py4lexis4.md
  • Cycles:
    • Cycle 1: iRODS HTTP fsspec implementation (Http2IrodsFileSystem)
    • Cycle 2: Session management (OWILIXSession wrapper)
    • Cycle 3: Repository package refactor (owilix/core/repository/)
    • Cycle 4: Configuration management (V3 migration, config commands)
    • Cycle 5: UI improvements (short/wide/markdown display formats)
  • Deliverables:
    • owilix/core/fsspec/http2irods.py - HTTP-based iRODS filesystem
    • owilix/core/repository/ - Modular repository package
    • owilix/core/manager/session.py - Unified session wrapper
    • Config V3 migration with owi config migrate
    • New display formats: owi remote ls --format=short|wide|markdown
    • docs/source/config.md - Configuration documentation

🟢 Documentation Alignment #

  • Branch: main (direct)
  • Completed: 2026-01-01
  • Description: Aligned documentation structure - added contributing section to Sphinx with symlinks to dev docs
  • Deliverables:
    • docs/source/contributing/ section
    • AI agent onboarding in docs/epics.md
    • Branch documentation template

🟢 AI Infrastructure Setup #

  • Branch: main (direct)
  • Completed: 2026-01-01
  • Description: Set up AI coding infrastructure for development with Antigravity
  • Deliverables:
    • .agent/workflows/ directory
    • Documentation structure
    • Coding guidelines

🟢 Web-graph Support #

  • Branch: (integrated)
  • Completed: Prior to v0.23.0
  • Description: Added graph analysis and reporting features

🟢 Docker Install #

  • Branch: (integrated)
  • Completed: Prior to v0.8.0
  • Description: Docker containerization support

Epic Template #

When creating a new epic, use this template:

### ⚪ [Epic Name]

- **Branch**: feature/<epic-name>
- **Started**: YYYY-MM-DD
- **Description**: Brief description
- **Cycles**:
  - Cycle 1: Goal
  - Cycle 2: Goal