Testing Guide #

Comprehensive testing documentation for OWILIX developers and AI agents.

For AI Agents: This is your testing reference. Use the Quick Verification section for fast checks, and CLI Smoke Tests for subprocess-based testing.

Quick Verification (Recommended for AI) #

Before committing changes, run these quick checks:

# turbo
# 1. Check imports work
uv run python -c "from owilix.core.manager import OWIlixManager; print('✅ Imports OK')"

# turbo
# 2. Run fast CLI smoke tests (~8s)
uv run pytest tests/owilix/cli/test_smoke.py -v -m "not integration"

# turbo
# 3. Run AI verification suite (structured output)
uv run python tests/owilix/cli/test_ai_verifiable.py

Latest Local Runs #

Latest observed local runs in this repository:

• uv run pytest -q Result on 2026-04-11: 471 passed, 2 skipped, 42 deselected, 2 warnings
• uv run pytest tests/owilix/cli/test_remote_search_cli.py tests/owilix/core/tasks/test_search.py -q Result on 2026-04-10: 24 passed
• uv run pytest tests/owilix/cli/test_smoke.py -q -m integration -k remote_search Result on 2026-04-10: passed in the current environment
• uv run pytest tests/owilix/core/test_sql_logging.py tests/owilix/core/db/test_executors.py -q Result on 2026-04-11: 22 passed, 2 deselected
• uv run pytest tests/owilix/cli/test_smoke.py -q -k 'admin_logs_location or admin_logs_sql' -o addopts='' Result on 2026-04-11: 2 passed

The remaining warning noise is the documented Rich/Jupyter cosmetic warning from tests/owilix/core/tasks/test_search.py.

SQL Debug Logging #

To capture executed SQL statements during CLI runs, enable the global SQL log:

uv run owi --sql-log query less -R lexis:latest --limit 10

The statements are written to:

~/.owi/.logs/sql.log

This log is opt-in and intended for debugging query construction and execution.

To inspect the logs from the CLI:

uv run owi admin logs location
uv run owi admin logs sql --lines 20
uv run owi admin logs events --lines 20

Test Organization #

tests/
├── owilix/
│   ├── cli/                    # CLI smoke tests (fast, subprocess-based)
│   │   ├── test_smoke.py
│   │   └── test_ai_verifiable.py  # AI-interpretable JSON output
│   └── core/
│       ├── db/                 # DuckDB executor & pool tests
│       │   ├── test_executors.py      # SQL query, pool, retry logic
│       │   ├── test_duckdb.py
│       │   └── test_copy_executor.py
│       ├── repository/         # Repository tests
│       │   ├── test_aggregate.py      # Multi-datacenter aggregation
│       │   ├── test_integration.py    # CLI integration (slow, network)
│       │   └── test_repository_legacy_ported.py
│       ├── tasks/              # Task function tests
│       │   ├── test_search.py         # Full-text search (SQL builder, remote)
│       │   └── test_query_unit.py     # query_less / query_sites / query_analyze
│       ├── fsspec/             # fsspec tests
│       │   ├── test_core_fsspec_unit.py
│       │   └── benchmark_async.py
│       └── test_metadata.py
└── data/                       # Test fixtures

By default, integration tests are deselected via addopts in pyproject.toml. The full unit suite runs in ~25 seconds.

Test Types & Timing #

Test Suite	Command	Duration	Network
Full unit suite	pytest tests/	~25s	No
CLI Smoke	pytest tests/owilix/cli/test_smoke.py	~8s	No
DB executor unit	pytest tests/owilix/core/db/	<1s	No
Repository Integration	pytest tests/owilix/core/repository/test_integration.py -m integration	5-10min	Yes

Expected Release Test Matrix #

For a normal client release, these are the tests that should be expected to run:

# 1. Full default gate
uv run pytest -q

# 2. Search/UI focused gate
uv run pytest tests/owilix/cli/test_remote_search_cli.py tests/owilix/core/tasks/test_search.py -q

# 3. Fast CLI smoke gate
uv run pytest tests/owilix/cli/test_smoke.py -q -m "not integration"

# 4. Selected network smoke checks when credentials/network are available
uv run pytest tests/owilix/cli/test_smoke.py -q -m integration -k "remote_doctor or remote_ls_latest or remote_search"

# 5. Broader integration pass before publishing when time permits
uv run pytest tests/ -m integration -v -s

Notes:

• query less --search integration is valid but heavier than remote search alone because it resolves collections and scans parquet after the DuckLake lookup.
• The docs build is also part of release hygiene when the docs dependency group is installed: uv sync --group docs && uv run sphinx-build -b html docs/source docs/build/html

Pytest Markers #

Registered in pyproject.toml:

# Skip network-dependent tests
uv run pytest tests/ -m "not integration"

# Run only integration tests
uv run pytest tests/ -m integration -v -s

# Skip slow tests
uv run pytest tests/ -m "not slow"

CLI Smoke Tests #

Fast, subprocess-based tests that run actual owi commands:

# tests/owilix/cli/test_smoke.py

def run_owi(*args, timeout=30):
    """Run owi command via subprocess."""
    result = subprocess.run(
        ["uv", "run", "owi"] + list(args),
        capture_output=True,
        text=True,
        timeout=timeout
    )
    return result.returncode, result.stdout, result.stderr

# Example test
def test_config_version(self):
    code, out, err = run_owi("config", "version")
    assert code == 0
    assert "Config version:" in out

AI-Verifiable Tests #

Tests with structured JSON output for AI parsing:

# Run verification suite
uv run python tests/owilix/cli/test_ai_verifiable.py

# Example output:
# ✅ owi --help (1.33s)
# ✅ owi config version (1.29s)
# --- SUMMARY ---
# {"total": 3, "passed": 3, "failed": 0, "all_passed": true}

The output includes --- AI RESULT --- blocks with structured JSON:

{
  "command": "owi config version",
  "exit_code": 0,
  "success": true,
  "elapsed_seconds": 1.29,
  "stdout_preview": "Config version: 3.0...",
  "contains": {
    "error": false,
    "traceback": false,
    "success_icon": true
  }
}

Integration Tests (Network Required) #

These tests invoke full CLI commands with network access:

# Run with output visible (-s flag important for progress)
uv run pytest tests/owilix/core/repository/test_integration.py -v -s

# Expected duration: 5-10 minutes
# Tests include: remote ls, remote pull, local ls

Progress indicators show timing during slow operations:

[0.0s] Starting remote ls test...
[5.2s] CLI completed with exit code 0
[5.2s] ✅ Remote ls test passed

Known Issues #

Import Errors #

• test_graph_commands.py requires owilix.core.ui (not yet implemented)
• Skip with: --ignore=tests/owilix/core/test_graph_commands.py

Mock Failures in fsspec #

2 tests fail due to mock subscripting issues:

• test_ls_returns_paths
• test_ls_with_detail

These are test issues, not code issues.

Adding New Tests #

Unit Tests (Fast) #

def test_my_feature():
    """Unit test - fast, no network."""
    result = my_function()
    assert result == expected

Integration Tests (Network) #

import pytest

@pytest.mark.integration
def test_remote_operation(capsys):
    """Integration test - requires network."""
    print("[0.0s] Starting test...")  # Progress with -s flag
    # ... test code ...
    print("[5.0s] ✅ Test passed")

Recommended Workflow #

1. Before each commit: Run smoke tests

   uv run pytest tests/owilix/cli/test_smoke.py -v -m "not integration"
   

2. Before merging: Run full test suite

   uv run pytest tests/ -v --ignore=tests/owilix/core/test_graph_commands.py
   

   For release work, also run the selected network smoke commands listed in the release matrix above.

3. For AI agents: Use the verification script

   uv run python tests/owilix/cli/test_ai_verifiable.py
   

Test Documentation #

Each test package has comprehensive docstrings in __init__.py files:

• tests/owilix/core/__init__.py - Overview of core tests
• tests/owilix/core/db/__init__.py - DuckDB executor tests and benchmarks
• tests/owilix/core/fsspec/__init__.py - Filesystem tests
• tests/owilix/core/repository/__init__.py - Repository tests

For detailed benchmarks and results, see docs/source/testing_and_benchmarks.md. For phased integration execution and optimization guidance, see docs/integration_test_plan.md. For current warning triage and tolerated cosmetic noise, see docs/test-status.md.

Database Executor Tests #

# turbo
# Run executor unit tests
uv run pytest tests/owilix/core/db/test_executors.py -v

# turbo  
# Run comprehensive benchmarks (sync vs async)
uv run python tests/owilix/core/db/benchmark_comprehensive.py