Owilix Coding Guidelines #

Guidelines for development to ensure consistent, maintainable, and AI-ready code.

---

1. Code Organization #

1.1 Module Structure #

Each module should have a comprehensive docstring:

"""
Module: owilix.core.repository

Purpose:
    Unified interface for dataset management across storage backends.

Key Concepts:
    - Dataset: Identified by UUID, carries metadata
    - Repository: Storage backend (Local, iRODS, S3)
    - AggregatedRepository: Merges multiple repositories

Dependencies:
    - fsspec: Filesystem abstraction
    - irods.session: iRODS connectivity
    - owilix.core.metadata: Dataset model

Exports:
    - AbstractRepository: Base class
    - AggregatedRepository: Multi-repo aggregator
    - FileBasedRepository: Local/S3 implementation
"""

1.2 File Naming #

Type	Convention	Example
Module	snake_case.py	repository.py
Class	PascalCase	AbstractRepository
Function	snake_case	list_datasets()
Constant	UPPER_SNAKE	DEFAULT_ACCESS

1.3 Import Order #

# 1. Standard library
import os
from datetime import datetime
from typing import List, Optional

# 2. Third-party
import pandas as pd
from pydantic import BaseModel

# 3. Local (absolute)
from owilix.core.metadata import Dataset
from owilix.core.utils import check_path

---

2. Type Hints & Pydantic #

2.1 Always Use Type Hints #

# ✓ Good
def list_datasets(access: str = "public", limit: int = 100) -> list[Dataset]:
    ...

# ✗ Bad
def list_datasets(access="public", limit=100):
    ...

2.2 Pydantic for Configuration #

from pydantic import BaseModel, Field, field_validator

class RepositoryConfig(BaseModel):
    """Configuration for a repository connection."""
    
    name: str = Field(..., description="Repository identifier")
    path: str = Field(..., description="Base path with {access} placeholder")
    protocol: str = Field(default="file", description="fsspec protocol")
    
    @field_validator("path")
    @classmethod
    def path_must_have_access(cls, v: str) -> str:
        if "{access}" not in v:
            raise ValueError("Path must contain {access} placeholder")
        return v

2.3 Shared Constants & Enums #

Avoid raw strings for shared options:

from typing import Literal

# Type alias for access levels
AccessType = Literal["public", "project", "private"]

# Or use Enum for more structure
from enum import Enum

class AccessLevel(str, Enum):
    PUBLIC = "public"
    PROJECT = "project"
    PRIVATE = "private"

---

3. Documentation Standards #

3.1 Module Docstrings #

See Section 1.1 for format.

3.2 Class Docstrings #

class Dataset:
    """
    Represents a data slice with metadata and file access.
    
    A dataset is identified by UUID and contains metadata 
    following a DataCite-like schema.
    
    Attributes:
        metadata: Dictionary with id, title, dates, etc.
        repository: Reference to containing repository.
        
    Example:
        dataset = repo.create(access="public", metadata={...})
        files = dataset.files()
    """

3.3 Function Docstrings #

def pull_dataset(
    dataset: Dataset,
    local_path: str,
    files_glob: str | None = None,
) -> list[str]:
    """
    Download dataset files to local path.
    
    Args:
        dataset: The dataset to download.
        local_path: Destination directory.
        files_glob: Optional glob pattern to filter files.
        
    Returns:
        List of downloaded file paths.
        
    Raises:
        IOError: If download fails.
        
    Example:
        paths = pull_dataset(ds, "/tmp/data", "*.parquet")
    """

3.4 Inline Annotations #

Use these tags for tracking:

Tag	Purpose	Example
TODO	Work to be done	# TODO: Add retry logic
FIXME	Known bug	# FIXME: Fails on empty input
EXTENSION	Future enhancement	# EXTENSION: Support async downloads

---

4. Error Handling #

4.1 Fail Fast with Context #

# ✓ Good: Descriptive error with context
def create_repository(protocol: str, path: str):
    SUPPORTED = ["file", "s3", "irods"]
    if protocol not in SUPPORTED:
        raise ValueError(
            f"Unsupported protocol: '{protocol}'. "
            f"Supported: {SUPPORTED}"
        )
    ...

# ✗ Bad: Silent fallback
def create_repository(protocol: str, path: str):
    if protocol not in ["file", "s3"]:
        protocol = "file"  # This hides bugs!
    ...

4.2 Chain Exceptions #

try:
    result = external_api.call()
except ExternalError as e:
    raise ValueError(f"API call failed: {e}") from e  # Preserve context

---

5. Testing #

5.1 Test Naming #

def test_repository_list_returns_datasets():
    """List returns Dataset objects."""
    ...

def test_repository_list_handles_empty_response():
    """Empty response returns empty list, not None."""
    ...

5.2 Test Structure #

import pytest
from owilix.core.repository import FileBasedRepository

class TestFileBasedRepository:
    """Tests for FileBasedRepository."""
    
    @pytest.fixture
    def repo(self, tmp_path):
        """Create test repository."""
        path = str(tmp_path / "{access}/data")
        return FileBasedRepository(None, path)
    
    def test_list_public_datasets(self, repo):
        """Should list datasets in public access."""
        datasets = repo.list(access="public")
        assert isinstance(datasets, list)

    @pytest.mark.integration
    def test_remote_connection(self):
        """Requires network - marked as integration."""
        ...

---

6. Development Environment #

6.1 Assumptions #

When developing and testing:

• User is logged in: Refresh token is available
• Remote servers may be slow: Use timeouts and retries
• Data can be large: Use --limit flags in tests

6.2 Running Locally #

# Install dependencies
uv sync

# Run CLI
uv run owi --help

# Run tests
uv run pytest tests/

# Build docs
cd docs && uv run sphinx-build -b html source/ build/html

6.3 Debug Logging #

import logging
logger = logging.getLogger("owilix")

# Use debug for detailed info
logger.debug(f"Listing datasets in {path}")

# Use info for important operations
logger.info(f"Downloaded {len(files)} files")

# Use warning for recoverable issues
logger.warning(f"Retry {attempt}/{max_retries} after {e}")

---

7. Quick Reference #

AI Context Checklist #

When working on a module, verify:

1. Module docstring: Purpose, concepts, dependencies
2. Type hints: All functions typed
3. Error handling: No silent fallbacks
4. Tests: Unit + integration as needed
5. Branch docs: Updated if in a feature branch

Code Review Checklist #

• Type hints on all functions
• Docstrings with examples
• Pydantic validation on config
• Tests for happy path + edge cases
• Logging for errors and state changes
• No silent fallbacks

---

See Also #

• docs/architecture.md - Project architecture
• .agent/workflows/ - Development workflows