# Query Slice Command

The `query slice` command is a powerful tool for creating new datasets (subsets) from existing local or remote datasets based on SQL queries. It leverages DuckDB to efficiently process parquet files and generate new datasets adhering to the OWI metadata schema.

## Usage

```bash
owi query slice [OPTIONS]
```

### Key Arguments

- **Target Source**:
    - `-L, --local SPECIFIER`: Slice from a local dataset.
    - `-R, --remote SPECIFIER`: Slice from a remote dataset (requires files to be accessible/pulled or creates empty structure for metadata slicing). *Note: Slicing usually operates on downloaded files.*

- **Filtering & Selection**:
    - `--where TEXT`: SQL WHERE clause to filter records (e.g., `url_suffix='de'`).
    - `--select TEXT`: SQL SELECT clause to choose columns (default: `*`).

- **Output Dataset**:
    - `--collection NAME`: Name of the collection for the new dataset.
    - `--access LEVEL`: Access level for the new dataset (default: `public`). Options: `public`, `project`, `user`.
    - `--id UUID`: Target an **existing dataset** by ID. If specified, the slice result is appended/merged into this dataset.

- **Execution Control**:
    - `--yes`: Skip confirmation prompts.
    - `--format FORMAT`: Output format (`table`, `json`, `jsonl`).

## Workflows

### 1. Creating a New Slice

To create a new dataset containing only German URLs from a source dataset:

```bash
owi query slice -L "all:latest" --where "url_suffix='de'" --collection "german_subset" --yes
```

This will:
1.  Discover files in the source dataset.
2.  Execute the DuckDB query `SELECT * FROM source WHERE url_suffix='de'`.
3.  Write the result to a new dataset under `~/.owi/public/german_subset/<NEW_ID>`.
4.  Infer metadata (file counts, size) and save it.

### 2. Updating (Appending to) a Slice

To add French URLs to the **same** dataset created above, use the `--id` flag:

```bash
# First, find the ID of the dataset created in step 1
owi local ls all/collectionName=german_subset

# Then slice and append
owi query slice -L "all:latest" --where "url_suffix='fr'" --collection "german_subset" --id <EXISTING_ID> --yes
```

### 3. Automated Slicing with JSON Output

For scripts and automation pipelines, use `--format json`. This outputs a structured JSON array containing details of the created/updated dataset, while suppressing progress bars.

```bash
owi query slice -L "all:latest" ... --format json
```

**Output Example:**

```json
[
  {
    "dataset": {
      "id": "15c682b7-7615-470e-8669-02fc667e3a62",
      "path": "/Users/user/.owi/public/german_subset/15c682b7-...",
      "collectionName": "german_subset",
      "access": "public",
      "title": "OWI-Open Web Index-german_subset...",
      "files": 150,
      "size": 104857600
    }
  }
]
```

## Technical Details

### Access Levels
The `access` parameter determines where the dataset is stored locally:
- `public`: `~/.owi/public/<collection>/`
- `project`: `~/.owi/project/<collection>/`
- `user`: `~/.owi/user/<collection>/`

*Fix Note*: Previous issues with `access` being ignored (defaulting to `None`) have been resolved.

### Data Provenance
The sliced dataset includes provenance information in its metadata, recording the source dataset ID, the query used (`select`, `where`), and the timestamp of the slice operation.

### CIFF Files
If the `--ciff` flag is enabled (default), the command also slices associated CIFF (Common Index File Format) index files. For every partition directory of the sliced dataset, each matching source `index.ciff.gz` is filtered down to the documents that survived the slice (matched by the parquet `id` column against the index `collection_docid`), and the non-empty results are merged into a single `index.ciff.gz` for that partition.

Notes:
- If none of the sliced document ids appear in the source indexes, a well-formed but empty index is written and a warning is printed (`sliced index ... contains 0 documents`). An empty index usually means the source index uses a different document identifier scheme than the parquet `id` column.
- If a partition has no matching source index, a warning is printed and no index is written for that partition.
- Requires `ciff-toolkit >= 0.2.2`.
- Use `--threads N` to control the parallelism of the index filtering.