tide-python-training
A training to familiarize TIDE analysts with parquet datasets and python
Star
Fork
Watch
tide-python-training CLAUDE.md
3.2 kB · 38 lines
Markdown
at main

Parquet + Polars Workshop #

Last verified: 2026-07-20

Teaching repo, not a software product. It teaches SAS-background analysts to inspect Parquet in VS Code and manipulate it with Polars. Audience-facing docs live in README.md; this file captures the invariants an editor must preserve.

Tech Stack #

  • Python 3.11+, single runtime dependency: polars==1.42.1 (pinned in pyproject.toml)
  • No test runner — every exercise/solution self-grades via assert
  • Synthetic data modelled on Sentinel Common Data Model (SCDM) v8.2.2

Commands #

  • pip install . — installs polars from pyproject.toml
  • python data/generate_data.py — regenerates the three gitignored Parquet files
  • python scripts/verify_scdm.py — validates generated data against the SCDM spec
  • python exercises/exNN_*.py / solutions/exNN_*.py — run a unit; success prints only All checks passed.

Structure (the teaching loop) #

Each unit is: lesson → demo → exercise → self-check → solution, numbered 02–06.

  • lessons/ — markdown, teaches each concept before it is used. 00 (Python survival kit) and 01 (Parquet in VS Code, Data Wrangler, no runnable script) have no demo/exercise/solution counterpart.
  • demos/ — runnable scripts (demo02–demo06) the lessons walk through
  • exercises/ — learner fills blanks; solutions/ — reference, must pass all assertions
  • data/generate_data.py produces the Parquet; data/*.parquet and output dirs are gitignored

Invariants (do not break when editing) #

  • Numbering is 1:1: lessons/NN ↔ demos/demoNN ↔ exercises/exNN ↔ solutions/exNN for NN in 02–06. There is no ex01 script — its content lives in lessons/01_parquet_in_vscode.md.
  • Assertion constants are data-derived (e.g. 15_698 filtered rows, dataset row counts). Changing a filter, the data generator, or an assertion in isolation desyncs the trio. Re-derive against freshly generated data and update exercise + solution together.
  • Blank convention: exercises use ____ (four underscores) for fill-in blanks in units 2–3. Units 4–6 use raise NotImplementedError exclusively (pattern hints in 4, tiered <details> hints in lesson 5's Your turn, task-only capstone in 6). Outcome signals (NotImplementedError / AssertionError / NameError/AttributeError on ____) are documented in lessons/00 and README "How to run the scripts" — keep those three in sync.
  • De-scaffolding progression is intentional: blanks → tiered hints → minimal capstone. Preserve it; don't re-scaffold unit 6 or strip hints from 2–3.
  • Scripts carry a # pattern: Imperative Shell header and a "Before you start / How to run" docstring block that points at the paired lesson and README rather than repeating platform commands. Keep new scripts consistent.
  • Polars only — never introduce pandas idioms. No SQL targeting Data Wrangler: it is low/no-code, SQL examples do not apply.

SCDM data (fabricated, no real records) #

Three tables keyed by PatID: demographic (1 row/patient), enrollment (1 row/span; DrugCov flags drug coverage), dispensing (1 row/fill). Unit 5(b) is a real Sentinel enrollment-coverage QC pattern (anti-join).