A training to familiarize TIDE analysts with parquet datasets and python

README.md

Parquet + Polars Workshop #

A hands-on workshop teaching SAS-background analysts to inspect Parquet files in VS Code and read/manipulate Parquet data with Polars in Python. By the end of this workshop, you will be able to replace common SAS data-step and PROC SQL patterns with concise Polars equivalents, and understand why Parquet is the modern columnar format of choice for analytics.

This repository serves as both the synchronous workshop materials and a standalone post-workshop reference. Every exercise is self-contained with built-in assertions that verify your solution.

Target audience: SAS analysts who are new to Python and want to work with Parquet files using Polars — a fast, type-safe DataFrame library that replaces pandas for analytics work.

Prerequisites #

Python 3.11+ — verify with python3 --version. On Windows with Git Bash, if python3 is not found, use python or the full path to your installation. Make sure Python is on your PATH during installation (check "Add to PATH").
VS Code — install from https://code.visualstudio.com/
Git Bash (Windows) — installed with Git for Windows. macOS and Linux users already have a terminal with bash.

Data Description #

This workshop uses synthetic data modeled on the Sentinel Common Data Model (SCDM) v8.2.2 — the same data standard used by the FDA Sentinel system for distributed query analytics. The data is entirely fabricated; no real patient records are included.

The three tables are:

Table	File	Row Grain	Key Columns
Demographic	`demographic.parquet`	One row per patient	`PatID`, `Birth_Date`, `Sex`, `Race`
Enrollment	`enrollment.parquet`	One row per enrollment span	`PatID`, `Enr_Start`, `Enr_End`, `DrugCov`
Dispensing	`dispensing.parquet`	One row per pharmacy dispensing	`PatID`, `RxDate`, `Rx`, `RxAmt`

PatID uniquely identifies a patient across all three tables. Enr_Start and Enr_End define the coverage window for each enrollment span. RxDate is the date a prescription was filled. DrugCov indicates whether drug (pharmacy) coverage was active during the enrollment span — a key field for coverage QC checks.

The SCDM specification is publicly available at the Sentinel CDM repository.

Exercise 0 — Environment Setup (~30 min) #

Step 1: Clone the repo #

git clone https://github.com/your-org/parquet-polars-training.git
cd parquet-polars-training

Step 2: Verify Python version #

python3 --version

You should see Python 3.11.x or higher.

Step 3: Create a virtual environment #

python3 -m venv .venv

Step 4: Activate the virtual environment #

On macOS and Linux:

source .venv/bin/activate

On Windows (Git Bash), the venv puts its scripts in Scripts/ instead of bin/:

source .venv/Scripts/activate

You should now see (.venv) at the start of your terminal prompt.

Step 5: Install dependencies #

pip install .

This reads pyproject.toml and installs a single package: polars==1.42.1.

Step 6: Generate the data #

python data/generate_data.py

You should see output like:

Data generation complete.
  demographic.parquet :   10,000 rows
  enrollment.parquet  :   14,221 rows
  dispensing.parquet  :  199,613 rows

Step 7: Verify the Parquet files exist #

ls -la data/*.parquet

You should see three files: demographic.parquet, enrollment.parquet, and dispensing.parquet.

New to Python? Before Exercise 1, work through lessons/00_python_survival_kit.md.

Troubleshooting #

`python3` not on PATH (Windows Git Bash) #

On Windows, if python3 is not found in Git Bash, try python instead. If neither works, Python may not be on your PATH. Reinstall from python.org and check "Add to PATH" during installation, then restart Git Bash.

pip behind a proxy #

If you are behind a corporate proxy, configure pip:

pip config set global.proxy <your-org-proxy-settings>

pip SSL certificate verification error #

If you see SSL: CERTIFICATE_VERIFY_FAILED due to a corporate firewall that intercepts HTTPS traffic:

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org .

Wrong interpreter selected in VS Code #

If VS Code shows a different Python interpreter (e.g., a system Python instead of your venv):

Press Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS)
Type and select Python: Select Interpreter
Choose the interpreter that shows .venv in its path

Course units #

New to Python? Start with Lesson 00: Python Survival Kit (~30 min) before diving into the units below.

Each unit follows the same loop: read the lesson → run the demo → do the exercise → check yourself → compare with the solution

Unit 1: Parquet in VS Code (~20 min) #

Before writing any Python, learn to visually inspect Parquet files in VS Code — the replacement for double-clicking a .sas7bdat file. Start with Lesson 01: Parquet in VS Code, then work through the inspection questions using the Data Wrangler extension.

Unit 2: Read and Inspect (~30 min) #

Start with Lesson 02: Read and Inspect, run demos/demo02.py, then complete exercises/ex02_read_and_inspect.py. Read a Parquet file into a Polars DataFrame and inspect it — the equivalent of PROC CONTENTS (schema), PROC PRINT (head), and PROC MEANS (describe) in a few lines of Python.

Unit 3: Filter and Select (~35 min) #

Start with Lesson 03: Filter and Select, run demos/demo03.py, then complete exercises/ex03_filter_select.py. Filter rows and select columns using the Polars expression model, learning pl.col(), expression chaining, and logical operators — the equivalents of SAS WHERE, KEEP, DROP, and PROC SORT.

Unit 4: Group By and Aggregate (~35 min) #

Start with Lesson 04: Group By and Aggregate, run demos/demo04.py, then complete exercises/ex04_groupby_agg.py. Group data and compute summary statistics, learning df.group_by().agg() — the equivalent of PROC SUMMARY / PROC MEANS with a CLASS statement or PROC SQL GROUP BY.

Unit 5: Joins (~40 min) #

Start with Lesson 05: Joins, run demos/demo05.py, then complete exercises/ex05_joins.py. Join multiple tables and check data coverage. Part (a) is a straightforward inner join (like a MERGE with IN= flags). Part (b) is a miniature Sentinel enrollment-coverage QC check: which dispensings have no matching drug-covered enrollment span? This mirrors a real Sentinel analytic pattern.

Unit 6: Write and Partition (~35 min) #

Start with Lesson 06: Write, Partition, Lazy, run demos/demo06.py, then complete exercises/ex06_write_and_partition.py. Write filtered data to new Parquet files and partition datasets by column value (e.g., year). This introduces lazy evaluation via scan_parquet.

How this course works #

This course teaches by doing. Each unit follows the same loop:

read the lesson → run the demo → do the exercise → check yourself → compare with the solution

Here's what each part does:

lessons/ — Teach every concept before you need it. Start with lessons/00_python_survival_kit.md if you're new to Python. Each lesson is a guided walkthrough with examples and a check-yourself section.
demos/ — Runnable Python scripts showing the lesson concepts in action. The lessons walk through these, so you'll understand how each piece works.
exercises/ — Your turn. Each exercise file has TODO blanks to fill in. Run the script — if you've done it right, you'll see All checks passed. and nothing else. If something is wrong, the script tells you what went wrong.
solutions/ — The complete, correct version. After you've made your best attempt, compare your solution with the reference solution.

Every exercise grades itself with assertions. You don't need an external test runner or a human grader — the script is the feedback mechanism.

How to run the scripts #

All scripts run from the repo root with the venv activated.

Activate the venv #

On macOS and Linux:

source .venv/bin/activate

On Windows (Git Bash):

source .venv/Scripts/activate

You should see (.venv) at the start of your terminal prompt.

Run any script #

python demos/demo02.py
python exercises/ex02_read_and_inspect.py
python solutions/ex02_read_and_inspect.py

Understand the output #

Every script prints one of these outcomes:

Outcome	Meaning	What to do next
`All checks passed.`	Your solution is correct.	Move to the next exercise.
`NotImplementedError: TODO ...`	You haven't started this TODO yet.	Look at the `TODO` block in the file and fill in the blank. See `lessons/00_python_survival_kit.md` "How the course grades you" for details.
`AssertionError: ...`	You've filled in the TODO, but the answer is wrong.	Check the error message, review the lesson, and try again. See `lessons/00_python_survival_kit.md` "How the course grades you" for details.
`NameError: name '____' is not defined` or `AttributeError: ... has no attribute '____'`	You missed a blank (the four underscores `____` are still in the file).	Find and fill in every `____` in the file. See `lessons/00_python_survival_kit.md` "How the course grades you" for a traceback walkthrough.

Note: Demo and exercise files often link here instead of repeating platform-specific commands.

SAS-to-Polars Mapping #

SAS Idiom	Polars Equivalent
`PROC CONTENTS`	`df.schema`, `df.columns`
`WHERE` (DATA/PROC)	`df.filter()`
`KEEP` / `DROP`	`df.select()`, `df.drop()`
`PROC SORT`	`df.sort()`
`PROC SUMMARY` / `PROC MEANS`	`df.group_by().agg()`
`MERGE` (with `IN=` flags)	`df.join(how="inner"/"left"/"anti")`
`PROC SQL SELECT`	`df.select()`, `df.filter()`
`DATA` step column creation	`df.with_columns()`

Each lesson includes a scoped, parameter-level version of this mapping for quick reference during the coding sections.

The Polars Expression Model #

The expression model — pl.col(), combining conditions with &/|, and why expressions beat WHERE strings — is taught in Lesson 03: Filter and Select.

Partitioning #

Hive-style partitioning — what RxYear=2022/ directories are and why they make big tables fast — is taught in Lesson 06: Write, Partition, Lazy.

A Note on Pandas #

You may have heard of pandas — the most popular Python DataFrame library. This workshop uses Polars instead. Polars is faster (written in Rust, multi-threaded), has a stricter type system (catches bugs earlier), and uses a lazy evaluation model that pandas lacks. The expression-based API is also more consistent and composable than pandas' mixed string/method approach.

If you are following along with external tutorials, do not mix pandas idioms with Polars code — the APIs are different. Everything in this workshop is self-contained in Polars.

Lazy Evaluation #

Eager vs lazy execution (read_parquet vs scan_parquet + .collect()) is taught in Lesson 06: Write, Partition, Lazy.

Repository Structure #

parquet-polars-training/
├── README.md                   ← you are here
├── pyproject.toml              ← project metadata + dependencies (polars==1.42.1)
├── .gitignore
├── data/
│   ├── generate_data.py        ← run this to create the Parquet files
│   ├── demographic.parquet     ← generated (gitignored)
│   ├── enrollment.parquet      ← generated (gitignored)
│   └── dispensing.parquet      ← generated (gitignored)
├── lessons/
│   ├── 00_python_survival_kit.md
│   ├── 01_parquet_in_vscode.md
│   ├── 02_read_and_inspect.md
│   ├── 03_filter_select.md
│   ├── 04_groupby_agg.md
│   ├── 05_joins.md
│   └── 06_write_partition_lazy.md
├── demos/
│   ├── demo02.py
│   ├── demo03.py
│   ├── demo04.py
│   ├── demo05.py
│   └── demo06.py
├── exercises/
│   ├── ex02_read_and_inspect.py
│   ├── ex03_filter_select.py
│   ├── ex04_groupby_agg.py
│   ├── ex05_joins.py
│   └── ex06_write_and_partition.py
├── solutions/
│   ├── ex02_read_and_inspect.py
│   ├── ex03_filter_select.py
│   ├── ex04_groupby_agg.py
│   ├── ex05_joins.py
│   └── ex06_write_and_partition.py
├── docs/
│   └── design-plans/
│       └── 2026-07-20-pedagogy-redesign.md
└── scripts/
    └── verify_scdm.py          ← validates generated data against SCDM spec

Exercises are progressively less scaffolded: units 2–3 use blanks to fill in, units 4–5 use tiered hints for more complex implementations, and unit 6 is a capstone with minimal scaffolding. Solutions are complete and pass all assertions. For outcome signals and how to interpret error messages, see ## How to run the scripts.