diff --git a/.gitignore b/.gitignore index bd295eb..8bfbf1a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ data/*.parquet __pycache__/ *.pyc +*.egg-info/ .venv/ venv/ data/dispensing_partitioned/ diff --git a/README.md b/README.md index 5cb8f51..07892c3 100644 --- a/README.md +++ b/README.md @@ -79,19 +79,28 @@ python3 -m venv .venv ### Step 4: Activate the virtual environment +On macOS and Linux: + ```bash source .venv/bin/activate ``` +On Windows (Git Bash), the venv puts its scripts in `Scripts/` instead of +`bin/`: + +```bash +source .venv/Scripts/activate +``` + You should now see `(.venv)` at the start of your terminal prompt. ### Step 5: Install dependencies ```bash -pip install -r requirements.txt +pip install . ``` -This installs a single package: `polars==1.42.1`. +This reads `pyproject.toml` and installs a single package: `polars==1.42.1`. ### Step 6: Generate the data @@ -142,7 +151,7 @@ If you see `SSL: CERTIFICATE_VERIFY_FAILED` due to a corporate firewall that intercepts HTTPS traffic: ```bash -pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org -r requirements.txt +pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org . ``` #### Wrong interpreter selected in VS Code @@ -323,7 +332,7 @@ perform a lazy aggregation across the partitioned dataset. ``` parquet-polars-training/ ├── README.md ← you are here -├── requirements.txt ← polars==1.42.1 +├── pyproject.toml ← project metadata + dependencies (polars==1.42.1) ├── .gitignore ├── data/ │ ├── generate_data.py ← run this to create the Parquet files diff --git a/exercises/ex01_vscode_explorer.md b/exercises/ex01_vscode_explorer.md index 1c1dddf..5ff6433 100644 --- a/exercises/ex01_vscode_explorer.md +++ b/exercises/ex01_vscode_explorer.md @@ -2,9 +2,9 @@ > **SAS framing:** This is the replacement for double-clicking a `.sas7bdat` file > in SAS Enterprise Guide — look first, code second. Before writing any Python, -> you should be able to open a data file and visually inspect its contents. You -> can also run SQL queries against Parquet files directly in VS Code, just like -> `PROC SQL` against a SAS dataset. +> you should be able to open a data file and visually inspect its contents. +> Data Wrangler gives you a point-and-click data grid with column statistics, +> much like the SAS viewtable window combined with `PROC FREQ` output. --- @@ -25,78 +25,77 @@ In VS Code, open the Extensions panel (Ctrl+Shift+X), then search for and instal After generating data (Exercise 0), you should have three files in `data/`: -1. Open `data/demographic.parquet` — click it in the Explorer sidebar. +1. Open `data/demographic.parquet` — right-click it in the Explorer sidebar + and choose **Open in Data Wrangler**. 2. Open `data/enrollment.parquet` 3. Open `data/dispensing.parquet` If the extension is working, you will see a data grid (not raw bytes). Take a moment to scroll through each file and get a feel for the columns and values. -## Step 3: Query Parquet files with SQL - -Both Data Wrangler and Data Preview let you run SQL queries against opened -Parquet files. This is the same concept as `PROC SQL` against a SAS dataset — -the table name is the file name (without the `.parquet` extension). - -Here is a worked example. Open `demographic.parquet`, then run this query: - -```sql --- PROC SQL equivalent: SELECT COUNT(*) AS n FROM demographic; -SELECT COUNT(*) AS row_count -FROM demographic; -``` - -You should get a result like `10000` — one row per patient. - -Try another one. This query counts how many patients are missing a -`PostalCode` value (the `IS NULL` check works just like in PROC SQL): - -```sql --- PROC SQL equivalent: SELECT COUNT(*) FROM demographic WHERE PostalCode IS NULL; -SELECT COUNT(*) AS missing_postal -FROM demographic -WHERE PostalCode IS NULL; -``` - -> **Tip:** If you are not sure where to type the query, look for a "Query" or -> "SQL" button in the Data Wrangler panel, or open a new SQL file -> (`File → New File`) and save it with a `.sql` extension, then select the -> Parquet file as the connection target. +## Step 3: Explore with column statistics + +Data Wrangler is a **low-code/no-code tool** — there is no SQL query panel. +Instead, everything you would ask with a small `PROC SQL` or `PROC FREQ` step +is answered by clicking around the interface: + +- **Row count** — shown in the summary bar (e.g., "10,000 rows"). This is your + `SELECT COUNT(*)`. +- **Column dtype** — shown in each column header next to the column name. This + is your `PROC CONTENTS`. +- **Column statistics panel** — click a column header to open a summary of that + column: value distribution, unique values, and missing-value count. This is + your `PROC FREQ` / `PROC MEANS`. +- **Filter and sort** — use the column header menus to filter rows or sort, + the way you would with a `WHERE` clause or `PROC SORT`. + +Here is a worked example. Open `demographic.parquet` in Data Wrangler and look +at the summary bar — you should see **10,000 rows**, one row per patient. Now +click the `PostalCode` column header: the statistics panel shows the count of +missing values, which answers "how many patients have no postal code?" without +writing a line of code. + +> **Note:** Data Wrangler can also generate Python (pandas) code for the +> operations you perform in the grid. Feel free to peek at it, but do not copy +> it into the exercises — this workshop uses Polars, and the two APIs do not +> mix. Treat Data Wrangler as a viewer here, not a code generator. ## Step 4: Answer the following questions -Use SQL queries (and the data grid view) to answer each question. Write your -answers below each question or in a separate document. +Use the data grid and column statistics panel to answer each question. Write +your answers below each question or in a separate document. ### Q1: How many rows are in `dispensing.parquet`? -Hint: `SELECT COUNT(*) FROM dispensing;` +Hint: Open the file in Data Wrangler and read the row count from the summary +bar. Your answer: ### Q2: What is the dtype of the `RxDate` column? Hint: Look at the column header in the data grid — the dtype is shown next to -the column name. You can also query the schema if the extension supports it. +the column name. Your answer: ### Q3: Which columns are in `enrollment.parquet`? -Hint: `SELECT * FROM enrollment LIMIT 5;` — then look at the column headers -in the result. +Hint: Open the file and read the column headers across the top of the grid. Your answer: ### Q4: What are the unique values of `Sex` in `demographic.parquet`? -Hint: `SELECT DISTINCT Sex FROM demographic;` +Hint: Click the `Sex` column header — the statistics panel lists the unique +values and their counts (like `PROC FREQ`). Your answer: ### Q5: What is the null count for `PostalCode` in `demographic.parquet`? -Hint: `SELECT COUNT(*) FROM demographic WHERE PostalCode IS NULL;` +Hint: Click the `PostalCode` column header and read the missing-value count +from the statistics panel. Your answer: @@ -105,7 +104,7 @@ Your answer: ## Why this matters In SAS, you would run `PROC CONTENTS` to see column metadata, `PROC PRINT` to -eyeball the data, and `PROC SQL` to query specific values. The VS Code -Parquet viewer gives you all three capabilities without writing a DATA step or -PROC. Use it to confirm your expectations before you start writing Polars -queries in the exercises that follow. +eyeball the data, and `PROC FREQ` to check value distributions. Data Wrangler +gives you all three capabilities without writing a DATA step or PROC. Use it to +confirm your expectations before you start writing Polars queries in the +exercises that follow. diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..5ea003f --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,15 @@ +[build-system] +requires = ["setuptools>=68"] +build-backend = "setuptools.build_meta" + +[project] +name = "parquet-polars-training" +version = "0.1.0" +description = "Parquet + Polars workshop for SAS analysts" +requires-python = ">=3.11" +dependencies = [ + "polars==1.42.1", +] + +[tool.setuptools] +packages = [] diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index 02463aa..0000000 --- a/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -polars==1.42.1