# data_processing Package

Data layer for NHS High-Cost Drug Patient Pathway Analysis Tool.

## Core Responsibilities

**Data Pipeline:** `Snowflake → Transforms → Pathway Generation → SQLite`

## Key Modules

**transforms.py** — Core data transformations (moved from tools/data.py):
- `patient_id()` — Creates UPID = Provider Code (first 3 chars) + PersonKey
- `drug_names()` — Standardizes drug names via drugnames.csv lookup
- `department_identification()` — 5-level fallback chain for directory assignment

**pathway_pipeline.py** — Pipeline orchestration:
- Processes 6 date filter combinations × 2 chart types (directory + indication)
- `fetch_and_transform_data()` — Snowflake fetch + UPID/drug/directory transforms
- `process_pathway_for_date_filter()` — Directory charts using `generate_icicle_chart()`
- `process_indication_pathway_for_date_filter()` — Indication charts using `generate_icicle_chart_indication()`
- `insert_pathway_records()` — SQLite insertion with parameterized queries

**diagnosis_lookup.py** — GP diagnosis matching:
- `get_patient_indication_groups()` — Batch queries Snowflake (500 patients at a time)
- Embeds ~148 Search_Term → Cluster_ID mappings as SQL CTE
- Returns most recent match per patient via `QUALIFY ROW_NUMBER()`

**database.py** — SQLite connection pooling and transaction management

**schema.py** — SQL schema definitions (reference tables + pathway_nodes)

**snowflake_connector.py** — Snowflake SSO integration via externalbrowser authenticator

**cache.py** — Query result caching with TTL-based invalidation

## Import Pattern

All imports use package names directly:
```python
from data_processing.transforms import patient_id, drug_names, department_identification
from data_processing.pathway_pipeline import process_all_date_filters
```