Initial commit before Ralph loop
This commit is contained in:
Andrew Charlwood
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(python*)",
|
||||
"Bash(git*)",
|
||||
"Bash(cd*)",
|
||||
"Bash(ls*)",
|
||||
"Bash(cat*)",
|
||||
"Bash(head*)",
|
||||
"Bash(tail*)",
|
||||
"Bash(mkdir*)",
|
||||
"Bash(touch*)",
|
||||
"Bash(rm*)",
|
||||
"Bash(mv*)",
|
||||
"Bash(cp*)",
|
||||
"Bash(timeout*)",
|
||||
"Bash(reflex*)",
|
||||
"Read",
|
||||
"Write",
|
||||
"Edit",
|
||||
"Glob",
|
||||
"Grep"
|
||||
],
|
||||
"deny": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"WebSearch",
|
||||
"Bash(wc:*)",
|
||||
"WebFetch(domain:flet.dev)",
|
||||
"WebFetch(domain:github.com)",
|
||||
"WebFetch(domain:docs.flet.dev)"
|
||||
]
|
||||
}
|
||||
}
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
assets/external/
|
||||
.states
|
||||
.web
|
||||
*.py[cod]
|
||||
# Python-generated files
|
||||
__pycache__/
|
||||
*.py[oc]
|
||||
build/
|
||||
dist/
|
||||
wheels/
|
||||
*.egg-info
|
||||
|
||||
# Virtual environments
|
||||
.venv
|
||||
|
||||
# Test and lint caches
|
||||
.coverage
|
||||
.mypy_cache/
|
||||
.pytest_cache/
|
||||
|
||||
# Data files (large)
|
||||
hcd_20250411.csv
|
||||
hcd_20250411.parquet
|
||||
|
||||
# IDE
|
||||
.idea
|
||||
|
||||
# Ignored experiments
|
||||
.ignore
|
||||
|
||||
# Ralph loop logs (keep directory via .gitkeep)
|
||||
logs/*.log
|
||||
logs/*.jsonl
|
||||
|
||||
# Reflex build artifacts (future)
|
||||
.web/
|
||||
.states/
|
||||
|
||||
# SQLite database (will contain local data)
|
||||
*.db
|
||||
*.sqlite
|
||||
|
||||
# Snowflake result cache
|
||||
data/cache/
|
||||
|
||||
# Uploaded data files
|
||||
data/uploads/
|
||||
|
||||
# Exported analysis results
|
||||
data/exports/
|
||||
|
||||
# Analysis output files
|
||||
output/*.html
|
||||
output/*.csv
|
||||
*.html
|
||||
|
||||
# VS Code workspace settings
|
||||
.vscode/
|
||||
|
||||
# User uploaded files
|
||||
uploaded_files/
|
||||
@@ -0,0 +1 @@
|
||||
3.10
|
||||
@@ -0,0 +1,302 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
NHS High-Cost Drug Patient Pathway Analysis Tool - a web-based application that analyzes secondary care patient treatment pathways. It processes clinical activity data to visualize hierarchical treatment patterns (Trust → Directory/Specialty → Drug → Patient pathway) as interactive Plotly icicle charts.
|
||||
|
||||
**Key Features:**
|
||||
- Multi-source data loading: CSV/Parquet files, SQLite database, Snowflake data warehouse
|
||||
- GP diagnosis integration for indication validation via SNOMED clusters
|
||||
- Interactive browser-based UI using Reflex framework
|
||||
- Real-time analysis with progress feedback
|
||||
|
||||
## Running the Application
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
# OR with uv
|
||||
uv sync
|
||||
|
||||
# Run the Reflex web application
|
||||
reflex run
|
||||
```
|
||||
|
||||
The application requires Python 3.10+ and runs on http://localhost:3000 by default.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Package Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── core/ # Core configuration and models
|
||||
│ ├── config.py # PathConfig dataclass for file paths
|
||||
│ ├── models.py # AnalysisFilters dataclass
|
||||
│ └── logging_config.py # Structured logging setup
|
||||
│
|
||||
├── data_processing/ # Data layer
|
||||
│ ├── database.py # SQLite connection management
|
||||
│ ├── schema.py # Database schema definitions
|
||||
│ ├── loader.py # DataLoader abstraction (CSV/SQLite)
|
||||
│ ├── patient_data.py # Patient data migration and loading
|
||||
│ ├── reference_data.py # Reference data migration
|
||||
│ ├── snowflake_connector.py # Snowflake integration
|
||||
│ ├── cache.py # Query result caching
|
||||
│ ├── data_source.py # Data source fallback chain
|
||||
│ └── diagnosis_lookup.py # GP diagnosis validation
|
||||
│
|
||||
├── analysis/ # Analysis pipeline
|
||||
│ ├── pathway_analyzer.py # prepare_data, calculate_statistics, build_hierarchy
|
||||
│ └── statistics.py # Statistical calculation functions
|
||||
│
|
||||
├── visualization/ # Chart generation
|
||||
│ └── plotly_generator.py # create_icicle_figure, save_figure_html
|
||||
│
|
||||
├── pathways_app/ # Reflex web application
|
||||
│ ├── pathways_app.py # State class and page components
|
||||
│ └── components/ # Layout and navigation components
|
||||
│
|
||||
├── tools/ # Legacy modules
|
||||
│ ├── dashboard_gui.py # Original analysis engine (being refactored)
|
||||
│ └── data.py # Data transformations (UPID, drug names, directory)
|
||||
│
|
||||
├── config/ # Configuration files
|
||||
│ └── snowflake.toml # Snowflake connection settings
|
||||
│
|
||||
├── data/ # Reference data and database
|
||||
│ ├── pathways.db # SQLite database
|
||||
│ └── *.csv # Reference data files
|
||||
│
|
||||
└── tests/ # Test suite
|
||||
├── conftest.py # Pytest fixtures
|
||||
└── test_*.py # Test modules
|
||||
```
|
||||
|
||||
### Core Module (`core/`)
|
||||
|
||||
- **PathConfig** - Dataclass encapsulating all file paths, with `validate()` method
|
||||
- **AnalysisFilters** - Dataclass for filter state (dates, drugs, trusts, directories)
|
||||
- **logging_config** - Structured logging with file and console output
|
||||
|
||||
### Data Processing Module (`data_processing/`)
|
||||
|
||||
**Database Management:**
|
||||
- `DatabaseManager` - SQLite connection pooling and transaction management
|
||||
- Tables: `ref_drug_names`, `ref_organizations`, `ref_directories`, `ref_drug_directory_map`, `ref_drug_indication_clusters`, `fact_interventions`, `mv_patient_treatment_summary`, `processed_files`
|
||||
|
||||
**Data Loaders:**
|
||||
- `FileDataLoader` - Loads from CSV/Parquet files
|
||||
- `SQLiteDataLoader` - Queries fact_interventions table
|
||||
- Factory function `get_loader()` selects appropriate loader
|
||||
|
||||
**Snowflake Integration:**
|
||||
- SSO authentication via `externalbrowser` authenticator
|
||||
- `fetch_activity_data(start_date, end_date, provider_codes)` method
|
||||
- Query caching with TTL-based invalidation
|
||||
- Fallback chain: cache → Snowflake → local files
|
||||
|
||||
**GP Diagnosis Validation:**
|
||||
- Uses pre-built SNOMED clusters from `ClinicalCodingClusterSnomedCodes`
|
||||
- `patient_has_indication(patient_pseudonym, cluster_ids)` checks GP records
|
||||
- `validate_indication(patient_pseudonym, drug_name)` returns full validation result
|
||||
- Adds `Indication_Source` column: "GP_SNOMED" | "HCD_SNOMED" | "NONE"
|
||||
|
||||
### Analysis Module (`analysis/`)
|
||||
|
||||
Refactored from the original 267-line `generate_graph()` function:
|
||||
|
||||
- **prepare_data()** - Filter DataFrame by date range, trusts, drugs, directories
|
||||
- **calculate_statistics()** - Compute frequency, cost, duration statistics
|
||||
- **build_hierarchy()** - Create Trust → Directory → Drug → Pathway structure
|
||||
- **prepare_chart_data()** - Format data for Plotly icicle chart
|
||||
|
||||
### Visualization Module (`visualization/`)
|
||||
|
||||
- **create_icicle_figure()** - Generate Plotly icicle chart figure
|
||||
- **save_figure_html()** - Save interactive HTML file
|
||||
- **open_figure_in_browser()** - Open chart in default browser
|
||||
|
||||
### Reflex Application (`pathways_app/`)
|
||||
|
||||
The `State` class manages all application state:
|
||||
- Filter variables: dates, drugs, trusts, directories
|
||||
- Reference data: available options loaded from CSV/SQLite
|
||||
- Analysis state: running flag, status messages, chart data
|
||||
- Data source state: file path, source type, row counts
|
||||
|
||||
### Legacy Modules (`tools/`)
|
||||
|
||||
Still used during transition:
|
||||
|
||||
- **tools/data.py** - Data transformation functions:
|
||||
- `patient_id()` - Creates UPID = Provider Code (first 3 chars) + PersonKey
|
||||
- `drug_names()` - Standardizes via drugnames.csv lookup
|
||||
- `department_identification()` - 5-level fallback chain for directory assignment
|
||||
|
||||
- **tools/dashboard_gui.py** - Original analysis engine (being replaced by `analysis/` module)
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Data Sources:
|
||||
CSV/Parquet file upload
|
||||
OR SQLite database query
|
||||
OR Snowflake fetch (with caching)
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────┐
|
||||
│ Data Transformations (tools/data.py) │
|
||||
│ → patient_id() creates UPID │
|
||||
│ → drug_names() standardizes names │
|
||||
│ → department_identification() → Dir │
|
||||
└──────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────┐
|
||||
│ Analysis Pipeline (analysis/) │
|
||||
│ → prepare_data() - filter by criteria │
|
||||
│ → calculate_statistics() │
|
||||
│ → build_hierarchy() │
|
||||
│ → prepare_chart_data() │
|
||||
└──────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────┐
|
||||
│ Visualization (visualization/) │
|
||||
│ → create_icicle_figure() │
|
||||
│ → Display in rx.plotly() component │
|
||||
└──────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Reference Data Files (`data/`)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `include.csv` | Drug filter list with default selections (Include=1) |
|
||||
| `defaultTrusts.csv` | NHS Trust list for filter |
|
||||
| `directory_list.csv` | Medical specialties/directories |
|
||||
| `drugnames.csv` | Drug name standardization mapping |
|
||||
| `org_codes.csv` | Provider code to organization name mapping |
|
||||
| `drug_directory_list.csv` | Valid drug-to-directory mappings (pipe-separated) |
|
||||
| `treatment_function_codes.csv` | NHS treatment function code mappings |
|
||||
| `drug_indication_clusters.csv` | Drug to SNOMED cluster mappings |
|
||||
| `ta-recommendations.xlsx` | NICE TA recommendations |
|
||||
| `pathways.db` | SQLite database with all tables |
|
||||
|
||||
### Key Patterns
|
||||
|
||||
**Department Identification Fallback Chain:**
|
||||
The `department_identification()` function has 5 levels of fallback:
|
||||
1. **SINGLE_VALID_DIR** - Drug has only one valid directory
|
||||
2. **EXTRACTED** - Extracted from Additional Detail/Description fields
|
||||
3. **CALCULATED_MOST_FREQ** - Most frequent valid directory for UPID/Drug
|
||||
4. **UPID_INFERENCE** - Inferred from other records with same UPID
|
||||
5. **UNDEFINED** - No directory could be determined
|
||||
|
||||
**Indication Validation Workflow:**
|
||||
1. Map drug → SNOMED cluster IDs (e.g., ADALIMUMAB → RARTH_COD, PSORIASIS_COD)
|
||||
2. Get all SNOMED codes for those clusters
|
||||
3. Check GP records (PrimaryCareClinicalCoding) for matching codes
|
||||
4. Report match/no-match with source tracking
|
||||
|
||||
**Data Source Fallback Chain:**
|
||||
1. Query cache for recent results
|
||||
2. Attempt Snowflake connection
|
||||
3. Fall back to SQLite database
|
||||
4. Fall back to CSV/Parquet files
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Reference Tables
|
||||
- `ref_drug_names` - Drug name standardization
|
||||
- `ref_organizations` - Provider code to name mapping
|
||||
- `ref_directories` - Valid directory names
|
||||
- `ref_drug_directory_map` - Valid drug-directory pairs
|
||||
- `ref_drug_indication_clusters` - Drug to SNOMED cluster mapping
|
||||
|
||||
### Fact Tables
|
||||
- `fact_interventions` - Patient intervention records (UPID, drug, date, cost, directory)
|
||||
|
||||
### Materialized Views
|
||||
- `mv_patient_treatment_summary` - Pre-aggregated patient statistics
|
||||
|
||||
### File Tracking
|
||||
- `processed_files` - Hash-based tracking for incremental loading
|
||||
|
||||
## Input Data Requirements
|
||||
|
||||
The input data (CSV/Parquet) must contain columns including:
|
||||
- `Provider Code`, `PersonKey` - Used to create UPID
|
||||
- `Drug Name`, `Intervention Date`, `Price Actual`
|
||||
- `OrganisationName`
|
||||
- Various `Additional Detail/Description` columns for directory extraction
|
||||
- `Treatment Function Code`
|
||||
|
||||
## Output
|
||||
|
||||
Interactive Plotly icicle chart showing:
|
||||
- Patient counts and percentages at each hierarchy level
|
||||
- Total and average costs
|
||||
- Treatment duration and dosing frequency information
|
||||
- Color gradient based on patient volume
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Run all tests with coverage
|
||||
python -m pytest tests/ -v --cov=core --cov=analysis
|
||||
|
||||
# Run specific test file
|
||||
python -m pytest tests/test_config.py -v
|
||||
|
||||
# Run specific test class
|
||||
python -m pytest tests/test_data_transformations.py::TestPatientId -v
|
||||
```
|
||||
|
||||
Test coverage includes:
|
||||
- PathConfig validation (23 tests)
|
||||
- AnalysisFilters validation (26 tests)
|
||||
- Data transformation functions (23 tests)
|
||||
- Directory assignment logic (19 tests)
|
||||
|
||||
## Configuration
|
||||
|
||||
### Snowflake Connection (`config/snowflake.toml`)
|
||||
|
||||
```toml
|
||||
[snowflake]
|
||||
account = "your-account"
|
||||
database = "DATA_HUB"
|
||||
schema = "CDM"
|
||||
warehouse = "your-warehouse"
|
||||
authenticator = "externalbrowser" # Required for NHS SSO
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Logs are written to `logs/` directory with structured format.
|
||||
Configure via `core/logging_config.py`.
|
||||
|
||||
## Development
|
||||
|
||||
### Adding New Data Sources
|
||||
|
||||
1. Create loader class implementing `DataLoader` protocol in `data_processing/loader.py`
|
||||
2. Add to factory function `get_loader()`
|
||||
3. Update `DataSourceManager` fallback chain if needed
|
||||
|
||||
### Adding New Analysis Features
|
||||
|
||||
1. Add statistical functions to `analysis/statistics.py`
|
||||
2. Integrate into pipeline in `analysis/pathway_analyzer.py`
|
||||
3. Update visualization in `visualization/plotly_generator.py`
|
||||
|
||||
### Adding New Reference Data
|
||||
|
||||
1. Add CSV file to `data/` directory
|
||||
2. Define schema in `data_processing/schema.py`
|
||||
3. Create migration function in `data_processing/reference_data.py`
|
||||
4. Add path to `PathConfig` in `core/config.py`
|
||||
@@ -0,0 +1,189 @@
|
||||
# Design System - HCD Analysis v2
|
||||
|
||||
This document defines the visual design language for the UI redesign. All components should reference these tokens for consistency.
|
||||
|
||||
## Color Palette
|
||||
|
||||
### Primary Blues (NHS-inspired, modernized)
|
||||
| Name | Hex | Usage |
|
||||
|------|-----|-------|
|
||||
| Heritage Blue | `#003087` | Deep headers, authoritative accents |
|
||||
| Primary Blue | `#0066CC` | Main actions, links, focus states |
|
||||
| Vibrant Blue | `#1E88E5` | Highlights, hover states, chart primary |
|
||||
| Sky Blue | `#4FC3F7` | Accents, progress bars, secondary elements |
|
||||
| Pale Blue | `#E3F2FD` | Subtle backgrounds, card tints |
|
||||
|
||||
### Neutrals (warm-tinted for clinical warmth)
|
||||
| Name | Hex | Usage |
|
||||
|------|-----|-------|
|
||||
| Slate 900 | `#1E293B` | Primary text |
|
||||
| Slate 700 | `#334155` | Secondary text |
|
||||
| Slate 500 | `#64748B` | Muted text, placeholders |
|
||||
| Slate 300 | `#CBD5E1` | Borders, dividers |
|
||||
| Slate 100 | `#F1F5F9` | Card backgrounds, hover states |
|
||||
| White | `#FFFFFF` | Page background |
|
||||
|
||||
### Semantic Colors
|
||||
| Name | Hex | Usage |
|
||||
|------|-----|-------|
|
||||
| Success | `#059669` | Positive states, confirmations |
|
||||
| Warning | `#D97706` | Caution states, alerts |
|
||||
| Error | `#DC2626` | Error states, destructive actions |
|
||||
| Info | `#0284C7` | Informational (matches primary family) |
|
||||
|
||||
### Chart Palette
|
||||
```
|
||||
Primary series: #003087, #0066CC, #1E88E5, #4FC3F7, #90CAF9
|
||||
Categorical: #0066CC, #059669, #D97706, #8B5CF6, #EC4899
|
||||
```
|
||||
|
||||
## Typography
|
||||
|
||||
**Font Family:** Inter (primary), system-ui (fallback)
|
||||
|
||||
| Style | Size | Weight | Tracking | Line Height | Usage |
|
||||
|-------|------|--------|----------|-------------|-------|
|
||||
| Display | 32px | 700 | -0.02em | 1.2 | Page titles |
|
||||
| Heading 1 | 24px | 600 | -0.01em | 1.3 | Section headers |
|
||||
| Heading 2 | 20px | 600 | normal | 1.4 | Card titles |
|
||||
| Heading 3 | 16px | 600 | normal | 1.4 | Subsections |
|
||||
| Body | 14px | 400 | normal | 1.5 | Default text |
|
||||
| Body Small | 13px | 400 | normal | 1.5 | Secondary info |
|
||||
| Caption | 12px | 500 | normal | 1.4 | Labels, metadata |
|
||||
| Mono | 13px | 400 | normal | 1.5 | Data values, codes (JetBrains Mono) |
|
||||
|
||||
## Spacing Scale
|
||||
|
||||
| Token | Value | Usage |
|
||||
|-------|-------|-------|
|
||||
| xs | 4px | Tight internal padding |
|
||||
| sm | 8px | Between related elements |
|
||||
| md | 12px | Standard gaps |
|
||||
| lg | 16px | Section padding |
|
||||
| xl | 24px | Card padding |
|
||||
| 2xl | 32px | Major section gaps |
|
||||
| 3xl | 48px | Page margins |
|
||||
|
||||
## Border Radius
|
||||
|
||||
| Token | Value | Usage |
|
||||
|-------|-------|-------|
|
||||
| sm | 4px | Small elements, inputs |
|
||||
| md | 8px | Buttons, small cards |
|
||||
| lg | 12px | Cards, modals |
|
||||
| xl | 16px | Large containers |
|
||||
| full | 9999px | Pills, avatars |
|
||||
|
||||
## Shadows
|
||||
|
||||
| Token | Value | Usage |
|
||||
|-------|-------|-------|
|
||||
| sm | `0 1px 2px rgba(0,0,0,0.05)` | Subtle elevation |
|
||||
| md | `0 1px 3px rgba(0,0,0,0.08)` | Cards at rest |
|
||||
| lg | `0 4px 6px rgba(0,0,0,0.1)` | Cards on hover, dropdowns |
|
||||
| xl | `0 10px 15px rgba(0,0,0,0.1)` | Modals, popovers |
|
||||
|
||||
## Component Specifications
|
||||
|
||||
### Cards
|
||||
- Background: White
|
||||
- Border: 1px Slate 300 (optional, or use shadow only)
|
||||
- Border radius: lg (12px)
|
||||
- Padding: xl (24px)
|
||||
- Shadow: md at rest, lg on hover
|
||||
- Hover: translateY(-2px) transition
|
||||
|
||||
### Buttons
|
||||
**Primary:**
|
||||
- Background: Primary Blue
|
||||
- Text: White
|
||||
- Border radius: md (8px)
|
||||
- Padding: 10px 20px
|
||||
- Hover: Vibrant Blue background, slight scale (1.02)
|
||||
|
||||
**Secondary:**
|
||||
- Background: White
|
||||
- Border: 1px Primary Blue
|
||||
- Text: Primary Blue
|
||||
- Hover: Pale Blue background
|
||||
|
||||
**Ghost:**
|
||||
- Background: transparent
|
||||
- Text: Primary Blue
|
||||
- Hover: Pale Blue background
|
||||
|
||||
### Form Controls
|
||||
- Height: 40px (inputs, selects)
|
||||
- Border: 1px Slate 300
|
||||
- Border radius: md (8px)
|
||||
- Focus: 2px Primary Blue ring
|
||||
- Placeholder: Slate 500
|
||||
|
||||
### Data Cards (KPIs)
|
||||
- Large mono number: 32-48px, Slate 900
|
||||
- Label: Caption size, Slate 500
|
||||
- Background: White or Pale Blue tint
|
||||
- Optional trend indicator or sparkline
|
||||
|
||||
## Layout
|
||||
|
||||
### Page Structure
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Logo + App Name [Chart Tabs] Data Freshness │ ← Top Bar (64px height)
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─ Filters ─────────────────────────────────────────────────┐ │ ← Filter Section
|
||||
│ │ Date ranges, dropdowns, filter controls │ │
|
||||
│ └───────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─ KPIs ────────────────────────────────────────────────────┐ │ ← KPI Row
|
||||
│ │ [ Metric 1 ] [ Metric 2 ] [ Metric 3 ] [ Metric 4 ] │ │
|
||||
│ └───────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─ Chart ───────────────────────────────────────────────────┐ │ ← Main Chart (fills remaining)
|
||||
│ │ │ │
|
||||
│ │ [ Interactive Visualization ] │ │
|
||||
│ │ │ │
|
||||
│ └───────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Responsive Breakpoints
|
||||
- Mobile: < 640px
|
||||
- Tablet: 640px - 1024px
|
||||
- Desktop: > 1024px
|
||||
|
||||
## Transitions
|
||||
|
||||
| Property | Duration | Easing |
|
||||
|----------|----------|--------|
|
||||
| Color, background | 150ms | ease-out |
|
||||
| Transform | 200ms | ease-out |
|
||||
| Shadow | 200ms | ease-out |
|
||||
| Opacity | 200ms | ease-in-out |
|
||||
|
||||
## Reflex Implementation Notes
|
||||
|
||||
### Using Design Tokens
|
||||
Create a `styles.py` module with these values as Python constants. Import throughout the app:
|
||||
|
||||
```python
|
||||
# Example structure
|
||||
class Colors:
|
||||
PRIMARY = "#0066CC"
|
||||
PRIMARY_DARK = "#003087"
|
||||
# etc.
|
||||
|
||||
class Spacing:
|
||||
XS = "4px"
|
||||
SM = "8px"
|
||||
# etc.
|
||||
```
|
||||
|
||||
### rx.theme Configuration
|
||||
Configure Reflex's theme provider with the color palette for consistent component styling.
|
||||
|
||||
### Custom CSS
|
||||
For styles not achievable via Reflex props, use `rx.style` or a custom CSS file.
|
||||
@@ -0,0 +1,199 @@
|
||||
# Implementation Plan - HCD Analysis UI Redesign
|
||||
|
||||
## Project Overview
|
||||
|
||||
Complete frontend redesign of the Patient Pathway Analysis tool. Replace the current multi-page sidebar layout with a modern, single-page dashboard featuring:
|
||||
- Instant reactive filtering with debounce
|
||||
- Interactive Plotly icicle chart that updates in real-time
|
||||
- NHS-inspired but bold, modern visual design
|
||||
- KPI metrics that respond to filter changes
|
||||
|
||||
**Design Reference:** See `DESIGN_SYSTEM.md` for color palette, typography, spacing, and component specs.
|
||||
|
||||
**Source Code:** The existing `pathways_app/pathways_app.py` contains the current implementation. Create a new `pathways_app/app_v2.py` for the redesign, leaving the original intact until verification.
|
||||
|
||||
## Quality Checks
|
||||
|
||||
Run after each task:
|
||||
|
||||
```bash
|
||||
# Syntax check
|
||||
python -m py_compile pathways_app/app_v2.py
|
||||
|
||||
# Import verification
|
||||
python -c "from pathways_app.app_v2 import app"
|
||||
|
||||
# Reflex compilation test
|
||||
cd pathways_app && timeout 60 python -m reflex run 2>&1 | head -30
|
||||
|
||||
# If compilation shows errors, fix before marking task complete
|
||||
```
|
||||
|
||||
## Phase 1: Foundation
|
||||
|
||||
### 1.1 Design Tokens Module
|
||||
- [ ] Create `pathways_app/styles.py` with design token classes:
|
||||
- `Colors` class with all palette colors as constants
|
||||
- `Typography` class with font sizes, weights
|
||||
- `Spacing` class with spacing scale
|
||||
- `Shadows` class with shadow values
|
||||
- `Radii` class with border radius values
|
||||
- [ ] Create helper functions for common style patterns (e.g., `card_style()`, `button_primary_style()`)
|
||||
- [ ] Verify imports work: `from pathways_app.styles import Colors, Spacing`
|
||||
|
||||
### 1.2 App Skeleton
|
||||
- [ ] Create `pathways_app/app_v2.py` with basic Reflex app structure
|
||||
- [ ] Define new `AppState` class with minimal state (placeholder for now)
|
||||
- [ ] Create single-page layout structure matching DESIGN_SYSTEM.md
|
||||
- [ ] Verify `reflex run` compiles and shows blank page with correct structure
|
||||
- [ ] Configure Reflex theme with design system colors
|
||||
|
||||
## Phase 2: Layout Components
|
||||
|
||||
### 2.1 Top Navigation Bar
|
||||
- [ ] Create `top_bar()` component:
|
||||
- Logo (use existing NHS person logo from assets)
|
||||
- App title "HCD Analysis"
|
||||
- Chart type tabs/pills (Icicle active, placeholders for future charts)
|
||||
- Data freshness indicator (right side): "12,450 records (2d ago)"
|
||||
- [ ] Style with Heritage Blue accents, clean typography
|
||||
- [ ] Fixed height: 64px
|
||||
- [ ] Verify renders correctly
|
||||
|
||||
### 2.2 Filter Section
|
||||
- [ ] Create `filter_section()` component with card styling
|
||||
- [ ] Add date range pickers:
|
||||
- "Initiated" range with enable/disable checkbox (default: disabled)
|
||||
- "Last Seen" range with enable/disable checkbox (default: enabled, last 6 months)
|
||||
- "To" date defaults to latest date in dataset
|
||||
- [ ] Add searchable multi-select dropdowns:
|
||||
- Drugs dropdown with search, select all, count display
|
||||
- Indications dropdown with search, select all, count display
|
||||
- Directorates dropdown with search, select all, count display
|
||||
- [ ] Implement debounced filter change handlers (300ms)
|
||||
- [ ] Style according to design system
|
||||
|
||||
### 2.3 KPI Row
|
||||
- [ ] Create `kpi_card()` component:
|
||||
- Large mono number (32-48px)
|
||||
- Label below (caption style)
|
||||
- Subtle background tint
|
||||
- [ ] Create `kpi_row()` component with responsive grid
|
||||
- [ ] Initially show: Unique Patients count
|
||||
- [ ] Leave space for future metrics (Drugs count, Total cost, Match rate)
|
||||
- [ ] KPIs should be reactive to filter state
|
||||
|
||||
### 2.4 Chart Container
|
||||
- [ ] Create `chart_section()` component
|
||||
- [ ] Full-width card with appropriate padding
|
||||
- [ ] Placeholder for Plotly chart (integrate in Phase 3)
|
||||
- [ ] Loading state with skeleton/spinner
|
||||
- [ ] Error state with friendly message
|
||||
|
||||
## Phase 3: State Management
|
||||
|
||||
### 3.1 Core State Variables
|
||||
- [ ] Define filter state variables in `AppState`:
|
||||
- `initiated_filter_enabled: bool = False`
|
||||
- `initiated_from: datetime`
|
||||
- `initiated_to: datetime`
|
||||
- `last_seen_filter_enabled: bool = True`
|
||||
- `last_seen_from: datetime` (default: 6 months ago)
|
||||
- `last_seen_to: datetime` (default: latest in dataset)
|
||||
- `selected_drugs: List[str]` (default: all)
|
||||
- `selected_indications: List[str]` (default: all)
|
||||
- `selected_directorates: List[str]` (default: all)
|
||||
- [ ] Define data state variables:
|
||||
- `data_loaded: bool`
|
||||
- `total_records: int`
|
||||
- `last_updated: datetime`
|
||||
- `filtered_data: pd.DataFrame` (or computed)
|
||||
- [ ] Define UI state variables:
|
||||
- `chart_loading: bool`
|
||||
- `error_message: str`
|
||||
|
||||
### 3.2 Data Loading
|
||||
- [ ] Create `load_data()` method that reads from SQLite
|
||||
- [ ] Populate available options for dropdowns (drugs, indications, directorates)
|
||||
- [ ] Detect latest date in dataset for "to" date defaults
|
||||
- [ ] Calculate total records and last updated timestamp
|
||||
- [ ] Call on app initialization
|
||||
|
||||
### 3.3 Filter Logic
|
||||
- [ ] Create `apply_filters()` computed method that filters the data based on current state
|
||||
- [ ] Handle initiated date filter (when enabled)
|
||||
- [ ] Handle last seen date filter (when enabled)
|
||||
- [ ] Handle drug/indication/directorate multi-select filters
|
||||
- [ ] Return filtered DataFrame
|
||||
|
||||
### 3.4 KPI Calculations
|
||||
- [ ] Create computed properties for KPI values:
|
||||
- `unique_patients: int` — COUNT(DISTINCT patient_id) from filtered data
|
||||
- (Future: drug count, total cost, indication match rate)
|
||||
- [ ] Ensure KPIs update reactively when filters change
|
||||
|
||||
## Phase 4: Interactive Chart
|
||||
|
||||
### 4.1 Chart Data Preparation
|
||||
- [ ] Create `prepare_chart_data()` method that transforms filtered data for Plotly icicle
|
||||
- [ ] Reuse/adapt logic from existing `pathway_analyzer.py`
|
||||
- [ ] Return data structure compatible with `plotly.express.icicle()`
|
||||
|
||||
### 4.2 Reactive Plotly Integration
|
||||
- [ ] Create `generate_icicle_chart()` computed property that returns Plotly figure
|
||||
- [ ] Configure chart colors using design system palette
|
||||
- [ ] Configure chart interactivity (zoom, pan, click, hover)
|
||||
- [ ] Set responsive sizing
|
||||
|
||||
### 4.3 Chart Component
|
||||
- [ ] Integrate `rx.plotly()` component in chart_section
|
||||
- [ ] Pass reactive figure from state
|
||||
- [ ] Handle loading states (show skeleton while computing)
|
||||
- [ ] Handle empty data state (friendly message)
|
||||
- [ ] Verify chart updates when filters change
|
||||
|
||||
## Phase 5: Polish & Verification
|
||||
|
||||
### 5.1 Visual Polish
|
||||
- [ ] Review all components against DESIGN_SYSTEM.md
|
||||
- [ ] Ensure consistent spacing throughout
|
||||
- [ ] Ensure consistent typography throughout
|
||||
- [ ] Add hover states and transitions to interactive elements
|
||||
- [ ] Test responsive behavior (resize browser)
|
||||
|
||||
### 5.2 Performance Optimization
|
||||
- [ ] Profile filter + chart update cycle
|
||||
- [ ] Ensure debounce is working correctly (not triggering on every keystroke)
|
||||
- [ ] Optimize any slow computed properties
|
||||
- [ ] Verify smooth 60fps interactions
|
||||
|
||||
### 5.3 Error Handling
|
||||
- [ ] Handle no data loaded state gracefully
|
||||
- [ ] Handle filter resulting in zero records
|
||||
- [ ] Handle any data loading errors
|
||||
- [ ] User-friendly error messages
|
||||
|
||||
### 5.4 Final Verification
|
||||
- [ ] Load real data from SQLite
|
||||
- [ ] Test all filter combinations
|
||||
- [ ] Verify KPIs update correctly
|
||||
- [ ] Verify chart updates correctly
|
||||
- [ ] Compare key metrics with original app to ensure correctness
|
||||
- [ ] Test with large dataset for performance
|
||||
|
||||
### 5.5 Cleanup
|
||||
- [ ] Remove or comment out old `pathways_app.py` code paths
|
||||
- [ ] Update any imports/references to use new app
|
||||
- [ ] Update README with new run instructions
|
||||
- [ ] Document any breaking changes
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
All tasks marked `[x]` AND:
|
||||
- [ ] App compiles without errors (`reflex run` succeeds)
|
||||
- [ ] All filters work with instant (debounced) updates
|
||||
- [ ] KPIs display correct numbers matching filter state
|
||||
- [ ] Icicle chart renders and updates reactively
|
||||
- [ ] Visual design matches DESIGN_SYSTEM.md
|
||||
- [ ] No console errors during normal operation
|
||||
- [ ] Verified with real patient data from SQLite
|
||||
@@ -0,0 +1,859 @@
|
||||
# Patient Pathway Analysis - Improvement Recommendations
|
||||
|
||||
This document outlines recommended improvements to modernize the Patient Pathway Analysis application, based on multi-domain expert analysis.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
| Area | Current State | Recommended Change | Priority |
|
||||
|------|--------------|-------------------|----------|
|
||||
| **GUI Framework** | CustomTkinter | **Reflex** (browser-based, native Plotly) | High |
|
||||
| **Data Storage** | CSV files (90MB+) | SQLite with caching | High |
|
||||
| **Data Source** | Manual CSV export | Direct Snowflake connection | Medium |
|
||||
| **Directory Assignment** | Multi-stage fallback | GP diagnosis codes as primary | Medium |
|
||||
| **Code Quality** | Monolithic, no types | Modular, typed, tested | Low |
|
||||
|
||||
---
|
||||
|
||||
## 1. GUI Framework: Replace CustomTkinter with Reflex or Flet
|
||||
|
||||
### What
|
||||
Replace the CustomTkinter-based GUI with a modern Python framework. Two strong options:
|
||||
- **[Reflex](https://reflex.dev)** - React-based, runs in browser
|
||||
- **[Flet](https://flet.dev)** - Flutter-based, native desktop or browser
|
||||
|
||||
### Why
|
||||
|
||||
Since Python is approved and standalone `.exe` distribution isn't required, **both frameworks are viable**.
|
||||
|
||||
| Criterion | CustomTkinter | Reflex | Flet |
|
||||
|-----------|---------------|--------|------|
|
||||
| UI paradigm | Native desktop | Browser (localhost) | Desktop or browser |
|
||||
| Component richness | Limited | 60+ React components | Material Design |
|
||||
| Styling | Manual/limited | Full CSS/Tailwind | Flutter theming |
|
||||
| Plotly integration | External HTML | **Native embed** | WebView needed |
|
||||
| State management | Manual | Automatic re-render | Manual updates |
|
||||
| Learning curve | Low | Moderate (React-like) | Low-moderate |
|
||||
| Community | Small | 22k+ GitHub stars | 12k+ GitHub stars |
|
||||
| Maturity | Stable | Active (v0.6+) | Active (v0.80+) |
|
||||
|
||||
### Recommendation: **Reflex**
|
||||
|
||||
Given that:
|
||||
1. Python is approved for users
|
||||
2. Standalone `.exe` not required
|
||||
3. **Interactive Plotly is required** (Reflex has native `rx.plotly()` component)
|
||||
|
||||
Reflex is now the better choice because:
|
||||
- **Native Plotly support** - no need to open external browser windows
|
||||
- **Modern React-based UI** - cleaner, more customizable
|
||||
- **Simpler state management** - automatic re-rendering on state changes
|
||||
- **Better for data apps** - designed for dashboards and data visualization
|
||||
|
||||
### How (Reflex)
|
||||
|
||||
**Basic app structure:**
|
||||
|
||||
```python
|
||||
import reflex as rx
|
||||
|
||||
class State(rx.State):
|
||||
"""Application state."""
|
||||
start_date: str = "2019-04-01"
|
||||
end_date: str = "2025-04-30"
|
||||
selected_drugs: list[str] = []
|
||||
selected_trusts: list[str] = []
|
||||
analysis_running: bool = False
|
||||
chart_data: dict = {}
|
||||
|
||||
async def run_analysis(self):
|
||||
self.analysis_running = True
|
||||
yield # Update UI
|
||||
|
||||
# Run analysis (async)
|
||||
df = await self.load_and_process_data()
|
||||
self.chart_data = generate_plotly_figure(df)
|
||||
|
||||
self.analysis_running = False
|
||||
|
||||
def index() -> rx.Component:
|
||||
return rx.box(
|
||||
rx.hstack(
|
||||
# Sidebar with filters
|
||||
rx.vstack(
|
||||
rx.date_picker(
|
||||
value=State.start_date,
|
||||
on_change=State.set_start_date,
|
||||
),
|
||||
rx.checkbox_group(
|
||||
items=drug_list,
|
||||
value=State.selected_drugs,
|
||||
on_change=State.set_selected_drugs,
|
||||
),
|
||||
rx.button(
|
||||
"Run Analysis",
|
||||
on_click=State.run_analysis,
|
||||
loading=State.analysis_running,
|
||||
),
|
||||
width="300px",
|
||||
),
|
||||
# Main content - interactive Plotly chart
|
||||
rx.plotly(data=State.chart_data, layout=chart_layout),
|
||||
width="100%",
|
||||
)
|
||||
)
|
||||
|
||||
app = rx.App()
|
||||
app.add_page(index)
|
||||
```
|
||||
|
||||
**Key components mapping:**
|
||||
|
||||
| Current Component | Reflex Equivalent |
|
||||
|-------------------|-------------------|
|
||||
| `CTkFrame` | `rx.box`, `rx.vstack`, `rx.hstack` |
|
||||
| `CTkButton` | `rx.button` |
|
||||
| `CTkCheckBox` | `rx.checkbox` |
|
||||
| `CTkSlider` | `rx.slider` |
|
||||
| `DateEntry` | `rx.date_picker` |
|
||||
| `CTkScrollableFrame` | `rx.scroll_area` |
|
||||
| `filedialog` | `rx.upload` |
|
||||
| Plotly HTML file | **`rx.plotly()`** - native embed! |
|
||||
|
||||
**Running the app:**
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install reflex
|
||||
|
||||
# Initialize (first time)
|
||||
reflex init
|
||||
|
||||
# Run development server
|
||||
reflex run
|
||||
# Opens http://localhost:3000 in browser
|
||||
```
|
||||
|
||||
**Background tasks with progress:**
|
||||
|
||||
```python
|
||||
class State(rx.State):
|
||||
progress: int = 0
|
||||
status: str = ""
|
||||
|
||||
async def run_analysis(self):
|
||||
self.status = "Loading data..."
|
||||
self.progress = 10
|
||||
yield
|
||||
|
||||
df = load_data()
|
||||
self.status = "Processing..."
|
||||
self.progress = 50
|
||||
yield
|
||||
|
||||
result = process_data(df)
|
||||
self.status = "Complete"
|
||||
self.progress = 100
|
||||
yield
|
||||
```
|
||||
|
||||
### Alternative: Flet
|
||||
|
||||
If you prefer a more desktop-like feel, Flet remains a good option:
|
||||
|
||||
```python
|
||||
import flet as ft
|
||||
|
||||
def main(page: ft.Page):
|
||||
page.title = "HCD Analysis"
|
||||
|
||||
async def run_analysis(e):
|
||||
# Background task
|
||||
page.run_task(do_analysis)
|
||||
|
||||
page.add(
|
||||
ft.Row([
|
||||
# Sidebar
|
||||
ft.Column([
|
||||
ft.DatePicker(),
|
||||
ft.ElevatedButton("Run", on_click=run_analysis),
|
||||
]),
|
||||
# Chart area (opens in browser for interactivity)
|
||||
ft.ElevatedButton("View Chart", on_click=open_chart),
|
||||
])
|
||||
)
|
||||
|
||||
ft.app(target=main) # Desktop window
|
||||
# OR
|
||||
ft.app(target=main, view=ft.WEB_BROWSER) # Browser
|
||||
```
|
||||
|
||||
### Effort Estimate
|
||||
- Learning Reflex basics: 2-3 days
|
||||
- Rewriting GUI: 1-2 weeks
|
||||
- Testing and polish: 3-5 days
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Storage: SQLite Architecture
|
||||
|
||||
### What
|
||||
Replace CSV-based data loading with a SQLite database that stores reference data in normalized tables and caches processed patient data.
|
||||
|
||||
### Why
|
||||
|
||||
| Aspect | Current (CSV) | SQLite |
|
||||
|--------|---------------|--------|
|
||||
| Startup time | 90MB+ file read + full processing | Load reference data once (< 1MB) |
|
||||
| Memory usage | Entire dataset in memory | Incremental queries |
|
||||
| Incremental updates | Full reprocess required | Only process new/changed records |
|
||||
| Query performance | Pandas groupby/merge | Indexed SQL with CTEs |
|
||||
| Data consistency | Multiple CSVs can drift | Single source of truth with FK constraints |
|
||||
| Caching | None | Materialized views |
|
||||
|
||||
**Expected improvements:**
|
||||
- 60-80% faster startup
|
||||
- 50-70% memory reduction
|
||||
- 90%+ time savings on incremental updates
|
||||
|
||||
### How
|
||||
|
||||
**Recommended schema (simplified):**
|
||||
|
||||
```sql
|
||||
-- Reference tables
|
||||
CREATE TABLE ref_drug_names (
|
||||
drug_name_raw TEXT PRIMARY KEY,
|
||||
drug_name_std TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE ref_organizations (
|
||||
org_code TEXT PRIMARY KEY,
|
||||
org_name TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE ref_directories (
|
||||
directory_id INTEGER PRIMARY KEY,
|
||||
directory_name TEXT UNIQUE NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE ref_drug_directory_map (
|
||||
drug_name_std TEXT,
|
||||
directory_id INTEGER,
|
||||
is_single_valid BOOLEAN DEFAULT FALSE,
|
||||
PRIMARY KEY (drug_name_std, directory_id)
|
||||
);
|
||||
|
||||
-- Patient data (fact table)
|
||||
CREATE TABLE fact_interventions (
|
||||
intervention_id INTEGER PRIMARY KEY,
|
||||
upid TEXT NOT NULL,
|
||||
provider_code TEXT,
|
||||
drug_name_std TEXT NOT NULL,
|
||||
intervention_date DATE NOT NULL,
|
||||
price_actual REAL,
|
||||
directory_id INTEGER,
|
||||
directory_assignment_method TEXT,
|
||||
data_load_batch_id INTEGER
|
||||
);
|
||||
|
||||
-- Critical indexes
|
||||
CREATE INDEX idx_upid ON fact_interventions(upid);
|
||||
CREATE INDEX idx_upid_drug ON fact_interventions(upid, drug_name_std);
|
||||
CREATE INDEX idx_intervention_date ON fact_interventions(intervention_date);
|
||||
|
||||
-- Materialized view for patient summaries (cached aggregations)
|
||||
CREATE TABLE mv_patient_treatment_summary (
|
||||
upid TEXT PRIMARY KEY,
|
||||
first_seen DATE,
|
||||
last_seen DATE,
|
||||
total_cost REAL,
|
||||
drug_count INTEGER,
|
||||
last_refresh TIMESTAMP
|
||||
);
|
||||
|
||||
-- File tracking for incremental updates
|
||||
CREATE TABLE processed_files (
|
||||
file_path TEXT PRIMARY KEY,
|
||||
file_hash TEXT NOT NULL,
|
||||
last_processed TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
**Migration strategy:**
|
||||
|
||||
1. **Phase 1**: Create schema, load reference tables from existing CSVs
|
||||
2. **Phase 2**: Develop incremental load scripts for patient data
|
||||
3. **Phase 3**: Build materialized views for aggregations
|
||||
4. **Phase 4**: Modify `dashboard_gui.py` to query SQLite instead of processing CSVs
|
||||
|
||||
**Key query replacing pandas aggregation:**
|
||||
|
||||
```sql
|
||||
-- Replaces ~200 lines of pandas groupby/merge
|
||||
WITH patient_drugs AS (
|
||||
SELECT
|
||||
upid,
|
||||
drug_name_std,
|
||||
MIN(intervention_date) as first_date,
|
||||
MAX(intervention_date) as last_date,
|
||||
COUNT(*) as intervention_count,
|
||||
SUM(price_actual) as drug_cost
|
||||
FROM fact_interventions
|
||||
WHERE intervention_date BETWEEN :start_date AND :end_date
|
||||
AND provider_code IN (:trust_filters)
|
||||
GROUP BY upid, drug_name_std
|
||||
)
|
||||
SELECT * FROM patient_drugs;
|
||||
```
|
||||
|
||||
### Effort Estimate
|
||||
- Schema design and setup: 2-3 days
|
||||
- Migration scripts: 3-4 days
|
||||
- Query optimization: 2-3 days
|
||||
- Integration testing: 2-3 days
|
||||
|
||||
---
|
||||
|
||||
## 3. Snowflake Integration
|
||||
|
||||
### What
|
||||
Enable direct download of HCD activity data from Snowflake servers, replacing manual CSV exports.
|
||||
|
||||
### Why
|
||||
- Eliminates manual export step
|
||||
- Enables date-range filtering at query level (faster)
|
||||
- Automatic caching with TTL
|
||||
- Graceful fallback to local files if Snowflake unavailable
|
||||
|
||||
### How
|
||||
|
||||
**Authentication: SSO Browser Login**
|
||||
|
||||
Using `externalbrowser` authenticator - opens system browser for SSO authentication:
|
||||
|
||||
```python
|
||||
import snowflake.connector
|
||||
|
||||
conn = snowflake.connector.connect(
|
||||
account="your_account.region",
|
||||
user="your.email@nhs.net",
|
||||
authenticator="externalbrowser",
|
||||
warehouse="ANALYTICS_WH",
|
||||
database="data_hub",
|
||||
schema="dwh"
|
||||
)
|
||||
```
|
||||
|
||||
**Note**: User will see browser popup on first connection each session.
|
||||
|
||||
**Configuration (`config/snowflake.toml`):**
|
||||
|
||||
```toml
|
||||
[snowflake]
|
||||
account = "your_account.region"
|
||||
warehouse = "ANALYTICS_WH"
|
||||
database = "DataWarehouse"
|
||||
schema = "dwh"
|
||||
|
||||
[query]
|
||||
default_timeout = 300
|
||||
chunk_size = 100000
|
||||
|
||||
[cache]
|
||||
enabled = true
|
||||
ttl_hours = 24
|
||||
directory = "./data/cache"
|
||||
```
|
||||
|
||||
**Core connector pattern:**
|
||||
|
||||
```python
|
||||
from snowflake.connector import connect
|
||||
|
||||
class SnowflakeConnector:
|
||||
def fetch_activity_data(self, start_date, end_date, provider_codes=None):
|
||||
query = """
|
||||
SELECT
|
||||
"Provider Code",
|
||||
"PersonKey",
|
||||
"ProductDescription" as "Drug Name",
|
||||
"Intervention Date",
|
||||
"Price Actual",
|
||||
-- ... other columns
|
||||
FROM DataWarehouse.dwh.FactHighCostDrugs
|
||||
WHERE "Intervention Date" BETWEEN :start_date AND :end_date
|
||||
"""
|
||||
|
||||
with self.connect() as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(query, {'start_date': start_date, 'end_date': end_date})
|
||||
return cursor.fetch_pandas_all()
|
||||
```
|
||||
|
||||
**Caching strategy:**
|
||||
|
||||
| Scenario | Action |
|
||||
|----------|--------|
|
||||
| Same date range within 24 hours | Use cache |
|
||||
| Date range includes today | Query Snowflake (data may be updating) |
|
||||
| User clicks "Refresh" | Query Snowflake |
|
||||
| Snowflake unavailable | Fallback to local CSV/Parquet |
|
||||
|
||||
**Data loader with fallback:**
|
||||
|
||||
```python
|
||||
class DataLoader:
|
||||
def load_data(self, start_date, end_date, force_refresh=False):
|
||||
# 1. Try cache
|
||||
if self.cache and not force_refresh:
|
||||
cached = self.cache.get(start_date, end_date)
|
||||
if cached is not None:
|
||||
return cached, "cache"
|
||||
|
||||
# 2. Try Snowflake
|
||||
try:
|
||||
df = self.snowflake.fetch_activity_data(start_date, end_date)
|
||||
self.cache.set(df, start_date, end_date)
|
||||
return df, "snowflake"
|
||||
except SnowflakeConnectionError:
|
||||
pass
|
||||
|
||||
# 3. Fallback to local files
|
||||
if self.fallback_file.exists():
|
||||
return pd.read_parquet(self.fallback_file), "local_file"
|
||||
|
||||
raise RuntimeError("No data source available")
|
||||
```
|
||||
|
||||
**Dependencies to add:**
|
||||
|
||||
```toml
|
||||
dependencies = [
|
||||
"snowflake-connector-python[pandas]>=3.12.0",
|
||||
"cryptography>=42.0.0",
|
||||
]
|
||||
```
|
||||
|
||||
### Effort Estimate
|
||||
- Snowflake connector setup: 2-3 days
|
||||
- Caching layer: 1-2 days
|
||||
- GUI integration (data source selector): 1-2 days
|
||||
- Testing with real data: 2-3 days
|
||||
|
||||
---
|
||||
|
||||
## 4. GP Diagnosis Code Integration
|
||||
|
||||
### What
|
||||
Use GP diagnosis codes as the **primary source** for directory/specialty assignment, with existing logic as fallback.
|
||||
|
||||
### Why
|
||||
- More accurate: Diagnosis directly indicates specialty
|
||||
- Reduces "Undefined" assignments
|
||||
- Leverages existing NHS data linkage
|
||||
- Maintains current logic as safety net
|
||||
|
||||
### How
|
||||
|
||||
**NHS diagnosis code landscape:**
|
||||
|
||||
| Code System | Usage | Notes |
|
||||
|-------------|-------|-------|
|
||||
| **SNOMED CT** | GP systems (mandatory since 2018) | Primary source |
|
||||
| **ICD-10** | Secondary care | Maps FROM SNOMED CT |
|
||||
| **Read Codes** | Legacy only | Historical records |
|
||||
|
||||
**New priority chain:**
|
||||
|
||||
```
|
||||
1. Drug has single valid directory → use that (unchanged)
|
||||
2. [NEW] GP diagnosis available → map SNOMED/ICD-10 to directory
|
||||
3. Extract from clinical data fields (existing)
|
||||
4. Most frequent for same patient/drug (existing)
|
||||
5. UPID-based inference (existing)
|
||||
6. Default to "Undefined" (existing)
|
||||
```
|
||||
|
||||
**ICD-10 to Directory mapping (examples):**
|
||||
|
||||
```python
|
||||
ICD10_TO_DIRECTORY = {
|
||||
# Neoplasms (Chapter II)
|
||||
"C": ["MEDICAL ONCOLOGY", "CLINICAL ONCOLOGY", "CLINICAL HAEMATOLOGY"],
|
||||
|
||||
# Blood diseases (Chapter III)
|
||||
"D5": ["CLINICAL HAEMATOLOGY"],
|
||||
"D6": ["CLINICAL HAEMATOLOGY"],
|
||||
|
||||
# Endocrine (Chapter IV)
|
||||
"E10": ["DIABETIC MEDICINE"], # Type 1 diabetes
|
||||
"E11": ["DIABETIC MEDICINE"], # Type 2 diabetes
|
||||
|
||||
# Eye (Chapter VII)
|
||||
"H0": ["OPHTHALMOLOGY"],
|
||||
"H1": ["OPHTHALMOLOGY"],
|
||||
"H2": ["OPHTHALMOLOGY"],
|
||||
"H3": ["OPHTHALMOLOGY"],
|
||||
|
||||
# Musculoskeletal (Chapter XIII)
|
||||
"M05": ["RHEUMATOLOGY"], # Rheumatoid arthritis
|
||||
"M06": ["RHEUMATOLOGY"],
|
||||
"M32": ["RHEUMATOLOGY"], # SLE
|
||||
|
||||
# Genitourinary (Chapter XIV)
|
||||
"N0": ["NEPHROLOGY"],
|
||||
"N1": ["NEPHROLOGY"],
|
||||
"N18": ["NEPHROLOGY"], # CKD
|
||||
}
|
||||
```
|
||||
|
||||
**Multi-diagnosis resolution:**
|
||||
|
||||
```python
|
||||
def resolve_directory_from_diagnoses(diagnoses, drug_valid_dirs):
|
||||
"""
|
||||
When patient has multiple diagnoses:
|
||||
1. Filter to diagnoses mapping to directories valid for this drug
|
||||
2. Oncology diagnoses take priority (ICD-10 chapter C)
|
||||
3. Use most recent active diagnosis
|
||||
4. Default to first alphabetically (deterministic)
|
||||
"""
|
||||
valid_matches = []
|
||||
|
||||
for dx in diagnoses:
|
||||
icd10_prefix = dx.icd10_code[:3]
|
||||
possible_dirs = ICD10_TO_DIRECTORY.get(icd10_prefix, [])
|
||||
matching = set(possible_dirs) & set(drug_valid_dirs)
|
||||
|
||||
if matching:
|
||||
valid_matches.append({
|
||||
'directories': matching,
|
||||
'is_oncology': dx.icd10_code.startswith('C'),
|
||||
'date': dx.diagnosis_date
|
||||
})
|
||||
|
||||
if not valid_matches:
|
||||
return None # Fall back to existing logic
|
||||
|
||||
# Oncology priority
|
||||
oncology = [m for m in valid_matches if m['is_oncology']]
|
||||
if oncology:
|
||||
return sorted(oncology[0]['directories'])[0]
|
||||
|
||||
# Most recent
|
||||
valid_matches.sort(key=lambda x: x['date'], reverse=True)
|
||||
return sorted(valid_matches[0]['directories'])[0]
|
||||
```
|
||||
|
||||
**Data source options:**
|
||||
|
||||
1. **Snowflake linked data** (recommended): Query `data_hub.dwh.DimClinicalCoding` joined via `PatientPseudo`
|
||||
2. **Local CSV cache**: Pre-extracted GP diagnosis data for offline use
|
||||
3. **Hybrid**: Cache with Snowflake refresh
|
||||
|
||||
**GP Diagnosis Query (confirm column names via Snowflake MCP):**
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
PatientPseudo,
|
||||
SNOMEDCode, -- or similar
|
||||
ICD10Code, -- may need mapping from SNOMED
|
||||
DiagnosisDate,
|
||||
DiagnosisStatus -- Active/Resolved if available
|
||||
FROM data_hub.dwh.DimClinicalCoding
|
||||
WHERE PatientPseudo IN (:patient_pseudo_list)
|
||||
ORDER BY DiagnosisDate DESC
|
||||
```
|
||||
|
||||
**New reference file needed (`./data/diagnosis_directory_map.csv`):**
|
||||
|
||||
```csv
|
||||
icd10_prefix,directory,priority,notes
|
||||
C,MEDICAL ONCOLOGY,1,All malignancies
|
||||
C81,CLINICAL HAEMATOLOGY,1,Hodgkin lymphoma
|
||||
C90,CLINICAL HAEMATOLOGY,1,Multiple myeloma
|
||||
E10,DIABETIC MEDICINE,1,Type 1 diabetes
|
||||
E11,DIABETIC MEDICINE,1,Type 2 diabetes
|
||||
G35,NEUROLOGY,1,Multiple sclerosis
|
||||
H0,OPHTHALMOLOGY,1,Eye disorders
|
||||
M05,RHEUMATOLOGY,1,Rheumatoid arthritis
|
||||
N18,NEPHROLOGY,1,Chronic kidney disease
|
||||
```
|
||||
|
||||
**Tracking assignment source (for audit):**
|
||||
|
||||
```python
|
||||
df['Directory_Source'] = pd.NA # New column
|
||||
|
||||
# After each assignment step:
|
||||
df.loc[assigned_mask, 'Directory_Source'] = 'DRUG_SINGLE' # Step 1
|
||||
df.loc[assigned_mask, 'Directory_Source'] = 'GP_DIAGNOSIS' # Step 2 (NEW)
|
||||
df.loc[assigned_mask, 'Directory_Source'] = 'CLINICAL_EXTRACT' # Step 3
|
||||
# ... etc
|
||||
```
|
||||
|
||||
### Prerequisites
|
||||
- Explore `data_hub.dwh.DimClinicalCoding` schema to confirm exact column names (use Snowflake MCP)
|
||||
- Map `PatientPseudo` to your HCD data (may need to add PatientPseudo to your data extract)
|
||||
- Obtain SNOMED CT to ICD-10 mapping table from NHS TRUD (if DimClinicalCoding only has SNOMED)
|
||||
|
||||
### Effort Estimate
|
||||
- Mapping table creation: 2-3 days
|
||||
- Snowflake GP query development: 2-3 days
|
||||
- Integration with existing logic: 2-3 days
|
||||
- Validation and testing: 3-5 days
|
||||
|
||||
---
|
||||
|
||||
## 5. Code Quality Improvements
|
||||
|
||||
### What
|
||||
Modernize the codebase with better structure, type hints, error handling, and testing.
|
||||
|
||||
### Why
|
||||
- `generate_graph()` is 267 lines with complexity >30
|
||||
- Zero type hints across entire codebase
|
||||
- Global variables create hidden state
|
||||
- No automated tests
|
||||
- Print statements instead of logging
|
||||
|
||||
### How
|
||||
|
||||
**Quick wins (implement first):**
|
||||
|
||||
1. **Replace global variables** with dataclass:
|
||||
```python
|
||||
@dataclass
|
||||
class AnalysisFilters:
|
||||
start_date: date
|
||||
end_date: date
|
||||
last_seen: date
|
||||
minimum_patients: int
|
||||
selected_trusts: list[str]
|
||||
selected_drugs: list[str]
|
||||
selected_directories: list[str]
|
||||
custom_title: str = ""
|
||||
|
||||
def validate(self) -> list[str]:
|
||||
errors = []
|
||||
if self.start_date >= self.end_date:
|
||||
errors.append("Start date must be before end date")
|
||||
return errors
|
||||
```
|
||||
|
||||
2. **Externalize configuration:**
|
||||
```python
|
||||
@dataclass
|
||||
class PathConfig:
|
||||
data_dir: Path = Path("./data")
|
||||
|
||||
@property
|
||||
def drug_names_file(self) -> Path:
|
||||
return self.data_dir / "include.csv"
|
||||
|
||||
@property
|
||||
def org_codes_file(self) -> Path:
|
||||
return self.data_dir / "org_codes.csv"
|
||||
|
||||
# ... etc for all 7 reference files
|
||||
|
||||
def validate(self) -> list[str]:
|
||||
"""Check all required files exist at startup."""
|
||||
errors = []
|
||||
for file_path in [self.drug_names_file, self.org_codes_file, ...]:
|
||||
if not file_path.exists():
|
||||
errors.append(f"Required file not found: {file_path}")
|
||||
return errors
|
||||
```
|
||||
|
||||
3. **Add logging:**
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
|
||||
handlers=[
|
||||
logging.FileHandler("./logs/analysis.log"),
|
||||
logging.StreamHandler()
|
||||
]
|
||||
)
|
||||
logger = logging.getLogger("PatientPathway")
|
||||
|
||||
# Replace all print() with:
|
||||
logger.info("Starting analysis...")
|
||||
logger.error(f"Failed to load file: {e}")
|
||||
```
|
||||
|
||||
4. **Extract `generate_graph()` into smaller functions:**
|
||||
```python
|
||||
def generate_graph(df, filters: AnalysisFilters, config: PathConfig):
|
||||
df = prepare_data(df, filters) # ~50 lines
|
||||
stats = calculate_statistics(df) # ~80 lines
|
||||
hierarchy = build_hierarchy(df, stats) # ~60 lines
|
||||
chart_data = prepare_chart_data(hierarchy) # ~40 lines
|
||||
return render_icicle_chart(chart_data, filters.custom_title) # ~40 lines
|
||||
```
|
||||
|
||||
**Recommended project structure:**
|
||||
|
||||
```
|
||||
project/
|
||||
├── gui.py # Entry point only
|
||||
├── core/
|
||||
│ ├── config.py # PathConfig, AnalysisFilters
|
||||
│ ├── models.py # Data models
|
||||
│ └── exceptions.py # Custom exceptions
|
||||
├── data_processing/
|
||||
│ ├── loader.py # File/Snowflake loading
|
||||
│ ├── transformer.py # Data transformations
|
||||
│ └── validator.py # Data validation
|
||||
├── analysis/
|
||||
│ ├── pathway_analyzer.py # Patient pathway calculations
|
||||
│ └── statistics.py # Statistical calculations
|
||||
├── visualization/
|
||||
│ └── plotly_generator.py # Graph generation
|
||||
└── tests/
|
||||
├── test_data_processing.py
|
||||
├── test_analysis.py
|
||||
└── test_config.py
|
||||
```
|
||||
|
||||
**Add development dependencies:**
|
||||
|
||||
```toml
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.0.0",
|
||||
"pytest-cov>=4.1.0",
|
||||
"mypy>=1.8.0",
|
||||
"black>=24.0.0",
|
||||
"ruff>=0.2.0",
|
||||
]
|
||||
```
|
||||
|
||||
**Priority tests to write:**
|
||||
|
||||
```python
|
||||
# tests/test_data_processing.py
|
||||
def test_drop_duplicate_treatments_ascending():
|
||||
"""Verify first intervention kept when ascending=True."""
|
||||
# ...
|
||||
|
||||
def test_drop_duplicate_treatments_descending():
|
||||
"""Verify last intervention kept when ascending=False."""
|
||||
# ...
|
||||
|
||||
# tests/test_config.py
|
||||
def test_path_config_validates_missing_files():
|
||||
"""Verify validation catches missing reference files."""
|
||||
# ...
|
||||
|
||||
def test_analysis_filters_validates_date_range():
|
||||
"""Verify start date must be before end date."""
|
||||
# ...
|
||||
```
|
||||
|
||||
### Effort Estimate
|
||||
- Dataclasses and config: 1-2 days
|
||||
- Logging setup: 0.5 days
|
||||
- Extract `generate_graph()`: 2-3 days
|
||||
- Add type hints (public API): 1-2 days
|
||||
- Basic test coverage: 2-3 days
|
||||
|
||||
---
|
||||
|
||||
## Implementation Roadmap
|
||||
|
||||
### Phase 1: Foundation (2-3 weeks)
|
||||
1. Create `PathConfig` and `AnalysisFilters` dataclasses
|
||||
2. Set up logging infrastructure
|
||||
3. Design and create SQLite schema
|
||||
4. Migrate reference data CSVs to SQLite
|
||||
|
||||
### Phase 2: Data Layer (2-3 weeks)
|
||||
1. Implement Snowflake connector with SSO browser auth
|
||||
2. Build caching layer with TTL
|
||||
3. Create data loader with fallback chain
|
||||
4. Migrate `dashboard_gui.py` to use SQLite queries
|
||||
|
||||
### Phase 3: Diagnosis Integration (2-3 weeks)
|
||||
1. Explore `data_hub.dwh.DimClinicalCoding` schema via Snowflake MCP
|
||||
2. Create ICD-10 to directory mapping table
|
||||
3. Implement GP diagnosis lookup using `PatientPseudo` linkage
|
||||
4. Integrate into `department_identification()` as step 2
|
||||
5. Add `Directory_Source` tracking column
|
||||
|
||||
### Phase 4: GUI Modernization (3-4 weeks)
|
||||
1. Learn Reflex fundamentals
|
||||
2. Recreate main window and navigation with `rx.vstack`/`rx.hstack`
|
||||
3. Implement filter panels (date pickers, checkbox groups)
|
||||
4. Integrate Plotly charts with native `rx.plotly()` component
|
||||
5. Test with `reflex run`
|
||||
|
||||
### Phase 5: Quality & Polish (1-2 weeks)
|
||||
1. Add type hints to public API
|
||||
2. Write priority unit tests
|
||||
3. Extract `generate_graph()` into smaller functions
|
||||
4. Documentation and cleanup
|
||||
|
||||
---
|
||||
|
||||
## Configuration Decisions
|
||||
|
||||
Based on requirements, the following decisions have been made:
|
||||
|
||||
| Question | Decision |
|
||||
|----------|----------|
|
||||
| **Snowflake auth** | SSO browser login (`authenticator='externalbrowser'`) |
|
||||
| **GP diagnosis data** | `data_hub.dwh.DimClinicalCoding` |
|
||||
| **Patient linkage** | Use `PatientPseudo` (anonymized identifier) - NOT UPID |
|
||||
| **Plotly interactivity** | Must be interactive - **Reflex has native `rx.plotly()` component** |
|
||||
| **Distribution** | Python script (`reflex run`) - no .exe needed |
|
||||
|
||||
### Implications
|
||||
|
||||
**Snowflake SSO**: Connection code becomes:
|
||||
```python
|
||||
conn = snowflake.connector.connect(
|
||||
account="your_account.region",
|
||||
user=os.environ.get("SNOWFLAKE_USER"),
|
||||
authenticator="externalbrowser", # Opens browser for SSO
|
||||
warehouse="ANALYTICS_WH",
|
||||
database="data_hub",
|
||||
schema="dwh"
|
||||
)
|
||||
```
|
||||
|
||||
**Patient Linkage**: The GP diagnosis query needs to join on `PatientPseudo`, not UPID:
|
||||
```sql
|
||||
SELECT
|
||||
cc.PatientPseudo,
|
||||
cc.SNOMEDCode, -- Confirm actual column names
|
||||
cc.ICD10Code,
|
||||
cc.DiagnosisDate
|
||||
FROM data_hub.dwh.DimClinicalCoding cc
|
||||
WHERE cc.PatientPseudo IN (:patient_list)
|
||||
```
|
||||
|
||||
**Note**: You'll need to confirm the exact column names in `DimClinicalCoding` - explore via Snowflake MCP or SQL client.
|
||||
|
||||
**Plotly Interactivity**: Reflex solves this elegantly with native embedding:
|
||||
```python
|
||||
# Interactive Plotly chart directly in the Reflex app
|
||||
rx.plotly(data=State.chart_data, layout=chart_layout)
|
||||
```
|
||||
Full interactivity (zoom, pan, hover tooltips) works in the browser-based app - no external HTML files needed.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Reflex Documentation](https://reflex.dev/docs/)
|
||||
- [Reflex Plotly Component](https://reflex.dev/docs/library/graphing/plotly/)
|
||||
- [Flet Documentation](https://flet.dev/docs/) (alternative)
|
||||
- [Snowflake Python Connector](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector)
|
||||
- [NHS SNOMED CT](https://digital.nhs.uk/services/terminology-and-classifications/snomed-ct)
|
||||
- [NHS ICD-10 Classifications](https://isd.digital.nhs.uk/trud/users/guest/filters/0/categories/28)
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
# Ralph Wiggum Loop - Reflex UI Redesign
|
||||
|
||||
You are operating inside an automated loop building a Reflex frontend application. Each iteration you receive fresh context — you have NO memory of previous iterations. Your only memory is the filesystem.
|
||||
|
||||
## First Actions Every Iteration
|
||||
|
||||
Read these files in this order before doing anything else:
|
||||
|
||||
1. `progress.txt` — What previous iterations accomplished, what's blocked, and what to do next. The most recent entry is most important.
|
||||
2. `IMPLEMENTATION_PLAN.md` — Task list with status markers, project overview, and completion criteria.
|
||||
3. `guardrails.md` — Known failure patterns to avoid. You MUST read and follow these.
|
||||
4. `DESIGN_SYSTEM.md` — Color palette, typography, spacing, and component specifications.
|
||||
|
||||
Then run `git log --oneline -5` to see recent commits.
|
||||
|
||||
## Narration
|
||||
|
||||
Narrate your work as you go. Your output is the only visibility the operator has into what's happening. For every significant action, explain what you're doing and why:
|
||||
|
||||
- **Reading files**: "Reading progress.txt to check what the last iteration accomplished..."
|
||||
- **Creating components**: "Creating the top_bar() component with logo, title, and chart tabs..."
|
||||
- **Debugging**: "Reflex compilation failed with TypeError. Checking the error — looks like rx.foreach issue..."
|
||||
- **Testing**: "Running reflex compile to verify the component renders..."
|
||||
- **Making decisions**: "The design system specifies Primary Blue #0066CC for buttons. Using that."
|
||||
- **Committing**: "Committing styles.py — design token module complete."
|
||||
|
||||
Do NOT just output a summary at the end. Narrate throughout. Think of this as a live log of your reasoning.
|
||||
|
||||
## Task Selection
|
||||
|
||||
Pick the highest-priority task that is READY to work on:
|
||||
|
||||
1. Read ALL tasks in IMPLEMENTATION_PLAN.md — understand the full picture
|
||||
2. Skip any marked `[x]` (complete) or `[B]` (blocked)
|
||||
3. Check progress.txt for guidance — if the previous iteration recommended a specific next task, prefer that unless it's blocked
|
||||
4. If no guidance exists, pick the first `[ ]` (ready) task in the first incomplete phase
|
||||
5. Mark your chosen task `[~]` (in progress) in IMPLEMENTATION_PLAN.md
|
||||
|
||||
If your chosen task turns out to be blocked during work:
|
||||
- Mark it `[B]` with a reason in IMPLEMENTATION_PLAN.md
|
||||
- Document the blocker in progress.txt
|
||||
- Move to the next ready task within this same iteration
|
||||
|
||||
## Development
|
||||
|
||||
Work on ONE task per iteration. Build incrementally and verify as you go.
|
||||
|
||||
### Code Patterns
|
||||
|
||||
- **Use design tokens**: Import from `pathways_app/styles.py` — never hardcode colors/spacing
|
||||
- **Reflex Vars in rx.foreach**: Use `.to(int)` for comparisons, `.to_string()` for text interpolation
|
||||
- **Component functions**: Each component should be a function returning `rx.Component`
|
||||
- **State class**: All reactive state goes in the `AppState` class
|
||||
- **Computed properties**: Use `@rx.var` decorator for derived values
|
||||
|
||||
### Verification Steps
|
||||
|
||||
After writing code, ALWAYS verify:
|
||||
|
||||
1. **Syntax check**: `python -m py_compile pathways_app/app_v2.py`
|
||||
2. **Import check**: `python -c "from pathways_app.app_v2 import app"`
|
||||
3. **Reflex compile**: Run `reflex run` briefly to check for compilation errors
|
||||
|
||||
If any step fails, fix the issue before proceeding.
|
||||
|
||||
## Validation Protocol
|
||||
|
||||
Every task MUST pass validation before being marked complete:
|
||||
|
||||
### Tier 1: Code Validation (MANDATORY)
|
||||
- Code compiles without Python syntax errors
|
||||
- Reflex compiles the app without errors
|
||||
- No TypeErrors, ImportErrors, or AttributeErrors
|
||||
|
||||
### Tier 2: Visual Validation (MANDATORY for UI tasks)
|
||||
- Component renders in the browser
|
||||
- Styling matches DESIGN_SYSTEM.md specifications
|
||||
- Responsive behavior works (if applicable)
|
||||
|
||||
### Tier 3: Functional Validation (MANDATORY for state/logic tasks)
|
||||
- State changes trigger expected UI updates
|
||||
- Computed properties return correct values
|
||||
- Filters produce expected data transformations
|
||||
|
||||
### Validation Failure
|
||||
|
||||
If any tier fails:
|
||||
- DO NOT mark the task complete
|
||||
- Document the failure details in progress.txt
|
||||
- Fix the issue within this iteration if possible
|
||||
- If you cannot fix it, mark the task `[B]` with details
|
||||
|
||||
## Quality Gates
|
||||
|
||||
Before marking ANY task `[x]`, ALL of these must be true:
|
||||
|
||||
1. Code is saved to the appropriate file(s)
|
||||
2. Tier 1 code validation passed
|
||||
3. Tier 2/3 validation passed (as applicable)
|
||||
4. Design tokens used — no hardcoded colors, fonts, or spacing
|
||||
5. All changes committed to git with a descriptive message
|
||||
|
||||
These are non-negotiable. A task that "feels done" but hasn't passed all gates is NOT done.
|
||||
|
||||
## Update Progress
|
||||
|
||||
After completing your work (whether the task succeeded, failed, or was blocked), append to progress.txt using this format:
|
||||
|
||||
```
|
||||
## Iteration [N] — [YYYY-MM-DD]
|
||||
### Task: [which task you worked on]
|
||||
### Status: COMPLETE | BLOCKED | IN PROGRESS
|
||||
### What was done:
|
||||
- [Specific actions taken]
|
||||
### Validation results:
|
||||
- Tier 1 (Code): [syntax check, import check, reflex compile]
|
||||
- Tier 2 (Visual): [what was checked visually, or N/A]
|
||||
- Tier 3 (Functional): [what logic was tested, or N/A]
|
||||
### Files changed:
|
||||
- [list of files created/modified]
|
||||
### Committed: [git hash] "[commit message]"
|
||||
### Patterns discovered:
|
||||
- [Any reusable learnings — Reflex quirks, component patterns]
|
||||
### Next iteration should:
|
||||
- [Explicit guidance for what the next fresh instance should do first]
|
||||
- [Note any context that would be lost without writing it here]
|
||||
### Blocked items:
|
||||
- [Any tasks that are blocked and why]
|
||||
```
|
||||
|
||||
If you discover a failure pattern that future iterations should avoid, add it to `guardrails.md`.
|
||||
|
||||
## Commit Changes
|
||||
|
||||
1. Stage changed files (styles.py, app_v2.py, etc.)
|
||||
2. Use a descriptive commit message referencing the task (e.g., "feat: create design tokens module")
|
||||
3. Commit after your task is validated and complete — one commit per logical unit of work
|
||||
4. If you updated progress.txt with a blocked status, commit that too
|
||||
|
||||
## Completion Check
|
||||
|
||||
If ALL tasks in IMPLEMENTATION_PLAN.md are marked `[x]`:
|
||||
|
||||
1. Run `reflex run` and verify the app works end-to-end
|
||||
2. Verify all completion criteria at the bottom of IMPLEMENTATION_PLAN.md are satisfied
|
||||
3. Only then output the completion signal on its own line:
|
||||
|
||||
```
|
||||
<promise>COMPLETE</promise>
|
||||
```
|
||||
|
||||
DO NOT output this string under any other circumstances.
|
||||
DO NOT output it if any task is still `[ ]` or `[B]` or `[~]`.
|
||||
DO NOT paraphrase, vary, or conditionally output this string.
|
||||
|
||||
## Rules
|
||||
|
||||
- Complete ONE task per iteration, then update progress and stop
|
||||
- ALWAYS read progress.txt, guardrails.md, and DESIGN_SYSTEM.md before starting work
|
||||
- **Use design tokens** — never hardcode hex colors, pixel values, or font names
|
||||
- **Reflex Var safety** — use `.to()` methods when working with Vars from rx.foreach or computed properties
|
||||
- Keep commits atomic and well-described
|
||||
- If stuck on the same issue for more than 2 attempts within one iteration, document it in progress.txt and move to the next ready task
|
||||
- When in doubt, check the existing `pathways_app.py` for patterns that work
|
||||
- The goal is a working, beautiful app — correctness and visual quality matter equally
|
||||
@@ -0,0 +1,229 @@
|
||||
# NHS High-Cost Drug Patient Pathway Analysis Tool
|
||||
|
||||
A web-based application for analyzing secondary care patient treatment pathways. It processes clinical activity data to visualize hierarchical treatment patterns (Trust → Directory/Specialty → Drug → Patient pathway) as interactive Plotly icicle charts.
|
||||
|
||||
## Features
|
||||
|
||||
- **Interactive Visualization**: Plotly icicle charts showing patient treatment hierarchies with cost and frequency statistics
|
||||
- **Multi-Source Data Loading**: CSV/Parquet files, SQLite database, or direct Snowflake integration
|
||||
- **GP Diagnosis Validation**: Validate patient indications against GP SNOMED codes via NHS Snowflake
|
||||
- **Modern Web Interface**: Browser-based UI using Reflex framework with NHS branding
|
||||
- **Flexible Filtering**: Filter by date range, NHS trusts, drugs, and medical directories
|
||||
- **Export Options**: Export charts as interactive HTML or data as CSV
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.10 or higher
|
||||
- pip or uv package manager
|
||||
|
||||
### Optional (for Snowflake integration)
|
||||
- `snowflake-connector-python` package
|
||||
- Access to NHS Snowflake data warehouse with SSO authentication
|
||||
|
||||
## Installation
|
||||
|
||||
### Using pip
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone <repository-url>
|
||||
cd patient-pathway-analysis
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### Using uv (recommended)
|
||||
|
||||
```bash
|
||||
# Install uv if not already installed
|
||||
pip install uv
|
||||
|
||||
# Sync dependencies
|
||||
uv sync
|
||||
```
|
||||
|
||||
### Install with test dependencies
|
||||
|
||||
```bash
|
||||
pip install -e ".[test]"
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Run the Web Application (Recommended)
|
||||
|
||||
```bash
|
||||
reflex run
|
||||
```
|
||||
|
||||
Open http://localhost:3000 in your browser.
|
||||
|
||||
## Usage
|
||||
|
||||
### Web Interface (Reflex)
|
||||
|
||||
1. **Load Data**: On the home page, select your data source:
|
||||
- **SQLite Database**: Uses pre-loaded data from `data/pathways.db`
|
||||
- **File Upload**: Drag and drop a CSV or Parquet file
|
||||
- **Snowflake**: Fetch data directly from NHS Snowflake (requires configuration)
|
||||
|
||||
2. **Configure Filters**:
|
||||
- Set date range (Start Date, End Date, Last Seen After)
|
||||
- Navigate to Drug/Trust/Directory selection pages using the sidebar
|
||||
- Use search boxes to find and select items
|
||||
- Set minimum patient threshold to filter small groups
|
||||
|
||||
3. **Run Analysis**: Click "Run Analysis" to generate the icicle chart
|
||||
|
||||
4. **Export Results**:
|
||||
- **Export HTML**: Save the interactive chart as a standalone HTML file
|
||||
- **Export CSV**: Export the filtered data as a CSV file
|
||||
|
||||
### Data Migration
|
||||
|
||||
To populate the SQLite database from CSV files:
|
||||
|
||||
```bash
|
||||
# Initialize database schema
|
||||
python -m data_processing.migrate
|
||||
|
||||
# Load reference data from CSV files
|
||||
python -m data_processing.migrate --reference-data --verify
|
||||
|
||||
# Load patient data from a CSV/Parquet file
|
||||
python -m data_processing.migrate --load-patient-data path/to/data.csv
|
||||
```
|
||||
|
||||
### Snowflake Configuration
|
||||
|
||||
To use Snowflake integration, edit `config/snowflake.toml`:
|
||||
|
||||
```toml
|
||||
[connection]
|
||||
account = "your-account-identifier"
|
||||
warehouse = "your-warehouse"
|
||||
database = "DATA_HUB"
|
||||
schema = "CDM"
|
||||
authenticator = "externalbrowser" # NHS SSO authentication
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── core/ # Core configuration and models
|
||||
├── data_processing/ # Data layer (SQLite, Snowflake, loaders)
|
||||
├── analysis/ # Analysis pipeline (refactored from generate_graph)
|
||||
├── visualization/ # Chart generation (Plotly)
|
||||
├── pathways_app/ # Reflex web application
|
||||
├── tools/ # Legacy modules (original analysis engine)
|
||||
├── config/ # Configuration files
|
||||
├── data/ # Reference data and SQLite database
|
||||
├── docs/ # Additional documentation
|
||||
└── tests/ # Test suite
|
||||
```
|
||||
|
||||
See `CLAUDE.md` for detailed architecture documentation.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [docs/USER_GUIDE.md](docs/USER_GUIDE.md) - End-user guide for using the web interface
|
||||
- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) - Production deployment guide (Docker, nginx, cloud)
|
||||
- [CLAUDE.md](CLAUDE.md) - Technical architecture documentation for developers
|
||||
|
||||
## Deployment
|
||||
|
||||
Quick production start:
|
||||
|
||||
```bash
|
||||
# Run in production mode
|
||||
reflex run --env prod
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python -m pytest tests/ -v
|
||||
|
||||
# Run with coverage
|
||||
python -m pytest tests/ -v --cov=core --cov=data_processing --cov=analysis
|
||||
|
||||
# Run only fast tests (exclude slow/integration)
|
||||
python -m pytest tests/ -v -m "not slow"
|
||||
```
|
||||
|
||||
## Reference Data Files
|
||||
|
||||
The `data/` directory contains essential reference files:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `include.csv` | Drug filter list with default selections |
|
||||
| `defaultTrusts.csv` | NHS Trust list for filtering |
|
||||
| `directory_list.csv` | Medical specialties/directories |
|
||||
| `drugnames.csv` | Drug name standardization mapping |
|
||||
| `org_codes.csv` | Provider code to organization name mapping |
|
||||
| `drug_directory_list.csv` | Valid drug-to-directory mappings |
|
||||
| `drug_indication_clusters.csv` | Drug to SNOMED cluster mappings |
|
||||
| `ta-recommendations.xlsx` | NICE TA recommendations |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Reflex compilation errors
|
||||
|
||||
If you encounter compilation errors when running `reflex run`:
|
||||
|
||||
```bash
|
||||
# Clear the build cache and restart
|
||||
rm -rf .web
|
||||
reflex run
|
||||
```
|
||||
|
||||
### Snowflake connection issues
|
||||
|
||||
1. Ensure `snowflake-connector-python` is installed:
|
||||
```bash
|
||||
pip install snowflake-connector-python
|
||||
```
|
||||
|
||||
2. Check that `config/snowflake.toml` has the correct account identifier
|
||||
|
||||
3. For SSO authentication, a browser window will open automatically
|
||||
|
||||
### SQLite database not found
|
||||
|
||||
If `data/pathways.db` doesn't exist, create it:
|
||||
|
||||
```bash
|
||||
python -m data_processing.migrate
|
||||
python -m data_processing.migrate --reference-data
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Code Quality
|
||||
|
||||
```bash
|
||||
# Type checking
|
||||
python -m mypy core/ data_processing/ analysis/ --ignore-missing-imports
|
||||
|
||||
# Run tests with coverage report
|
||||
python -m pytest tests/ -v --cov=core --cov=data_processing --cov-report=html
|
||||
```
|
||||
|
||||
### Adding New Reference Data
|
||||
|
||||
1. Add CSV file to `data/` directory
|
||||
2. Define schema in `data_processing/schema.py`
|
||||
3. Create migration function in `data_processing/reference_data.py`
|
||||
4. Add path to `PathConfig` in `core/config.py`
|
||||
|
||||
## License
|
||||
|
||||
Internal NHS use only. Not for distribution.
|
||||
|
||||
## Support
|
||||
|
||||
For questions or issues, contact the Medicines Intelligence team.
|
||||
@@ -0,0 +1,192 @@
|
||||
# Snowflake Reference
|
||||
|
||||
Essential database context for querying NHS data. Read this every iteration when working with Snowflake.
|
||||
|
||||
---
|
||||
|
||||
## Snowflake MCP Server
|
||||
|
||||
Use `mcp__snowflake-mcp__*` functions to explore schema and test queries.
|
||||
|
||||
### Schema Discovery (USE THESE FIRST)
|
||||
- `test_connection()` - Verify connectivity
|
||||
- `list_databases()` - List accessible databases
|
||||
- `list_schemas(database_name)` - List schemas in a database
|
||||
- `list_tables(database, schema)` - List tables with descriptions
|
||||
- `list_views(schema_name, database)` - List views with descriptions
|
||||
- `describe_table(table_name, database)` - Get detailed table schema
|
||||
- `describe_query(query, database)` - Preview query output columns without execution
|
||||
|
||||
### Query Execution
|
||||
- `read_data(query, database, max_rows)` - Execute SELECT queries with row limits
|
||||
- `read_data_paginated(query, database, page_size, page)` - Paginated results with total count
|
||||
- `read_data_pandas(query, database, max_rows, output_format)` - Results in pandas-friendly formats
|
||||
|
||||
### Async Query Support (long-running queries)
|
||||
- `execute_async(query, database)` - Submit asynchronously, returns query_id
|
||||
- `get_query_status(query_id, database)` - Check status
|
||||
- `get_async_results(query_id, database, max_rows)` - Retrieve results
|
||||
|
||||
### Usage Guidelines
|
||||
- **ALWAYS** verify table structures and column names via MCP before writing queries
|
||||
- Test with small result sets (`LIMIT 20`) before full execution
|
||||
- Use `describe_query` to preview complex query outputs before running
|
||||
- Use async queries for operations expected to take >30 seconds
|
||||
|
||||
---
|
||||
|
||||
## Database Overview
|
||||
|
||||
| Database | Purpose |
|
||||
|----------|---------|
|
||||
| `DATA_HUB` | **Analyst-curated** data warehouse - primary source for most queries |
|
||||
| `PRIMARY_CARE` | Raw extracts from EMIS and TPP clinical systems |
|
||||
| `NATIONAL` | NHS England national datasets (SUS, ECDS, MHSDS, etc.) |
|
||||
| `FACTS_AND_DIMENSIONS_ALL_DATA` | External reference data (BNF, SNOMED, QOF clusters) |
|
||||
| `REPORTING_DATASETS_ICB` | Reporting outputs and analyst workspaces (includes SCRATCHPAD) |
|
||||
|
||||
**Avoid**: `SYSTEM` database.
|
||||
|
||||
---
|
||||
|
||||
## Key Tables and Views
|
||||
|
||||
### DATA_HUB.DWH (Dimensions)
|
||||
|
||||
| View | Purpose | Key Columns |
|
||||
|------|---------|-------------|
|
||||
| `DimMedicineAndDevice` | Master medication/device reference | `ProductSnomedCode`, `TherapeuticMoietySnomedCode` (VTM), `BNFParagraphCode`, `StrengthDescription`, `ProductDescription` |
|
||||
| `DimPerson` | Patient demographics | `PatientPseudonym`, `PersonKey`, `CurrentGeneralPractice`, `IsCurrentNWRegistered`, `YearMonthBirth` |
|
||||
| `DimSnomedCode` | SNOMED code descriptions | `SnomedCode`, `SnomedDescription` |
|
||||
| `DimOrganisationAndSite` | GP practices and NHS orgs | `SiteCode`, `OrganisationName`, `OrganisationSubType`, `IsSiteNorfolkAndWaveney`, `IsSiteActive` |
|
||||
| `DimDate` | Date dimension | |
|
||||
| `DimCondition` | Clinical conditions | Long-term condition flags |
|
||||
| `DimDeprivation` | Deprivation rankings by area | |
|
||||
|
||||
**CRITICAL**:
|
||||
- `ProductDescription` is the correct column for product names. `ProductName` does NOT exist.
|
||||
- `IsLatest` does NOT exist in `DimMedicineAndDevice`.
|
||||
|
||||
### DATA_HUB.CDM (Common Data Model)
|
||||
|
||||
| View | Purpose | Key Columns |
|
||||
|------|---------|-------------|
|
||||
| `Acute__Conmon__PatientLevelDrugs` | HCD activity data | `PseudoNHSNoLinked`, `InterventionDate`, `DrugName`, `Price Actual` |
|
||||
|
||||
**Note**: HCD `PseudoNHSNoLinked` = GP `PatientPseudonym` for patient linkage.
|
||||
|
||||
### DATA_HUB.PHM (Population Health Management)
|
||||
|
||||
| View | Purpose | Key Columns |
|
||||
|------|---------|-------------|
|
||||
| `PrimaryCareClinicalCoding` | **Unified** clinical coding (EMIS + TPP, no duplicates) | `PatientPseudonym`, `SNOMEDCode`, `EventDateTime`, `NumericValue` |
|
||||
| `PrimaryCareMedication` | **Unified** medication data (EMIS + TPP, no duplicates) | `PatientPseudonym`, `SNOMEDCode`, `DateMedicationStart`, `Quantity` |
|
||||
| `ClinicalCodingClusterSnomedCodes` | SNOMED codes grouped by cluster | `ClusterId`, `SnomedCode` |
|
||||
| `PersonCohort` | Pre-defined patient cohorts | |
|
||||
|
||||
**Prefer DATA_HUB.PHM unified views** over raw PRIMARY_CARE tables.
|
||||
|
||||
---
|
||||
|
||||
## Patient Identifiers
|
||||
|
||||
| Identifier | Source | Usage |
|
||||
|------------|--------|-------|
|
||||
| `PatientPseudonym` | DATA_HUB, NATIONAL | Primary - use for most joins |
|
||||
| `PseudoNHSNoLinked` | DATA_HUB.CDM (HCD data) | Links to PatientPseudonym |
|
||||
| `PersonKey` | DATA_HUB.DWH.DimPerson | Integer key for person dimension |
|
||||
|
||||
### Standard Join Patterns
|
||||
```sql
|
||||
-- HCD Activity to GP Diagnosis
|
||||
FROM DATA_HUB.CDM."Acute__Conmon__PatientLevelDrugs" hcd
|
||||
LEFT JOIN DATA_HUB.PHM."PrimaryCareClinicalCoding" pcc
|
||||
ON hcd."PseudoNHSNoLinked" = pcc."PatientPseudonym"
|
||||
|
||||
-- Activity to Person Demographics
|
||||
FROM DATA_HUB.CDM."Acute__Conmon__PatientLevelDrugs" hcd
|
||||
INNER JOIN DATA_HUB.DWH."DimPerson" dp
|
||||
ON hcd."PseudoNHSNoLinked" = dp."PatientPseudonym"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL: Registered Population Filter
|
||||
|
||||
**ALWAYS** apply when counting patients:
|
||||
|
||||
```sql
|
||||
WHERE dp."IsCurrentNWRegistered" = 'Yes'
|
||||
AND dp."CurrentGeneralPractice" <> '*'
|
||||
```
|
||||
|
||||
Without this filter, counts will be ~2x inflated (includes deceased, deregistered, out-of-area patients).
|
||||
|
||||
---
|
||||
|
||||
## Query Development Patterns
|
||||
|
||||
### Clinical Condition Detection (GP SNOMED Clusters)
|
||||
```sql
|
||||
-- Get all SNOMED codes for a clinical cluster
|
||||
SELECT "SnomedCode"
|
||||
FROM DATA_HUB.PHM."ClinicalCodingClusterSnomedCodes"
|
||||
WHERE "ClusterId" = 'RARTH_COD' -- Rheumatoid arthritis
|
||||
|
||||
-- Check if patient has condition
|
||||
SELECT DISTINCT pcc."PatientPseudonym"
|
||||
FROM DATA_HUB.PHM."PrimaryCareClinicalCoding" pcc
|
||||
WHERE pcc."SNOMEDCode" IN (SELECT "SnomedCode" FROM cluster_codes)
|
||||
AND pcc."PatientPseudonym" IS NOT NULL
|
||||
```
|
||||
|
||||
### Available SNOMED Clusters for HCD Indications
|
||||
- `RARTH_COD` (155 codes) - Rheumatoid arthritis
|
||||
- `PSORIASIS_COD` (116 codes) - Psoriasis
|
||||
- `CROHNS_COD` (93 codes) - Crohn's disease
|
||||
- `ULCCOLITIS_COD` (62 codes) - Ulcerative colitis
|
||||
- `MS_COD` (44 codes) - Multiple sclerosis
|
||||
- `DM_COD` / `DMTYPE1_COD` / `DMTYPE2AUDIT_COD` - Diabetes
|
||||
|
||||
### Sample HCD Activity Query
|
||||
```sql
|
||||
SELECT
|
||||
hcd."PseudoNHSNoLinked" AS PatientPseudonym,
|
||||
hcd."DrugName",
|
||||
hcd."InterventionDate",
|
||||
hcd."Provider Code",
|
||||
hcd."OrganisationName"
|
||||
FROM DATA_HUB.CDM."Acute__Conmon__PatientLevelDrugs" hcd
|
||||
WHERE hcd."InterventionDate" >= '2024-01-01'
|
||||
LIMIT 20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Snowflake SQL Syntax
|
||||
|
||||
- Double-quote identifiers: `"PatientPseudonym"`
|
||||
- Date literals: `'2025-04-01'::DATE`
|
||||
- Date functions: `DATEADD('MONTH', -3, date)`, `DATEDIFF('YEAR', d1, d2)`, `LAST_DAY(date)`
|
||||
- Boolean: `TRUE`/`FALSE`
|
||||
- No `TOP N` - use `LIMIT N`
|
||||
- `COALESCE()`, `NULLIF()`, `GREATEST()` work as expected
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Column not found errors
|
||||
1. Use `describe_table(table_name, database)` to get actual column names
|
||||
2. Remember: Snowflake identifiers are case-sensitive when quoted
|
||||
3. Common mistakes: `ProductName` (wrong) vs `ProductDescription` (correct)
|
||||
|
||||
### Empty results
|
||||
1. Check patient identifier filtering (`IS NOT NULL`)
|
||||
2. Check date ranges
|
||||
3. Test with `LIMIT 20` first to see sample data
|
||||
|
||||
### Slow queries
|
||||
1. Add `LIMIT` during development
|
||||
2. Use `describe_query` to validate structure before execution
|
||||
3. Consider async execution for large result sets
|
||||
@@ -0,0 +1,50 @@
|
||||
"""
|
||||
Analysis package for patient pathway processing.
|
||||
|
||||
This package contains refactored functions from the original generate_graph() pipeline:
|
||||
- pathway_analyzer: Main analysis pipeline with prepare_data, calculate_statistics, build_hierarchy
|
||||
- statistics: Statistical calculation functions (costs, frequencies, durations)
|
||||
"""
|
||||
|
||||
from analysis.pathway_analyzer import (
|
||||
prepare_data,
|
||||
calculate_statistics,
|
||||
build_hierarchy,
|
||||
prepare_chart_data,
|
||||
generate_icicle_chart,
|
||||
)
|
||||
|
||||
from analysis.statistics import (
|
||||
count_consecutive_values,
|
||||
calculate_drug_costs,
|
||||
calculate_dosing_frequency,
|
||||
calculate_drug_frequency_row,
|
||||
calculate_cost_per_patient_per_annum,
|
||||
calculate_treatment_duration,
|
||||
calculate_pathway_proportion,
|
||||
aggregate_patient_costs,
|
||||
aggregate_drug_frequencies,
|
||||
format_treatment_statistics,
|
||||
remove_nan_values,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Pathway analysis pipeline
|
||||
"prepare_data",
|
||||
"calculate_statistics",
|
||||
"build_hierarchy",
|
||||
"prepare_chart_data",
|
||||
"generate_icicle_chart",
|
||||
# Statistical calculations
|
||||
"count_consecutive_values",
|
||||
"calculate_drug_costs",
|
||||
"calculate_dosing_frequency",
|
||||
"calculate_drug_frequency_row",
|
||||
"calculate_cost_per_patient_per_annum",
|
||||
"calculate_treatment_duration",
|
||||
"calculate_pathway_proportion",
|
||||
"aggregate_patient_costs",
|
||||
"aggregate_drug_frequencies",
|
||||
"format_treatment_statistics",
|
||||
"remove_nan_values",
|
||||
]
|
||||
@@ -0,0 +1,751 @@
|
||||
"""
|
||||
Patient pathway analysis pipeline.
|
||||
|
||||
This module contains functions extracted from the original generate_graph() function
|
||||
to improve maintainability and testability. The functions follow this pipeline:
|
||||
|
||||
1. prepare_data() - Apply filters, create composite keys, load reference data
|
||||
2. calculate_statistics() - Calculate patient costs, drug frequencies, treatment durations
|
||||
3. build_hierarchy() - Build the Trust → Directory → Drug → Pathway hierarchy
|
||||
4. prepare_chart_data() - Finalize data for Plotly icicle chart
|
||||
|
||||
The generate_icicle_chart() function orchestrates the full pipeline.
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from core import PathConfig, default_paths
|
||||
from core.logging_config import get_logger
|
||||
from analysis.statistics import (
|
||||
count_consecutive_values,
|
||||
calculate_drug_costs,
|
||||
calculate_dosing_frequency,
|
||||
calculate_cost_per_patient_per_annum,
|
||||
remove_nan_values,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def prepare_data(
|
||||
df: pd.DataFrame,
|
||||
trust_filter: list[str],
|
||||
drug_filter: list[str],
|
||||
directory_filter: list[str],
|
||||
paths: Optional[PathConfig] = None,
|
||||
) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
|
||||
"""
|
||||
Prepare data for analysis by applying filters and loading reference data.
|
||||
|
||||
Args:
|
||||
df: DataFrame with processed patient intervention data
|
||||
trust_filter: List of trust names to include
|
||||
drug_filter: List of drug names to include
|
||||
directory_filter: List of directories to include
|
||||
paths: PathConfig for file paths (uses default if None)
|
||||
|
||||
Returns:
|
||||
Tuple of (filtered_df, org_codes_df, directory_df) or (None, None, None) if no data
|
||||
"""
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
df["UPIDTreatment"] = df["UPID"] + df["Drug Name"]
|
||||
|
||||
org_codes = pd.read_csv(paths.org_codes_csv, index_col=1)
|
||||
df["Provider Code"] = df["Provider Code"].map(org_codes["Name"])
|
||||
|
||||
df = df[
|
||||
(df["Provider Code"].isin(trust_filter))
|
||||
& (df["Drug Name"].isin(drug_filter))
|
||||
& (df["Directory"].isin(directory_filter))
|
||||
]
|
||||
|
||||
if len(df) == 0:
|
||||
logger.warning("No data found for selected filters.")
|
||||
return None, None, None
|
||||
|
||||
directory_df = df[["UPID", "Directory"]].drop_duplicates("UPID").set_index("UPID")
|
||||
|
||||
logger.info("Filtering unrelated interventions")
|
||||
return df, org_codes, directory_df
|
||||
|
||||
|
||||
def _count_list_values(x):
|
||||
"""Count consecutive occurrences of each value in a sorted list."""
|
||||
return count_consecutive_values(x)
|
||||
|
||||
|
||||
def _sum_list_values(x):
|
||||
"""Calculate sum of price_actual for each drug's portion of the list."""
|
||||
return calculate_drug_costs(x["Drug Name"], x["Price Actual"])
|
||||
|
||||
|
||||
def _start_date_drug(start_dates_df: pd.DataFrame, x: pd.Series) -> list:
|
||||
"""Get start dates for each drug in a patient's treatment."""
|
||||
drug_count = x.notnull().sum()
|
||||
date_string = []
|
||||
for d in range(drug_count):
|
||||
UPID_date_var = str(x.name) + str(x[d])
|
||||
date = start_dates_df.loc[UPID_date_var, "Intervention Date"]
|
||||
date_string.append(date)
|
||||
return date_string
|
||||
|
||||
|
||||
def _end_date_drug(end_dates_df: pd.DataFrame, x: pd.Series) -> list:
|
||||
"""Get end dates for each drug in a patient's treatment."""
|
||||
drug_count = x.notnull().sum()
|
||||
date_string = []
|
||||
for d in range(drug_count - 1):
|
||||
UPID_date_var = str(x.name) + str(x[d])
|
||||
date = end_dates_df.loc[UPID_date_var, "Intervention Date"]
|
||||
date_string.append(date)
|
||||
return date_string
|
||||
|
||||
|
||||
def _drug_frequency_average(x: pd.Series) -> list[float]:
|
||||
"""Calculate average dosing frequency for each drug."""
|
||||
drug_count = x.index.str.contains("drug_").sum()
|
||||
freq = []
|
||||
for d in range(drug_count):
|
||||
freq_val = x.get(f"freq_{d}", 0)
|
||||
if pd.isna(freq_val):
|
||||
freq_val = 0
|
||||
else:
|
||||
freq_val = int(freq_val)
|
||||
|
||||
if freq_val > 1:
|
||||
start_date = x.get(f"start_date_{d}")
|
||||
end_date = x.get(f"end_date_{d}")
|
||||
if pd.notna(start_date) and pd.notna(end_date):
|
||||
freq_calc = calculate_dosing_frequency(freq_val, start_date, end_date)
|
||||
else:
|
||||
freq_calc = 0.0
|
||||
else:
|
||||
freq_calc = 0.0
|
||||
freq.append(freq_calc)
|
||||
return freq
|
||||
|
||||
|
||||
def _drop_duplicate_treatments(df: pd.DataFrame, ascending: bool) -> pd.DataFrame:
|
||||
"""Drop duplicate treatments keeping first/last based on date sort order."""
|
||||
df_sorted = df.sort_values(by=["Intervention Date"], ascending=ascending)
|
||||
df_treatment_steps = df_sorted.drop_duplicates(subset="UPIDTreatment", keep="first")
|
||||
if not ascending:
|
||||
df_treatment_steps = df_treatment_steps.sort_values(by=["Intervention Date"], ascending=True)
|
||||
return df_treatment_steps
|
||||
|
||||
|
||||
def calculate_statistics(
|
||||
df: pd.DataFrame,
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
last_seen_date: str,
|
||||
title: str,
|
||||
) -> tuple[pd.DataFrame, pd.DataFrame, str]:
|
||||
"""
|
||||
Calculate patient statistics: costs, drug frequencies, treatment durations.
|
||||
|
||||
Args:
|
||||
df: Filtered DataFrame from prepare_data()
|
||||
start_date: Start date for patient initiation filter
|
||||
end_date: End date for patient initiation filter
|
||||
last_seen_date: Filter for patients last seen after this date
|
||||
title: Chart title (auto-generated if empty)
|
||||
|
||||
Returns:
|
||||
Tuple of (patient_info_df, date_df, final_title) or (None, None, "") if no valid data
|
||||
"""
|
||||
cost_df = df[["UPID", "Price Actual"]]
|
||||
total_costs = pd.DataFrame(cost_df.groupby("UPID").sum())
|
||||
total_costs.rename(columns={"Price Actual": "Total cost"}, inplace=True)
|
||||
|
||||
df_end_dates = _drop_duplicate_treatments(df, False)
|
||||
df1_unique = _drop_duplicate_treatments(df, True)
|
||||
logger.info("Identifying unique patients and interventions used")
|
||||
|
||||
df_drug_freq = (
|
||||
df.groupby("UPID")
|
||||
.agg({"Drug Name": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
df_drug_cost = (
|
||||
df.groupby("UPID")
|
||||
.agg({"Price Actual": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
df_drug_freq["Price Actual"] = df_drug_freq.index.map(df_drug_cost["Price Actual"])
|
||||
df_drug_freq["Drug Name"] = df_drug_freq["Drug Name"].apply(_count_list_values)
|
||||
df_drug_freq["Drug cost total"] = df_drug_freq.apply(lambda x: _sum_list_values(x), axis=1)
|
||||
|
||||
df_drugs = (
|
||||
df1_unique.groupby("UPID")
|
||||
.agg({"Drug Name": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
df_dates = (
|
||||
df1_unique.groupby("UPID")
|
||||
.agg({"Intervention Date": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
df_end_dates_grouped = (
|
||||
df_end_dates.groupby("UPID")
|
||||
.agg({"Intervention Date": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Calculating each unique patient's intervention average frequency, cost and duration of each intervention"
|
||||
)
|
||||
|
||||
df_dates_unwrapped = pd.DataFrame(
|
||||
df_dates["Intervention Date"].values.tolist(), index=df_dates.index
|
||||
).add_prefix("date_")
|
||||
df_end_dates_unwrapped = pd.DataFrame(
|
||||
df_end_dates_grouped["Intervention Date"].values.tolist(),
|
||||
index=df_end_dates_grouped.index,
|
||||
).add_prefix("date_end_")
|
||||
df_drugs_unwrapped = pd.DataFrame(
|
||||
df_drugs["Drug Name"].values.tolist(), index=df_drugs.index
|
||||
).add_prefix("drug_")
|
||||
|
||||
df_freq_unwrapped = pd.DataFrame(
|
||||
df_drug_freq["Drug Name"].values.tolist(), index=df_drug_freq.index
|
||||
).add_prefix("freq_")
|
||||
|
||||
start_dates = (
|
||||
df[["UPIDTreatment", "Intervention Date"]]
|
||||
.sort_values(by=["Intervention Date"], ascending=True)
|
||||
.drop_duplicates(subset="UPIDTreatment")
|
||||
.set_index("UPIDTreatment")
|
||||
)
|
||||
end_dates = (
|
||||
df[["UPIDTreatment", "Intervention Date"]]
|
||||
.sort_values(by=["Intervention Date"], ascending=False)
|
||||
.drop_duplicates(subset="UPIDTreatment")
|
||||
.set_index("UPIDTreatment")
|
||||
)
|
||||
|
||||
df_drugs_unwrapped["start_dates"] = df_drugs_unwrapped.apply(
|
||||
lambda x: _start_date_drug(start_dates, x), axis=1
|
||||
)
|
||||
df_start_dates_unwrapped = pd.DataFrame(
|
||||
df_drugs_unwrapped["start_dates"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("start_date_")
|
||||
df_drugs_unwrapped.drop(["start_dates"], inplace=True, axis=1)
|
||||
|
||||
df_drugs_unwrapped["end_dates"] = df_drugs_unwrapped.apply(
|
||||
lambda x: _start_date_drug(end_dates, x), axis=1
|
||||
)
|
||||
df_end_dates_unwrapped_2 = pd.DataFrame(
|
||||
df_drugs_unwrapped["end_dates"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("end_date_")
|
||||
df_drugs_unwrapped.drop(["end_dates"], inplace=True, axis=1)
|
||||
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_start_dates_unwrapped, left_index=True, right_index=True
|
||||
)
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_end_dates_unwrapped_2, left_index=True, right_index=True
|
||||
)
|
||||
|
||||
df_freq_for_merge = pd.DataFrame(
|
||||
df_drug_freq["Drug Name"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("freq_")
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_freq_for_merge, left_index=True, right_index=True
|
||||
)
|
||||
df_drugs_unwrapped["frequency"] = df_drugs_unwrapped.apply(
|
||||
lambda x: _drug_frequency_average(x), axis=1
|
||||
)
|
||||
|
||||
df_spacing_unwrapped = pd.DataFrame(
|
||||
df_drugs_unwrapped["frequency"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("spacing_")
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_spacing_unwrapped, left_index=True, right_index=True
|
||||
)
|
||||
|
||||
df_cost_unwrapped = pd.DataFrame(
|
||||
df_drug_freq["Drug cost total"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("total_cost_drug_")
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_cost_unwrapped, left_index=True, right_index=True
|
||||
)
|
||||
df_drugs_unwrapped.drop(["frequency"], inplace=True, axis=1)
|
||||
|
||||
df_drugs_unwrapped.insert(0, "First seen", df_dates_unwrapped.min(axis=1))
|
||||
df_drugs_unwrapped.insert(1, "Last seen", df_end_dates_unwrapped.max(axis=1))
|
||||
|
||||
patient_info = df.drop_duplicates(subset="UPID", keep="first").set_index("UPID")
|
||||
patient_info = pd.merge(patient_info, df_drugs_unwrapped, left_index=True, right_index=True)
|
||||
patient_info = pd.merge(patient_info, df_freq_unwrapped, left_index=True, right_index=True)
|
||||
patient_info = pd.merge(patient_info, total_costs, left_index=True, right_index=True)
|
||||
|
||||
patient_info = patient_info[
|
||||
(patient_info["First seen"] >= str(start_date))
|
||||
& (patient_info["First seen"] < str(end_date))
|
||||
]
|
||||
|
||||
if title == "":
|
||||
title = f"Patients initiated from {start_date} to {end_date}"
|
||||
|
||||
patient_info = patient_info[patient_info["Last seen"] > str(last_seen_date)]
|
||||
|
||||
patient_info["drug_0"] = patient_info["drug_0"].replace("N/A", np.nan)
|
||||
patient_info.dropna(subset=["drug_0"], inplace=True)
|
||||
|
||||
if len(patient_info) == 0:
|
||||
logger.warning("No patients remaining after date filters.")
|
||||
return None, None, ""
|
||||
|
||||
patient_info["Days treated"] = patient_info["Last seen"] - patient_info["First seen"]
|
||||
date_df = patient_info[["First seen", "Last seen", "Days treated"]]
|
||||
|
||||
return patient_info, date_df, title
|
||||
|
||||
|
||||
def _row_function(row: pd.Series) -> str:
|
||||
"""Build composite parent-label-id string for hierarchy."""
|
||||
ids = ""
|
||||
parents = "N&WICS"
|
||||
count = row.count()
|
||||
for c in range(count):
|
||||
v = row[c]
|
||||
if type(v) != str:
|
||||
v = row[c + 1]
|
||||
if c == count - 1:
|
||||
ids = parents + " - " + v
|
||||
continue
|
||||
parents += " - " + v
|
||||
label = row[count - 1]
|
||||
value = parents + "," + label + "," + ids
|
||||
return value
|
||||
|
||||
|
||||
def _remove_nan_string(y) -> list:
|
||||
"""Remove 'nan' strings from list."""
|
||||
return remove_nan_values(y)
|
||||
|
||||
|
||||
def _list_to_string(x: pd.Series) -> str:
|
||||
"""Format drug statistics into readable string."""
|
||||
list_parts = x.ids.split(" - ")
|
||||
drug_list = list_parts[len(list_parts) - len(x.average_cost) :]
|
||||
ret_string = ""
|
||||
for y in range(len(x.average_cost)):
|
||||
if (
|
||||
(round(x.average_spacing[y], 0) > 1)
|
||||
and (round(x.average_administered[y], 0) > 2.5)
|
||||
and (int(x.value) > 0)
|
||||
):
|
||||
string = (
|
||||
f"<br><b>{drug_list[y]}</b><br>On average given "
|
||||
f"{round(x.average_administered[y], 1)} times with a "
|
||||
f"{round(int(x.average_spacing[y]) / 7, 1)} weekly interval ("
|
||||
f"{round((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1), 0)} weeks total treatment length)"
|
||||
)
|
||||
else:
|
||||
string = (
|
||||
f"<br><b>{drug_list[y]}</b><br>On average given "
|
||||
f"{round(x.average_administered[y], 1)} times with a "
|
||||
f"{round(int(x.average_spacing[y]) / 7, 1)} weekly interval ("
|
||||
f"{round((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1), 0)} weeks total treatment length)"
|
||||
)
|
||||
ret_string += string
|
||||
return ret_string
|
||||
|
||||
|
||||
def _min_max_treatment_dates(ice_df: pd.DataFrame, row: pd.Series) -> str:
|
||||
"""Get min/max dates for a pathway."""
|
||||
ids = row["ids"]
|
||||
min_max = ice_df[ice_df["ids"].str.contains(ids, regex=False)]
|
||||
if len(min_max) == 0:
|
||||
return "N/A,N/A"
|
||||
|
||||
# Handle NaT (Not a Time) values
|
||||
first_seen_min = min_max["First seen"].min()
|
||||
last_seen_max = min_max["Last seen"].max()
|
||||
|
||||
if pd.isna(first_seen_min):
|
||||
min_date = "N/A"
|
||||
else:
|
||||
min_date = str(first_seen_min.strftime("%Y-%m-%d"))
|
||||
|
||||
if pd.isna(last_seen_max):
|
||||
max_date = "N/A"
|
||||
else:
|
||||
max_date = str(last_seen_max.strftime("%Y-%m-%d"))
|
||||
|
||||
return f"{min_date},{max_date}"
|
||||
|
||||
|
||||
def _cost_pp_pa(x: pd.Series) -> str:
|
||||
"""Calculate cost per patient per annum."""
|
||||
result = calculate_cost_per_patient_per_annum(x["costpp"], x["avg_days"])
|
||||
if result is not None:
|
||||
return str(round(result, 2))
|
||||
else:
|
||||
return "N/A"
|
||||
|
||||
|
||||
def build_hierarchy(
|
||||
patient_info: pd.DataFrame,
|
||||
date_df: pd.DataFrame,
|
||||
df: pd.DataFrame,
|
||||
org_codes: pd.DataFrame,
|
||||
directory_df: pd.DataFrame,
|
||||
total_costs: pd.DataFrame,
|
||||
df_drugs_unwrapped: pd.DataFrame,
|
||||
) -> pd.DataFrame:
|
||||
"""
|
||||
Build the hierarchical structure for the icicle chart.
|
||||
|
||||
Args:
|
||||
patient_info: DataFrame with calculated patient statistics
|
||||
date_df: DataFrame with first/last seen dates
|
||||
df: Original filtered DataFrame
|
||||
org_codes: Organization codes lookup
|
||||
directory_df: Directory assignments by UPID
|
||||
total_costs: Total costs by UPID
|
||||
df_drugs_unwrapped: Drug data with dates and frequencies unwrapped
|
||||
|
||||
Returns:
|
||||
DataFrame with parents, ids, labels, value, colour for icicle chart
|
||||
"""
|
||||
number_of_drugs = np.count_nonzero(patient_info.columns.str.startswith("drug_"))
|
||||
final_drug_index = patient_info.columns.to_list().index("drug_" + str(number_of_drugs - 1))
|
||||
|
||||
upid_drugs_df = patient_info.iloc[
|
||||
:, (final_drug_index - number_of_drugs + 1) : final_drug_index + 1
|
||||
]
|
||||
upid_drugs_df = upid_drugs_df.copy()
|
||||
|
||||
upid_drugs_df.insert(0, "Trust", upid_drugs_df.index.str[:3])
|
||||
upid_drugs_df.insert(1, "Directory", upid_drugs_df.index)
|
||||
|
||||
upid_drugs_df["Trust"] = upid_drugs_df["Trust"].map(org_codes["Name"])
|
||||
upid_drugs_df["Directory"] = upid_drugs_df["Directory"].map(directory_df["Directory"])
|
||||
|
||||
upid_drugs_df["value"] = upid_drugs_df.apply(lambda x: _row_function(x), axis=1)
|
||||
upid_drugs_df = pd.merge(upid_drugs_df, date_df, left_index=True, right_index=True)
|
||||
|
||||
upid_drugs_df["ids"] = upid_drugs_df["value"].str.split(",").str[2]
|
||||
|
||||
avg_treatment_dfs = pd.DataFrame(
|
||||
upid_drugs_df.groupby("ids", as_index=False)["Days treated"].mean()
|
||||
).set_index("ids")
|
||||
value_dfs = pd.DataFrame(
|
||||
upid_drugs_df.groupby("value", as_index=False).size()
|
||||
).reset_index()
|
||||
first_seen_treatment_dfs = pd.DataFrame(
|
||||
upid_drugs_df.groupby("ids", as_index=False)["First seen"].min()
|
||||
).set_index("ids")
|
||||
last_seen_treatment_dfs = pd.DataFrame(
|
||||
upid_drugs_df.groupby("ids", as_index=False)["Last seen"].max()
|
||||
).set_index("ids")
|
||||
|
||||
upid_drugs_df["Cost"] = upid_drugs_df.index.map(total_costs["Total cost"])
|
||||
cost_dfs = pd.DataFrame(
|
||||
upid_drugs_df.groupby("value", as_index=False)["Cost"].sum()
|
||||
).set_index("value", drop=True)
|
||||
|
||||
upid_drugs_df = pd.merge(upid_drugs_df, df_drugs_unwrapped, left_index=True, right_index=True)
|
||||
|
||||
spacing_average = pd.DataFrame(
|
||||
upid_drugs_df.groupby("value", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if "spacing_" in col]
|
||||
].mean()
|
||||
).set_index("value", drop=True)
|
||||
spacing_average = spacing_average.round()
|
||||
spacing_average["combined"] = spacing_average.values.tolist()
|
||||
spacing_average["ids"] = spacing_average.index
|
||||
spacing_average["ids"] = spacing_average["ids"].str.split(",").str[2]
|
||||
spacing_average.set_index("ids", inplace=True)
|
||||
|
||||
cost_average = pd.DataFrame(
|
||||
upid_drugs_df.groupby("value", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if "total_cost_drug_" in col]
|
||||
].mean()
|
||||
).set_index("value", drop=True)
|
||||
cost_average = cost_average.round(2)
|
||||
cost_average["combined"] = cost_average.values.tolist()
|
||||
cost_average["ids"] = cost_average.index
|
||||
cost_average["ids"] = cost_average["ids"].str.split(",").str[2]
|
||||
cost_average.set_index("ids", inplace=True)
|
||||
|
||||
freq_average = pd.DataFrame(
|
||||
upid_drugs_df.groupby("ids", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if "freq_" in col]
|
||||
].mean()
|
||||
).set_index("ids", drop=True)
|
||||
freq_average["combined"] = freq_average.values.tolist()
|
||||
|
||||
num = cost_dfs._get_numeric_data()
|
||||
num[num < 0] = 0
|
||||
|
||||
value_dfs["Cost"] = value_dfs["value"].map(cost_dfs["Cost"])
|
||||
|
||||
ice_df = pd.DataFrame()
|
||||
ice_df[["parents", "labels", "ids"]] = value_dfs["value"].str.split(",", expand=True)
|
||||
|
||||
ice_df["average_administered"] = ice_df["ids"].map(freq_average["combined"])
|
||||
ice_df["cost"] = value_dfs["Cost"]
|
||||
ice_df["value"] = value_dfs["size"]
|
||||
|
||||
ice_df["average_cost"] = ice_df["ids"].map(cost_average["combined"])
|
||||
ice_df["average_cost"] = ice_df["average_cost"].apply(_remove_nan_string)
|
||||
|
||||
ice_df["average_spacing"] = ice_df["ids"].map(spacing_average["combined"])
|
||||
ice_df["average_spacing"] = ice_df["average_spacing"].apply(_remove_nan_string)
|
||||
ice_df["average_spacing"] = ice_df.apply(lambda x: _list_to_string(x), axis=1)
|
||||
ice_df["average_spacing"] = ice_df["average_spacing"].str.replace("nan", "N/A")
|
||||
|
||||
logger.info("Building graph dataframe structure.")
|
||||
|
||||
new_row = pd.DataFrame(
|
||||
{"parents": "", "ids": "N&WICS", "labels": "N&WICS", "value": 0, "cost": 0}, index=[0]
|
||||
)
|
||||
ice_df = pd.concat(objs=[ice_df, new_row], ignore_index=True, axis=0)
|
||||
|
||||
l_df = pd.DataFrame()
|
||||
ice_df2 = pd.DataFrame()
|
||||
l3 = [x for x in ice_df.parents.unique() if x not in ice_df.ids]
|
||||
while len(l3) > 1:
|
||||
for l in l3:
|
||||
z = l.rfind("-")
|
||||
if z > 0:
|
||||
l_dict = {
|
||||
"parents": l[: z - 1],
|
||||
"ids": l,
|
||||
"value": 0,
|
||||
"labels": l[z + 2 :],
|
||||
"cost": 0,
|
||||
}
|
||||
l_df = pd.concat([l_df, pd.DataFrame(l_dict, index=[0])], ignore_index=True)
|
||||
ice_df2 = pd.concat([ice_df, l_df], ignore_index=True)
|
||||
l3 = [x for x in ice_df2.parents.unique() if x not in ice_df2.ids.unique()]
|
||||
if len(ice_df2) > 0:
|
||||
ice_df = ice_df2.drop_duplicates("ids")
|
||||
|
||||
ice_df["level"] = ice_df["ids"].str.count("-")
|
||||
ice_df = ice_df[~ice_df["labels"].isin(["COST", "CHARGE", "N/A"])]
|
||||
ice_df.sort_values(by=["level"], ascending=False, inplace=True, ignore_index=True)
|
||||
|
||||
for index, row in ice_df.iterrows():
|
||||
lookup_index = ice_df.index[ice_df["ids"] == row["parents"]]
|
||||
ice_df.loc[lookup_index, "value"] = (
|
||||
ice_df.loc[lookup_index, "value"] + ice_df.loc[index, "value"]
|
||||
)
|
||||
ice_df.loc[lookup_index, "cost"] = (
|
||||
ice_df.loc[lookup_index, "cost"] + ice_df.loc[index, "cost"]
|
||||
)
|
||||
|
||||
colour_df = pd.DataFrame(ice_df.groupby(["parents"])["value"].sum())
|
||||
ice_df["colour"] = ice_df["parents"].map(colour_df["value"])
|
||||
ice_df["colour"] = ice_df["value"] / ice_df["colour"]
|
||||
|
||||
ice_df["costpp"] = ice_df["cost"] / ice_df["value"]
|
||||
ice_df["avg_days"] = ice_df["ids"].map(avg_treatment_dfs["Days treated"])
|
||||
ice_df["First seen"] = ice_df["ids"].map(first_seen_treatment_dfs["First seen"])
|
||||
ice_df["Last seen"] = ice_df["ids"].map(last_seen_treatment_dfs["Last seen"])
|
||||
|
||||
ice_df["dates"] = ice_df.apply(lambda x: _min_max_treatment_dates(ice_df, x), axis=1)
|
||||
ice_df[["First seen (Parent)", "Last seen (Parent)"]] = ice_df["dates"].str.split(
|
||||
",", expand=True
|
||||
)
|
||||
|
||||
ice_df["First seen"] = pd.to_datetime(ice_df["First seen"])
|
||||
ice_df["Last seen"] = pd.to_datetime(ice_df["Last seen"])
|
||||
ice_df["cost_pp_pa"] = ice_df.apply(lambda x: _cost_pp_pa(x), axis=1)
|
||||
|
||||
return ice_df
|
||||
|
||||
|
||||
def prepare_chart_data(
|
||||
ice_df: pd.DataFrame,
|
||||
minimum_num_patients: int,
|
||||
) -> pd.DataFrame:
|
||||
"""
|
||||
Prepare final chart data by applying patient threshold filter.
|
||||
|
||||
Args:
|
||||
ice_df: DataFrame from build_hierarchy()
|
||||
minimum_num_patients: Minimum number of patients to include a pathway
|
||||
|
||||
Returns:
|
||||
Filtered DataFrame ready for chart generation
|
||||
"""
|
||||
ice_df = ice_df[ice_df["value"] >= minimum_num_patients]
|
||||
logger.info("Generating graph.")
|
||||
return ice_df
|
||||
|
||||
|
||||
def generate_icicle_chart(
|
||||
df: pd.DataFrame,
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
last_seen_date: str,
|
||||
trust_filter: list[str],
|
||||
drug_filter: list[str],
|
||||
directory_filter: list[str],
|
||||
minimum_num_patients: int,
|
||||
title: str = "",
|
||||
paths: Optional[PathConfig] = None,
|
||||
) -> tuple[pd.DataFrame, str]:
|
||||
"""
|
||||
Generate icicle chart data using the refactored pipeline.
|
||||
|
||||
This is the main entry point that orchestrates the full analysis pipeline.
|
||||
|
||||
Args:
|
||||
df: DataFrame with processed patient intervention data
|
||||
start_date: Start date for patient initiation filter
|
||||
end_date: End date for patient initiation filter
|
||||
last_seen_date: Filter for patients last seen after this date
|
||||
trust_filter: List of trust names to include
|
||||
drug_filter: List of drug names to include
|
||||
directory_filter: List of directories to include
|
||||
minimum_num_patients: Minimum number of patients to include a pathway
|
||||
title: Chart title (auto-generated if empty)
|
||||
paths: PathConfig for file paths (uses default if None)
|
||||
|
||||
Returns:
|
||||
Tuple of (ice_df for chart, final_title) or (None, "") if no data
|
||||
"""
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
result = prepare_data(df, trust_filter, drug_filter, directory_filter, paths)
|
||||
if result[0] is None:
|
||||
return None, ""
|
||||
filtered_df, org_codes, directory_df = result
|
||||
|
||||
cost_df = filtered_df[["UPID", "Price Actual"]]
|
||||
total_costs = pd.DataFrame(cost_df.groupby("UPID").sum())
|
||||
total_costs.rename(columns={"Price Actual": "Total cost"}, inplace=True)
|
||||
|
||||
result = calculate_statistics(filtered_df, start_date, end_date, last_seen_date, title)
|
||||
if result[0] is None:
|
||||
return None, ""
|
||||
patient_info, date_df, final_title = result
|
||||
|
||||
df_drug_freq = (
|
||||
filtered_df.groupby("UPID")
|
||||
.agg({"Drug Name": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
df_drug_cost = (
|
||||
filtered_df.groupby("UPID")
|
||||
.agg({"Price Actual": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
df_drug_freq["Price Actual"] = df_drug_freq.index.map(df_drug_cost["Price Actual"])
|
||||
df_drug_freq["Drug Name"] = df_drug_freq["Drug Name"].apply(_count_list_values)
|
||||
df_drug_freq["Drug cost total"] = df_drug_freq.apply(lambda x: _sum_list_values(x), axis=1)
|
||||
|
||||
df1_unique = _drop_duplicate_treatments(filtered_df, True)
|
||||
df_drugs = (
|
||||
df1_unique.groupby("UPID")
|
||||
.agg({"Drug Name": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
df_dates = (
|
||||
df1_unique.groupby("UPID")
|
||||
.agg({"Intervention Date": lambda x: list(x)})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
|
||||
df_dates_unwrapped = pd.DataFrame(
|
||||
df_dates["Intervention Date"].values.tolist(), index=df_dates.index
|
||||
).add_prefix("date_")
|
||||
df_drugs_unwrapped = pd.DataFrame(
|
||||
df_drugs["Drug Name"].values.tolist(), index=df_drugs.index
|
||||
).add_prefix("drug_")
|
||||
|
||||
start_dates = (
|
||||
filtered_df[["UPIDTreatment", "Intervention Date"]]
|
||||
.sort_values(by=["Intervention Date"], ascending=True)
|
||||
.drop_duplicates(subset="UPIDTreatment")
|
||||
.set_index("UPIDTreatment")
|
||||
)
|
||||
end_dates = (
|
||||
filtered_df[["UPIDTreatment", "Intervention Date"]]
|
||||
.sort_values(by=["Intervention Date"], ascending=False)
|
||||
.drop_duplicates(subset="UPIDTreatment")
|
||||
.set_index("UPIDTreatment")
|
||||
)
|
||||
|
||||
df_drugs_unwrapped["start_dates"] = df_drugs_unwrapped.apply(
|
||||
lambda x: _start_date_drug(start_dates, x), axis=1
|
||||
)
|
||||
df_start_dates_unwrapped = pd.DataFrame(
|
||||
df_drugs_unwrapped["start_dates"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("start_date_")
|
||||
df_drugs_unwrapped.drop(["start_dates"], inplace=True, axis=1)
|
||||
|
||||
df_drugs_unwrapped["end_dates"] = df_drugs_unwrapped.apply(
|
||||
lambda x: _start_date_drug(end_dates, x), axis=1
|
||||
)
|
||||
df_end_dates_unwrapped_2 = pd.DataFrame(
|
||||
df_drugs_unwrapped["end_dates"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("end_date_")
|
||||
df_drugs_unwrapped.drop(["end_dates"], inplace=True, axis=1)
|
||||
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_start_dates_unwrapped, left_index=True, right_index=True
|
||||
)
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_end_dates_unwrapped_2, left_index=True, right_index=True
|
||||
)
|
||||
|
||||
df_freq_for_merge = pd.DataFrame(
|
||||
df_drug_freq["Drug Name"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("freq_")
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_freq_for_merge, left_index=True, right_index=True
|
||||
)
|
||||
df_drugs_unwrapped["frequency"] = df_drugs_unwrapped.apply(
|
||||
lambda x: _drug_frequency_average(x), axis=1
|
||||
)
|
||||
|
||||
df_spacing_unwrapped = pd.DataFrame(
|
||||
df_drugs_unwrapped["frequency"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("spacing_")
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_spacing_unwrapped, left_index=True, right_index=True
|
||||
)
|
||||
|
||||
df_cost_unwrapped = pd.DataFrame(
|
||||
df_drug_freq["Drug cost total"].values.tolist(), index=df_drugs_unwrapped.index
|
||||
).add_prefix("total_cost_drug_")
|
||||
df_drugs_unwrapped = pd.merge(
|
||||
df_drugs_unwrapped, df_cost_unwrapped, left_index=True, right_index=True
|
||||
)
|
||||
df_drugs_unwrapped.drop(["frequency"], inplace=True, axis=1)
|
||||
|
||||
ice_df = build_hierarchy(
|
||||
patient_info,
|
||||
date_df,
|
||||
filtered_df,
|
||||
org_codes,
|
||||
directory_df,
|
||||
total_costs,
|
||||
df_drugs_unwrapped,
|
||||
)
|
||||
|
||||
ice_df = prepare_chart_data(ice_df, minimum_num_patients)
|
||||
|
||||
return ice_df, final_title
|
||||
@@ -0,0 +1,330 @@
|
||||
"""
|
||||
Statistical calculation functions for patient pathway analysis.
|
||||
|
||||
This module contains functions for calculating:
|
||||
- Drug frequency counts and averages
|
||||
- Cost aggregations (total, per patient, per annum)
|
||||
- Treatment duration calculations
|
||||
- Dosing interval calculations
|
||||
|
||||
These functions are extracted from the analysis pipeline to enable:
|
||||
- Independent testing
|
||||
- Reuse across different analysis contexts
|
||||
- Clearer separation of concerns
|
||||
"""
|
||||
|
||||
from itertools import groupby
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
def count_consecutive_values(values: list) -> list[int]:
|
||||
"""
|
||||
Count consecutive occurrences of each value in a sorted list.
|
||||
|
||||
Used to count how many times each drug was administered.
|
||||
|
||||
Args:
|
||||
values: List of values (typically drug names)
|
||||
|
||||
Returns:
|
||||
List of counts for each unique value in sorted order
|
||||
|
||||
Example:
|
||||
>>> count_consecutive_values(['A', 'A', 'B', 'A'])
|
||||
[3, 1] # 'A' appears 3 times, 'B' appears 1 time (sorted)
|
||||
"""
|
||||
return [len(list(group)) for key, group in groupby(sorted(values))]
|
||||
|
||||
|
||||
def calculate_drug_costs(drug_counts: list[int], prices: list[float]) -> list[float]:
|
||||
"""
|
||||
Calculate total cost for each drug based on counts and prices.
|
||||
|
||||
Splits the price list based on drug administration counts and sums
|
||||
each drug's portion.
|
||||
|
||||
Args:
|
||||
drug_counts: List of administration counts per drug (from count_consecutive_values)
|
||||
prices: List of individual administration prices (Price Actual values)
|
||||
|
||||
Returns:
|
||||
List of total costs per drug
|
||||
|
||||
Example:
|
||||
>>> calculate_drug_costs([3, 2], [100, 100, 100, 200, 200])
|
||||
[300.0, 400.0] # Drug 1: 3x$100 = $300, Drug 2: 2x$200 = $400
|
||||
"""
|
||||
sum_list = []
|
||||
cumulative = 0
|
||||
for count in drug_counts:
|
||||
drug_cost = sum(prices[cumulative:cumulative + count])
|
||||
sum_list.append(float(drug_cost))
|
||||
cumulative += count
|
||||
return sum_list
|
||||
|
||||
|
||||
def calculate_dosing_frequency(
|
||||
freq: int,
|
||||
start_date: pd.Timestamp,
|
||||
end_date: pd.Timestamp,
|
||||
) -> float:
|
||||
"""
|
||||
Calculate average dosing interval in days.
|
||||
|
||||
Computes the average number of days between administrations.
|
||||
|
||||
Args:
|
||||
freq: Number of administrations
|
||||
start_date: First administration date
|
||||
end_date: Last administration date
|
||||
|
||||
Returns:
|
||||
Average days between administrations, or 0 if only one dose
|
||||
|
||||
Example:
|
||||
>>> start = pd.Timestamp('2024-01-01')
|
||||
>>> end = pd.Timestamp('2024-01-22')
|
||||
>>> calculate_dosing_frequency(4, start, end)
|
||||
7.0 # 21 days / (4-1) = 7 days between doses
|
||||
"""
|
||||
if freq <= 1:
|
||||
return 0.0
|
||||
|
||||
duration_days = (end_date - start_date) / np.timedelta64(1, "D")
|
||||
if duration_days <= 0:
|
||||
return 0.0
|
||||
|
||||
return duration_days / (freq - 1)
|
||||
|
||||
|
||||
def calculate_drug_frequency_row(row: pd.Series) -> list[float]:
|
||||
"""
|
||||
Calculate average dosing frequency for each drug in a patient's treatment.
|
||||
|
||||
Used with DataFrame.apply() on rows containing drug_*, freq_*, start_date_*, end_date_* columns.
|
||||
|
||||
Args:
|
||||
row: Series with drug names, frequencies, start dates, and end dates
|
||||
|
||||
Returns:
|
||||
List of average dosing intervals (days) for each drug
|
||||
"""
|
||||
drug_count = row.index.str.contains("drug_").sum()
|
||||
frequencies = []
|
||||
|
||||
for d in range(drug_count):
|
||||
freq_col = f"freq_{d}"
|
||||
start_col = f"start_date_{d}"
|
||||
end_col = f"end_date_{d}"
|
||||
|
||||
freq = row.get(freq_col, 0)
|
||||
if freq is None or pd.isna(freq):
|
||||
freq = 0
|
||||
else:
|
||||
freq = int(freq)
|
||||
|
||||
if freq > 1:
|
||||
start_date = row.get(start_col)
|
||||
end_date = row.get(end_col)
|
||||
|
||||
if pd.notna(start_date) and pd.notna(end_date):
|
||||
interval = calculate_dosing_frequency(freq, start_date, end_date)
|
||||
else:
|
||||
interval = 0.0
|
||||
else:
|
||||
interval = 0.0
|
||||
|
||||
frequencies.append(interval)
|
||||
|
||||
return frequencies
|
||||
|
||||
|
||||
def calculate_cost_per_patient_per_annum(
|
||||
total_cost: float,
|
||||
days_treated: Optional[pd.Timedelta],
|
||||
) -> Optional[float]:
|
||||
"""
|
||||
Calculate annualized cost per patient.
|
||||
|
||||
Normalizes costs to a per-year basis to enable comparison across
|
||||
patients with different treatment durations.
|
||||
|
||||
Args:
|
||||
total_cost: Total cost for the patient
|
||||
days_treated: Treatment duration as timedelta
|
||||
|
||||
Returns:
|
||||
Annualized cost, or None if days_treated is 0 or None
|
||||
|
||||
Example:
|
||||
>>> calculate_cost_per_patient_per_annum(5000, pd.Timedelta(days=182.5))
|
||||
10000.0 # Half year treatment, so annual cost is 2x
|
||||
"""
|
||||
if days_treated is None or pd.isna(days_treated):
|
||||
return None
|
||||
|
||||
days = days_treated / np.timedelta64(1, "D") if hasattr(days_treated, '__truediv__') else float(days_treated)
|
||||
|
||||
if days <= 0:
|
||||
return None
|
||||
|
||||
return total_cost / (days / 365)
|
||||
|
||||
|
||||
def calculate_treatment_duration(
|
||||
first_seen: pd.Timestamp,
|
||||
last_seen: pd.Timestamp,
|
||||
) -> pd.Timedelta:
|
||||
"""
|
||||
Calculate treatment duration from first to last seen dates.
|
||||
|
||||
Args:
|
||||
first_seen: Date of first treatment
|
||||
last_seen: Date of last treatment
|
||||
|
||||
Returns:
|
||||
Duration as timedelta
|
||||
"""
|
||||
return last_seen - first_seen
|
||||
|
||||
|
||||
def calculate_pathway_proportion(value: int, parent_value: int) -> float:
|
||||
"""
|
||||
Calculate proportion of parent value for color scaling.
|
||||
|
||||
Used to determine color intensity in the icicle chart based on
|
||||
what proportion of the parent category this pathway represents.
|
||||
|
||||
Args:
|
||||
value: Patient count for this pathway
|
||||
parent_value: Total patient count for the parent category
|
||||
|
||||
Returns:
|
||||
Proportion (0.0 to 1.0)
|
||||
"""
|
||||
if parent_value <= 0:
|
||||
return 0.0
|
||||
return value / parent_value
|
||||
|
||||
|
||||
def aggregate_patient_costs(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Calculate total cost per patient (UPID).
|
||||
|
||||
Args:
|
||||
df: DataFrame with UPID and Price Actual columns
|
||||
|
||||
Returns:
|
||||
DataFrame indexed by UPID with Total cost column
|
||||
"""
|
||||
cost_df = df[["UPID", "Price Actual"]]
|
||||
total_costs = cost_df.groupby("UPID").sum()
|
||||
total_costs.rename(columns={"Price Actual": "Total cost"}, inplace=True)
|
||||
return total_costs
|
||||
|
||||
|
||||
def aggregate_drug_frequencies(df: pd.DataFrame) -> pd.DataFrame:
|
||||
"""
|
||||
Calculate drug administration frequency per patient.
|
||||
|
||||
Groups by UPID and returns counts of each drug's administrations.
|
||||
|
||||
Args:
|
||||
df: DataFrame with UPID and Drug Name columns
|
||||
|
||||
Returns:
|
||||
DataFrame indexed by UPID with Drug Name as list of counts
|
||||
"""
|
||||
return (
|
||||
df.groupby("UPID")
|
||||
.agg({"Drug Name": lambda x: count_consecutive_values(list(x))})
|
||||
.reset_index()
|
||||
.set_index("UPID")
|
||||
)
|
||||
|
||||
|
||||
def calculate_average_spacing_for_pathway(
|
||||
upid_drugs_df: pd.DataFrame,
|
||||
pathway_value: str,
|
||||
) -> list[float]:
|
||||
"""
|
||||
Calculate average dosing spacing for a treatment pathway.
|
||||
|
||||
Groups patients by pathway and calculates mean spacing for each drug position.
|
||||
|
||||
Args:
|
||||
upid_drugs_df: DataFrame with patient pathway data and spacing columns
|
||||
pathway_value: Pathway identifier string
|
||||
|
||||
Returns:
|
||||
List of average spacing values (days) for each drug in pathway
|
||||
"""
|
||||
spacing_cols = [col for col in upid_drugs_df.columns if col.startswith("spacing_")]
|
||||
|
||||
pathway_data = upid_drugs_df[upid_drugs_df["value"] == pathway_value]
|
||||
|
||||
if len(pathway_data) == 0:
|
||||
return []
|
||||
|
||||
averages = pathway_data[spacing_cols].mean()
|
||||
return [round(v, 0) if pd.notna(v) else 0.0 for v in averages.tolist()]
|
||||
|
||||
|
||||
def format_treatment_statistics(
|
||||
drug_names: list[str],
|
||||
average_administered: list[float],
|
||||
average_spacing: list[float],
|
||||
average_cost: list[float],
|
||||
) -> str:
|
||||
"""
|
||||
Format drug treatment statistics into a readable string for chart display.
|
||||
|
||||
Creates an HTML-formatted string with drug name, average administrations,
|
||||
dosing interval, and total treatment length.
|
||||
|
||||
Args:
|
||||
drug_names: List of drug names in treatment sequence
|
||||
average_administered: Average number of administrations per drug
|
||||
average_spacing: Average days between doses per drug
|
||||
average_cost: Average cost per drug
|
||||
|
||||
Returns:
|
||||
HTML-formatted string for chart hover text
|
||||
"""
|
||||
ret_string = ""
|
||||
|
||||
for i, drug_name in enumerate(drug_names):
|
||||
admin_count = average_administered[i] if i < len(average_administered) else 0
|
||||
spacing_days = average_spacing[i] if i < len(average_spacing) else 0
|
||||
|
||||
# Convert to weeks
|
||||
spacing_weeks = spacing_days / 7 if spacing_days > 0 else 0
|
||||
total_weeks = spacing_weeks * admin_count if admin_count > 0 else 0
|
||||
|
||||
string = (
|
||||
f"<br><b>{drug_name}</b><br>On average given "
|
||||
f"{round(admin_count, 1)} times with a "
|
||||
f"{round(spacing_weeks, 1)} weekly interval ("
|
||||
f"{round(total_weeks, 0)} weeks total treatment length)"
|
||||
)
|
||||
ret_string += string
|
||||
|
||||
return ret_string
|
||||
|
||||
|
||||
def remove_nan_values(values: list) -> list:
|
||||
"""
|
||||
Remove NaN string values from a list.
|
||||
|
||||
Used to clean up aggregated statistics that may contain 'nan' strings.
|
||||
|
||||
Args:
|
||||
values: List potentially containing 'nan' strings
|
||||
|
||||
Returns:
|
||||
Filtered list without 'nan' strings
|
||||
"""
|
||||
return [x for x in values if str(x).lower() != "nan"]
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 4.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
@@ -0,0 +1,268 @@
|
||||
"""
|
||||
Configuration module for Patient Pathway Analysis.
|
||||
|
||||
This module provides access to configuration settings loaded from TOML files.
|
||||
Primary configuration file: config/snowflake.toml
|
||||
|
||||
Usage:
|
||||
from config import load_snowflake_config, SnowflakeConfig
|
||||
|
||||
config = load_snowflake_config()
|
||||
print(config.connection.account)
|
||||
print(config.cache.ttl_seconds)
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
import tomllib # Python 3.11+ built-in TOML parser
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConnectionConfig:
|
||||
"""Snowflake connection settings."""
|
||||
account: str = ""
|
||||
warehouse: str = "ANALYST_WH"
|
||||
database: str = "DATA_HUB"
|
||||
schema: str = "DWH"
|
||||
authenticator: str = "externalbrowser"
|
||||
user: str = ""
|
||||
role: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class TimeoutConfig:
|
||||
"""Timeout settings for Snowflake operations."""
|
||||
connection_timeout: int = 30
|
||||
query_timeout: int = 300
|
||||
login_timeout: int = 120
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheConfig:
|
||||
"""Cache settings for Snowflake query results."""
|
||||
enabled: bool = True
|
||||
directory: str = "data/cache"
|
||||
ttl_seconds: int = 86400 # 24 hours
|
||||
ttl_current_data_seconds: int = 3600 # 1 hour
|
||||
max_size_mb: int = 500
|
||||
|
||||
|
||||
@dataclass
|
||||
class TableReference:
|
||||
"""Reference to a Snowflake table or view."""
|
||||
database: str = ""
|
||||
schema: str = ""
|
||||
view: str = ""
|
||||
table: str = ""
|
||||
key_columns: list = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def fully_qualified_name(self) -> str:
|
||||
"""Return the fully qualified table/view name."""
|
||||
obj_name = self.table or self.view
|
||||
if not obj_name:
|
||||
return ""
|
||||
if self.database and self.schema:
|
||||
return f'"{self.database}"."{self.schema}"."{obj_name}"'
|
||||
elif self.schema:
|
||||
return f'"{self.schema}"."{obj_name}"'
|
||||
else:
|
||||
return f'"{obj_name}"'
|
||||
|
||||
|
||||
@dataclass
|
||||
class TablesConfig:
|
||||
"""Configuration for commonly used tables."""
|
||||
activity: TableReference = field(default_factory=TableReference)
|
||||
patient: TableReference = field(default_factory=TableReference)
|
||||
medication: TableReference = field(default_factory=TableReference)
|
||||
organization: TableReference = field(default_factory=TableReference)
|
||||
|
||||
|
||||
@dataclass
|
||||
class QueryConfig:
|
||||
"""Query execution settings."""
|
||||
quote_identifiers: bool = True
|
||||
test_limit: int = 20
|
||||
max_rows: int = 100000
|
||||
chunk_size: int = 10000
|
||||
|
||||
|
||||
@dataclass
|
||||
class SnowflakeConfig:
|
||||
"""Complete Snowflake configuration."""
|
||||
connection: ConnectionConfig = field(default_factory=ConnectionConfig)
|
||||
timeouts: TimeoutConfig = field(default_factory=TimeoutConfig)
|
||||
cache: CacheConfig = field(default_factory=CacheConfig)
|
||||
tables: TablesConfig = field(default_factory=TablesConfig)
|
||||
query: QueryConfig = field(default_factory=QueryConfig)
|
||||
|
||||
def validate(self) -> list[str]:
|
||||
"""
|
||||
Validate the configuration.
|
||||
|
||||
Returns:
|
||||
List of error messages (empty if valid).
|
||||
"""
|
||||
errors = []
|
||||
|
||||
if not self.connection.account:
|
||||
errors.append("Snowflake account is not configured (connection.account)")
|
||||
|
||||
if not self.connection.warehouse:
|
||||
errors.append("Snowflake warehouse is not configured (connection.warehouse)")
|
||||
|
||||
if self.connection.authenticator not in ("externalbrowser", "snowflake", "oauth", "okta"):
|
||||
errors.append(f"Invalid authenticator: {self.connection.authenticator}")
|
||||
|
||||
if self.cache.ttl_seconds < 0:
|
||||
errors.append("Cache TTL must be non-negative")
|
||||
|
||||
if self.query.max_rows < 1:
|
||||
errors.append("max_rows must be at least 1")
|
||||
|
||||
return errors
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Return True if minimum required settings are present."""
|
||||
return bool(self.connection.account)
|
||||
|
||||
|
||||
def _parse_table_reference(data: dict) -> TableReference:
|
||||
"""Parse a table reference from TOML data."""
|
||||
return TableReference(
|
||||
database=data.get("database", ""),
|
||||
schema=data.get("schema", ""),
|
||||
view=data.get("view", ""),
|
||||
table=data.get("table", ""),
|
||||
key_columns=data.get("key_columns", []),
|
||||
)
|
||||
|
||||
|
||||
def load_snowflake_config(config_path: Optional[Path] = None) -> SnowflakeConfig:
|
||||
"""
|
||||
Load Snowflake configuration from TOML file.
|
||||
|
||||
Args:
|
||||
config_path: Path to the TOML config file. Defaults to config/snowflake.toml
|
||||
relative to the project root.
|
||||
|
||||
Returns:
|
||||
SnowflakeConfig dataclass with all settings.
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If the config file doesn't exist.
|
||||
tomllib.TOMLDecodeError: If the TOML is invalid.
|
||||
"""
|
||||
if config_path is None:
|
||||
# Default to config/snowflake.toml relative to this file's directory
|
||||
config_path = Path(__file__).parent / "snowflake.toml"
|
||||
|
||||
if not config_path.exists():
|
||||
# Return default config if file doesn't exist
|
||||
return SnowflakeConfig()
|
||||
|
||||
with open(config_path, "rb") as f:
|
||||
data = tomllib.load(f)
|
||||
|
||||
# Parse connection settings
|
||||
conn_data = data.get("connection", {})
|
||||
connection = ConnectionConfig(
|
||||
account=conn_data.get("account", ""),
|
||||
warehouse=conn_data.get("warehouse", "ANALYST_WH"),
|
||||
database=conn_data.get("database", "DATA_HUB"),
|
||||
schema=conn_data.get("schema", "DWH"),
|
||||
authenticator=conn_data.get("authenticator", "externalbrowser"),
|
||||
user=conn_data.get("user", ""),
|
||||
role=conn_data.get("role", ""),
|
||||
)
|
||||
|
||||
# Parse timeout settings
|
||||
timeout_data = data.get("timeouts", {})
|
||||
timeouts = TimeoutConfig(
|
||||
connection_timeout=timeout_data.get("connection_timeout", 30),
|
||||
query_timeout=timeout_data.get("query_timeout", 300),
|
||||
login_timeout=timeout_data.get("login_timeout", 120),
|
||||
)
|
||||
|
||||
# Parse cache settings
|
||||
cache_data = data.get("cache", {})
|
||||
cache = CacheConfig(
|
||||
enabled=cache_data.get("enabled", True),
|
||||
directory=cache_data.get("directory", "data/cache"),
|
||||
ttl_seconds=cache_data.get("ttl_seconds", 86400),
|
||||
ttl_current_data_seconds=cache_data.get("ttl_current_data_seconds", 3600),
|
||||
max_size_mb=cache_data.get("max_size_mb", 500),
|
||||
)
|
||||
|
||||
# Parse table references
|
||||
tables_data = data.get("tables", {})
|
||||
tables = TablesConfig(
|
||||
activity=_parse_table_reference(tables_data.get("activity", {})),
|
||||
patient=_parse_table_reference(tables_data.get("patient", {})),
|
||||
medication=_parse_table_reference(tables_data.get("medication", {})),
|
||||
organization=_parse_table_reference(tables_data.get("organization", {})),
|
||||
)
|
||||
|
||||
# Parse query settings
|
||||
query_data = data.get("query", {})
|
||||
query = QueryConfig(
|
||||
quote_identifiers=query_data.get("quote_identifiers", True),
|
||||
test_limit=query_data.get("test_limit", 20),
|
||||
max_rows=query_data.get("max_rows", 100000),
|
||||
chunk_size=query_data.get("chunk_size", 10000),
|
||||
)
|
||||
|
||||
return SnowflakeConfig(
|
||||
connection=connection,
|
||||
timeouts=timeouts,
|
||||
cache=cache,
|
||||
tables=tables,
|
||||
query=query,
|
||||
)
|
||||
|
||||
|
||||
# Module-level cached config (loaded on first access)
|
||||
_cached_config: Optional[SnowflakeConfig] = None
|
||||
|
||||
|
||||
def get_snowflake_config() -> SnowflakeConfig:
|
||||
"""
|
||||
Get the Snowflake configuration (cached after first load).
|
||||
|
||||
Returns:
|
||||
SnowflakeConfig dataclass with all settings.
|
||||
"""
|
||||
global _cached_config
|
||||
if _cached_config is None:
|
||||
_cached_config = load_snowflake_config()
|
||||
return _cached_config
|
||||
|
||||
|
||||
def reload_snowflake_config() -> SnowflakeConfig:
|
||||
"""
|
||||
Reload the Snowflake configuration from disk.
|
||||
|
||||
Returns:
|
||||
SnowflakeConfig dataclass with all settings.
|
||||
"""
|
||||
global _cached_config
|
||||
_cached_config = load_snowflake_config()
|
||||
return _cached_config
|
||||
|
||||
|
||||
# Export public API
|
||||
__all__ = [
|
||||
"SnowflakeConfig",
|
||||
"ConnectionConfig",
|
||||
"TimeoutConfig",
|
||||
"CacheConfig",
|
||||
"TableReference",
|
||||
"TablesConfig",
|
||||
"QueryConfig",
|
||||
"load_snowflake_config",
|
||||
"get_snowflake_config",
|
||||
"reload_snowflake_config",
|
||||
]
|
||||
@@ -0,0 +1,128 @@
|
||||
# Snowflake Configuration for NHS Patient Pathway Analysis
|
||||
#
|
||||
# This file contains connection settings for the Snowflake data warehouse.
|
||||
# IMPORTANT: This file should NOT be committed to version control if it contains
|
||||
# sensitive information. However, with externalbrowser auth, no passwords are stored.
|
||||
#
|
||||
# For NHS SSO authentication, the 'externalbrowser' authenticator opens a browser
|
||||
# window for authentication via NHS identity management.
|
||||
|
||||
[connection]
|
||||
# Snowflake account identifier (e.g., "xy12345.uk-south.azure")
|
||||
# Ask your Snowflake administrator for the correct account name
|
||||
account = ""
|
||||
|
||||
# Default warehouse to use for queries
|
||||
# Common options: ANALYST_WH, COMPUTE_WH
|
||||
warehouse = "ANALYST_WH"
|
||||
|
||||
# Default database for queries
|
||||
# DATA_HUB is the primary analyst-curated data warehouse
|
||||
database = "DATA_HUB"
|
||||
|
||||
# Default schema (optional, can be overridden per query)
|
||||
schema = "DWH"
|
||||
|
||||
# Authentication method
|
||||
# "externalbrowser" opens browser for NHS SSO (required for NHS environments)
|
||||
# Other options: "snowflake" (username/password), "oauth", "okta"
|
||||
authenticator = "externalbrowser"
|
||||
|
||||
# User principal (email address for externalbrowser auth)
|
||||
# Leave empty to use current Windows user or prompt
|
||||
user = ""
|
||||
|
||||
# Role to use (optional, uses default role if empty)
|
||||
role = ""
|
||||
|
||||
[timeouts]
|
||||
# Connection timeout in seconds
|
||||
connection_timeout = 30
|
||||
|
||||
# Query execution timeout in seconds (for long-running queries)
|
||||
# Set to 0 for no timeout
|
||||
query_timeout = 300
|
||||
|
||||
# Login timeout in seconds (for SSO browser auth)
|
||||
login_timeout = 120
|
||||
|
||||
[cache]
|
||||
# Enable result caching
|
||||
enabled = true
|
||||
|
||||
# Cache directory (relative to project root or absolute path)
|
||||
# Defaults to data/cache/ if not specified
|
||||
directory = "data/cache"
|
||||
|
||||
# Time-to-live for cached results in seconds
|
||||
# 24 hours for historical data (86400 seconds)
|
||||
ttl_seconds = 86400
|
||||
|
||||
# TTL for data that includes today's date (shorter)
|
||||
ttl_current_data_seconds = 3600
|
||||
|
||||
# Maximum cache size in MB (oldest entries removed when exceeded)
|
||||
max_size_mb = 500
|
||||
|
||||
[databases]
|
||||
# Quick reference for database purposes (read-only documentation)
|
||||
# DATA_HUB = "Analyst-curated data warehouse - primary source for most queries"
|
||||
# PRIMARY_CARE = "Raw extracts from EMIS and TPP clinical systems"
|
||||
# NATIONAL = "NHS England national datasets (SUS, ECDS, MHSDS, etc.)"
|
||||
# FACTS_AND_DIMENSIONS_ALL_DATA = "External reference data (BNF, SNOMED, QOF clusters)"
|
||||
# REPORTING_DATASETS_ICB = "Reporting outputs and analyst workspaces"
|
||||
|
||||
# Tables commonly used for high-cost drug analysis
|
||||
[tables.activity]
|
||||
# Main activity data source (high-cost drug interventions)
|
||||
# Acute__Conmon__PatientLevelDrugs contains patient-level high-cost drug data
|
||||
database = "DATA_HUB"
|
||||
schema = "CDM"
|
||||
table = "Acute__Conmon__PatientLevelDrugs"
|
||||
key_columns = [
|
||||
"PseudoNHSNoLinked", # Pseudonymised NHS number for patient linking
|
||||
"ProviderCode", # NHS provider code (e.g., RM1, RGP)
|
||||
"LocalPatientID", # Local patient identifier within provider
|
||||
"InterventionDate", # Date of drug intervention
|
||||
"DrugName", # Drug name (raw, needs standardization)
|
||||
"DrugSNOMEDCode", # SNOMED code for drug
|
||||
"PriceActual", # Actual cost of intervention
|
||||
"TreatmentFunctionCode", # NHS treatment function code
|
||||
"TreatmentFunctionDesc", # Treatment function description
|
||||
"AdditionalDetail1", # Additional details (used for directory identification)
|
||||
]
|
||||
|
||||
[tables.patient]
|
||||
# Patient demographics
|
||||
database = "DATA_HUB"
|
||||
schema = "DWH"
|
||||
view = "DimPerson"
|
||||
key_columns = ["PatientPseudonym", "PersonKey", "CurrentGeneralPractice"]
|
||||
|
||||
[tables.medication]
|
||||
# Medication reference data
|
||||
database = "DATA_HUB"
|
||||
schema = "DWH"
|
||||
view = "DimMedicineAndDevice"
|
||||
key_columns = ["ProductSnomedCode", "TherapeuticMoietySnomedCode", "ProductDescription"]
|
||||
|
||||
[tables.organization]
|
||||
# NHS organizations and GP practices
|
||||
database = "DATA_HUB"
|
||||
schema = "DWH"
|
||||
view = "DimOrganisationAndSite"
|
||||
key_columns = ["SiteCode", "OrganisationName"]
|
||||
|
||||
[query]
|
||||
# Default query behaviors
|
||||
# Always double-quote identifiers for case-sensitivity
|
||||
quote_identifiers = true
|
||||
|
||||
# Default row limit for test queries
|
||||
test_limit = 20
|
||||
|
||||
# Maximum rows to fetch in a single query (prevents runaway queries)
|
||||
max_rows = 100000
|
||||
|
||||
# Chunk size for large result sets
|
||||
chunk_size = 10000
|
||||
@@ -0,0 +1,17 @@
|
||||
"""
|
||||
Core module for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Contains configuration, models, and shared utilities used across the application.
|
||||
"""
|
||||
|
||||
from core.config import PathConfig, default_paths
|
||||
from core.models import AnalysisFilters
|
||||
from core.logging_config import setup_logging, get_logger
|
||||
|
||||
__all__ = [
|
||||
"PathConfig",
|
||||
"default_paths",
|
||||
"AnalysisFilters",
|
||||
"setup_logging",
|
||||
"get_logger",
|
||||
]
|
||||
+197
@@ -0,0 +1,197 @@
|
||||
"""
|
||||
Configuration module for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Contains PathConfig dataclass for centralizing all file path references.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class PathConfig:
|
||||
"""
|
||||
Centralizes all file paths used across the application.
|
||||
|
||||
Provides a single source of truth for file locations, making it easier to:
|
||||
- Change the data directory location
|
||||
- Support different environments (development, production)
|
||||
- Validate that required files exist
|
||||
|
||||
Attributes:
|
||||
base_dir: Root directory of the application (defaults to current working directory)
|
||||
data_dir: Directory containing reference data files
|
||||
images_dir: Directory containing UI assets and fonts
|
||||
"""
|
||||
|
||||
base_dir: Path = field(default_factory=Path.cwd)
|
||||
_data_dir: Optional[Path] = field(default=None, repr=False)
|
||||
_images_dir: Optional[Path] = field(default=None, repr=False)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
"""Set default subdirectories relative to base_dir if not provided."""
|
||||
if self._data_dir is None:
|
||||
self._data_dir = self.base_dir / "data"
|
||||
if self._images_dir is None:
|
||||
self._images_dir = self.base_dir / "images"
|
||||
|
||||
@property
|
||||
def data_dir(self) -> Path:
|
||||
"""Directory containing reference data files."""
|
||||
# _data_dir is always set after __post_init__
|
||||
assert self._data_dir is not None
|
||||
return self._data_dir
|
||||
|
||||
@property
|
||||
def images_dir(self) -> Path:
|
||||
"""Directory containing UI assets and fonts."""
|
||||
# _images_dir is always set after __post_init__
|
||||
assert self._images_dir is not None
|
||||
return self._images_dir
|
||||
|
||||
# Reference data files (read-only lookups)
|
||||
@property
|
||||
def drugnames_csv(self) -> Path:
|
||||
"""Drug name standardization mapping."""
|
||||
return self.data_dir / "drugnames.csv"
|
||||
|
||||
@property
|
||||
def directory_list_csv(self) -> Path:
|
||||
"""Medical specialties/directories list."""
|
||||
return self.data_dir / "directory_list.csv"
|
||||
|
||||
@property
|
||||
def treatment_function_codes_csv(self) -> Path:
|
||||
"""NHS treatment function code mappings."""
|
||||
return self.data_dir / "treatment_function_codes.csv"
|
||||
|
||||
@property
|
||||
def drug_directory_list_csv(self) -> Path:
|
||||
"""Valid drug-to-directory mappings (pipe-separated)."""
|
||||
return self.data_dir / "drug_directory_list.csv"
|
||||
|
||||
@property
|
||||
def org_codes_csv(self) -> Path:
|
||||
"""Provider code to organization name mapping."""
|
||||
return self.data_dir / "org_codes.csv"
|
||||
|
||||
@property
|
||||
def include_csv(self) -> Path:
|
||||
"""Drug filter list with default selections."""
|
||||
return self.data_dir / "include.csv"
|
||||
|
||||
@property
|
||||
def default_trusts_csv(self) -> Path:
|
||||
"""NHS Trust list for filter."""
|
||||
return self.data_dir / "defaultTrusts.csv"
|
||||
|
||||
# Output/diagnostic files
|
||||
@property
|
||||
def na_directory_rows_csv(self) -> Path:
|
||||
"""Exported rows with unresolved Directory for diagnostics."""
|
||||
return self.data_dir / "na_directory_rows.csv"
|
||||
|
||||
@property
|
||||
def ta_recommendations_xlsx(self) -> Path:
|
||||
"""NICE TA recommendations (downloaded from web)."""
|
||||
return self.data_dir / "ta-recommendations.xlsx"
|
||||
|
||||
# UI assets
|
||||
@property
|
||||
def font_medium(self) -> Path:
|
||||
"""AvenirLTStd-Medium font file."""
|
||||
return self.images_dir / "AvenirLTStd-Medium.ttf"
|
||||
|
||||
@property
|
||||
def font_roman(self) -> Path:
|
||||
"""AvenirLTStd-Roman font file."""
|
||||
return self.images_dir / "AvenirLTStd-Roman.ttf"
|
||||
|
||||
@property
|
||||
def logo_ico(self) -> Path:
|
||||
"""Application icon."""
|
||||
return self.images_dir / "logo.ico"
|
||||
|
||||
@property
|
||||
def logo_png(self) -> Path:
|
||||
"""Application logo."""
|
||||
return self.images_dir / "logo.png"
|
||||
|
||||
def validate(self) -> list[str]:
|
||||
"""
|
||||
Validate that required files and directories exist.
|
||||
|
||||
Returns:
|
||||
List of error messages. Empty list means all validations passed.
|
||||
"""
|
||||
errors = []
|
||||
|
||||
# Check directories exist
|
||||
if not self.data_dir.exists():
|
||||
errors.append(f"Data directory not found: {self.data_dir}")
|
||||
if not self.images_dir.exists():
|
||||
errors.append(f"Images directory not found: {self.images_dir}")
|
||||
|
||||
# Check required reference files
|
||||
required_files = [
|
||||
(self.drugnames_csv, "Drug names mapping"),
|
||||
(self.directory_list_csv, "Directory list"),
|
||||
(self.treatment_function_codes_csv, "Treatment function codes"),
|
||||
(self.drug_directory_list_csv, "Drug-directory mapping"),
|
||||
(self.org_codes_csv, "Organization codes"),
|
||||
(self.include_csv, "Drug include list"),
|
||||
(self.default_trusts_csv, "Default trusts"),
|
||||
]
|
||||
|
||||
for file_path, description in required_files:
|
||||
if not file_path.exists():
|
||||
errors.append(f"{description} not found: {file_path}")
|
||||
|
||||
return errors
|
||||
|
||||
def validate_fonts(self) -> list[str]:
|
||||
"""
|
||||
Validate that font files exist (for GUI mode).
|
||||
|
||||
Returns:
|
||||
List of error messages. Empty list means all validations passed.
|
||||
"""
|
||||
errors = []
|
||||
|
||||
font_files = [
|
||||
(self.font_medium, "Medium font"),
|
||||
(self.font_roman, "Roman font"),
|
||||
]
|
||||
|
||||
for file_path, description in font_files:
|
||||
if not file_path.exists():
|
||||
errors.append(f"{description} not found: {file_path}")
|
||||
|
||||
return errors
|
||||
|
||||
def as_legacy_paths(self) -> dict[str, str]:
|
||||
"""
|
||||
Return paths as strings with './' prefix for backwards compatibility.
|
||||
|
||||
This method eases migration by providing paths in the format
|
||||
currently used throughout the codebase.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping path names to legacy-format string paths.
|
||||
"""
|
||||
return {
|
||||
"drugnames_csv": f"./{self.drugnames_csv.relative_to(self.base_dir)}",
|
||||
"directory_list_csv": f"./{self.directory_list_csv.relative_to(self.base_dir)}",
|
||||
"treatment_function_codes_csv": f"./{self.treatment_function_codes_csv.relative_to(self.base_dir)}",
|
||||
"drug_directory_list_csv": f"./{self.drug_directory_list_csv.relative_to(self.base_dir)}",
|
||||
"org_codes_csv": f"./{self.org_codes_csv.relative_to(self.base_dir)}",
|
||||
"include_csv": f"./{self.include_csv.relative_to(self.base_dir)}",
|
||||
"default_trusts_csv": f"./{self.default_trusts_csv.relative_to(self.base_dir)}",
|
||||
"na_directory_rows_csv": f"./{self.na_directory_rows_csv.relative_to(self.base_dir)}",
|
||||
"ta_recommendations_xlsx": f"./{self.ta_recommendations_xlsx.relative_to(self.base_dir)}",
|
||||
}
|
||||
|
||||
|
||||
# Default instance for application-wide use
|
||||
default_paths = PathConfig()
|
||||
@@ -0,0 +1,121 @@
|
||||
"""
|
||||
Logging configuration for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Provides structured logging setup with console and optional file handlers.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
# Default log format: timestamp, level, module name, message
|
||||
DEFAULT_FORMAT = "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
|
||||
DEFAULT_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
|
||||
|
||||
# Simplified format for console output (used when redirecting to GUI)
|
||||
SIMPLE_FORMAT = "%(message)s"
|
||||
|
||||
|
||||
def setup_logging(
|
||||
level: int = logging.INFO,
|
||||
log_dir: Optional[Path] = None,
|
||||
console: bool = True,
|
||||
file_logging: bool = False,
|
||||
simple_console: bool = False,
|
||||
) -> logging.Logger:
|
||||
"""
|
||||
Configure application-wide logging.
|
||||
|
||||
Args:
|
||||
level: Logging level (default: INFO)
|
||||
log_dir: Directory for log files (default: ./logs/)
|
||||
console: Whether to log to console/stdout (default: True)
|
||||
file_logging: Whether to log to file (default: False)
|
||||
simple_console: Use simplified format for console (just message, no timestamp)
|
||||
|
||||
Returns:
|
||||
Root logger configured for the application
|
||||
|
||||
Usage:
|
||||
# Basic setup - console only
|
||||
logger = setup_logging()
|
||||
|
||||
# With file logging
|
||||
logger = setup_logging(file_logging=True)
|
||||
|
||||
# Debug mode
|
||||
logger = setup_logging(level=logging.DEBUG)
|
||||
|
||||
# GUI mode - simple format for stdout capture
|
||||
logger = setup_logging(simple_console=True)
|
||||
"""
|
||||
# Get root logger for the application
|
||||
root_logger = logging.getLogger("pathways")
|
||||
|
||||
# Clear any existing handlers to avoid duplicates on re-initialization
|
||||
root_logger.handlers.clear()
|
||||
|
||||
root_logger.setLevel(level)
|
||||
|
||||
# Console handler
|
||||
if console:
|
||||
console_handler = logging.StreamHandler(sys.stdout)
|
||||
console_handler.setLevel(level)
|
||||
|
||||
if simple_console:
|
||||
console_format = logging.Formatter(SIMPLE_FORMAT)
|
||||
else:
|
||||
console_format = logging.Formatter(DEFAULT_FORMAT, datefmt=DEFAULT_DATE_FORMAT)
|
||||
|
||||
console_handler.setFormatter(console_format)
|
||||
root_logger.addHandler(console_handler)
|
||||
|
||||
# File handler
|
||||
if file_logging:
|
||||
if log_dir is None:
|
||||
log_dir = Path("./logs")
|
||||
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
log_filename = f"pathways_{datetime.now().strftime('%Y%m%d_%H%M%S')}.log"
|
||||
log_path = log_dir / log_filename
|
||||
|
||||
file_handler = logging.FileHandler(log_path, encoding="utf-8")
|
||||
file_handler.setLevel(level)
|
||||
file_handler.setFormatter(
|
||||
logging.Formatter(DEFAULT_FORMAT, datefmt=DEFAULT_DATE_FORMAT)
|
||||
)
|
||||
root_logger.addHandler(file_handler)
|
||||
|
||||
return root_logger
|
||||
|
||||
|
||||
def get_logger(name: str) -> logging.Logger:
|
||||
"""
|
||||
Get a logger for a specific module.
|
||||
|
||||
Args:
|
||||
name: Module name (typically __name__)
|
||||
|
||||
Returns:
|
||||
Logger instance configured as child of root pathways logger
|
||||
|
||||
Usage:
|
||||
from core.logging_config import get_logger
|
||||
logger = get_logger(__name__)
|
||||
logger.info("Processing started")
|
||||
logger.error("Something went wrong")
|
||||
"""
|
||||
# Create child logger under the pathways namespace
|
||||
if name.startswith("pathways."):
|
||||
return logging.getLogger(name)
|
||||
return logging.getLogger(f"pathways.{name}")
|
||||
|
||||
|
||||
# Module-level loggers for common components
|
||||
data_logger = get_logger("data")
|
||||
dashboard_logger = get_logger("dashboard")
|
||||
gui_logger = get_logger("gui")
|
||||
+140
@@ -0,0 +1,140 @@
|
||||
"""
|
||||
Data models for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Contains dataclasses for encapsulating application state and filter parameters.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnalysisFilters:
|
||||
"""
|
||||
Encapsulates all filter state for the analysis pipeline.
|
||||
|
||||
Replaces the individual parameters currently passed to generate_graph()
|
||||
and the global state managed in the GUI. This provides:
|
||||
- Type safety for filter values
|
||||
- Validation of filter combinations
|
||||
- Easy serialization for caching/persistence
|
||||
- Clear interface between GUI and analysis engine
|
||||
|
||||
Attributes:
|
||||
start_date: Patient initiated start date (treatment pathway start)
|
||||
end_date: Patient initiated end date (treatment pathway start cutoff)
|
||||
last_seen_date: Minimum last seen date (filters out patients not seen recently)
|
||||
trusts: List of NHS Trust names to include (empty = all)
|
||||
drugs: List of drug names to include (empty = all)
|
||||
directories: List of medical directories/specialties to include (empty = all)
|
||||
custom_title: Optional custom title for the graph (blank = auto-generated)
|
||||
minimum_patients: Minimum number of patients for a pathway to be included
|
||||
output_dir: Directory where output files should be saved
|
||||
"""
|
||||
|
||||
start_date: date
|
||||
end_date: date
|
||||
last_seen_date: date
|
||||
trusts: list[str] = field(default_factory=list)
|
||||
drugs: list[str] = field(default_factory=list)
|
||||
directories: list[str] = field(default_factory=list)
|
||||
custom_title: str = ""
|
||||
minimum_patients: int = 0
|
||||
output_dir: Optional[Path] = None
|
||||
|
||||
def validate(self) -> list[str]:
|
||||
"""
|
||||
Validate filter configuration for logical consistency.
|
||||
|
||||
Returns:
|
||||
List of error messages. Empty list means all validations passed.
|
||||
"""
|
||||
errors = []
|
||||
|
||||
# Date range validation
|
||||
if self.end_date < self.start_date:
|
||||
errors.append(
|
||||
f"End date ({self.end_date}) cannot be before start date ({self.start_date})"
|
||||
)
|
||||
|
||||
if self.last_seen_date > self.end_date:
|
||||
errors.append(
|
||||
f"Last seen date ({self.last_seen_date}) is after end date ({self.end_date}), "
|
||||
"which would exclude all patients"
|
||||
)
|
||||
|
||||
# Minimum patients validation
|
||||
if self.minimum_patients < 0:
|
||||
errors.append(
|
||||
f"Minimum patients ({self.minimum_patients}) cannot be negative"
|
||||
)
|
||||
|
||||
# Output directory validation
|
||||
if self.output_dir is not None and not self.output_dir.exists():
|
||||
errors.append(f"Output directory does not exist: {self.output_dir}")
|
||||
|
||||
# Filter list validation (warn if empty but don't error)
|
||||
# Empty lists are valid and mean "include all"
|
||||
|
||||
return errors
|
||||
|
||||
@property
|
||||
def has_trust_filter(self) -> bool:
|
||||
"""Check if any trust filter is applied."""
|
||||
return len(self.trusts) > 0
|
||||
|
||||
@property
|
||||
def has_drug_filter(self) -> bool:
|
||||
"""Check if any drug filter is applied."""
|
||||
return len(self.drugs) > 0
|
||||
|
||||
@property
|
||||
def has_directory_filter(self) -> bool:
|
||||
"""Check if any directory filter is applied."""
|
||||
return len(self.directories) > 0
|
||||
|
||||
@property
|
||||
def title(self) -> str:
|
||||
"""
|
||||
Return the display title for the graph.
|
||||
|
||||
If custom_title is set, use it. Otherwise, generate a default title
|
||||
based on the date range.
|
||||
"""
|
||||
if self.custom_title:
|
||||
return self.custom_title
|
||||
return f"Patients initiated from {self.start_date} to {self.end_date}"
|
||||
|
||||
def summary(self) -> str:
|
||||
"""
|
||||
Return a human-readable summary of the filter configuration.
|
||||
|
||||
Useful for logging and display in the GUI.
|
||||
"""
|
||||
lines = [
|
||||
f"Date range: {self.start_date} to {self.end_date}",
|
||||
f"Last seen after: {self.last_seen_date}",
|
||||
f"Minimum patients: {self.minimum_patients}",
|
||||
]
|
||||
|
||||
if self.trusts:
|
||||
lines.append(f"Trusts: {len(self.trusts)} selected")
|
||||
else:
|
||||
lines.append("Trusts: All")
|
||||
|
||||
if self.drugs:
|
||||
lines.append(f"Drugs: {len(self.drugs)} selected")
|
||||
else:
|
||||
lines.append("Drugs: All")
|
||||
|
||||
if self.directories:
|
||||
lines.append(f"Directories: {len(self.directories)} selected")
|
||||
else:
|
||||
lines.append("Directories: All")
|
||||
|
||||
if self.custom_title:
|
||||
lines.append(f"Custom title: {self.custom_title}")
|
||||
|
||||
return "\n".join(lines)
|
||||
Binary file not shown.
@@ -0,0 +1,273 @@
|
||||
"""
|
||||
Data processing module for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Contains SQLite database management, data loaders, and Snowflake integration.
|
||||
Handles the migration from CSV-based storage to SQLite for improved performance.
|
||||
|
||||
Submodules:
|
||||
database: SQLite connection management and schema definitions
|
||||
loader: Data loading abstractions (CSV, SQLite, Snowflake)
|
||||
snowflake_connector: Snowflake integration with SSO authentication
|
||||
"""
|
||||
|
||||
from data_processing.database import (
|
||||
DatabaseConfig,
|
||||
DatabaseManager,
|
||||
default_db_config,
|
||||
default_db_manager,
|
||||
)
|
||||
from data_processing.schema import (
|
||||
# Reference table schemas
|
||||
REF_DRUG_NAMES_SCHEMA,
|
||||
REF_ORGANIZATIONS_SCHEMA,
|
||||
REF_DIRECTORIES_SCHEMA,
|
||||
REF_DRUG_DIRECTORY_MAP_SCHEMA,
|
||||
REF_DRUG_INDICATION_CLUSTERS_SCHEMA,
|
||||
REFERENCE_TABLES_SCHEMA,
|
||||
# Fact table schemas
|
||||
FACT_INTERVENTIONS_SCHEMA,
|
||||
FACT_TABLES_SCHEMA,
|
||||
# Materialized view schemas
|
||||
MV_PATIENT_TREATMENT_SUMMARY_SCHEMA,
|
||||
MATERIALIZED_VIEWS_SCHEMA,
|
||||
# File tracking schemas
|
||||
PROCESSED_FILES_SCHEMA,
|
||||
FILE_TRACKING_SCHEMA,
|
||||
# Combined schema
|
||||
ALL_TABLES_SCHEMA,
|
||||
# Reference table functions
|
||||
create_reference_tables,
|
||||
drop_reference_tables,
|
||||
get_reference_table_counts,
|
||||
verify_reference_tables_exist,
|
||||
# Fact table functions
|
||||
create_fact_tables,
|
||||
drop_fact_tables,
|
||||
get_fact_table_counts,
|
||||
verify_fact_tables_exist,
|
||||
# File tracking functions
|
||||
create_file_tracking_tables,
|
||||
drop_file_tracking_tables,
|
||||
get_file_tracking_counts,
|
||||
verify_file_tracking_tables_exist,
|
||||
# Combined functions
|
||||
create_all_tables,
|
||||
drop_all_tables,
|
||||
get_all_table_counts,
|
||||
verify_all_tables_exist,
|
||||
)
|
||||
|
||||
# Reference data migration functions
|
||||
from data_processing.reference_data import (
|
||||
MigrationResult,
|
||||
migrate_drug_names,
|
||||
get_drug_name_counts,
|
||||
verify_drug_names_migration,
|
||||
migrate_organizations,
|
||||
get_organization_counts,
|
||||
verify_organizations_migration,
|
||||
migrate_directories,
|
||||
get_directory_counts,
|
||||
verify_directories_migration,
|
||||
migrate_drug_directory_map,
|
||||
get_drug_directory_map_counts,
|
||||
verify_drug_directory_map_migration,
|
||||
migrate_drug_indication_clusters,
|
||||
get_drug_indication_cluster_counts,
|
||||
verify_drug_indication_clusters_migration,
|
||||
)
|
||||
|
||||
# Data loader abstractions
|
||||
from data_processing.loader import (
|
||||
DataLoader,
|
||||
FileDataLoader,
|
||||
SQLiteDataLoader,
|
||||
LoadResult,
|
||||
get_loader,
|
||||
REQUIRED_COLUMNS,
|
||||
OPTIONAL_COLUMNS,
|
||||
)
|
||||
|
||||
# Patient data migration functions
|
||||
from data_processing.patient_data import (
|
||||
PatientDataLoadResult,
|
||||
load_patient_data,
|
||||
get_patient_data_stats,
|
||||
list_processed_files,
|
||||
calculate_file_hash,
|
||||
# Materialized view functions
|
||||
MVRefreshResult,
|
||||
refresh_patient_treatment_summary,
|
||||
get_patient_summary_stats,
|
||||
verify_mv_consistency,
|
||||
)
|
||||
|
||||
# Snowflake connector
|
||||
from data_processing.snowflake_connector import (
|
||||
SnowflakeConnector,
|
||||
SnowflakeConnectionError,
|
||||
SnowflakeNotConfiguredError,
|
||||
SnowflakeNotAvailableError,
|
||||
ConnectionInfo,
|
||||
get_connector,
|
||||
reset_connector,
|
||||
is_snowflake_available,
|
||||
is_snowflake_configured,
|
||||
SNOWFLAKE_AVAILABLE,
|
||||
)
|
||||
|
||||
# Query result caching
|
||||
from data_processing.cache import (
|
||||
QueryCache,
|
||||
CacheEntry,
|
||||
CacheStats,
|
||||
get_cache,
|
||||
reset_cache,
|
||||
is_cache_enabled,
|
||||
)
|
||||
|
||||
# Data source management with fallback chain
|
||||
from data_processing.data_source import (
|
||||
DataSourceType,
|
||||
DataSourceResult,
|
||||
SourceStatus,
|
||||
DataSourceManager,
|
||||
get_data_source_manager,
|
||||
get_data,
|
||||
reset_data_source_manager,
|
||||
)
|
||||
|
||||
# Diagnosis lookup (GP diagnosis validation)
|
||||
from data_processing.diagnosis_lookup import (
|
||||
ClusterSnomedCodes,
|
||||
IndicationValidationResult,
|
||||
DrugIndicationMatchRate,
|
||||
get_drug_clusters,
|
||||
get_drug_cluster_ids,
|
||||
get_cluster_snomed_codes,
|
||||
patient_has_indication,
|
||||
validate_indication,
|
||||
get_indication_match_rate,
|
||||
batch_validate_indications,
|
||||
get_available_clusters,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Database management
|
||||
"DatabaseConfig",
|
||||
"DatabaseManager",
|
||||
"default_db_config",
|
||||
"default_db_manager",
|
||||
# Reference table schemas
|
||||
"REF_DRUG_NAMES_SCHEMA",
|
||||
"REF_ORGANIZATIONS_SCHEMA",
|
||||
"REF_DIRECTORIES_SCHEMA",
|
||||
"REF_DRUG_DIRECTORY_MAP_SCHEMA",
|
||||
"REF_DRUG_INDICATION_CLUSTERS_SCHEMA",
|
||||
"REFERENCE_TABLES_SCHEMA",
|
||||
# Fact table schemas
|
||||
"FACT_INTERVENTIONS_SCHEMA",
|
||||
"FACT_TABLES_SCHEMA",
|
||||
# Materialized view schemas
|
||||
"MV_PATIENT_TREATMENT_SUMMARY_SCHEMA",
|
||||
"MATERIALIZED_VIEWS_SCHEMA",
|
||||
# File tracking schemas
|
||||
"PROCESSED_FILES_SCHEMA",
|
||||
"FILE_TRACKING_SCHEMA",
|
||||
# Combined schema
|
||||
"ALL_TABLES_SCHEMA",
|
||||
# Reference table functions
|
||||
"create_reference_tables",
|
||||
"drop_reference_tables",
|
||||
"get_reference_table_counts",
|
||||
"verify_reference_tables_exist",
|
||||
# Fact table functions
|
||||
"create_fact_tables",
|
||||
"drop_fact_tables",
|
||||
"get_fact_table_counts",
|
||||
"verify_fact_tables_exist",
|
||||
# File tracking functions
|
||||
"create_file_tracking_tables",
|
||||
"drop_file_tracking_tables",
|
||||
"get_file_tracking_counts",
|
||||
"verify_file_tracking_tables_exist",
|
||||
# Combined functions
|
||||
"create_all_tables",
|
||||
"drop_all_tables",
|
||||
"get_all_table_counts",
|
||||
"verify_all_tables_exist",
|
||||
# Reference data migration
|
||||
"MigrationResult",
|
||||
"migrate_drug_names",
|
||||
"get_drug_name_counts",
|
||||
"verify_drug_names_migration",
|
||||
"migrate_organizations",
|
||||
"get_organization_counts",
|
||||
"verify_organizations_migration",
|
||||
"migrate_directories",
|
||||
"get_directory_counts",
|
||||
"verify_directories_migration",
|
||||
"migrate_drug_directory_map",
|
||||
"get_drug_directory_map_counts",
|
||||
"verify_drug_directory_map_migration",
|
||||
"migrate_drug_indication_clusters",
|
||||
"get_drug_indication_cluster_counts",
|
||||
"verify_drug_indication_clusters_migration",
|
||||
# Data loader abstractions
|
||||
"DataLoader",
|
||||
"FileDataLoader",
|
||||
"SQLiteDataLoader",
|
||||
"LoadResult",
|
||||
"get_loader",
|
||||
"REQUIRED_COLUMNS",
|
||||
"OPTIONAL_COLUMNS",
|
||||
# Patient data migration
|
||||
"PatientDataLoadResult",
|
||||
"load_patient_data",
|
||||
"get_patient_data_stats",
|
||||
"list_processed_files",
|
||||
"calculate_file_hash",
|
||||
# Materialized view functions
|
||||
"MVRefreshResult",
|
||||
"refresh_patient_treatment_summary",
|
||||
"get_patient_summary_stats",
|
||||
"verify_mv_consistency",
|
||||
# Snowflake connector
|
||||
"SnowflakeConnector",
|
||||
"SnowflakeConnectionError",
|
||||
"SnowflakeNotConfiguredError",
|
||||
"SnowflakeNotAvailableError",
|
||||
"ConnectionInfo",
|
||||
"get_connector",
|
||||
"reset_connector",
|
||||
"is_snowflake_available",
|
||||
"is_snowflake_configured",
|
||||
"SNOWFLAKE_AVAILABLE",
|
||||
# Query result caching
|
||||
"QueryCache",
|
||||
"CacheEntry",
|
||||
"CacheStats",
|
||||
"get_cache",
|
||||
"reset_cache",
|
||||
"is_cache_enabled",
|
||||
# Data source management with fallback chain
|
||||
"DataSourceType",
|
||||
"DataSourceResult",
|
||||
"SourceStatus",
|
||||
"DataSourceManager",
|
||||
"get_data_source_manager",
|
||||
"get_data",
|
||||
"reset_data_source_manager",
|
||||
# Diagnosis lookup
|
||||
"ClusterSnomedCodes",
|
||||
"IndicationValidationResult",
|
||||
"DrugIndicationMatchRate",
|
||||
"get_drug_clusters",
|
||||
"get_drug_cluster_ids",
|
||||
"get_cluster_snomed_codes",
|
||||
"patient_has_indication",
|
||||
"validate_indication",
|
||||
"get_indication_match_rate",
|
||||
"batch_validate_indications",
|
||||
"get_available_clusters",
|
||||
]
|
||||
@@ -0,0 +1,553 @@
|
||||
"""
|
||||
Query result caching module for NHS Patient Pathway Analysis.
|
||||
|
||||
Provides file-based caching for Snowflake query results with TTL-based invalidation.
|
||||
Supports different TTLs for historical data vs data including the current date.
|
||||
|
||||
Cache keys are generated from query hashes. Results are stored as compressed JSON.
|
||||
|
||||
Usage:
|
||||
from data_processing.cache import QueryCache, get_cache
|
||||
|
||||
cache = get_cache()
|
||||
|
||||
# Check for cached result
|
||||
result = cache.get(query, params)
|
||||
if result is None:
|
||||
# Execute query and cache result
|
||||
result = execute_query(query, params)
|
||||
cache.set(query, params, result, includes_current_data=False)
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, date
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
|
||||
from config import get_snowflake_config, CacheConfig
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheEntry:
|
||||
"""Metadata for a cached query result."""
|
||||
cache_key: str
|
||||
query_hash: str
|
||||
created_at: datetime
|
||||
expires_at: datetime
|
||||
includes_current_data: bool
|
||||
row_count: int
|
||||
file_size_bytes: int
|
||||
file_path: Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheStats:
|
||||
"""Statistics about the cache."""
|
||||
enabled: bool
|
||||
cache_dir: Path
|
||||
total_entries: int
|
||||
total_size_mb: float
|
||||
max_size_mb: int
|
||||
oldest_entry: Optional[datetime]
|
||||
newest_entry: Optional[datetime]
|
||||
hit_count: int
|
||||
miss_count: int
|
||||
|
||||
|
||||
class QueryCache:
|
||||
"""
|
||||
File-based cache for Snowflake query results.
|
||||
|
||||
Results are stored as gzipped JSON files with TTL-based expiration.
|
||||
Supports different TTLs for historical vs current data.
|
||||
|
||||
Attributes:
|
||||
config: CacheConfig with cache settings
|
||||
cache_dir: Path to cache directory
|
||||
"""
|
||||
|
||||
def __init__(self, config: Optional[CacheConfig] = None, base_path: Optional[Path] = None):
|
||||
"""
|
||||
Initialize the query cache.
|
||||
|
||||
Args:
|
||||
config: Optional CacheConfig. If not provided, loads from snowflake.toml
|
||||
base_path: Base path for relative cache directory. Defaults to cwd.
|
||||
"""
|
||||
if config is None:
|
||||
sf_config = get_snowflake_config()
|
||||
config = sf_config.cache
|
||||
|
||||
self._config = config
|
||||
self._base_path = base_path or Path.cwd()
|
||||
|
||||
# Resolve cache directory
|
||||
cache_dir = Path(config.directory)
|
||||
if not cache_dir.is_absolute():
|
||||
cache_dir = self._base_path / cache_dir
|
||||
self._cache_dir = cache_dir
|
||||
|
||||
# Stats tracking (in-memory only, reset on restart)
|
||||
self._hit_count = 0
|
||||
self._miss_count = 0
|
||||
|
||||
# Ensure cache directory exists if enabled
|
||||
if self._config.enabled:
|
||||
self._cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@property
|
||||
def config(self) -> CacheConfig:
|
||||
"""Return the cache configuration."""
|
||||
return self._config
|
||||
|
||||
@property
|
||||
def cache_dir(self) -> Path:
|
||||
"""Return the cache directory path."""
|
||||
return self._cache_dir
|
||||
|
||||
@property
|
||||
def is_enabled(self) -> bool:
|
||||
"""Return True if caching is enabled."""
|
||||
return self._config.enabled
|
||||
|
||||
def _generate_cache_key(self, query: str, params: Optional[tuple] = None) -> str:
|
||||
"""
|
||||
Generate a cache key from query and parameters.
|
||||
|
||||
Uses SHA256 hash of query + params to create unique key.
|
||||
"""
|
||||
# Normalize query (strip whitespace, lowercase)
|
||||
normalized_query = " ".join(query.lower().split())
|
||||
|
||||
# Combine query and params
|
||||
key_content = normalized_query
|
||||
if params:
|
||||
key_content += "|" + "|".join(str(p) for p in params)
|
||||
|
||||
# Hash to create key
|
||||
hash_obj = hashlib.sha256(key_content.encode("utf-8"))
|
||||
return hash_obj.hexdigest()[:32] # Use first 32 chars for readability
|
||||
|
||||
def _get_cache_file_path(self, cache_key: str) -> Path:
|
||||
"""Get the file path for a cache entry."""
|
||||
return self._cache_dir / f"{cache_key}.json.gz"
|
||||
|
||||
def _get_meta_file_path(self, cache_key: str) -> Path:
|
||||
"""Get the metadata file path for a cache entry."""
|
||||
return self._cache_dir / f"{cache_key}.meta.json"
|
||||
|
||||
def _is_expired(self, meta: dict) -> bool:
|
||||
"""Check if a cache entry is expired based on its metadata."""
|
||||
expires_at = datetime.fromisoformat(meta["expires_at"])
|
||||
return datetime.now() > expires_at
|
||||
|
||||
def get(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None,
|
||||
check_expiry: bool = True
|
||||
) -> Optional[list[dict]]:
|
||||
"""
|
||||
Get a cached query result.
|
||||
|
||||
Args:
|
||||
query: SQL query string
|
||||
params: Optional query parameters
|
||||
check_expiry: If True, returns None for expired entries
|
||||
|
||||
Returns:
|
||||
Cached result as list of dicts, or None if not cached/expired
|
||||
"""
|
||||
if not self.is_enabled:
|
||||
self._miss_count += 1
|
||||
return None
|
||||
|
||||
cache_key = self._generate_cache_key(query, params)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
meta_file = self._get_meta_file_path(cache_key)
|
||||
|
||||
# Check if files exist
|
||||
if not cache_file.exists() or not meta_file.exists():
|
||||
self._miss_count += 1
|
||||
logger.debug(f"Cache miss (not found): {cache_key}")
|
||||
return None
|
||||
|
||||
# Load and check metadata
|
||||
try:
|
||||
with open(meta_file, "r", encoding="utf-8") as f:
|
||||
meta = json.load(f)
|
||||
|
||||
if check_expiry and self._is_expired(meta):
|
||||
self._miss_count += 1
|
||||
logger.debug(f"Cache miss (expired): {cache_key}")
|
||||
return None
|
||||
|
||||
# Load cached data
|
||||
with gzip.open(cache_file, "rt", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
self._hit_count += 1
|
||||
logger.info(f"Cache hit: {cache_key} ({meta['row_count']} rows)")
|
||||
return data
|
||||
|
||||
except (json.JSONDecodeError, KeyError, OSError) as e:
|
||||
logger.warning(f"Cache read error for {cache_key}: {e}")
|
||||
self._miss_count += 1
|
||||
# Clean up corrupted entry
|
||||
self._delete_entry(cache_key)
|
||||
return None
|
||||
|
||||
def set(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple],
|
||||
data: list[dict],
|
||||
includes_current_data: bool = False,
|
||||
custom_ttl_seconds: Optional[int] = None
|
||||
) -> Optional[CacheEntry]:
|
||||
"""
|
||||
Cache a query result.
|
||||
|
||||
Args:
|
||||
query: SQL query string
|
||||
params: Optional query parameters
|
||||
data: Query result as list of dicts
|
||||
includes_current_data: If True, uses shorter TTL for current data
|
||||
custom_ttl_seconds: Optional custom TTL (overrides config)
|
||||
|
||||
Returns:
|
||||
CacheEntry with metadata, or None if caching disabled/failed
|
||||
"""
|
||||
if not self.is_enabled:
|
||||
return None
|
||||
|
||||
cache_key = self._generate_cache_key(query, params)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
meta_file = self._get_meta_file_path(cache_key)
|
||||
|
||||
# Determine TTL
|
||||
if custom_ttl_seconds is not None:
|
||||
ttl = custom_ttl_seconds
|
||||
elif includes_current_data:
|
||||
ttl = self._config.ttl_current_data_seconds
|
||||
else:
|
||||
ttl = self._config.ttl_seconds
|
||||
|
||||
now = datetime.now()
|
||||
expires_at = datetime.fromtimestamp(now.timestamp() + ttl)
|
||||
|
||||
try:
|
||||
# Write compressed data
|
||||
with gzip.open(cache_file, "wt", encoding="utf-8", compresslevel=6) as f:
|
||||
json.dump(data, f, default=str)
|
||||
|
||||
file_size = cache_file.stat().st_size
|
||||
|
||||
# Write metadata
|
||||
meta = {
|
||||
"cache_key": cache_key,
|
||||
"query_hash": hashlib.sha256(query.encode()).hexdigest()[:16],
|
||||
"created_at": now.isoformat(),
|
||||
"expires_at": expires_at.isoformat(),
|
||||
"includes_current_data": includes_current_data,
|
||||
"row_count": len(data),
|
||||
"file_size_bytes": file_size,
|
||||
"ttl_seconds": ttl,
|
||||
}
|
||||
|
||||
with open(meta_file, "w", encoding="utf-8") as f:
|
||||
json.dump(meta, f, indent=2)
|
||||
|
||||
logger.info(f"Cached {len(data)} rows as {cache_key} (expires in {ttl}s)")
|
||||
|
||||
# Check if we need to enforce size limit
|
||||
self._enforce_size_limit()
|
||||
|
||||
return CacheEntry(
|
||||
cache_key=cache_key,
|
||||
query_hash=str(meta["query_hash"]),
|
||||
created_at=now,
|
||||
expires_at=expires_at,
|
||||
includes_current_data=includes_current_data,
|
||||
row_count=len(data),
|
||||
file_size_bytes=file_size,
|
||||
file_path=cache_file,
|
||||
)
|
||||
|
||||
except (OSError, TypeError) as e:
|
||||
logger.error(f"Failed to cache result: {e}")
|
||||
return None
|
||||
|
||||
def invalidate(self, query: str, params: Optional[tuple] = None) -> bool:
|
||||
"""
|
||||
Invalidate a specific cache entry.
|
||||
|
||||
Args:
|
||||
query: SQL query string
|
||||
params: Optional query parameters
|
||||
|
||||
Returns:
|
||||
True if entry was deleted, False if not found
|
||||
"""
|
||||
cache_key = self._generate_cache_key(query, params)
|
||||
return self._delete_entry(cache_key)
|
||||
|
||||
def _delete_entry(self, cache_key: str) -> bool:
|
||||
"""Delete a cache entry by key."""
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
meta_file = self._get_meta_file_path(cache_key)
|
||||
|
||||
deleted = False
|
||||
|
||||
if cache_file.exists():
|
||||
cache_file.unlink()
|
||||
deleted = True
|
||||
|
||||
if meta_file.exists():
|
||||
meta_file.unlink()
|
||||
deleted = True
|
||||
|
||||
if deleted:
|
||||
logger.debug(f"Deleted cache entry: {cache_key}")
|
||||
|
||||
return deleted
|
||||
|
||||
def clear(self) -> int:
|
||||
"""
|
||||
Clear all cache entries.
|
||||
|
||||
Returns:
|
||||
Number of entries deleted
|
||||
"""
|
||||
if not self._cache_dir.exists():
|
||||
return 0
|
||||
|
||||
count = 0
|
||||
for file in self._cache_dir.glob("*.json*"):
|
||||
try:
|
||||
file.unlink()
|
||||
count += 1
|
||||
except OSError as e:
|
||||
logger.warning(f"Failed to delete {file}: {e}")
|
||||
|
||||
# Reset stats
|
||||
self._hit_count = 0
|
||||
self._miss_count = 0
|
||||
|
||||
logger.info(f"Cleared {count} cache files")
|
||||
return count // 2 # Divide by 2 since we have .json.gz and .meta.json
|
||||
|
||||
def clear_expired(self) -> int:
|
||||
"""
|
||||
Remove expired cache entries.
|
||||
|
||||
Returns:
|
||||
Number of expired entries deleted
|
||||
"""
|
||||
if not self._cache_dir.exists():
|
||||
return 0
|
||||
|
||||
count = 0
|
||||
for meta_file in self._cache_dir.glob("*.meta.json"):
|
||||
try:
|
||||
with open(meta_file, "r", encoding="utf-8") as f:
|
||||
meta = json.load(f)
|
||||
|
||||
if self._is_expired(meta):
|
||||
cache_key = meta_file.stem.replace(".meta", "")
|
||||
self._delete_entry(cache_key)
|
||||
count += 1
|
||||
except (OSError, json.JSONDecodeError):
|
||||
# Delete corrupted metadata files
|
||||
cache_key = meta_file.stem.replace(".meta", "")
|
||||
self._delete_entry(cache_key)
|
||||
count += 1
|
||||
|
||||
logger.info(f"Cleared {count} expired cache entries")
|
||||
return count
|
||||
|
||||
def _get_total_size_mb(self) -> float:
|
||||
"""Calculate total cache size in MB."""
|
||||
if not self._cache_dir.exists():
|
||||
return 0.0
|
||||
|
||||
total_bytes = sum(
|
||||
f.stat().st_size
|
||||
for f in self._cache_dir.glob("*")
|
||||
if f.is_file()
|
||||
)
|
||||
return total_bytes / (1024 * 1024)
|
||||
|
||||
def _enforce_size_limit(self) -> int:
|
||||
"""
|
||||
Enforce cache size limit by removing oldest entries.
|
||||
|
||||
Returns:
|
||||
Number of entries removed
|
||||
"""
|
||||
max_size_mb = self._config.max_size_mb
|
||||
current_size_mb = self._get_total_size_mb()
|
||||
|
||||
if current_size_mb <= max_size_mb:
|
||||
return 0
|
||||
|
||||
# Get all entries sorted by creation time
|
||||
entries = []
|
||||
for meta_file in self._cache_dir.glob("*.meta.json"):
|
||||
try:
|
||||
with open(meta_file, "r", encoding="utf-8") as f:
|
||||
meta = json.load(f)
|
||||
entries.append((
|
||||
meta_file.stem.replace(".meta", ""),
|
||||
datetime.fromisoformat(meta["created_at"]),
|
||||
meta.get("file_size_bytes", 0)
|
||||
))
|
||||
except (OSError, json.JSONDecodeError, KeyError):
|
||||
# Clean up corrupted entry
|
||||
cache_key = meta_file.stem.replace(".meta", "")
|
||||
self._delete_entry(cache_key)
|
||||
|
||||
# Sort by creation time (oldest first)
|
||||
entries.sort(key=lambda x: x[1])
|
||||
|
||||
# Remove oldest entries until under limit
|
||||
removed = 0
|
||||
size_to_remove_bytes = (current_size_mb - max_size_mb * 0.9) * 1024 * 1024 # Target 90% of limit
|
||||
removed_bytes = 0
|
||||
|
||||
for cache_key, created_at, file_size in entries:
|
||||
if removed_bytes >= size_to_remove_bytes:
|
||||
break
|
||||
|
||||
self._delete_entry(cache_key)
|
||||
removed_bytes += file_size
|
||||
removed += 1
|
||||
|
||||
logger.info(f"Removed {removed} cache entries to enforce size limit")
|
||||
return removed
|
||||
|
||||
def get_stats(self) -> CacheStats:
|
||||
"""Get cache statistics."""
|
||||
if not self._cache_dir.exists():
|
||||
return CacheStats(
|
||||
enabled=self.is_enabled,
|
||||
cache_dir=self._cache_dir,
|
||||
total_entries=0,
|
||||
total_size_mb=0.0,
|
||||
max_size_mb=self._config.max_size_mb,
|
||||
oldest_entry=None,
|
||||
newest_entry=None,
|
||||
hit_count=self._hit_count,
|
||||
miss_count=self._miss_count,
|
||||
)
|
||||
|
||||
entries = []
|
||||
for meta_file in self._cache_dir.glob("*.meta.json"):
|
||||
try:
|
||||
with open(meta_file, "r", encoding="utf-8") as f:
|
||||
meta = json.load(f)
|
||||
entries.append(datetime.fromisoformat(meta["created_at"]))
|
||||
except (OSError, json.JSONDecodeError, KeyError):
|
||||
pass
|
||||
|
||||
oldest = min(entries) if entries else None
|
||||
newest = max(entries) if entries else None
|
||||
|
||||
return CacheStats(
|
||||
enabled=self.is_enabled,
|
||||
cache_dir=self._cache_dir,
|
||||
total_entries=len(entries),
|
||||
total_size_mb=self._get_total_size_mb(),
|
||||
max_size_mb=self._config.max_size_mb,
|
||||
oldest_entry=oldest,
|
||||
newest_entry=newest,
|
||||
hit_count=self._hit_count,
|
||||
miss_count=self._miss_count,
|
||||
)
|
||||
|
||||
def list_entries(self) -> list[CacheEntry]:
|
||||
"""List all cache entries with metadata."""
|
||||
if not self._cache_dir.exists():
|
||||
return []
|
||||
|
||||
entries = []
|
||||
for meta_file in self._cache_dir.glob("*.meta.json"):
|
||||
try:
|
||||
with open(meta_file, "r", encoding="utf-8") as f:
|
||||
meta = json.load(f)
|
||||
|
||||
cache_key = meta["cache_key"]
|
||||
entries.append(CacheEntry(
|
||||
cache_key=cache_key,
|
||||
query_hash=meta.get("query_hash", ""),
|
||||
created_at=datetime.fromisoformat(meta["created_at"]),
|
||||
expires_at=datetime.fromisoformat(meta["expires_at"]),
|
||||
includes_current_data=meta.get("includes_current_data", False),
|
||||
row_count=meta.get("row_count", 0),
|
||||
file_size_bytes=meta.get("file_size_bytes", 0),
|
||||
file_path=self._get_cache_file_path(cache_key),
|
||||
))
|
||||
except (OSError, json.JSONDecodeError, KeyError):
|
||||
pass
|
||||
|
||||
# Sort by creation time (newest first)
|
||||
entries.sort(key=lambda x: x.created_at, reverse=True)
|
||||
return entries
|
||||
|
||||
|
||||
# Module-level singleton
|
||||
_default_cache: Optional[QueryCache] = None
|
||||
|
||||
|
||||
def get_cache(config: Optional[CacheConfig] = None) -> QueryCache:
|
||||
"""
|
||||
Get a QueryCache instance (creates singleton on first call).
|
||||
|
||||
Args:
|
||||
config: Optional CacheConfig. If provided, creates new cache with
|
||||
this config. If None, uses/creates default cache.
|
||||
|
||||
Returns:
|
||||
QueryCache instance
|
||||
"""
|
||||
global _default_cache
|
||||
|
||||
if config is not None:
|
||||
# Custom config requested, create new cache
|
||||
return QueryCache(config)
|
||||
|
||||
if _default_cache is None:
|
||||
_default_cache = QueryCache()
|
||||
|
||||
return _default_cache
|
||||
|
||||
|
||||
def reset_cache() -> None:
|
||||
"""Reset the default cache singleton."""
|
||||
global _default_cache
|
||||
_default_cache = None
|
||||
|
||||
|
||||
def is_cache_enabled() -> bool:
|
||||
"""Return True if caching is enabled in configuration."""
|
||||
config = get_snowflake_config()
|
||||
return config.cache.enabled
|
||||
|
||||
|
||||
# Export public API
|
||||
__all__ = [
|
||||
"QueryCache",
|
||||
"CacheEntry",
|
||||
"CacheStats",
|
||||
"get_cache",
|
||||
"reset_cache",
|
||||
"is_cache_enabled",
|
||||
]
|
||||
@@ -0,0 +1,968 @@
|
||||
"""
|
||||
Unified data access layer with fallback chain for NHS Patient Pathway Analysis.
|
||||
|
||||
Provides a high-level interface that automatically selects the best available data source:
|
||||
1. Cache - Returns cached results if valid and not expired
|
||||
2. Snowflake - Queries Snowflake warehouse if configured and connected
|
||||
3. Local - Falls back to SQLite database or CSV/Parquet files
|
||||
|
||||
The fallback chain handles connection errors, missing configurations, and
|
||||
unavailable services gracefully, always attempting to provide data from
|
||||
some source.
|
||||
|
||||
Usage:
|
||||
from data_processing.data_source import DataSourceManager, get_data
|
||||
|
||||
# Simple usage with automatic source selection
|
||||
result = get_data(
|
||||
start_date=date(2024, 1, 1),
|
||||
end_date=date(2024, 12, 31),
|
||||
trusts=["TRUST A", "TRUST B"],
|
||||
)
|
||||
|
||||
# Or with explicit source preference
|
||||
manager = DataSourceManager()
|
||||
result = manager.get_data(
|
||||
start_date=date(2024, 1, 1),
|
||||
end_date=date(2024, 12, 31),
|
||||
preferred_source="snowflake",
|
||||
)
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Optional, Callable
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class DataSourceType(Enum):
|
||||
"""Enumeration of available data sources."""
|
||||
CACHE = "cache"
|
||||
SNOWFLAKE = "snowflake"
|
||||
SQLITE = "sqlite"
|
||||
FILE = "file"
|
||||
|
||||
|
||||
@dataclass
|
||||
class DataSourceResult:
|
||||
"""Result from data source query.
|
||||
|
||||
Attributes:
|
||||
df: The loaded DataFrame with patient intervention data
|
||||
source_type: Which data source was used
|
||||
source_detail: Additional details about the source (e.g., file path, query hash)
|
||||
row_count: Number of rows returned
|
||||
cached: Whether the result came from cache
|
||||
from_fallback: Whether a fallback source was used
|
||||
load_time_seconds: Time taken to load data
|
||||
warnings: Any warnings generated during loading
|
||||
"""
|
||||
df: pd.DataFrame
|
||||
source_type: DataSourceType
|
||||
source_detail: str = ""
|
||||
row_count: int = 0
|
||||
cached: bool = False
|
||||
from_fallback: bool = False
|
||||
load_time_seconds: float = 0.0
|
||||
warnings: list[str] = field(default_factory=list)
|
||||
|
||||
def __post_init__(self):
|
||||
if self.row_count == 0 and self.df is not None:
|
||||
self.row_count = len(self.df)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SourceStatus:
|
||||
"""Status of a data source.
|
||||
|
||||
Attributes:
|
||||
source_type: The type of data source
|
||||
available: Whether the source is available
|
||||
configured: Whether the source is properly configured
|
||||
message: Status message explaining the state
|
||||
last_checked: When the status was last checked
|
||||
"""
|
||||
source_type: DataSourceType
|
||||
available: bool = False
|
||||
configured: bool = False
|
||||
message: str = ""
|
||||
last_checked: Optional[datetime] = None
|
||||
|
||||
|
||||
class DataSourceManager:
|
||||
"""
|
||||
Manages data access with automatic fallback between sources.
|
||||
|
||||
The manager attempts to retrieve data from sources in order of preference:
|
||||
1. Cache (if enabled and has valid cached data)
|
||||
2. Snowflake (if configured and connected)
|
||||
3. SQLite (if database exists with data)
|
||||
4. Local files (CSV/Parquet)
|
||||
|
||||
Attributes:
|
||||
cache_enabled: Whether to use caching
|
||||
local_file_path: Path to local CSV/Parquet file (optional fallback)
|
||||
sqlite_db_path: Path to SQLite database (optional)
|
||||
|
||||
Example:
|
||||
manager = DataSourceManager()
|
||||
|
||||
# Check what sources are available
|
||||
status = manager.check_all_sources()
|
||||
for s in status:
|
||||
print(f"{s.source_type.value}: {s.message}")
|
||||
|
||||
# Get data with automatic fallback
|
||||
result = manager.get_data(
|
||||
start_date=date(2024, 1, 1),
|
||||
end_date=date(2024, 6, 30),
|
||||
)
|
||||
print(f"Got {result.row_count} rows from {result.source_type.value}")
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cache_enabled: bool = True,
|
||||
local_file_path: Optional[Path | str] = None,
|
||||
sqlite_db_path: Optional[Path | str] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the data source manager.
|
||||
|
||||
Args:
|
||||
cache_enabled: Whether to check cache before querying (default True)
|
||||
local_file_path: Path to local CSV/Parquet file for file fallback
|
||||
sqlite_db_path: Path to SQLite database (uses default if None)
|
||||
"""
|
||||
self._cache_enabled = cache_enabled
|
||||
self._local_file_path = Path(local_file_path) if local_file_path else None
|
||||
self._sqlite_db_path = Path(sqlite_db_path) if sqlite_db_path else None
|
||||
self._source_status: dict[DataSourceType, SourceStatus] = {}
|
||||
|
||||
@property
|
||||
def cache_enabled(self) -> bool:
|
||||
"""Return whether caching is enabled."""
|
||||
return self._cache_enabled
|
||||
|
||||
@cache_enabled.setter
|
||||
def cache_enabled(self, value: bool):
|
||||
"""Set whether caching is enabled."""
|
||||
self._cache_enabled = value
|
||||
|
||||
def _check_cache_status(self) -> SourceStatus:
|
||||
"""Check if cache is available."""
|
||||
try:
|
||||
from data_processing.cache import is_cache_enabled, get_cache
|
||||
|
||||
if not is_cache_enabled():
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.CACHE,
|
||||
available=False,
|
||||
configured=False,
|
||||
message="Cache disabled in configuration",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
cache = get_cache()
|
||||
stats = cache.get_stats()
|
||||
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.CACHE,
|
||||
available=True,
|
||||
configured=True,
|
||||
message=f"Cache enabled ({stats.total_entries} entries, {stats.total_size_mb:.1f}MB)",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
except Exception as e:
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.CACHE,
|
||||
available=False,
|
||||
configured=False,
|
||||
message=f"Cache error: {e}",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
def _check_snowflake_status(self) -> SourceStatus:
|
||||
"""Check if Snowflake is available and configured."""
|
||||
try:
|
||||
from data_processing.snowflake_connector import (
|
||||
is_snowflake_available,
|
||||
is_snowflake_configured,
|
||||
)
|
||||
|
||||
if not is_snowflake_available():
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SNOWFLAKE,
|
||||
available=False,
|
||||
configured=False,
|
||||
message="snowflake-connector-python not installed",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
if not is_snowflake_configured():
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SNOWFLAKE,
|
||||
available=True,
|
||||
configured=False,
|
||||
message="Snowflake account not configured in config/snowflake.toml",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SNOWFLAKE,
|
||||
available=True,
|
||||
configured=True,
|
||||
message="Snowflake configured and ready",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
except Exception as e:
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SNOWFLAKE,
|
||||
available=False,
|
||||
configured=False,
|
||||
message=f"Snowflake error: {e}",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
def _check_sqlite_status(self) -> SourceStatus:
|
||||
"""Check if SQLite database is available with data."""
|
||||
try:
|
||||
from data_processing.database import default_db_manager, default_db_config
|
||||
|
||||
db_path = self._sqlite_db_path or Path(default_db_config.db_path)
|
||||
|
||||
if not db_path.exists():
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SQLITE,
|
||||
available=False,
|
||||
configured=True,
|
||||
message=f"Database not found: {db_path}",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
from data_processing.database import DatabaseManager, DatabaseConfig
|
||||
|
||||
config = DatabaseConfig(db_path=db_path)
|
||||
manager = DatabaseManager(config)
|
||||
|
||||
if not manager.table_exists("fact_interventions"):
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SQLITE,
|
||||
available=False,
|
||||
configured=True,
|
||||
message="fact_interventions table not found",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
count = manager.get_table_count("fact_interventions")
|
||||
if count == 0:
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SQLITE,
|
||||
available=False,
|
||||
configured=True,
|
||||
message="fact_interventions table is empty",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SQLITE,
|
||||
available=True,
|
||||
configured=True,
|
||||
message=f"SQLite database ready ({count:,} rows)",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
except Exception as e:
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.SQLITE,
|
||||
available=False,
|
||||
configured=False,
|
||||
message=f"SQLite error: {e}",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
def _check_file_status(self) -> SourceStatus:
|
||||
"""Check if local file is available."""
|
||||
if self._local_file_path is None:
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.FILE,
|
||||
available=False,
|
||||
configured=False,
|
||||
message="No local file path configured",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
if not self._local_file_path.exists():
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.FILE,
|
||||
available=False,
|
||||
configured=True,
|
||||
message=f"File not found: {self._local_file_path}",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
size_mb = self._local_file_path.stat().st_size / (1024 * 1024)
|
||||
return SourceStatus(
|
||||
source_type=DataSourceType.FILE,
|
||||
available=True,
|
||||
configured=True,
|
||||
message=f"Local file ready: {self._local_file_path.name} ({size_mb:.1f}MB)",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
def check_source_status(self, source_type: DataSourceType) -> SourceStatus:
|
||||
"""
|
||||
Check the status of a specific data source.
|
||||
|
||||
Args:
|
||||
source_type: The type of source to check
|
||||
|
||||
Returns:
|
||||
SourceStatus with current availability information
|
||||
"""
|
||||
if source_type == DataSourceType.CACHE:
|
||||
return self._check_cache_status()
|
||||
elif source_type == DataSourceType.SNOWFLAKE:
|
||||
return self._check_snowflake_status()
|
||||
elif source_type == DataSourceType.SQLITE:
|
||||
return self._check_sqlite_status()
|
||||
elif source_type == DataSourceType.FILE:
|
||||
return self._check_file_status()
|
||||
else:
|
||||
return SourceStatus(
|
||||
source_type=source_type,
|
||||
available=False,
|
||||
configured=False,
|
||||
message=f"Unknown source type: {source_type}",
|
||||
last_checked=datetime.now(),
|
||||
)
|
||||
|
||||
def check_all_sources(self) -> list[SourceStatus]:
|
||||
"""
|
||||
Check the status of all data sources.
|
||||
|
||||
Returns:
|
||||
List of SourceStatus for each source type
|
||||
"""
|
||||
statuses = []
|
||||
for source_type in DataSourceType:
|
||||
status = self.check_source_status(source_type)
|
||||
self._source_status[source_type] = status
|
||||
statuses.append(status)
|
||||
return statuses
|
||||
|
||||
def _build_cache_key_params(
|
||||
self,
|
||||
start_date: Optional[date],
|
||||
end_date: Optional[date],
|
||||
trusts: Optional[list[str]],
|
||||
drugs: Optional[list[str]],
|
||||
directories: Optional[list[str]],
|
||||
) -> tuple[str, tuple]:
|
||||
"""Build a cache-compatible query string and params for the filter criteria."""
|
||||
# Create a canonical representation for caching
|
||||
query_parts = ["SELECT * FROM activity_data"]
|
||||
params = []
|
||||
|
||||
conditions = []
|
||||
if start_date:
|
||||
conditions.append("start_date >= ?")
|
||||
params.append(str(start_date))
|
||||
if end_date:
|
||||
conditions.append("end_date <= ?")
|
||||
params.append(str(end_date))
|
||||
if trusts:
|
||||
placeholders = ",".join(["?"] * len(trusts))
|
||||
conditions.append(f"trust IN ({placeholders})")
|
||||
params.extend(sorted(trusts))
|
||||
if drugs:
|
||||
placeholders = ",".join(["?"] * len(drugs))
|
||||
conditions.append(f"drug IN ({placeholders})")
|
||||
params.extend(sorted(drugs))
|
||||
if directories:
|
||||
placeholders = ",".join(["?"] * len(directories))
|
||||
conditions.append(f"directory IN ({placeholders})")
|
||||
params.extend(sorted(directories))
|
||||
|
||||
if conditions:
|
||||
query_parts.append("WHERE " + " AND ".join(conditions))
|
||||
|
||||
query = " ".join(query_parts)
|
||||
return query, tuple(params)
|
||||
|
||||
def _try_cache(
|
||||
self,
|
||||
start_date: Optional[date],
|
||||
end_date: Optional[date],
|
||||
trusts: Optional[list[str]],
|
||||
drugs: Optional[list[str]],
|
||||
directories: Optional[list[str]],
|
||||
) -> Optional[DataSourceResult]:
|
||||
"""Try to get data from cache."""
|
||||
if not self._cache_enabled:
|
||||
return None
|
||||
|
||||
try:
|
||||
from data_processing.cache import get_cache
|
||||
|
||||
cache = get_cache()
|
||||
if not cache.is_enabled:
|
||||
return None
|
||||
|
||||
query, params = self._build_cache_key_params(
|
||||
start_date, end_date, trusts, drugs, directories
|
||||
)
|
||||
|
||||
cached_data = cache.get(query, params)
|
||||
if cached_data is None:
|
||||
logger.debug("Cache miss")
|
||||
return None
|
||||
|
||||
# Convert cached data back to DataFrame
|
||||
df = pd.DataFrame(cached_data)
|
||||
|
||||
# Convert date columns
|
||||
if 'Intervention Date' in df.columns:
|
||||
df['Intervention Date'] = pd.to_datetime(df['Intervention Date'])
|
||||
|
||||
logger.info(f"Cache hit: {len(df)} rows")
|
||||
|
||||
return DataSourceResult(
|
||||
df=df,
|
||||
source_type=DataSourceType.CACHE,
|
||||
source_detail=f"cache_key={query[:50]}...",
|
||||
row_count=len(df),
|
||||
cached=True,
|
||||
from_fallback=False,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache lookup failed: {e}")
|
||||
return None
|
||||
|
||||
def _try_snowflake(
|
||||
self,
|
||||
start_date: Optional[date],
|
||||
end_date: Optional[date],
|
||||
trusts: Optional[list[str]],
|
||||
drugs: Optional[list[str]],
|
||||
directories: Optional[list[str]],
|
||||
progress_callback: Optional[Callable[[int, int], None]] = None,
|
||||
) -> Optional[DataSourceResult]:
|
||||
"""Try to get data from Snowflake."""
|
||||
import time
|
||||
|
||||
try:
|
||||
from data_processing.snowflake_connector import (
|
||||
is_snowflake_available,
|
||||
is_snowflake_configured,
|
||||
get_connector,
|
||||
SnowflakeConnectionError,
|
||||
)
|
||||
|
||||
if not is_snowflake_available():
|
||||
logger.debug("Snowflake connector not installed")
|
||||
return None
|
||||
|
||||
if not is_snowflake_configured():
|
||||
logger.debug("Snowflake not configured")
|
||||
return None
|
||||
|
||||
# Get connector and fetch data
|
||||
connector = get_connector()
|
||||
logger.info("Fetching data from Snowflake...")
|
||||
start_time = time.time()
|
||||
|
||||
# Fetch activity data from Snowflake
|
||||
# Note: provider_codes filter not directly supported yet - would need trust name to code mapping
|
||||
rows = connector.fetch_activity_data(
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
provider_codes=None, # TODO: map trust names to provider codes if needed
|
||||
)
|
||||
|
||||
if not rows:
|
||||
logger.warning("Snowflake returned no data")
|
||||
return None
|
||||
|
||||
# Convert to DataFrame
|
||||
df = pd.DataFrame(rows)
|
||||
load_time = time.time() - start_time
|
||||
|
||||
logger.info(f"Snowflake loaded {len(df)} rows in {load_time:.2f}s")
|
||||
|
||||
# Apply local transformations to match expected format
|
||||
# (patient_id, drug_names, department_identification)
|
||||
from tools.data import patient_id, drug_names, department_identification
|
||||
from core import default_paths
|
||||
|
||||
df = patient_id(df)
|
||||
df = drug_names(df, paths=default_paths)
|
||||
df = department_identification(df, paths=default_paths)
|
||||
|
||||
# Apply additional filters if provided
|
||||
if trusts and 'OrganisationName' in df.columns:
|
||||
df = df[df['OrganisationName'].isin(trusts)]
|
||||
if drugs and 'Drug Name' in df.columns:
|
||||
df = df[df['Drug Name'].isin(drugs)]
|
||||
if directories and 'Directory' in df.columns:
|
||||
df = df[df['Directory'].isin(directories)]
|
||||
|
||||
return DataSourceResult(
|
||||
df=df,
|
||||
source_type=DataSourceType.SNOWFLAKE,
|
||||
source_detail="DATA_HUB.CDM.Acute__Conmon__PatientLevelDrugs",
|
||||
row_count=len(df),
|
||||
cached=False,
|
||||
from_fallback=False,
|
||||
load_time_seconds=load_time,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Snowflake query failed: {e}")
|
||||
return None
|
||||
|
||||
def _try_sqlite(
|
||||
self,
|
||||
start_date: Optional[date],
|
||||
end_date: Optional[date],
|
||||
trusts: Optional[list[str]],
|
||||
drugs: Optional[list[str]],
|
||||
directories: Optional[list[str]],
|
||||
) -> Optional[DataSourceResult]:
|
||||
"""Try to get data from SQLite."""
|
||||
import time
|
||||
|
||||
try:
|
||||
from data_processing.loader import SQLiteDataLoader
|
||||
|
||||
# Determine database path
|
||||
db_path = self._sqlite_db_path
|
||||
if db_path is None:
|
||||
from data_processing.database import default_db_config
|
||||
db_path = Path(default_db_config.db_path)
|
||||
|
||||
loader = SQLiteDataLoader(
|
||||
db_path=db_path,
|
||||
date_range=(start_date, end_date) if start_date and end_date else None,
|
||||
trusts=trusts,
|
||||
drugs=drugs,
|
||||
directories=directories,
|
||||
)
|
||||
|
||||
# Check if source is valid
|
||||
is_valid, msg = loader.validate_source()
|
||||
if not is_valid:
|
||||
logger.debug(f"SQLite not available: {msg}")
|
||||
return None
|
||||
|
||||
start_time = time.time()
|
||||
result = loader.load()
|
||||
load_time = time.time() - start_time
|
||||
|
||||
logger.info(f"SQLite loaded {result.row_count} rows in {load_time:.2f}s")
|
||||
|
||||
return DataSourceResult(
|
||||
df=result.df,
|
||||
source_type=DataSourceType.SQLITE,
|
||||
source_detail=str(db_path),
|
||||
row_count=result.row_count,
|
||||
cached=False,
|
||||
from_fallback=False,
|
||||
load_time_seconds=load_time,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"SQLite query failed: {e}")
|
||||
return None
|
||||
|
||||
def _try_file(
|
||||
self,
|
||||
start_date: Optional[date],
|
||||
end_date: Optional[date],
|
||||
trusts: Optional[list[str]],
|
||||
drugs: Optional[list[str]],
|
||||
directories: Optional[list[str]],
|
||||
) -> Optional[DataSourceResult]:
|
||||
"""Try to get data from local file."""
|
||||
import time
|
||||
|
||||
if self._local_file_path is None:
|
||||
logger.debug("No local file configured")
|
||||
return None
|
||||
|
||||
try:
|
||||
from data_processing.loader import FileDataLoader
|
||||
|
||||
loader = FileDataLoader(file_path=self._local_file_path)
|
||||
|
||||
is_valid, msg = loader.validate_source()
|
||||
if not is_valid:
|
||||
logger.debug(f"Local file not available: {msg}")
|
||||
return None
|
||||
|
||||
start_time = time.time()
|
||||
result = loader.load()
|
||||
df = result.df
|
||||
|
||||
# Apply filters (file loader loads all data, then we filter)
|
||||
if start_date and 'Intervention Date' in df.columns:
|
||||
df = df[df['Intervention Date'] >= pd.Timestamp(start_date)]
|
||||
if end_date and 'Intervention Date' in df.columns:
|
||||
df = df[df['Intervention Date'] < pd.Timestamp(end_date)]
|
||||
if trusts and 'OrganisationName' in df.columns:
|
||||
df = df[df['OrganisationName'].isin(trusts)]
|
||||
if drugs and 'Drug Name' in df.columns:
|
||||
df = df[df['Drug Name'].isin(drugs)]
|
||||
if directories and 'Directory' in df.columns:
|
||||
df = df[df['Directory'].isin(directories)]
|
||||
|
||||
load_time = time.time() - start_time
|
||||
|
||||
logger.info(f"File loaded and filtered: {len(df)} rows in {load_time:.2f}s")
|
||||
|
||||
return DataSourceResult(
|
||||
df=df,
|
||||
source_type=DataSourceType.FILE,
|
||||
source_detail=str(self._local_file_path),
|
||||
row_count=len(df),
|
||||
cached=False,
|
||||
from_fallback=True,
|
||||
load_time_seconds=load_time,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"File load failed: {e}")
|
||||
return None
|
||||
|
||||
def get_data(
|
||||
self,
|
||||
start_date: Optional[date] = None,
|
||||
end_date: Optional[date] = None,
|
||||
trusts: Optional[list[str]] = None,
|
||||
drugs: Optional[list[str]] = None,
|
||||
directories: Optional[list[str]] = None,
|
||||
preferred_source: Optional[str] = None,
|
||||
skip_cache: bool = False,
|
||||
progress_callback: Optional[Callable[[int, int], None]] = None,
|
||||
) -> DataSourceResult:
|
||||
"""
|
||||
Get patient intervention data from the best available source.
|
||||
|
||||
The fallback chain is: Cache → Snowflake → SQLite → File
|
||||
|
||||
Args:
|
||||
start_date: Optional start date for filtering (inclusive)
|
||||
end_date: Optional end date for filtering (exclusive)
|
||||
trusts: Optional list of trust names to filter
|
||||
drugs: Optional list of drug names to filter
|
||||
directories: Optional list of directories to filter
|
||||
preferred_source: Optional preferred source ("snowflake", "sqlite", "file")
|
||||
skip_cache: If True, bypass cache and query source directly
|
||||
progress_callback: Optional callback(current, total) for progress updates
|
||||
|
||||
Returns:
|
||||
DataSourceResult with the loaded data and metadata
|
||||
|
||||
Raises:
|
||||
ValueError: If no data source is available or all sources fail
|
||||
"""
|
||||
import time
|
||||
start_time = time.time()
|
||||
warnings = []
|
||||
|
||||
# If preferred source specified, try that first
|
||||
if preferred_source:
|
||||
preferred = preferred_source.lower()
|
||||
if preferred == "snowflake":
|
||||
result = self._try_snowflake(
|
||||
start_date, end_date, trusts, drugs, directories, progress_callback
|
||||
)
|
||||
if result:
|
||||
result.load_time_seconds = time.time() - start_time
|
||||
return result
|
||||
warnings.append("Preferred source 'snowflake' unavailable")
|
||||
|
||||
elif preferred == "sqlite":
|
||||
result = self._try_sqlite(
|
||||
start_date, end_date, trusts, drugs, directories
|
||||
)
|
||||
if result:
|
||||
result.load_time_seconds = time.time() - start_time
|
||||
return result
|
||||
warnings.append("Preferred source 'sqlite' unavailable")
|
||||
|
||||
elif preferred == "file":
|
||||
result = self._try_file(
|
||||
start_date, end_date, trusts, drugs, directories
|
||||
)
|
||||
if result:
|
||||
result.load_time_seconds = time.time() - start_time
|
||||
return result
|
||||
warnings.append("Preferred source 'file' unavailable")
|
||||
|
||||
# Standard fallback chain: cache → snowflake → sqlite → file
|
||||
|
||||
# 1. Try cache first (unless skipped)
|
||||
if not skip_cache:
|
||||
result = self._try_cache(
|
||||
start_date, end_date, trusts, drugs, directories
|
||||
)
|
||||
if result:
|
||||
result.load_time_seconds = time.time() - start_time
|
||||
return result
|
||||
|
||||
# 2. Try Snowflake
|
||||
result = self._try_snowflake(
|
||||
start_date, end_date, trusts, drugs, directories, progress_callback
|
||||
)
|
||||
if result:
|
||||
# Cache the result for future queries
|
||||
if self._cache_enabled:
|
||||
self._cache_result(
|
||||
result.df,
|
||||
start_date, end_date, trusts, drugs, directories,
|
||||
includes_current_data=end_date is None or end_date >= date.today()
|
||||
)
|
||||
result.load_time_seconds = time.time() - start_time
|
||||
return result
|
||||
|
||||
# 3. Try SQLite
|
||||
result = self._try_sqlite(
|
||||
start_date, end_date, trusts, drugs, directories
|
||||
)
|
||||
if result:
|
||||
result.from_fallback = True # Mark as fallback since Snowflake wasn't used
|
||||
result.load_time_seconds = time.time() - start_time
|
||||
if warnings:
|
||||
result.warnings.extend(warnings)
|
||||
return result
|
||||
|
||||
# 4. Try local file
|
||||
result = self._try_file(
|
||||
start_date, end_date, trusts, drugs, directories
|
||||
)
|
||||
if result:
|
||||
result.from_fallback = True
|
||||
result.load_time_seconds = time.time() - start_time
|
||||
if warnings:
|
||||
result.warnings.extend(warnings)
|
||||
return result
|
||||
|
||||
# All sources failed
|
||||
source_status = self.check_all_sources()
|
||||
status_msg = "; ".join(
|
||||
f"{s.source_type.value}: {s.message}" for s in source_status
|
||||
)
|
||||
raise ValueError(f"No data source available. Status: {status_msg}")
|
||||
|
||||
def _cache_result(
|
||||
self,
|
||||
df: pd.DataFrame,
|
||||
start_date: Optional[date],
|
||||
end_date: Optional[date],
|
||||
trusts: Optional[list[str]],
|
||||
drugs: Optional[list[str]],
|
||||
directories: Optional[list[str]],
|
||||
includes_current_data: bool = False,
|
||||
) -> bool:
|
||||
"""Cache a query result for future use."""
|
||||
try:
|
||||
from data_processing.cache import get_cache
|
||||
|
||||
cache = get_cache()
|
||||
if not cache.is_enabled:
|
||||
return False
|
||||
|
||||
query, params = self._build_cache_key_params(
|
||||
start_date, end_date, trusts, drugs, directories
|
||||
)
|
||||
|
||||
# Convert DataFrame to list of dicts for caching
|
||||
# Convert datetime columns to strings for JSON serialization
|
||||
df_copy = df.copy()
|
||||
for col in df_copy.columns:
|
||||
if pd.api.types.is_datetime64_any_dtype(df_copy[col]):
|
||||
df_copy[col] = df_copy[col].astype(str)
|
||||
|
||||
data = df_copy.to_dict(orient='records')
|
||||
|
||||
entry = cache.set(
|
||||
query, params, data,
|
||||
includes_current_data=includes_current_data
|
||||
)
|
||||
|
||||
if entry:
|
||||
logger.info(f"Cached {len(data)} rows (key={entry.cache_key[:16]}...)")
|
||||
return True
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to cache result: {e}")
|
||||
return False
|
||||
|
||||
def clear_cache(self) -> int:
|
||||
"""
|
||||
Clear all cached data.
|
||||
|
||||
Returns:
|
||||
Number of cache entries cleared
|
||||
"""
|
||||
try:
|
||||
from data_processing.cache import get_cache
|
||||
cache = get_cache()
|
||||
return cache.clear()
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clear cache: {e}")
|
||||
return 0
|
||||
|
||||
def refresh_from_snowflake(
|
||||
self,
|
||||
start_date: Optional[date] = None,
|
||||
end_date: Optional[date] = None,
|
||||
trusts: Optional[list[str]] = None,
|
||||
drugs: Optional[list[str]] = None,
|
||||
directories: Optional[list[str]] = None,
|
||||
progress_callback: Optional[Callable[[int, int], None]] = None,
|
||||
) -> DataSourceResult:
|
||||
"""
|
||||
Force a refresh from Snowflake, bypassing cache and other sources.
|
||||
|
||||
This method specifically queries Snowflake and will fail if Snowflake
|
||||
is not available or not configured.
|
||||
|
||||
Args:
|
||||
start_date: Optional start date for filtering
|
||||
end_date: Optional end date for filtering
|
||||
trusts: Optional list of trust names
|
||||
drugs: Optional list of drug names
|
||||
directories: Optional list of directories
|
||||
progress_callback: Optional progress callback
|
||||
|
||||
Returns:
|
||||
DataSourceResult from Snowflake
|
||||
|
||||
Raises:
|
||||
ValueError: If Snowflake is not available or query fails
|
||||
"""
|
||||
from data_processing.snowflake_connector import (
|
||||
is_snowflake_available,
|
||||
is_snowflake_configured,
|
||||
)
|
||||
|
||||
if not is_snowflake_available():
|
||||
raise ValueError("Snowflake connector not installed")
|
||||
|
||||
if not is_snowflake_configured():
|
||||
raise ValueError("Snowflake not configured - edit config/snowflake.toml")
|
||||
|
||||
result = self._try_snowflake(
|
||||
start_date, end_date, trusts, drugs, directories, progress_callback
|
||||
)
|
||||
|
||||
if result is None:
|
||||
raise ValueError("Snowflake query failed - check logs for details")
|
||||
|
||||
# Cache the fresh result
|
||||
if self._cache_enabled:
|
||||
self._cache_result(
|
||||
result.df,
|
||||
start_date, end_date, trusts, drugs, directories,
|
||||
includes_current_data=end_date is None or end_date >= date.today()
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# Module-level singleton and convenience functions
|
||||
_default_manager: Optional[DataSourceManager] = None
|
||||
|
||||
|
||||
def get_data_source_manager(
|
||||
cache_enabled: bool = True,
|
||||
local_file_path: Optional[Path | str] = None,
|
||||
sqlite_db_path: Optional[Path | str] = None,
|
||||
) -> DataSourceManager:
|
||||
"""
|
||||
Get a DataSourceManager instance.
|
||||
|
||||
Args:
|
||||
cache_enabled: Whether to enable caching
|
||||
local_file_path: Optional path to local CSV/Parquet file
|
||||
sqlite_db_path: Optional path to SQLite database
|
||||
|
||||
Returns:
|
||||
DataSourceManager instance
|
||||
"""
|
||||
global _default_manager
|
||||
|
||||
# If custom paths provided, create a new manager
|
||||
if local_file_path or sqlite_db_path:
|
||||
return DataSourceManager(
|
||||
cache_enabled=cache_enabled,
|
||||
local_file_path=local_file_path,
|
||||
sqlite_db_path=sqlite_db_path,
|
||||
)
|
||||
|
||||
# Otherwise use/create singleton
|
||||
if _default_manager is None:
|
||||
_default_manager = DataSourceManager(cache_enabled=cache_enabled)
|
||||
|
||||
return _default_manager
|
||||
|
||||
|
||||
def get_data(
|
||||
start_date: Optional[date] = None,
|
||||
end_date: Optional[date] = None,
|
||||
trusts: Optional[list[str]] = None,
|
||||
drugs: Optional[list[str]] = None,
|
||||
directories: Optional[list[str]] = None,
|
||||
preferred_source: Optional[str] = None,
|
||||
skip_cache: bool = False,
|
||||
) -> DataSourceResult:
|
||||
"""
|
||||
Convenience function to get data using the default manager.
|
||||
|
||||
Args:
|
||||
start_date: Optional start date for filtering
|
||||
end_date: Optional end date for filtering
|
||||
trusts: Optional list of trust names
|
||||
drugs: Optional list of drug names
|
||||
directories: Optional list of directories
|
||||
preferred_source: Optional preferred source
|
||||
skip_cache: If True, bypass cache
|
||||
|
||||
Returns:
|
||||
DataSourceResult with loaded data
|
||||
"""
|
||||
manager = get_data_source_manager()
|
||||
return manager.get_data(
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
trusts=trusts,
|
||||
drugs=drugs,
|
||||
directories=directories,
|
||||
preferred_source=preferred_source,
|
||||
skip_cache=skip_cache,
|
||||
)
|
||||
|
||||
|
||||
def reset_data_source_manager() -> None:
|
||||
"""Reset the default data source manager singleton."""
|
||||
global _default_manager
|
||||
_default_manager = None
|
||||
|
||||
|
||||
# Export public API
|
||||
__all__ = [
|
||||
"DataSourceType",
|
||||
"DataSourceResult",
|
||||
"SourceStatus",
|
||||
"DataSourceManager",
|
||||
"get_data_source_manager",
|
||||
"get_data",
|
||||
"reset_data_source_manager",
|
||||
]
|
||||
@@ -0,0 +1,239 @@
|
||||
"""
|
||||
SQLite database connection management for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Provides connection management, schema initialization, and common database operations.
|
||||
Uses context manager pattern for safe resource handling.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import Optional, Generator, Literal
|
||||
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class DatabaseConfig:
|
||||
"""
|
||||
Configuration for SQLite database location and connection parameters.
|
||||
|
||||
Attributes:
|
||||
db_path: Path to the SQLite database file
|
||||
timeout: Connection timeout in seconds (default: 30)
|
||||
isolation_level: Transaction isolation level (default: None for autocommit)
|
||||
"""
|
||||
|
||||
DEFAULT_DB_NAME = "pathways.db"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
db_path: Optional[Path] = None,
|
||||
data_dir: Optional[Path] = None,
|
||||
timeout: float = 30.0,
|
||||
isolation_level: Optional[Literal['DEFERRED', 'EXCLUSIVE', 'IMMEDIATE']] = None
|
||||
):
|
||||
"""
|
||||
Initialize database configuration.
|
||||
|
||||
Args:
|
||||
db_path: Full path to database file. If None, uses data_dir/DEFAULT_DB_NAME.
|
||||
data_dir: Directory to place database in. Defaults to ./data/
|
||||
timeout: Connection timeout in seconds.
|
||||
isolation_level: Transaction isolation level. None = autocommit.
|
||||
"""
|
||||
if db_path is not None:
|
||||
self.db_path = Path(db_path)
|
||||
elif data_dir is not None:
|
||||
self.db_path = Path(data_dir) / self.DEFAULT_DB_NAME
|
||||
else:
|
||||
self.db_path = Path("./data") / self.DEFAULT_DB_NAME
|
||||
|
||||
self.timeout = timeout
|
||||
self.isolation_level = isolation_level
|
||||
|
||||
def validate(self) -> list[str]:
|
||||
"""
|
||||
Validate database configuration.
|
||||
|
||||
Returns:
|
||||
List of error messages. Empty list means configuration is valid.
|
||||
"""
|
||||
errors = []
|
||||
|
||||
# Check parent directory exists
|
||||
parent_dir = self.db_path.parent
|
||||
if not parent_dir.exists():
|
||||
errors.append(f"Database directory does not exist: {parent_dir}")
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
class DatabaseManager:
|
||||
"""
|
||||
Manages SQLite database connections and operations.
|
||||
|
||||
Provides context manager for safe connection handling and methods
|
||||
for common database operations.
|
||||
|
||||
Usage:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
# Using context manager (recommended)
|
||||
with db_manager.get_connection() as conn:
|
||||
cursor = conn.execute("SELECT * FROM ref_drug_names")
|
||||
results = cursor.fetchall()
|
||||
|
||||
# Or get a managed connection for longer operations
|
||||
conn = db_manager.connect()
|
||||
try:
|
||||
# ... do work ...
|
||||
finally:
|
||||
conn.close()
|
||||
"""
|
||||
|
||||
def __init__(self, config: Optional[DatabaseConfig] = None):
|
||||
"""
|
||||
Initialize the database manager.
|
||||
|
||||
Args:
|
||||
config: Database configuration. If None, uses default configuration.
|
||||
"""
|
||||
self.config = config or DatabaseConfig()
|
||||
self._connection: Optional[sqlite3.Connection] = None
|
||||
|
||||
@property
|
||||
def db_path(self) -> Path:
|
||||
"""Path to the SQLite database file."""
|
||||
return self.config.db_path
|
||||
|
||||
@property
|
||||
def exists(self) -> bool:
|
||||
"""Check if the database file exists."""
|
||||
return self.db_path.exists()
|
||||
|
||||
def connect(self) -> sqlite3.Connection:
|
||||
"""
|
||||
Create a new database connection.
|
||||
|
||||
Returns:
|
||||
sqlite3.Connection: New database connection.
|
||||
|
||||
Note:
|
||||
The caller is responsible for closing the connection.
|
||||
Consider using get_connection() context manager instead.
|
||||
"""
|
||||
conn = sqlite3.connect(
|
||||
str(self.db_path),
|
||||
timeout=self.config.timeout,
|
||||
isolation_level=self.config.isolation_level
|
||||
)
|
||||
# Enable foreign key support
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
# Return rows as sqlite3.Row for dict-like access
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
@contextmanager
|
||||
def get_connection(self) -> Generator[sqlite3.Connection, None, None]:
|
||||
"""
|
||||
Context manager for database connections.
|
||||
|
||||
Yields:
|
||||
sqlite3.Connection: Database connection.
|
||||
|
||||
Example:
|
||||
with db_manager.get_connection() as conn:
|
||||
conn.execute("INSERT INTO table VALUES (?)", (value,))
|
||||
conn.commit()
|
||||
"""
|
||||
conn = self.connect()
|
||||
try:
|
||||
yield conn
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
@contextmanager
|
||||
def get_transaction(self) -> Generator[sqlite3.Connection, None, None]:
|
||||
"""
|
||||
Context manager for transactional operations.
|
||||
|
||||
Automatically commits on success, rolls back on exception.
|
||||
|
||||
Yields:
|
||||
sqlite3.Connection: Database connection in transaction mode.
|
||||
|
||||
Example:
|
||||
with db_manager.get_transaction() as conn:
|
||||
conn.execute("INSERT INTO table VALUES (?)", (value1,))
|
||||
conn.execute("INSERT INTO other_table VALUES (?)", (value2,))
|
||||
# Auto-commits if no exception
|
||||
"""
|
||||
conn = sqlite3.connect(
|
||||
str(self.db_path),
|
||||
timeout=self.config.timeout,
|
||||
isolation_level="DEFERRED" # Explicit transaction mode
|
||||
)
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
yield conn
|
||||
conn.commit()
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def execute_script(self, sql_script: str) -> None:
|
||||
"""
|
||||
Execute a SQL script (multiple statements).
|
||||
|
||||
Args:
|
||||
sql_script: SQL script containing one or more statements.
|
||||
"""
|
||||
with self.get_connection() as conn:
|
||||
conn.executescript(sql_script)
|
||||
logger.info("Executed SQL script successfully")
|
||||
|
||||
def table_exists(self, table_name: str) -> bool:
|
||||
"""
|
||||
Check if a table exists in the database.
|
||||
|
||||
Args:
|
||||
table_name: Name of the table to check.
|
||||
|
||||
Returns:
|
||||
True if the table exists, False otherwise.
|
||||
"""
|
||||
with self.get_connection() as conn:
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?",
|
||||
(table_name,)
|
||||
)
|
||||
return cursor.fetchone() is not None
|
||||
|
||||
def get_table_count(self, table_name: str) -> int:
|
||||
"""
|
||||
Get the row count for a table.
|
||||
|
||||
Args:
|
||||
table_name: Name of the table.
|
||||
|
||||
Returns:
|
||||
Number of rows in the table.
|
||||
"""
|
||||
with self.get_connection() as conn:
|
||||
# Use parameterized table name via string formatting (safe since we control table_name)
|
||||
cursor = conn.execute(f"SELECT COUNT(*) FROM {table_name}")
|
||||
result = cursor.fetchone()
|
||||
return result[0] if result else 0
|
||||
|
||||
|
||||
# Default instance for application-wide use
|
||||
default_db_config = DatabaseConfig()
|
||||
default_db_manager = DatabaseManager(default_db_config)
|
||||
@@ -0,0 +1,581 @@
|
||||
"""
|
||||
Diagnosis lookup module for NHS Patient Pathway Analysis.
|
||||
|
||||
Provides functions to validate patient indications by checking GP diagnosis records
|
||||
against SNOMED cluster codes. Uses the drug-to-cluster mapping from
|
||||
drug_indication_clusters.csv and queries Snowflake for SNOMED codes and GP records.
|
||||
|
||||
Key workflow:
|
||||
1. Get drug's valid indication clusters from local mapping
|
||||
2. Get all SNOMED codes for those clusters from Snowflake
|
||||
3. Check if patient has any of those SNOMED codes in GP records
|
||||
4. Report indication validation status
|
||||
|
||||
IMPORTANT: HCD activity data indication codes are UNRELIABLE. This module uses
|
||||
GP/Primary Care data (PrimaryCareClinicalCoding) as the authoritative source.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Callable, Any, cast
|
||||
import csv
|
||||
|
||||
from core.logging_config import get_logger
|
||||
from data_processing.database import DatabaseManager, default_db_manager
|
||||
from data_processing.snowflake_connector import (
|
||||
SnowflakeConnector,
|
||||
get_connector,
|
||||
is_snowflake_available,
|
||||
is_snowflake_configured,
|
||||
SNOWFLAKE_AVAILABLE,
|
||||
)
|
||||
from data_processing.cache import get_cache, is_cache_enabled
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClusterSnomedCodes:
|
||||
"""SNOMED codes for a clinical coding cluster."""
|
||||
cluster_id: str
|
||||
cluster_description: str
|
||||
snomed_codes: list[str] = field(default_factory=list)
|
||||
snomed_descriptions: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def code_count(self) -> int:
|
||||
return len(self.snomed_codes)
|
||||
|
||||
|
||||
@dataclass
|
||||
class IndicationValidationResult:
|
||||
"""Result of validating a patient's indication for a drug."""
|
||||
patient_pseudonym: str
|
||||
drug_name: str
|
||||
has_valid_indication: bool
|
||||
matched_cluster_id: Optional[str] = None
|
||||
matched_snomed_code: Optional[str] = None
|
||||
matched_snomed_description: Optional[str] = None
|
||||
checked_clusters: list[str] = field(default_factory=list)
|
||||
total_codes_checked: int = 0
|
||||
source: str = "GP_SNOMED" # GP_SNOMED | NONE
|
||||
error_message: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class DrugIndicationMatchRate:
|
||||
"""Match rate statistics for a drug's indication validation."""
|
||||
drug_name: str
|
||||
total_patients: int
|
||||
patients_with_indication: int
|
||||
patients_without_indication: int
|
||||
match_rate: float # 0.0 to 1.0
|
||||
clusters_checked: list[str] = field(default_factory=list)
|
||||
sample_unmatched: list[str] = field(default_factory=list) # Sample patient IDs
|
||||
|
||||
|
||||
def get_drug_clusters(
|
||||
drug_name: str,
|
||||
db_manager: Optional[DatabaseManager] = None
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Get all SNOMED cluster mappings for a drug from local SQLite.
|
||||
|
||||
Args:
|
||||
drug_name: Drug name to look up (case-insensitive)
|
||||
db_manager: Optional DatabaseManager (defaults to default_db_manager)
|
||||
|
||||
Returns:
|
||||
List of dicts with keys: drug_name, indication, cluster_id,
|
||||
cluster_description, nice_ta_reference
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = default_db_manager
|
||||
|
||||
query = """
|
||||
SELECT drug_name, indication, cluster_id, cluster_description, nice_ta_reference
|
||||
FROM ref_drug_indication_clusters
|
||||
WHERE UPPER(drug_name) = UPPER(?)
|
||||
ORDER BY indication, cluster_id
|
||||
"""
|
||||
|
||||
try:
|
||||
with db_manager.get_connection() as conn:
|
||||
cursor = conn.execute(query, (drug_name,))
|
||||
rows = cursor.fetchall()
|
||||
|
||||
results = []
|
||||
for row in rows:
|
||||
results.append({
|
||||
"drug_name": row["drug_name"],
|
||||
"indication": row["indication"],
|
||||
"cluster_id": row["cluster_id"],
|
||||
"cluster_description": row["cluster_description"],
|
||||
"nice_ta_reference": row["nice_ta_reference"],
|
||||
})
|
||||
|
||||
logger.debug(f"Found {len(results)} cluster mappings for drug '{drug_name}'")
|
||||
return results
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting clusters for drug '{drug_name}': {e}")
|
||||
return []
|
||||
|
||||
|
||||
def get_drug_cluster_ids(
|
||||
drug_name: str,
|
||||
db_manager: Optional[DatabaseManager] = None
|
||||
) -> list[str]:
|
||||
"""
|
||||
Get unique cluster IDs for a drug.
|
||||
|
||||
Args:
|
||||
drug_name: Drug name to look up
|
||||
db_manager: Optional DatabaseManager
|
||||
|
||||
Returns:
|
||||
List of unique cluster IDs
|
||||
"""
|
||||
clusters = get_drug_clusters(drug_name, db_manager)
|
||||
return list(set(c["cluster_id"] for c in clusters))
|
||||
|
||||
|
||||
def get_cluster_snomed_codes(
|
||||
cluster_id: str,
|
||||
connector: Optional[SnowflakeConnector] = None,
|
||||
use_cache: bool = True,
|
||||
) -> ClusterSnomedCodes:
|
||||
"""
|
||||
Get all SNOMED codes for a cluster from Snowflake.
|
||||
|
||||
Queries the ClinicalCodingClusterSnomedCodes table to get all SNOMED codes
|
||||
that belong to the specified cluster.
|
||||
|
||||
Args:
|
||||
cluster_id: Cluster ID to look up (e.g., 'RARTH_COD', 'PSORIASIS_COD')
|
||||
connector: Optional SnowflakeConnector (defaults to singleton)
|
||||
use_cache: Whether to use cached results (default True)
|
||||
|
||||
Returns:
|
||||
ClusterSnomedCodes with list of SNOMED codes and descriptions
|
||||
"""
|
||||
if not SNOWFLAKE_AVAILABLE:
|
||||
logger.warning("Snowflake connector not available")
|
||||
return ClusterSnomedCodes(cluster_id=cluster_id, cluster_description="")
|
||||
|
||||
if not is_snowflake_configured():
|
||||
logger.warning("Snowflake not configured - cannot get cluster codes")
|
||||
return ClusterSnomedCodes(cluster_id=cluster_id, cluster_description="")
|
||||
|
||||
# Check cache first
|
||||
cache_key = f"cluster_snomed_{cluster_id}"
|
||||
if use_cache and is_cache_enabled():
|
||||
cache = get_cache()
|
||||
cached = cache.get(cache_key)
|
||||
if cached is not None and len(cached) > 0:
|
||||
logger.debug(f"Using cached SNOMED codes for cluster '{cluster_id}'")
|
||||
cached_dict = cached[0] # First element is our data dict
|
||||
return ClusterSnomedCodes(
|
||||
cluster_id=cluster_id,
|
||||
cluster_description=str(cached_dict.get("description", "")),
|
||||
snomed_codes=list(cached_dict.get("codes", [])),
|
||||
snomed_descriptions=dict(cached_dict.get("descriptions", {})),
|
||||
)
|
||||
|
||||
if connector is None:
|
||||
connector = get_connector()
|
||||
|
||||
query = '''
|
||||
SELECT DISTINCT
|
||||
"Cluster_ID",
|
||||
"Cluster_Description",
|
||||
"SNOMEDCode",
|
||||
"SNOMEDDescription"
|
||||
FROM DATA_HUB.PHM."ClinicalCodingClusterSnomedCodes"
|
||||
WHERE "Cluster_ID" = %s
|
||||
ORDER BY "SNOMEDCode"
|
||||
'''
|
||||
|
||||
try:
|
||||
results = connector.execute_dict(query, (cluster_id,))
|
||||
|
||||
if not results:
|
||||
logger.warning(f"No SNOMED codes found for cluster '{cluster_id}'")
|
||||
return ClusterSnomedCodes(cluster_id=cluster_id, cluster_description="")
|
||||
|
||||
codes = []
|
||||
descriptions = {}
|
||||
description = results[0].get("Cluster_Description", "") if results else ""
|
||||
|
||||
for row in results:
|
||||
code = row.get("SNOMEDCode")
|
||||
if code:
|
||||
codes.append(code)
|
||||
descriptions[code] = row.get("SNOMEDDescription", "")
|
||||
|
||||
logger.info(f"Found {len(codes)} SNOMED codes for cluster '{cluster_id}'")
|
||||
|
||||
# Cache the results (using query-based cache with fake params)
|
||||
if use_cache and is_cache_enabled():
|
||||
cache = get_cache()
|
||||
cache_data = [{
|
||||
"description": description,
|
||||
"codes": codes,
|
||||
"descriptions": descriptions,
|
||||
}]
|
||||
cache.set(cache_key, None, cache_data) # type: ignore[arg-type]
|
||||
|
||||
return ClusterSnomedCodes(
|
||||
cluster_id=cluster_id,
|
||||
cluster_description=description,
|
||||
snomed_codes=codes,
|
||||
snomed_descriptions=descriptions,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting SNOMED codes for cluster '{cluster_id}': {e}")
|
||||
return ClusterSnomedCodes(cluster_id=cluster_id, cluster_description="")
|
||||
|
||||
|
||||
def patient_has_indication(
|
||||
patient_pseudonym: str,
|
||||
cluster_ids: list[str],
|
||||
connector: Optional[SnowflakeConnector] = None,
|
||||
before_date: Optional[date] = None,
|
||||
) -> tuple[bool, Optional[str], Optional[str], Optional[str]]:
|
||||
"""
|
||||
Check if a patient has any SNOMED codes from the specified clusters in GP records.
|
||||
|
||||
Args:
|
||||
patient_pseudonym: Patient's pseudonymised NHS number
|
||||
cluster_ids: List of cluster IDs to check against
|
||||
connector: Optional SnowflakeConnector
|
||||
before_date: Optional date - only check diagnoses before this date
|
||||
|
||||
Returns:
|
||||
Tuple of (has_indication, matched_cluster_id, matched_snomed_code, matched_description)
|
||||
"""
|
||||
if not SNOWFLAKE_AVAILABLE or not is_snowflake_configured():
|
||||
return False, None, None, None
|
||||
|
||||
if not cluster_ids:
|
||||
return False, None, None, None
|
||||
|
||||
if connector is None:
|
||||
connector = get_connector()
|
||||
|
||||
# Build placeholders for cluster IDs
|
||||
placeholders = ", ".join(["%s"] * len(cluster_ids))
|
||||
|
||||
# Query to check if patient has any matching SNOMED code
|
||||
query = f'''
|
||||
SELECT
|
||||
pc."SNOMEDCode",
|
||||
cc."Cluster_ID",
|
||||
cc."SNOMEDDescription"
|
||||
FROM DATA_HUB.PHM."PrimaryCareClinicalCoding" pc
|
||||
INNER JOIN DATA_HUB.PHM."ClinicalCodingClusterSnomedCodes" cc
|
||||
ON pc."SNOMEDCode" = cc."SNOMEDCode"
|
||||
WHERE pc."PatientPseudonym" = %s
|
||||
AND cc."Cluster_ID" IN ({placeholders})
|
||||
'''
|
||||
|
||||
params = [patient_pseudonym] + cluster_ids
|
||||
|
||||
if before_date:
|
||||
query += ' AND pc."EventDateTime" < %s'
|
||||
params.append(before_date.isoformat())
|
||||
|
||||
query += ' LIMIT 1'
|
||||
|
||||
try:
|
||||
results = connector.execute_dict(query, tuple(params))
|
||||
|
||||
if results:
|
||||
row = results[0]
|
||||
return (
|
||||
True,
|
||||
row.get("Cluster_ID"),
|
||||
row.get("SNOMEDCode"),
|
||||
row.get("SNOMEDDescription"),
|
||||
)
|
||||
|
||||
return False, None, None, None
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking indication for patient '{patient_pseudonym}': {e}")
|
||||
return False, None, None, None
|
||||
|
||||
|
||||
def validate_indication(
|
||||
patient_pseudonym: str,
|
||||
drug_name: str,
|
||||
connector: Optional[SnowflakeConnector] = None,
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
before_date: Optional[date] = None,
|
||||
) -> IndicationValidationResult:
|
||||
"""
|
||||
Validate that a patient has an appropriate indication for a drug.
|
||||
|
||||
Full validation workflow:
|
||||
1. Get drug's valid indication clusters from local mapping
|
||||
2. Check if patient has any matching SNOMED codes in GP records
|
||||
3. Return detailed validation result
|
||||
|
||||
Args:
|
||||
patient_pseudonym: Patient's pseudonymised NHS number
|
||||
drug_name: Drug name to validate indication for
|
||||
connector: Optional SnowflakeConnector
|
||||
db_manager: Optional DatabaseManager
|
||||
before_date: Optional date - only check diagnoses before this date
|
||||
|
||||
Returns:
|
||||
IndicationValidationResult with validation details
|
||||
"""
|
||||
result = IndicationValidationResult(
|
||||
patient_pseudonym=patient_pseudonym,
|
||||
drug_name=drug_name,
|
||||
has_valid_indication=False,
|
||||
)
|
||||
|
||||
# Step 1: Get drug's cluster mappings
|
||||
cluster_ids = get_drug_cluster_ids(drug_name, db_manager)
|
||||
|
||||
if not cluster_ids:
|
||||
result.error_message = f"No cluster mappings found for drug '{drug_name}'"
|
||||
result.source = "NONE"
|
||||
return result
|
||||
|
||||
result.checked_clusters = cluster_ids
|
||||
|
||||
# Step 2: Check Snowflake availability
|
||||
if not SNOWFLAKE_AVAILABLE:
|
||||
result.error_message = "Snowflake connector not installed"
|
||||
result.source = "NONE"
|
||||
return result
|
||||
|
||||
if not is_snowflake_configured():
|
||||
result.error_message = "Snowflake not configured"
|
||||
result.source = "NONE"
|
||||
return result
|
||||
|
||||
# Step 3: Check patient GP records
|
||||
has_indication, matched_cluster, matched_code, matched_desc = patient_has_indication(
|
||||
patient_pseudonym=patient_pseudonym,
|
||||
cluster_ids=cluster_ids,
|
||||
connector=connector,
|
||||
before_date=before_date,
|
||||
)
|
||||
|
||||
result.has_valid_indication = has_indication
|
||||
result.matched_cluster_id = matched_cluster
|
||||
result.matched_snomed_code = matched_code
|
||||
result.matched_snomed_description = matched_desc
|
||||
result.source = "GP_SNOMED" if has_indication else "NONE"
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def get_indication_match_rate(
|
||||
drug_name: str,
|
||||
patient_pseudonyms: list[str],
|
||||
connector: Optional[SnowflakeConnector] = None,
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
sample_unmatched_count: int = 10,
|
||||
) -> DrugIndicationMatchRate:
|
||||
"""
|
||||
Calculate indication match rate for a drug across a list of patients.
|
||||
|
||||
Args:
|
||||
drug_name: Drug name to check
|
||||
patient_pseudonyms: List of patient pseudonymised NHS numbers
|
||||
connector: Optional SnowflakeConnector
|
||||
db_manager: Optional DatabaseManager
|
||||
sample_unmatched_count: Number of unmatched patient IDs to include in sample
|
||||
|
||||
Returns:
|
||||
DrugIndicationMatchRate with match statistics
|
||||
"""
|
||||
if connector is None and SNOWFLAKE_AVAILABLE and is_snowflake_configured():
|
||||
connector = get_connector()
|
||||
|
||||
cluster_ids = get_drug_cluster_ids(drug_name, db_manager)
|
||||
|
||||
total = len(patient_pseudonyms)
|
||||
matched = 0
|
||||
unmatched = 0
|
||||
sample_unmatched: list[str] = []
|
||||
|
||||
if not cluster_ids:
|
||||
logger.warning(f"No cluster mappings for drug '{drug_name}' - all patients will be unmatched")
|
||||
return DrugIndicationMatchRate(
|
||||
drug_name=drug_name,
|
||||
total_patients=total,
|
||||
patients_with_indication=0,
|
||||
patients_without_indication=total,
|
||||
match_rate=0.0,
|
||||
clusters_checked=[],
|
||||
sample_unmatched=patient_pseudonyms[:sample_unmatched_count],
|
||||
)
|
||||
|
||||
for i, pseudonym in enumerate(patient_pseudonyms):
|
||||
if i > 0 and i % 100 == 0:
|
||||
logger.info(f"Validating indications: {i}/{total} ({100*i/total:.1f}%)")
|
||||
|
||||
has_indication, _, _, _ = patient_has_indication(
|
||||
patient_pseudonym=pseudonym,
|
||||
cluster_ids=cluster_ids,
|
||||
connector=connector,
|
||||
)
|
||||
|
||||
if has_indication:
|
||||
matched += 1
|
||||
else:
|
||||
unmatched += 1
|
||||
if len(sample_unmatched) < sample_unmatched_count:
|
||||
sample_unmatched.append(pseudonym)
|
||||
|
||||
match_rate = matched / total if total > 0 else 0.0
|
||||
|
||||
logger.info(f"Indication match rate for '{drug_name}': {100*match_rate:.1f}% ({matched}/{total})")
|
||||
|
||||
return DrugIndicationMatchRate(
|
||||
drug_name=drug_name,
|
||||
total_patients=total,
|
||||
patients_with_indication=matched,
|
||||
patients_without_indication=unmatched,
|
||||
match_rate=match_rate,
|
||||
clusters_checked=cluster_ids,
|
||||
sample_unmatched=sample_unmatched,
|
||||
)
|
||||
|
||||
|
||||
def batch_validate_indications(
|
||||
patient_drug_pairs: list[tuple[str, str]],
|
||||
connector: Optional[SnowflakeConnector] = None,
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
progress_callback: Optional[Callable[[int, int], None]] = None,
|
||||
) -> list[IndicationValidationResult]:
|
||||
"""
|
||||
Validate indications for multiple patient-drug pairs efficiently.
|
||||
|
||||
Args:
|
||||
patient_drug_pairs: List of (patient_pseudonym, drug_name) tuples
|
||||
connector: Optional SnowflakeConnector
|
||||
db_manager: Optional DatabaseManager
|
||||
progress_callback: Optional callback(current, total) for progress updates
|
||||
|
||||
Returns:
|
||||
List of IndicationValidationResult for each pair
|
||||
"""
|
||||
results = []
|
||||
total = len(patient_drug_pairs)
|
||||
|
||||
# Cache cluster lookups by drug
|
||||
drug_clusters_cache = {}
|
||||
|
||||
for i, (pseudonym, drug_name) in enumerate(patient_drug_pairs):
|
||||
if progress_callback:
|
||||
progress_callback(i + 1, total)
|
||||
|
||||
# Get clusters from cache or lookup
|
||||
drug_upper = drug_name.upper()
|
||||
if drug_upper not in drug_clusters_cache:
|
||||
drug_clusters_cache[drug_upper] = get_drug_cluster_ids(drug_name, db_manager)
|
||||
|
||||
cluster_ids = drug_clusters_cache[drug_upper]
|
||||
|
||||
if not cluster_ids:
|
||||
results.append(IndicationValidationResult(
|
||||
patient_pseudonym=pseudonym,
|
||||
drug_name=drug_name,
|
||||
has_valid_indication=False,
|
||||
source="NONE",
|
||||
error_message=f"No cluster mappings for drug '{drug_name}'",
|
||||
))
|
||||
continue
|
||||
|
||||
# Check patient indication
|
||||
has_indication, matched_cluster, matched_code, matched_desc = patient_has_indication(
|
||||
patient_pseudonym=pseudonym,
|
||||
cluster_ids=cluster_ids,
|
||||
connector=connector,
|
||||
)
|
||||
|
||||
results.append(IndicationValidationResult(
|
||||
patient_pseudonym=pseudonym,
|
||||
drug_name=drug_name,
|
||||
has_valid_indication=has_indication,
|
||||
matched_cluster_id=matched_cluster,
|
||||
matched_snomed_code=matched_code,
|
||||
matched_snomed_description=matched_desc,
|
||||
checked_clusters=cluster_ids,
|
||||
source="GP_SNOMED" if has_indication else "NONE",
|
||||
))
|
||||
|
||||
matched_count = sum(1 for r in results if r.has_valid_indication)
|
||||
logger.info(f"Batch validation complete: {matched_count}/{total} ({100*matched_count/total:.1f}%) with valid indications")
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def get_available_clusters(
|
||||
connector: Optional[SnowflakeConnector] = None,
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Get list of all available SNOMED clusters from Snowflake.
|
||||
|
||||
Returns:
|
||||
List of dicts with cluster_id, cluster_description, code_count
|
||||
"""
|
||||
if not SNOWFLAKE_AVAILABLE or not is_snowflake_configured():
|
||||
logger.warning("Snowflake not available - cannot list clusters")
|
||||
return []
|
||||
|
||||
if connector is None:
|
||||
connector = get_connector()
|
||||
|
||||
query = '''
|
||||
SELECT
|
||||
"Cluster_ID",
|
||||
"Cluster_Description",
|
||||
COUNT(DISTINCT "SNOMEDCode") as code_count
|
||||
FROM DATA_HUB.PHM."ClinicalCodingClusterSnomedCodes"
|
||||
GROUP BY "Cluster_ID", "Cluster_Description"
|
||||
ORDER BY "Cluster_ID"
|
||||
'''
|
||||
|
||||
try:
|
||||
results = connector.execute_dict(query)
|
||||
|
||||
clusters = []
|
||||
for row in results:
|
||||
clusters.append({
|
||||
"cluster_id": row.get("Cluster_ID"),
|
||||
"cluster_description": row.get("Cluster_Description"),
|
||||
"code_count": row.get("code_count", 0),
|
||||
})
|
||||
|
||||
logger.info(f"Found {len(clusters)} available SNOMED clusters")
|
||||
return clusters
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting available clusters: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# Export public API
|
||||
__all__ = [
|
||||
"ClusterSnomedCodes",
|
||||
"IndicationValidationResult",
|
||||
"DrugIndicationMatchRate",
|
||||
"get_drug_clusters",
|
||||
"get_drug_cluster_ids",
|
||||
"get_cluster_snomed_codes",
|
||||
"patient_has_indication",
|
||||
"validate_indication",
|
||||
"get_indication_match_rate",
|
||||
"batch_validate_indications",
|
||||
"get_available_clusters",
|
||||
]
|
||||
@@ -0,0 +1,399 @@
|
||||
"""
|
||||
Data loader abstractions for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Provides a unified interface for loading patient intervention data from:
|
||||
- CSV/Parquet files (current behavior)
|
||||
- SQLite database (new, faster approach)
|
||||
- Snowflake (future, direct from warehouse)
|
||||
|
||||
The DataLoader ABC defines the contract for all loader implementations.
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from core import PathConfig, default_paths
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoadResult:
|
||||
"""Result of a data load operation.
|
||||
|
||||
Attributes:
|
||||
df: The loaded DataFrame with processed patient intervention data
|
||||
source: Description of the data source (e.g., "csv:/path/to/file.csv", "sqlite:fact_interventions")
|
||||
row_count: Number of rows loaded
|
||||
columns: List of column names in the DataFrame
|
||||
load_time_seconds: Time taken to load the data
|
||||
"""
|
||||
df: pd.DataFrame
|
||||
source: str
|
||||
row_count: int
|
||||
columns: list[str] = field(default_factory=list)
|
||||
load_time_seconds: float = 0.0
|
||||
|
||||
def __post_init__(self):
|
||||
if not self.columns:
|
||||
self.columns = list(self.df.columns)
|
||||
|
||||
|
||||
# Expected columns in a processed DataFrame
|
||||
# These are the columns that generate_graph() expects to receive
|
||||
REQUIRED_COLUMNS = [
|
||||
"UPID", # Unique Patient ID (Provider Code prefix + PersonKey)
|
||||
"Drug Name", # Standardized drug name
|
||||
"Intervention Date", # Date of intervention
|
||||
"Price Actual", # Cost of intervention
|
||||
"OrganisationName", # NHS Trust name
|
||||
"Directory", # Medical specialty/directory
|
||||
"Provider Code", # NHS provider code
|
||||
"PersonKey", # Patient identifier within provider
|
||||
]
|
||||
|
||||
# Additional columns that are useful but not strictly required
|
||||
OPTIONAL_COLUMNS = [
|
||||
"UPIDTreatment", # UPID + Drug Name combo (created by generate_graph)
|
||||
"Treatment Function Code", # NHS treatment function code
|
||||
"Additional Detail 1",
|
||||
"Additional Detail 2",
|
||||
"Additional Detail 3",
|
||||
"Additional Detail 4",
|
||||
"Additional Detail 5",
|
||||
]
|
||||
|
||||
|
||||
class DataLoader(ABC):
|
||||
"""Abstract base class for data loaders.
|
||||
|
||||
All data loaders must implement the load() method which returns
|
||||
a DataFrame ready for use by generate_graph().
|
||||
|
||||
The returned DataFrame must contain REQUIRED_COLUMNS at minimum.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def load(self) -> LoadResult:
|
||||
"""Load and process patient intervention data.
|
||||
|
||||
Returns:
|
||||
LoadResult containing the processed DataFrame and metadata.
|
||||
The DataFrame must contain all REQUIRED_COLUMNS.
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If the data source doesn't exist
|
||||
ValueError: If the data is malformed or missing required columns
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def validate_source(self) -> tuple[bool, str]:
|
||||
"""Check if the data source is valid and accessible.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, message).
|
||||
If is_valid is False, message explains the issue.
|
||||
"""
|
||||
pass
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def source_description(self) -> str:
|
||||
"""Human-readable description of the data source."""
|
||||
pass
|
||||
|
||||
def validate_dataframe(self, df: pd.DataFrame) -> tuple[bool, list[str]]:
|
||||
"""Validate that a DataFrame has all required columns.
|
||||
|
||||
Args:
|
||||
df: DataFrame to validate
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, missing_columns).
|
||||
If is_valid is False, missing_columns lists what's missing.
|
||||
"""
|
||||
missing = [col for col in REQUIRED_COLUMNS if col not in df.columns]
|
||||
return len(missing) == 0, missing
|
||||
|
||||
|
||||
class FileDataLoader(DataLoader):
|
||||
"""Loads data from CSV or Parquet files.
|
||||
|
||||
This replicates the current behavior of dashboard_gui.main():
|
||||
1. Read CSV or Parquet file
|
||||
2. Apply patient_id() transformation
|
||||
3. Convert dates
|
||||
4. Apply drug_names() standardization
|
||||
5. Clean organization names
|
||||
6. Apply department_identification()
|
||||
|
||||
Args:
|
||||
file_path: Path to the CSV or Parquet file
|
||||
paths: PathConfig for reference data file locations (uses default_paths if None)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path: Path | str,
|
||||
paths: Optional[PathConfig] = None,
|
||||
):
|
||||
self.file_path = Path(file_path)
|
||||
self.paths = paths or default_paths
|
||||
|
||||
def validate_source(self) -> tuple[bool, str]:
|
||||
"""Check if the file exists and has a supported extension."""
|
||||
if not self.file_path.exists():
|
||||
return False, f"File not found: {self.file_path}"
|
||||
|
||||
ext = self.file_path.suffix.lower()
|
||||
if ext not in ('.csv', '.parquet'):
|
||||
return False, f"Unsupported file type: {ext}. Must be .csv or .parquet"
|
||||
|
||||
return True, "OK"
|
||||
|
||||
@property
|
||||
def source_description(self) -> str:
|
||||
return f"file:{self.file_path}"
|
||||
|
||||
def load(self) -> LoadResult:
|
||||
"""Load and process data from CSV or Parquet file.
|
||||
|
||||
Applies the same transformation pipeline as the original
|
||||
dashboard_gui.main() function.
|
||||
"""
|
||||
import time
|
||||
from tools import data
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Validate source before loading
|
||||
is_valid, msg = self.validate_source()
|
||||
if not is_valid:
|
||||
raise FileNotFoundError(msg)
|
||||
|
||||
# Read file based on extension
|
||||
ext = self.file_path.suffix.lower()
|
||||
logger.info(f"Reading {ext} file: {self.file_path}")
|
||||
|
||||
if ext == '.csv':
|
||||
df_raw = pd.read_csv(self.file_path, low_memory=False)
|
||||
else: # .parquet
|
||||
df_raw = pd.read_parquet(self.file_path)
|
||||
|
||||
logger.info(f"File read successfully. {len(df_raw)} rows.")
|
||||
|
||||
# Apply transformations (same as dashboard_gui.main())
|
||||
df = data.patient_id(df_raw)
|
||||
logger.info("Patient ID processing complete.")
|
||||
|
||||
df['Intervention Date'] = pd.to_datetime(df['Intervention Date'], format="%Y-%m-%d")
|
||||
logger.info("Date conversion complete.")
|
||||
|
||||
# Preserve original drug name before standardization (for SQLite storage)
|
||||
df['Drug Name Raw'] = df['Drug Name'].copy()
|
||||
|
||||
df = data.drug_names(df, self.paths)
|
||||
logger.info("Drug name processing complete.")
|
||||
|
||||
df['OrganisationName'] = df['OrganisationName'].str.replace(',', '')
|
||||
logger.info("Organisation name cleaning complete.")
|
||||
|
||||
df = data.department_identification(df, self.paths)
|
||||
logger.info("Department identification complete.")
|
||||
|
||||
# Validate result
|
||||
is_valid, missing = self.validate_dataframe(df)
|
||||
if not is_valid:
|
||||
raise ValueError(f"Processed DataFrame missing required columns: {missing}")
|
||||
|
||||
load_time = time.time() - start_time
|
||||
logger.info(f"Data loading complete. {len(df)} rows in {load_time:.2f}s")
|
||||
|
||||
return LoadResult(
|
||||
df=df,
|
||||
source=self.source_description,
|
||||
row_count=len(df),
|
||||
load_time_seconds=load_time,
|
||||
)
|
||||
|
||||
|
||||
class SQLiteDataLoader(DataLoader):
|
||||
"""Loads data from SQLite fact_interventions table.
|
||||
|
||||
This provides faster loading by reading pre-processed data from SQLite
|
||||
instead of re-processing CSV files each time.
|
||||
|
||||
The SQLite database must have been populated by the migration scripts.
|
||||
|
||||
Args:
|
||||
db_path: Path to the SQLite database (uses default if None)
|
||||
date_range: Optional tuple of (start_date, end_date) to filter data
|
||||
trusts: Optional list of trust names to filter
|
||||
drugs: Optional list of drug names to filter
|
||||
directories: Optional list of directories to filter
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
db_path: Optional[Path | str] = None,
|
||||
date_range: Optional[tuple[date, date]] = None,
|
||||
trusts: Optional[list[str]] = None,
|
||||
drugs: Optional[list[str]] = None,
|
||||
directories: Optional[list[str]] = None,
|
||||
):
|
||||
from data_processing.database import default_db_config
|
||||
|
||||
self.db_path = Path(db_path) if db_path else Path(default_db_config.db_path)
|
||||
self.date_range = date_range
|
||||
self.trusts = trusts
|
||||
self.drugs = drugs
|
||||
self.directories = directories
|
||||
|
||||
def validate_source(self) -> tuple[bool, str]:
|
||||
"""Check if the database exists and has the fact_interventions table."""
|
||||
if not self.db_path.exists():
|
||||
return False, f"Database not found: {self.db_path}"
|
||||
|
||||
# Check if fact_interventions table exists
|
||||
from data_processing.database import DatabaseManager, DatabaseConfig
|
||||
|
||||
config = DatabaseConfig(db_path=self.db_path)
|
||||
manager = DatabaseManager(config)
|
||||
|
||||
if not manager.table_exists("fact_interventions"):
|
||||
return False, "fact_interventions table not found in database"
|
||||
|
||||
count = manager.get_table_count("fact_interventions")
|
||||
if count == 0:
|
||||
return False, "fact_interventions table is empty"
|
||||
|
||||
return True, f"OK ({count:,} rows available)"
|
||||
|
||||
@property
|
||||
def source_description(self) -> str:
|
||||
return f"sqlite:{self.db_path}"
|
||||
|
||||
def load(self) -> LoadResult:
|
||||
"""Load data from SQLite fact_interventions table.
|
||||
|
||||
Maps SQLite column names to the expected DataFrame column names.
|
||||
Applies optional filters for date range, trusts, drugs, directories.
|
||||
"""
|
||||
import time
|
||||
from data_processing.database import DatabaseManager, DatabaseConfig
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Validate source
|
||||
is_valid, msg = self.validate_source()
|
||||
if not is_valid:
|
||||
raise FileNotFoundError(msg)
|
||||
|
||||
logger.info(f"Loading data from SQLite: {self.db_path}")
|
||||
|
||||
# Build query with optional filters
|
||||
query = """
|
||||
SELECT
|
||||
upid AS "UPID",
|
||||
provider_code AS "Provider Code",
|
||||
person_key AS "PersonKey",
|
||||
drug_name_std AS "Drug Name",
|
||||
intervention_date AS "Intervention Date",
|
||||
price_actual AS "Price Actual",
|
||||
org_name AS "OrganisationName",
|
||||
directory AS "Directory",
|
||||
treatment_function_code AS "Treatment Function Code",
|
||||
additional_detail_1 AS "Additional Detail 1",
|
||||
additional_detail_2 AS "Additional Detail 2",
|
||||
additional_detail_3 AS "Additional Detail 3",
|
||||
additional_detail_4 AS "Additional Detail 4",
|
||||
additional_detail_5 AS "Additional Detail 5"
|
||||
FROM fact_interventions
|
||||
WHERE 1=1
|
||||
"""
|
||||
params = []
|
||||
|
||||
if self.date_range:
|
||||
start, end = self.date_range
|
||||
query += " AND intervention_date >= ? AND intervention_date < ?"
|
||||
params.extend([str(start), str(end)])
|
||||
|
||||
if self.trusts:
|
||||
placeholders = ','.join('?' * len(self.trusts))
|
||||
query += f" AND org_name IN ({placeholders})"
|
||||
params.extend(self.trusts)
|
||||
|
||||
if self.drugs:
|
||||
placeholders = ','.join('?' * len(self.drugs))
|
||||
query += f" AND drug_name_std IN ({placeholders})"
|
||||
params.extend(self.drugs)
|
||||
|
||||
if self.directories:
|
||||
placeholders = ','.join('?' * len(self.directories))
|
||||
query += f" AND directory IN ({placeholders})"
|
||||
params.extend(self.directories)
|
||||
|
||||
# Execute query
|
||||
config = DatabaseConfig(db_path=self.db_path)
|
||||
manager = DatabaseManager(config)
|
||||
|
||||
with manager.get_connection() as conn:
|
||||
df = pd.read_sql_query(query, conn, params=params)
|
||||
|
||||
# Convert intervention_date to datetime
|
||||
df['Intervention Date'] = pd.to_datetime(df['Intervention Date'])
|
||||
|
||||
logger.info(f"Loaded {len(df)} rows from SQLite")
|
||||
|
||||
# Validate result
|
||||
is_valid, missing = self.validate_dataframe(df)
|
||||
if not is_valid:
|
||||
raise ValueError(f"SQLite data missing required columns: {missing}")
|
||||
|
||||
load_time = time.time() - start_time
|
||||
logger.info(f"SQLite data loading complete. {len(df)} rows in {load_time:.2f}s")
|
||||
|
||||
return LoadResult(
|
||||
df=df,
|
||||
source=self.source_description,
|
||||
row_count=len(df),
|
||||
load_time_seconds=load_time,
|
||||
)
|
||||
|
||||
|
||||
def get_loader(
|
||||
source: str | Path,
|
||||
paths: Optional[PathConfig] = None,
|
||||
**kwargs
|
||||
) -> DataLoader:
|
||||
"""Factory function to create the appropriate DataLoader.
|
||||
|
||||
Args:
|
||||
source: Either a file path (CSV/Parquet) or "sqlite" for database
|
||||
paths: PathConfig for reference data (used by FileDataLoader)
|
||||
**kwargs: Additional arguments passed to the loader constructor
|
||||
|
||||
Returns:
|
||||
Appropriate DataLoader instance
|
||||
|
||||
Examples:
|
||||
>>> loader = get_loader("data/activity.csv")
|
||||
>>> loader = get_loader("data/activity.parquet")
|
||||
>>> loader = get_loader("sqlite")
|
||||
>>> loader = get_loader("sqlite", date_range=(date(2024, 1, 1), date(2024, 12, 31)))
|
||||
"""
|
||||
source_str = str(source).lower()
|
||||
|
||||
if source_str == "sqlite":
|
||||
return SQLiteDataLoader(**kwargs)
|
||||
|
||||
# Assume it's a file path
|
||||
path = Path(source)
|
||||
return FileDataLoader(file_path=path, paths=paths)
|
||||
@@ -0,0 +1,593 @@
|
||||
"""
|
||||
Database migration script for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Provides functions to initialize the SQLite database schema and CLI interface
|
||||
for running migrations from the command line.
|
||||
|
||||
Usage:
|
||||
# Initialize database (creates all tables)
|
||||
python -m data_processing.migrate
|
||||
|
||||
# Drop existing tables and reinitialize
|
||||
python -m data_processing.migrate --drop-existing
|
||||
|
||||
# Show current database status
|
||||
python -m data_processing.migrate --status
|
||||
|
||||
# Migrate all reference data from CSV files
|
||||
python -m data_processing.migrate --reference-data
|
||||
|
||||
# Migrate reference data with verification
|
||||
python -m data_processing.migrate --reference-data --verify
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from core.logging_config import setup_logging, get_logger
|
||||
from data_processing.database import DatabaseManager, DatabaseConfig
|
||||
from core import PathConfig, default_paths
|
||||
from data_processing.schema import (
|
||||
create_all_tables,
|
||||
drop_all_tables,
|
||||
verify_all_tables_exist,
|
||||
get_all_table_counts,
|
||||
)
|
||||
from data_processing.reference_data import (
|
||||
MigrationResult,
|
||||
migrate_drug_names,
|
||||
migrate_organizations,
|
||||
migrate_directories,
|
||||
migrate_drug_directory_map,
|
||||
migrate_drug_indication_clusters,
|
||||
verify_drug_names_migration,
|
||||
verify_organizations_migration,
|
||||
verify_directories_migration,
|
||||
verify_drug_directory_map_migration,
|
||||
verify_drug_indication_clusters_migration,
|
||||
)
|
||||
from data_processing.patient_data import (
|
||||
load_patient_data,
|
||||
refresh_patient_treatment_summary,
|
||||
get_patient_data_stats,
|
||||
verify_mv_consistency,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def initialize_database(
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
drop_existing: bool = False,
|
||||
confirm_drop: bool = True
|
||||
) -> bool:
|
||||
"""
|
||||
Initialize the database with all required tables.
|
||||
|
||||
Creates all tables defined in the schema (reference tables, fact tables,
|
||||
materialized views, and file tracking tables). Uses IF NOT EXISTS so
|
||||
safe to run multiple times.
|
||||
|
||||
Args:
|
||||
db_manager: DatabaseManager instance. Uses default if not provided.
|
||||
drop_existing: If True, drops all existing tables before creating.
|
||||
confirm_drop: If True and drop_existing=True, prompts for confirmation.
|
||||
Set to False for non-interactive use.
|
||||
|
||||
Returns:
|
||||
True if initialization succeeded, False otherwise.
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
logger.info(f"Initializing database at: {db_manager.db_path}")
|
||||
|
||||
# Handle drop existing with confirmation
|
||||
if drop_existing:
|
||||
if confirm_drop:
|
||||
print(f"\nWARNING: This will delete ALL data from the database:")
|
||||
print(f" {db_manager.db_path}\n")
|
||||
response = input("Are you sure you want to continue? (yes/no): ")
|
||||
if response.lower() not in ("yes", "y"):
|
||||
print("Operation cancelled.")
|
||||
return False
|
||||
|
||||
if db_manager.exists:
|
||||
logger.warning("Dropping existing tables...")
|
||||
with db_manager.get_connection() as conn:
|
||||
drop_all_tables(conn)
|
||||
conn.commit()
|
||||
logger.info("Existing tables dropped")
|
||||
else:
|
||||
logger.info("Database does not exist yet, nothing to drop")
|
||||
|
||||
# Create all tables
|
||||
try:
|
||||
with db_manager.get_transaction() as conn:
|
||||
create_all_tables(conn)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create tables: {e}")
|
||||
return False
|
||||
|
||||
# Verify all tables were created
|
||||
with db_manager.get_connection() as conn:
|
||||
missing = verify_all_tables_exist(conn)
|
||||
|
||||
if missing:
|
||||
logger.error(f"Table creation failed. Missing tables: {missing}")
|
||||
return False
|
||||
|
||||
logger.info("All tables created successfully")
|
||||
return True
|
||||
|
||||
|
||||
def migrate_all_reference_data(
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
paths: Optional[PathConfig] = None,
|
||||
verify: bool = False
|
||||
) -> tuple[bool, list[MigrationResult]]:
|
||||
"""
|
||||
Run all reference data migrations from CSV files to SQLite tables.
|
||||
|
||||
Migrations are run in order:
|
||||
1. Drug names (drugnames.csv → ref_drug_names)
|
||||
2. Organizations (org_codes.csv → ref_organizations)
|
||||
3. Directories (directory_list.csv → ref_directories)
|
||||
4. Drug-directory mappings (drug_directory_list.csv → ref_drug_directory_map)
|
||||
|
||||
Args:
|
||||
db_manager: DatabaseManager instance. Uses default if not provided.
|
||||
paths: PathConfig instance for locating CSV files. Uses default if not provided.
|
||||
verify: If True, runs verification after each migration.
|
||||
|
||||
Returns:
|
||||
Tuple of (all_success: bool, results: list of MigrationResult)
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
results: list[MigrationResult] = []
|
||||
all_success = True
|
||||
|
||||
# Define migrations in order
|
||||
# Note: drug_indication_clusters uses a different signature (csv_path instead of paths)
|
||||
migrations = [
|
||||
("Drug names", migrate_drug_names, verify_drug_names_migration if verify else None, True),
|
||||
("Organizations", migrate_organizations, verify_organizations_migration if verify else None, True),
|
||||
("Directories", migrate_directories, verify_directories_migration if verify else None, True),
|
||||
("Drug-directory map", migrate_drug_directory_map, verify_drug_directory_map_migration if verify else None, True),
|
||||
("Drug indication clusters", migrate_drug_indication_clusters, verify_drug_indication_clusters_migration if verify else None, False),
|
||||
]
|
||||
|
||||
logger.info(f"Starting reference data migrations ({len(migrations)} tables)")
|
||||
|
||||
for name, migrate_fn, verify_fn, uses_paths in migrations:
|
||||
logger.info(f"Migrating: {name}...")
|
||||
|
||||
# Run migration (some use paths parameter, some use csv_path)
|
||||
if uses_paths:
|
||||
result = migrate_fn(db_manager=db_manager, paths=paths) # type: ignore[operator]
|
||||
else:
|
||||
# Drug indication clusters uses csv_path instead of paths
|
||||
result = migrate_fn(db_manager=db_manager) # type: ignore[operator]
|
||||
results.append(result)
|
||||
|
||||
if not result.success:
|
||||
logger.error(f"Migration failed: {name} - {result.error_message}")
|
||||
all_success = False
|
||||
continue
|
||||
|
||||
logger.info(f" {result}")
|
||||
|
||||
# Run verification if requested
|
||||
if verify_fn is not None:
|
||||
logger.info(f" Verifying {name}...")
|
||||
if uses_paths:
|
||||
verified, verify_msg = verify_fn(db_manager=db_manager, paths=paths) # type: ignore[call-arg]
|
||||
else:
|
||||
verified, verify_msg = verify_fn(db_manager=db_manager) # type: ignore[call-arg]
|
||||
if verified:
|
||||
logger.info(f" OK: {verify_msg}")
|
||||
else:
|
||||
logger.error(f" FAILED: Verification failed: {verify_msg}")
|
||||
all_success = False
|
||||
|
||||
# Summary
|
||||
successful = sum(1 for r in results if r.success)
|
||||
logger.info(f"Reference data migrations complete: {successful}/{len(results)} succeeded")
|
||||
|
||||
return all_success, results
|
||||
|
||||
|
||||
def print_migration_summary(results: list[MigrationResult]) -> None:
|
||||
"""Print a summary of migration results to stdout."""
|
||||
print("\n=== Reference Data Migration Summary ===\n")
|
||||
|
||||
for result in results:
|
||||
status = "[OK]" if result.success else "[FAILED]"
|
||||
print(f"{status} {result.table_name}")
|
||||
if result.success:
|
||||
print(f" Read: {result.rows_read}, Inserted: {result.rows_inserted}, Skipped: {result.rows_skipped}")
|
||||
else:
|
||||
print(f" Error: {result.error_message}")
|
||||
|
||||
successful = sum(1 for r in results if r.success)
|
||||
print(f"\nTotal: {successful}/{len(results)} migrations succeeded")
|
||||
print()
|
||||
|
||||
|
||||
def create_progress_reporter(description: str = "Loading", width: int = 40):
|
||||
"""
|
||||
Create a progress callback that prints a progress bar to stdout.
|
||||
|
||||
Args:
|
||||
description: Label to show before the progress bar.
|
||||
width: Width of the progress bar in characters.
|
||||
|
||||
Returns:
|
||||
Callback function(current, total) that prints progress.
|
||||
"""
|
||||
last_percent = [-1] # Use list to allow mutation in closure
|
||||
|
||||
def report_progress(current: int, total: int) -> None:
|
||||
"""Print a progress bar showing current/total progress."""
|
||||
if total == 0:
|
||||
percent = 100
|
||||
else:
|
||||
percent = int(100 * current / total)
|
||||
|
||||
# Only update display when percentage changes (avoid excessive output)
|
||||
if percent == last_percent[0]:
|
||||
return
|
||||
last_percent[0] = percent
|
||||
|
||||
filled = int(width * current / total) if total > 0 else width
|
||||
bar = "=" * filled + "-" * (width - filled)
|
||||
|
||||
# Use carriage return to overwrite the line
|
||||
sys.stdout.write(f"\r{description}: [{bar}] {percent:3d}% ({current:,}/{total:,})")
|
||||
sys.stdout.flush()
|
||||
|
||||
# Print newline when complete
|
||||
if current >= total:
|
||||
print()
|
||||
|
||||
return report_progress
|
||||
|
||||
|
||||
def load_patient_data_cli(
|
||||
file_path: Path,
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
paths: Optional[PathConfig] = None,
|
||||
force: bool = False,
|
||||
refresh_mv: bool = True
|
||||
) -> bool:
|
||||
"""
|
||||
Load patient data from file with CLI progress reporting.
|
||||
|
||||
Args:
|
||||
file_path: Path to CSV or Parquet file.
|
||||
db_manager: DatabaseManager instance. Uses default if not provided.
|
||||
paths: PathConfig for reference data. Uses default if not provided.
|
||||
force: If True, re-process even if file hash matches.
|
||||
refresh_mv: If True, refresh the materialized view after loading.
|
||||
|
||||
Returns:
|
||||
True if loading succeeded, False otherwise.
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
print(f"\n=== Loading Patient Data ===\n")
|
||||
print(f"File: {file_path}")
|
||||
|
||||
# Check file exists
|
||||
if not file_path.exists():
|
||||
print(f"ERROR: File not found: {file_path}")
|
||||
return False
|
||||
|
||||
# Calculate and display file info
|
||||
file_size_mb = file_path.stat().st_size / (1024 * 1024)
|
||||
print(f"Size: {file_size_mb:.1f} MB")
|
||||
print()
|
||||
|
||||
# Create progress callback
|
||||
progress_callback = create_progress_reporter("Loading rows", width=40)
|
||||
|
||||
# Load the data
|
||||
result = load_patient_data(
|
||||
file_path=file_path,
|
||||
db_manager=db_manager,
|
||||
paths=paths,
|
||||
batch_size=5000,
|
||||
force=force,
|
||||
progress_callback=progress_callback
|
||||
)
|
||||
|
||||
# Print result
|
||||
print()
|
||||
if result.was_already_processed:
|
||||
print("File already processed (same hash). Skipping.")
|
||||
print(f"Use --force to re-process.")
|
||||
elif result.success:
|
||||
print(f"Loaded {result.rows_inserted:,} rows in {result.load_time_seconds:.1f}s")
|
||||
if result.rows_skipped > 0:
|
||||
print(f"Skipped {result.rows_skipped:,} rows (missing UPID or date)")
|
||||
else:
|
||||
print(f"FAILED: {result.error_message}")
|
||||
return False
|
||||
|
||||
# Refresh materialized view if requested
|
||||
if refresh_mv and result.success and not result.was_already_processed:
|
||||
print()
|
||||
print("Refreshing materialized view...")
|
||||
mv_progress = create_progress_reporter("Processing patients", width=40)
|
||||
mv_result = refresh_patient_treatment_summary(
|
||||
db_manager=db_manager,
|
||||
progress_callback=mv_progress
|
||||
)
|
||||
|
||||
if mv_result.success:
|
||||
print(f"MV refreshed: {mv_result.patients_processed:,} patients in {mv_result.refresh_time_seconds:.1f}s")
|
||||
|
||||
# Verify consistency
|
||||
consistent, msg = verify_mv_consistency(db_manager)
|
||||
if consistent:
|
||||
print(f"MV verification: OK")
|
||||
else:
|
||||
print(f"MV verification: FAILED - {msg}")
|
||||
else:
|
||||
print(f"MV refresh FAILED: {mv_result.error_message}")
|
||||
|
||||
# Print summary statistics
|
||||
print()
|
||||
print("=== Patient Data Summary ===")
|
||||
stats = get_patient_data_stats(db_manager)
|
||||
print(f" Total rows: {stats['total_rows']:,}")
|
||||
print(f" Unique patients: {stats['unique_patients']:,}")
|
||||
print(f" Unique drugs: {stats['unique_drugs']:,}")
|
||||
print(f" Unique organizations: {stats['unique_organizations']:,}")
|
||||
if stats['date_range'][0] and stats['date_range'][1]:
|
||||
print(f" Date range: {stats['date_range'][0]} to {stats['date_range'][1]}")
|
||||
print()
|
||||
|
||||
return result.success
|
||||
|
||||
|
||||
def get_database_status(db_manager: Optional[DatabaseManager] = None) -> dict:
|
||||
"""
|
||||
Get the current status of the database.
|
||||
|
||||
Returns:
|
||||
Dictionary with database status information:
|
||||
- exists: Whether the database file exists
|
||||
- path: Path to the database file
|
||||
- size_bytes: Size of database file (if exists)
|
||||
- tables: Dictionary of table names to row counts
|
||||
- missing_tables: List of expected tables that don't exist
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
status = {
|
||||
"exists": db_manager.exists,
|
||||
"path": str(db_manager.db_path),
|
||||
"size_bytes": None,
|
||||
"tables": {},
|
||||
"missing_tables": [],
|
||||
}
|
||||
|
||||
if db_manager.exists:
|
||||
status["size_bytes"] = db_manager.db_path.stat().st_size
|
||||
|
||||
with db_manager.get_connection() as conn:
|
||||
status["missing_tables"] = verify_all_tables_exist(conn)
|
||||
|
||||
# Get counts for existing tables
|
||||
try:
|
||||
status["tables"] = get_all_table_counts(conn)
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not get table counts: {e}")
|
||||
|
||||
return status
|
||||
|
||||
|
||||
def print_database_status(db_manager: Optional[DatabaseManager] = None) -> None:
|
||||
"""Print database status to stdout in a human-readable format."""
|
||||
status = get_database_status(db_manager)
|
||||
|
||||
print("\n=== Database Status ===\n")
|
||||
print(f"Path: {status['path']}")
|
||||
print(f"Exists: {status['exists']}")
|
||||
|
||||
if status["exists"]:
|
||||
size_kb = (status["size_bytes"] or 0) / 1024
|
||||
print(f"Size: {size_kb:.1f} KB")
|
||||
|
||||
if status["missing_tables"]:
|
||||
print(f"\nMissing tables: {', '.join(status['missing_tables'])}")
|
||||
else:
|
||||
print("\nAll expected tables exist.")
|
||||
|
||||
if status["tables"]:
|
||||
print("\nTable row counts:")
|
||||
for table, count in sorted(status["tables"].items()):
|
||||
print(f" {table}: {count:,} rows")
|
||||
else:
|
||||
print("\nDatabase does not exist. Run migration to create it.")
|
||||
|
||||
print()
|
||||
|
||||
|
||||
def main():
|
||||
"""CLI entry point for database migration."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Initialize NHS Pathways Analysis SQLite database schema",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python -m data_processing.migrate # Initialize database
|
||||
python -m data_processing.migrate --status # Show database status
|
||||
python -m data_processing.migrate --drop-existing # Reset database
|
||||
python -m data_processing.migrate --reference-data # Migrate reference data
|
||||
python -m data_processing.migrate --reference-data --verify # With verification
|
||||
python -m data_processing.migrate --load-patient-data data.parquet # Load patient data
|
||||
python -m data_processing.migrate --load-patient-data data.csv --force # Force reload
|
||||
python -m data_processing.migrate --db-path ./data/test.db # Custom path
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--status",
|
||||
action="store_true",
|
||||
help="Show current database status and exit"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--drop-existing",
|
||||
action="store_true",
|
||||
help="Drop all existing tables before creating (WARNING: deletes data)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reference-data",
|
||||
action="store_true",
|
||||
help="Migrate all reference data from CSV files to SQLite tables"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verify",
|
||||
action="store_true",
|
||||
help="Verify migrated data matches CSV sources (use with --reference-data)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--db-path",
|
||||
type=Path,
|
||||
help="Path to database file (default: ./data/pathways.db)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--yes", "-y",
|
||||
action="store_true",
|
||||
help="Skip confirmation prompts (for non-interactive use)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose", "-v",
|
||||
action="store_true",
|
||||
help="Enable verbose logging"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--load-patient-data",
|
||||
type=Path,
|
||||
metavar="FILE",
|
||||
help="Load patient data from CSV or Parquet file with progress reporting"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="Force re-processing even if file hash matches (use with --load-patient-data)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-refresh-mv",
|
||||
action="store_true",
|
||||
help="Skip materialized view refresh after loading (use with --load-patient-data)"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Set up logging
|
||||
log_level = "DEBUG" if args.verbose else "INFO"
|
||||
setup_logging(level=log_level, simple_console=True)
|
||||
|
||||
# Create database manager with optional custom path
|
||||
if args.db_path:
|
||||
config = DatabaseConfig(db_path=args.db_path)
|
||||
db_manager = DatabaseManager(config)
|
||||
else:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
# Handle --status
|
||||
if args.status:
|
||||
print_database_status(db_manager)
|
||||
return 0
|
||||
|
||||
# Validate configuration
|
||||
config_errors = db_manager.config.validate()
|
||||
if config_errors:
|
||||
for error in config_errors:
|
||||
logger.error(error)
|
||||
return 1
|
||||
|
||||
# Handle --reference-data (migrate reference data from CSV to SQLite)
|
||||
if args.reference_data:
|
||||
# Ensure database exists with tables first
|
||||
if not db_manager.exists:
|
||||
print("Database does not exist. Initializing schema first...")
|
||||
success = initialize_database(db_manager=db_manager)
|
||||
if not success:
|
||||
print("\nDatabase initialization failed. Check logs for details.")
|
||||
return 1
|
||||
|
||||
# Run reference data migrations
|
||||
success, results = migrate_all_reference_data(
|
||||
db_manager=db_manager,
|
||||
paths=default_paths,
|
||||
verify=args.verify
|
||||
)
|
||||
|
||||
print_migration_summary(results)
|
||||
print_database_status(db_manager)
|
||||
|
||||
if success:
|
||||
print("Reference data migration completed successfully.")
|
||||
return 0
|
||||
else:
|
||||
print("Reference data migration completed with errors. Check logs for details.")
|
||||
return 1
|
||||
|
||||
# Handle --load-patient-data (load patient data from CSV/Parquet)
|
||||
if args.load_patient_data:
|
||||
# Ensure database exists with tables first
|
||||
if not db_manager.exists:
|
||||
print("Database does not exist. Initializing schema first...")
|
||||
success = initialize_database(db_manager=db_manager)
|
||||
if not success:
|
||||
print("\nDatabase initialization failed. Check logs for details.")
|
||||
return 1
|
||||
|
||||
# Load patient data with progress reporting
|
||||
success = load_patient_data_cli(
|
||||
file_path=args.load_patient_data,
|
||||
db_manager=db_manager,
|
||||
paths=default_paths,
|
||||
force=args.force,
|
||||
refresh_mv=not args.no_refresh_mv
|
||||
)
|
||||
|
||||
if success:
|
||||
print("Patient data load completed successfully.")
|
||||
return 0
|
||||
else:
|
||||
print("Patient data load failed. Check logs for details.")
|
||||
return 1
|
||||
|
||||
# Run schema migration (default behavior)
|
||||
success = initialize_database(
|
||||
db_manager=db_manager,
|
||||
drop_existing=args.drop_existing,
|
||||
confirm_drop=not args.yes
|
||||
)
|
||||
|
||||
if success:
|
||||
print("\nDatabase initialized successfully.")
|
||||
print_database_status(db_manager)
|
||||
return 0
|
||||
else:
|
||||
print("\nDatabase initialization failed. Check logs for details.")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,890 @@
|
||||
"""
|
||||
Patient data migration functions for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Provides functions to load patient intervention data from CSV/Parquet files
|
||||
into the SQLite fact_interventions table. Supports:
|
||||
- Batch processing for large files
|
||||
- File hash tracking for incremental updates
|
||||
- Progress reporting during loading
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import sqlite3
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from core import PathConfig, default_paths
|
||||
from core.logging_config import get_logger
|
||||
from data_processing.database import DatabaseManager
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PatientDataLoadResult:
|
||||
"""Results from a patient data load operation."""
|
||||
file_path: str
|
||||
file_hash: str
|
||||
rows_read: int
|
||||
rows_inserted: int
|
||||
rows_skipped: int
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
load_time_seconds: float = 0.0
|
||||
was_already_processed: bool = False
|
||||
|
||||
def __str__(self) -> str:
|
||||
if self.was_already_processed:
|
||||
return f"{self.file_path}: Already processed (same hash)"
|
||||
elif self.success:
|
||||
return (
|
||||
f"{self.file_path}: Loaded {self.rows_inserted:,} rows "
|
||||
f"in {self.load_time_seconds:.1f}s"
|
||||
)
|
||||
else:
|
||||
return f"{self.file_path}: FAILED - {self.error_message}"
|
||||
|
||||
|
||||
def calculate_file_hash(file_path: Path) -> str:
|
||||
"""
|
||||
Calculate SHA256 hash of a file.
|
||||
|
||||
Uses chunked reading to handle large files efficiently.
|
||||
|
||||
Args:
|
||||
file_path: Path to the file.
|
||||
|
||||
Returns:
|
||||
Hex string of SHA256 hash.
|
||||
"""
|
||||
sha256_hash = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(8192), b""):
|
||||
sha256_hash.update(chunk)
|
||||
return sha256_hash.hexdigest()
|
||||
|
||||
|
||||
def check_file_processed(
|
||||
conn: sqlite3.Connection,
|
||||
file_path: str,
|
||||
file_hash: str
|
||||
) -> tuple[bool, Optional[str]]:
|
||||
"""
|
||||
Check if a file has already been processed with the same hash.
|
||||
|
||||
Args:
|
||||
conn: Database connection.
|
||||
file_path: Full path to the file.
|
||||
file_hash: SHA256 hash of the file.
|
||||
|
||||
Returns:
|
||||
Tuple of (is_processed, old_hash).
|
||||
- If is_processed is True and old_hash == file_hash, file is unchanged.
|
||||
- If is_processed is True and old_hash != file_hash, file has changed.
|
||||
- If is_processed is False, file is new.
|
||||
"""
|
||||
cursor = conn.execute(
|
||||
"SELECT file_hash, status FROM processed_files WHERE file_path = ?",
|
||||
(file_path,)
|
||||
)
|
||||
result = cursor.fetchone()
|
||||
|
||||
if result is None:
|
||||
return False, None
|
||||
|
||||
old_hash = result["file_hash"]
|
||||
status = result["status"]
|
||||
|
||||
# Only consider it processed if status is success and hash matches
|
||||
if status == "success" and old_hash == file_hash:
|
||||
return True, old_hash
|
||||
|
||||
return False, old_hash
|
||||
|
||||
|
||||
def record_file_processing_start(
|
||||
conn: sqlite3.Connection,
|
||||
file_path: str,
|
||||
file_hash: str,
|
||||
file_size: int,
|
||||
file_modified: datetime
|
||||
) -> None:
|
||||
"""
|
||||
Record that we're starting to process a file.
|
||||
|
||||
Args:
|
||||
conn: Database connection.
|
||||
file_path: Full path to the file.
|
||||
file_hash: SHA256 hash of the file.
|
||||
file_size: File size in bytes.
|
||||
file_modified: File modification timestamp.
|
||||
"""
|
||||
file_name = Path(file_path).name
|
||||
now = datetime.now().isoformat()
|
||||
|
||||
conn.execute("""
|
||||
INSERT INTO processed_files (
|
||||
file_path, file_name, file_hash, file_size_bytes,
|
||||
file_modified_at, status, first_processed_at, last_processed_at
|
||||
) VALUES (?, ?, ?, ?, ?, 'processing', ?, ?)
|
||||
ON CONFLICT(file_path) DO UPDATE SET
|
||||
file_hash = excluded.file_hash,
|
||||
file_size_bytes = excluded.file_size_bytes,
|
||||
file_modified_at = excluded.file_modified_at,
|
||||
status = 'processing',
|
||||
last_processed_at = excluded.last_processed_at,
|
||||
error_message = NULL
|
||||
""", (file_path, file_name, file_hash, file_size, file_modified.isoformat(), now, now))
|
||||
|
||||
|
||||
def record_file_processing_complete(
|
||||
conn: sqlite3.Connection,
|
||||
file_path: str,
|
||||
row_count: int,
|
||||
duration_seconds: float,
|
||||
success: bool,
|
||||
error_message: Optional[str] = None
|
||||
) -> None:
|
||||
"""
|
||||
Record that file processing has completed.
|
||||
|
||||
Args:
|
||||
conn: Database connection.
|
||||
file_path: Full path to the file.
|
||||
row_count: Number of rows processed.
|
||||
duration_seconds: Time taken to process.
|
||||
success: Whether processing was successful.
|
||||
error_message: Error message if failed.
|
||||
"""
|
||||
status = "success" if success else "error"
|
||||
|
||||
conn.execute("""
|
||||
UPDATE processed_files
|
||||
SET status = ?,
|
||||
row_count = ?,
|
||||
processing_duration_seconds = ?,
|
||||
error_message = ?,
|
||||
last_processed_at = ?
|
||||
WHERE file_path = ?
|
||||
""", (status, row_count, duration_seconds, error_message, datetime.now().isoformat(), file_path))
|
||||
|
||||
|
||||
def load_dataframe_to_sqlite(
|
||||
df: pd.DataFrame,
|
||||
conn: sqlite3.Connection,
|
||||
source_file: str,
|
||||
batch_size: int = 5000,
|
||||
progress_callback: Optional[Callable[[int, int], None]] = None
|
||||
) -> int:
|
||||
"""
|
||||
Load a processed DataFrame into fact_interventions table.
|
||||
|
||||
Args:
|
||||
df: Processed DataFrame with required columns (from FileDataLoader).
|
||||
conn: Database connection.
|
||||
source_file: Source file path for tracking.
|
||||
batch_size: Number of rows to insert per batch.
|
||||
progress_callback: Optional callback(rows_inserted, total_rows) for progress updates.
|
||||
|
||||
Returns:
|
||||
Number of rows inserted.
|
||||
"""
|
||||
# Store the original drug names before processing (for rows where mapping doesn't exist)
|
||||
# The drug_names() transformation sets Drug Name to NULL when no mapping exists.
|
||||
# We need to preserve the original for those cases.
|
||||
|
||||
# Insert SQL columns - always include drug_name_raw
|
||||
insert_columns = [
|
||||
"upid", "provider_code", "person_key",
|
||||
"drug_name_raw", "drug_name_std",
|
||||
"intervention_date", "price_actual",
|
||||
"org_name", "directory",
|
||||
"treatment_function_code",
|
||||
"additional_detail_1", "additional_detail_2", "additional_detail_3",
|
||||
"additional_detail_4", "additional_detail_5",
|
||||
"source_file"
|
||||
]
|
||||
placeholders = ",".join(["?"] * len(insert_columns))
|
||||
insert_sql = f"""
|
||||
INSERT INTO fact_interventions ({",".join(insert_columns)})
|
||||
VALUES ({placeholders})
|
||||
"""
|
||||
|
||||
rows_inserted = 0
|
||||
rows_skipped = 0
|
||||
total_rows = len(df)
|
||||
|
||||
# Process in batches
|
||||
for batch_start in range(0, total_rows, batch_size):
|
||||
batch_end = min(batch_start + batch_size, total_rows)
|
||||
batch_df = df.iloc[batch_start:batch_end]
|
||||
|
||||
# Prepare batch data
|
||||
batch_data = []
|
||||
for _, row in batch_df.iterrows():
|
||||
# Skip rows missing required fields
|
||||
if pd.isna(row.get("UPID")) or pd.isna(row.get("Intervention Date")):
|
||||
rows_skipped += 1
|
||||
continue
|
||||
# Get drug names - raw and standardized
|
||||
drug_name_raw = row.get("Drug Name Raw") if "Drug Name Raw" in df.columns else None
|
||||
drug_name_std = row.get("Drug Name")
|
||||
|
||||
# If drug_name_std is NULL, use the raw drug name (uppercase)
|
||||
# This handles cases where the drug isn't in the drugnames.csv mapping
|
||||
if pd.isna(drug_name_std):
|
||||
if drug_name_raw is not None and not pd.isna(drug_name_raw):
|
||||
drug_name_std = str(drug_name_raw).upper().strip()
|
||||
else:
|
||||
drug_name_std = "UNKNOWN"
|
||||
|
||||
# Also clean up raw drug name for storage
|
||||
if drug_name_raw is not None and not pd.isna(drug_name_raw):
|
||||
drug_name_raw = str(drug_name_raw).strip()
|
||||
|
||||
# Get other values with null handling
|
||||
def get_value(col_name):
|
||||
if col_name not in df.columns:
|
||||
return None
|
||||
val = row[col_name]
|
||||
if pd.isna(val):
|
||||
return None
|
||||
elif hasattr(val, "strftime"):
|
||||
return val.strftime("%Y-%m-%d")
|
||||
return val
|
||||
|
||||
row_data = (
|
||||
get_value("UPID"),
|
||||
get_value("Provider Code"),
|
||||
get_value("PersonKey"),
|
||||
drug_name_raw,
|
||||
drug_name_std,
|
||||
get_value("Intervention Date"),
|
||||
get_value("Price Actual") or 0,
|
||||
get_value("OrganisationName"),
|
||||
get_value("Directory"),
|
||||
get_value("Treatment Function Code"),
|
||||
get_value("Additional Detail 1"),
|
||||
get_value("Additional Detail 2"),
|
||||
get_value("Additional Detail 3"),
|
||||
get_value("Additional Detail 4"),
|
||||
get_value("Additional Detail 5"),
|
||||
source_file
|
||||
)
|
||||
batch_data.append(row_data)
|
||||
|
||||
# Execute batch insert
|
||||
conn.executemany(insert_sql, batch_data)
|
||||
rows_inserted += len(batch_data)
|
||||
|
||||
# Report progress
|
||||
if progress_callback:
|
||||
progress_callback(rows_inserted, total_rows)
|
||||
|
||||
if rows_skipped > 0:
|
||||
logger.info(f"Skipped {rows_skipped:,} rows with missing UPID or Intervention Date")
|
||||
|
||||
return rows_inserted
|
||||
|
||||
|
||||
def delete_file_data(conn: sqlite3.Connection, source_file: str) -> int:
|
||||
"""
|
||||
Delete all data from a specific source file.
|
||||
|
||||
Used when re-processing a changed file.
|
||||
|
||||
Args:
|
||||
conn: Database connection.
|
||||
source_file: Source file path.
|
||||
|
||||
Returns:
|
||||
Number of rows deleted.
|
||||
"""
|
||||
cursor = conn.execute(
|
||||
"DELETE FROM fact_interventions WHERE source_file = ?",
|
||||
(source_file,)
|
||||
)
|
||||
return cursor.rowcount
|
||||
|
||||
|
||||
def load_patient_data(
|
||||
file_path: Path | str,
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
paths: Optional[PathConfig] = None,
|
||||
batch_size: int = 5000,
|
||||
force: bool = False,
|
||||
progress_callback: Optional[Callable[[int, int], None]] = None
|
||||
) -> PatientDataLoadResult:
|
||||
"""
|
||||
Load patient data from CSV/Parquet file into fact_interventions table.
|
||||
|
||||
This is the main entry point for loading patient data. It:
|
||||
1. Calculates file hash to detect changes
|
||||
2. Checks if file was already processed (skip if unchanged)
|
||||
3. Loads and transforms data using FileDataLoader
|
||||
4. Inserts data into SQLite in batches
|
||||
5. Records processing status in processed_files table
|
||||
|
||||
Args:
|
||||
file_path: Path to CSV or Parquet file.
|
||||
db_manager: DatabaseManager instance. Uses default if not provided.
|
||||
paths: PathConfig for reference data. Uses default if not provided.
|
||||
batch_size: Number of rows to insert per batch (default: 5000).
|
||||
force: If True, re-process even if file hash matches.
|
||||
progress_callback: Optional callback(rows_inserted, total_rows) for progress.
|
||||
|
||||
Returns:
|
||||
PatientDataLoadResult with loading statistics.
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
file_path = Path(file_path)
|
||||
file_path_str = str(file_path.absolute())
|
||||
|
||||
logger.info(f"Starting patient data load from {file_path}")
|
||||
start_time = time.time()
|
||||
|
||||
# Check file exists
|
||||
if not file_path.exists():
|
||||
error_msg = f"File not found: {file_path}"
|
||||
logger.error(error_msg)
|
||||
return PatientDataLoadResult(
|
||||
file_path=file_path_str,
|
||||
file_hash="",
|
||||
rows_read=0,
|
||||
rows_inserted=0,
|
||||
rows_skipped=0,
|
||||
success=False,
|
||||
error_message=error_msg
|
||||
)
|
||||
|
||||
# Calculate file hash
|
||||
logger.info("Calculating file hash...")
|
||||
file_hash = calculate_file_hash(file_path)
|
||||
file_size = file_path.stat().st_size
|
||||
file_modified = datetime.fromtimestamp(file_path.stat().st_mtime)
|
||||
|
||||
logger.info(f"File hash: {file_hash[:16]}... Size: {file_size:,} bytes")
|
||||
|
||||
# Check if already processed
|
||||
if not force:
|
||||
with db_manager.get_connection() as conn:
|
||||
is_processed, old_hash = check_file_processed(conn, file_path_str, file_hash)
|
||||
if is_processed:
|
||||
logger.info(f"File already processed with same hash, skipping")
|
||||
return PatientDataLoadResult(
|
||||
file_path=file_path_str,
|
||||
file_hash=file_hash,
|
||||
rows_read=0,
|
||||
rows_inserted=0,
|
||||
rows_skipped=0,
|
||||
success=True,
|
||||
was_already_processed=True
|
||||
)
|
||||
elif old_hash is not None:
|
||||
logger.info(f"File hash changed, will re-process (old: {old_hash[:16]}...)")
|
||||
|
||||
try:
|
||||
# Use FileDataLoader to load and transform data
|
||||
from data_processing.loader import FileDataLoader
|
||||
|
||||
loader = FileDataLoader(file_path, paths)
|
||||
logger.info("Loading and transforming data...")
|
||||
result = loader.load()
|
||||
df = result.df
|
||||
rows_read = result.row_count
|
||||
|
||||
logger.info(f"Loaded {rows_read:,} rows, starting SQLite insert...")
|
||||
|
||||
# Load into SQLite
|
||||
with db_manager.get_transaction() as conn:
|
||||
# Record that we're starting
|
||||
record_file_processing_start(conn, file_path_str, file_hash, file_size, file_modified)
|
||||
|
||||
# Delete any existing data from this file (for re-processing)
|
||||
deleted = delete_file_data(conn, file_path_str)
|
||||
if deleted > 0:
|
||||
logger.info(f"Deleted {deleted:,} existing rows from previous load")
|
||||
|
||||
# Insert new data
|
||||
rows_inserted = load_dataframe_to_sqlite(
|
||||
df, conn, file_path_str, batch_size, progress_callback
|
||||
)
|
||||
|
||||
# Record success
|
||||
load_time = time.time() - start_time
|
||||
record_file_processing_complete(
|
||||
conn, file_path_str, rows_inserted, load_time, True
|
||||
)
|
||||
|
||||
logger.info(f"Successfully loaded {rows_inserted:,} rows in {load_time:.1f}s")
|
||||
|
||||
return PatientDataLoadResult(
|
||||
file_path=file_path_str,
|
||||
file_hash=file_hash,
|
||||
rows_read=rows_read,
|
||||
rows_inserted=rows_inserted,
|
||||
rows_skipped=rows_read - rows_inserted,
|
||||
success=True,
|
||||
load_time_seconds=load_time
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
load_time = time.time() - start_time
|
||||
error_msg = str(e)
|
||||
logger.error(f"Failed to load patient data: {error_msg}")
|
||||
|
||||
# Record failure
|
||||
try:
|
||||
with db_manager.get_connection() as conn:
|
||||
record_file_processing_complete(
|
||||
conn, file_path_str, 0, load_time, False, error_msg
|
||||
)
|
||||
except Exception:
|
||||
pass # Don't fail on failure to record failure
|
||||
|
||||
return PatientDataLoadResult(
|
||||
file_path=file_path_str,
|
||||
file_hash=file_hash if 'file_hash' in dir() else "",
|
||||
rows_read=0,
|
||||
rows_inserted=0,
|
||||
rows_skipped=0,
|
||||
success=False,
|
||||
error_message=error_msg,
|
||||
load_time_seconds=load_time
|
||||
)
|
||||
|
||||
|
||||
def get_patient_data_stats(db_manager: Optional[DatabaseManager] = None) -> dict:
|
||||
"""
|
||||
Get statistics about patient data in fact_interventions.
|
||||
|
||||
Returns:
|
||||
Dictionary with statistics about the loaded data.
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
stats = {}
|
||||
|
||||
with db_manager.get_connection() as conn:
|
||||
# Total rows
|
||||
cursor = conn.execute("SELECT COUNT(*) FROM fact_interventions")
|
||||
stats["total_rows"] = cursor.fetchone()[0]
|
||||
|
||||
# Unique patients
|
||||
cursor = conn.execute("SELECT COUNT(DISTINCT upid) FROM fact_interventions")
|
||||
stats["unique_patients"] = cursor.fetchone()[0]
|
||||
|
||||
# Unique drugs
|
||||
cursor = conn.execute("SELECT COUNT(DISTINCT drug_name_std) FROM fact_interventions")
|
||||
stats["unique_drugs"] = cursor.fetchone()[0]
|
||||
|
||||
# Unique organizations
|
||||
cursor = conn.execute("SELECT COUNT(DISTINCT org_name) FROM fact_interventions")
|
||||
stats["unique_organizations"] = cursor.fetchone()[0]
|
||||
|
||||
# Date range
|
||||
cursor = conn.execute("""
|
||||
SELECT MIN(intervention_date), MAX(intervention_date)
|
||||
FROM fact_interventions
|
||||
""")
|
||||
result = cursor.fetchone()
|
||||
stats["date_range"] = (result[0], result[1]) if result else (None, None)
|
||||
|
||||
# Processed files
|
||||
cursor = conn.execute("""
|
||||
SELECT COUNT(*), SUM(row_count)
|
||||
FROM processed_files WHERE status = 'success'
|
||||
""")
|
||||
result = cursor.fetchone()
|
||||
stats["processed_files"] = result[0] if result else 0
|
||||
stats["processed_rows"] = result[1] if result and result[1] else 0
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def list_processed_files(db_manager: Optional[DatabaseManager] = None) -> list[dict]:
|
||||
"""
|
||||
List all processed files and their status.
|
||||
|
||||
Returns:
|
||||
List of dictionaries with file processing information.
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
files = []
|
||||
|
||||
with db_manager.get_connection() as conn:
|
||||
cursor = conn.execute("""
|
||||
SELECT file_path, file_name, file_hash, file_size_bytes,
|
||||
row_count, status, error_message,
|
||||
first_processed_at, last_processed_at, processing_duration_seconds
|
||||
FROM processed_files
|
||||
ORDER BY last_processed_at DESC
|
||||
""")
|
||||
|
||||
for row in cursor.fetchall():
|
||||
files.append({
|
||||
"file_path": row["file_path"],
|
||||
"file_name": row["file_name"],
|
||||
"file_hash": row["file_hash"],
|
||||
"file_size_bytes": row["file_size_bytes"],
|
||||
"row_count": row["row_count"],
|
||||
"status": row["status"],
|
||||
"error_message": row["error_message"],
|
||||
"first_processed_at": row["first_processed_at"],
|
||||
"last_processed_at": row["last_processed_at"],
|
||||
"processing_duration_seconds": row["processing_duration_seconds"],
|
||||
})
|
||||
|
||||
return files
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Materialized View Refresh Functions
|
||||
# =============================================================================
|
||||
|
||||
@dataclass
|
||||
class MVRefreshResult:
|
||||
"""Results from refreshing the patient treatment summary materialized view."""
|
||||
patients_processed: int
|
||||
rows_inserted: int
|
||||
refresh_time_seconds: float
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
def __str__(self) -> str:
|
||||
if self.success:
|
||||
return (
|
||||
f"Refreshed MV: {self.patients_processed:,} patients "
|
||||
f"in {self.refresh_time_seconds:.1f}s"
|
||||
)
|
||||
else:
|
||||
return f"MV refresh FAILED: {self.error_message}"
|
||||
|
||||
|
||||
def refresh_patient_treatment_summary(
|
||||
db_manager: Optional[DatabaseManager] = None,
|
||||
progress_callback: Optional[Callable[[int, int], None]] = None
|
||||
) -> MVRefreshResult:
|
||||
"""
|
||||
Refresh the mv_patient_treatment_summary materialized view.
|
||||
|
||||
This computes per-patient aggregations from fact_interventions:
|
||||
- First/last seen dates
|
||||
- Total cost, average cost per intervention
|
||||
- Intervention count, unique drug count
|
||||
- Drug sequence (chronological, pipe-separated)
|
||||
- Drug counts, costs, and date ranges (as JSON)
|
||||
|
||||
The MV is fully rebuilt (truncate and re-insert) for simplicity.
|
||||
This typically takes 30-60 seconds for ~35,000 patients.
|
||||
|
||||
Args:
|
||||
db_manager: DatabaseManager instance. Uses default if not provided.
|
||||
progress_callback: Optional callback(patients_done, total_patients).
|
||||
|
||||
Returns:
|
||||
MVRefreshResult with refresh statistics.
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
logger.info("Starting materialized view refresh...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
with db_manager.get_transaction() as conn:
|
||||
# Step 1: Get total patient count for progress reporting
|
||||
cursor = conn.execute("SELECT COUNT(DISTINCT upid) FROM fact_interventions")
|
||||
total_patients = cursor.fetchone()[0]
|
||||
logger.info(f"Processing {total_patients:,} unique patients")
|
||||
|
||||
if total_patients == 0:
|
||||
logger.warning("No patient data in fact_interventions, MV will be empty")
|
||||
return MVRefreshResult(
|
||||
patients_processed=0,
|
||||
rows_inserted=0,
|
||||
refresh_time_seconds=time.time() - start_time,
|
||||
success=True
|
||||
)
|
||||
|
||||
# Step 2: Clear existing MV data
|
||||
conn.execute("DELETE FROM mv_patient_treatment_summary")
|
||||
logger.info("Cleared existing MV data")
|
||||
|
||||
# Step 3: Compute aggregations using SQL CTEs
|
||||
# This is more efficient than processing row-by-row in Python
|
||||
refresh_sql = """
|
||||
WITH patient_aggs AS (
|
||||
-- Basic aggregations per patient
|
||||
SELECT
|
||||
upid,
|
||||
MIN(org_name) as org_name,
|
||||
MIN(directory) as directory,
|
||||
MIN(intervention_date) as first_seen_date,
|
||||
MAX(intervention_date) as last_seen_date,
|
||||
JULIANDAY(MAX(intervention_date)) - JULIANDAY(MIN(intervention_date)) as days_treated,
|
||||
SUM(price_actual) as total_cost,
|
||||
AVG(price_actual) as avg_cost_per_intervention,
|
||||
COUNT(*) as intervention_count,
|
||||
COUNT(DISTINCT drug_name_std) as unique_drug_count,
|
||||
COUNT(*) as source_row_count
|
||||
FROM fact_interventions
|
||||
GROUP BY upid
|
||||
),
|
||||
drug_sequences AS (
|
||||
-- Drug sequence per patient (chronological order, pipe-separated)
|
||||
SELECT
|
||||
upid,
|
||||
GROUP_CONCAT(drug_name_std, '|') as drug_sequence
|
||||
FROM (
|
||||
SELECT DISTINCT
|
||||
upid,
|
||||
drug_name_std,
|
||||
MIN(intervention_date) as first_date
|
||||
FROM fact_interventions
|
||||
GROUP BY upid, drug_name_std
|
||||
ORDER BY upid, first_date
|
||||
)
|
||||
GROUP BY upid
|
||||
),
|
||||
drug_counts AS (
|
||||
-- JSON object of drug counts per patient
|
||||
SELECT
|
||||
upid,
|
||||
'{' || GROUP_CONCAT('"' || drug_name_std || '": ' || cnt, ', ') || '}' as drug_counts_json
|
||||
FROM (
|
||||
SELECT
|
||||
upid,
|
||||
drug_name_std,
|
||||
COUNT(*) as cnt
|
||||
FROM fact_interventions
|
||||
GROUP BY upid, drug_name_std
|
||||
)
|
||||
GROUP BY upid
|
||||
),
|
||||
drug_costs AS (
|
||||
-- JSON object of drug costs per patient
|
||||
SELECT
|
||||
upid,
|
||||
'{' || GROUP_CONCAT('"' || drug_name_std || '": ' || ROUND(total_cost, 2), ', ') || '}' as drug_costs_json
|
||||
FROM (
|
||||
SELECT
|
||||
upid,
|
||||
drug_name_std,
|
||||
SUM(price_actual) as total_cost
|
||||
FROM fact_interventions
|
||||
GROUP BY upid, drug_name_std
|
||||
)
|
||||
GROUP BY upid
|
||||
),
|
||||
drug_dates AS (
|
||||
-- JSON object of drug date ranges per patient
|
||||
SELECT
|
||||
upid,
|
||||
'{' || GROUP_CONCAT('"' || drug_name_std || '": {"first": "' || first_date || '", "last": "' || last_date || '"}', ', ') || '}' as drug_date_ranges_json
|
||||
FROM (
|
||||
SELECT
|
||||
upid,
|
||||
drug_name_std,
|
||||
MIN(intervention_date) as first_date,
|
||||
MAX(intervention_date) as last_date
|
||||
FROM fact_interventions
|
||||
GROUP BY upid, drug_name_std
|
||||
)
|
||||
GROUP BY upid
|
||||
)
|
||||
INSERT INTO mv_patient_treatment_summary (
|
||||
upid, org_name, directory,
|
||||
first_seen_date, last_seen_date, days_treated,
|
||||
total_cost, avg_cost_per_intervention,
|
||||
intervention_count, unique_drug_count,
|
||||
drug_sequence, drug_counts_json, drug_costs_json, drug_date_ranges_json,
|
||||
source_row_count, computed_at
|
||||
)
|
||||
SELECT
|
||||
pa.upid,
|
||||
pa.org_name,
|
||||
pa.directory,
|
||||
pa.first_seen_date,
|
||||
pa.last_seen_date,
|
||||
CAST(pa.days_treated AS INTEGER),
|
||||
pa.total_cost,
|
||||
pa.avg_cost_per_intervention,
|
||||
pa.intervention_count,
|
||||
pa.unique_drug_count,
|
||||
ds.drug_sequence,
|
||||
dc.drug_counts_json,
|
||||
dco.drug_costs_json,
|
||||
dd.drug_date_ranges_json,
|
||||
pa.source_row_count,
|
||||
CURRENT_TIMESTAMP
|
||||
FROM patient_aggs pa
|
||||
LEFT JOIN drug_sequences ds ON pa.upid = ds.upid
|
||||
LEFT JOIN drug_counts dc ON pa.upid = dc.upid
|
||||
LEFT JOIN drug_costs dco ON pa.upid = dco.upid
|
||||
LEFT JOIN drug_dates dd ON pa.upid = dd.upid
|
||||
"""
|
||||
|
||||
logger.info("Executing MV refresh query...")
|
||||
conn.execute(refresh_sql)
|
||||
|
||||
# Get actual rows inserted
|
||||
cursor = conn.execute("SELECT COUNT(*) FROM mv_patient_treatment_summary")
|
||||
rows_inserted = cursor.fetchone()[0]
|
||||
|
||||
refresh_time = time.time() - start_time
|
||||
logger.info(f"MV refresh complete: {rows_inserted:,} rows in {refresh_time:.1f}s")
|
||||
|
||||
# Report progress if callback provided
|
||||
if progress_callback:
|
||||
progress_callback(rows_inserted, total_patients)
|
||||
|
||||
return MVRefreshResult(
|
||||
patients_processed=total_patients,
|
||||
rows_inserted=rows_inserted,
|
||||
refresh_time_seconds=refresh_time,
|
||||
success=True
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
refresh_time = time.time() - start_time
|
||||
error_msg = str(e)
|
||||
logger.error(f"MV refresh failed: {error_msg}")
|
||||
return MVRefreshResult(
|
||||
patients_processed=0,
|
||||
rows_inserted=0,
|
||||
refresh_time_seconds=refresh_time,
|
||||
success=False,
|
||||
error_message=error_msg
|
||||
)
|
||||
|
||||
|
||||
def get_patient_summary_stats(db_manager: Optional[DatabaseManager] = None) -> dict:
|
||||
"""
|
||||
Get statistics about the patient treatment summary MV.
|
||||
|
||||
Returns:
|
||||
Dictionary with MV statistics.
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
stats = {}
|
||||
|
||||
with db_manager.get_connection() as conn:
|
||||
# Total rows
|
||||
cursor = conn.execute("SELECT COUNT(*) FROM mv_patient_treatment_summary")
|
||||
stats["total_patients"] = cursor.fetchone()[0]
|
||||
|
||||
if stats["total_patients"] == 0:
|
||||
return stats
|
||||
|
||||
# Aggregated statistics
|
||||
cursor = conn.execute("""
|
||||
SELECT
|
||||
SUM(total_cost) as total_cost_all,
|
||||
AVG(total_cost) as avg_cost_per_patient,
|
||||
SUM(intervention_count) as total_interventions,
|
||||
AVG(intervention_count) as avg_interventions_per_patient,
|
||||
AVG(unique_drug_count) as avg_drugs_per_patient,
|
||||
AVG(days_treated) as avg_days_treated,
|
||||
MIN(first_seen_date) as earliest_date,
|
||||
MAX(last_seen_date) as latest_date,
|
||||
MAX(computed_at) as last_refresh
|
||||
FROM mv_patient_treatment_summary
|
||||
""")
|
||||
result = cursor.fetchone()
|
||||
|
||||
stats["total_cost"] = result[0] if result[0] else 0
|
||||
stats["avg_cost_per_patient"] = result[1] if result[1] else 0
|
||||
stats["total_interventions"] = result[2] if result[2] else 0
|
||||
stats["avg_interventions_per_patient"] = result[3] if result[3] else 0
|
||||
stats["avg_drugs_per_patient"] = result[4] if result[4] else 0
|
||||
stats["avg_days_treated"] = result[5] if result[5] else 0
|
||||
stats["date_range"] = (result[6], result[7])
|
||||
stats["last_refresh"] = result[8]
|
||||
|
||||
# Unique directories in MV
|
||||
cursor = conn.execute("SELECT COUNT(DISTINCT directory) FROM mv_patient_treatment_summary")
|
||||
stats["unique_directories"] = cursor.fetchone()[0]
|
||||
|
||||
# Unique organizations in MV
|
||||
cursor = conn.execute("SELECT COUNT(DISTINCT org_name) FROM mv_patient_treatment_summary")
|
||||
stats["unique_organizations"] = cursor.fetchone()[0]
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
def verify_mv_consistency(db_manager: Optional[DatabaseManager] = None) -> tuple[bool, str]:
|
||||
"""
|
||||
Verify that the MV is consistent with fact_interventions.
|
||||
|
||||
Checks that:
|
||||
- Patient counts match
|
||||
- Total cost sums match
|
||||
- Intervention counts match
|
||||
|
||||
Returns:
|
||||
Tuple of (is_consistent, message).
|
||||
"""
|
||||
if db_manager is None:
|
||||
db_manager = DatabaseManager()
|
||||
|
||||
with db_manager.get_connection() as conn:
|
||||
# Get fact table counts
|
||||
cursor = conn.execute("""
|
||||
SELECT
|
||||
COUNT(DISTINCT upid) as patients,
|
||||
SUM(price_actual) as total_cost,
|
||||
COUNT(*) as interventions
|
||||
FROM fact_interventions
|
||||
""")
|
||||
fact_row = cursor.fetchone()
|
||||
fact_patients = fact_row[0] or 0
|
||||
fact_cost = fact_row[1] or 0
|
||||
fact_interventions = fact_row[2] or 0
|
||||
|
||||
# Get MV counts
|
||||
cursor = conn.execute("""
|
||||
SELECT
|
||||
COUNT(*) as patients,
|
||||
SUM(total_cost) as total_cost,
|
||||
SUM(intervention_count) as interventions
|
||||
FROM mv_patient_treatment_summary
|
||||
""")
|
||||
mv_row = cursor.fetchone()
|
||||
mv_patients = mv_row[0] or 0
|
||||
mv_cost = mv_row[1] or 0
|
||||
mv_interventions = mv_row[2] or 0
|
||||
|
||||
# Compare
|
||||
issues = []
|
||||
|
||||
if fact_patients != mv_patients:
|
||||
issues.append(f"Patient count mismatch: fact={fact_patients:,}, mv={mv_patients:,}")
|
||||
|
||||
if mv_interventions != fact_interventions:
|
||||
issues.append(f"Intervention count mismatch: fact={fact_interventions:,}, mv={mv_interventions:,}")
|
||||
|
||||
# Allow small floating point differences in cost
|
||||
cost_diff = abs(fact_cost - mv_cost)
|
||||
if cost_diff > 0.01:
|
||||
issues.append(f"Cost mismatch: fact={fact_cost:,.2f}, mv={mv_cost:,.2f}, diff={cost_diff:.2f}")
|
||||
|
||||
if issues:
|
||||
return False, "; ".join(issues)
|
||||
|
||||
return True, f"MV consistent: {mv_patients:,} patients, {mv_interventions:,} interventions, £{mv_cost:,.2f} total"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,665 @@
|
||||
"""
|
||||
SQLite schema definitions for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
Contains SQL strings for creating reference tables, fact tables, and indexes.
|
||||
Schema design supports:
|
||||
- Reference data from CSV files (drug names, organizations, directories)
|
||||
- Drug-directory mappings with single-valid-directory flag
|
||||
- Patient intervention facts with proper indexing
|
||||
- Cached aggregations for performance
|
||||
- File tracking for incremental updates
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
import sqlite3
|
||||
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Reference Table Schemas
|
||||
# =============================================================================
|
||||
|
||||
REF_DRUG_NAMES_SCHEMA = """
|
||||
-- Mapping from raw drug names (as they appear in source data) to standardized names
|
||||
-- Source: data/drugnames.csv
|
||||
CREATE TABLE IF NOT EXISTS ref_drug_names (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
raw_name TEXT NOT NULL UNIQUE,
|
||||
standard_name TEXT NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Index for fast lookups during data transformation
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_names_raw ON ref_drug_names(raw_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_names_standard ON ref_drug_names(standard_name);
|
||||
"""
|
||||
|
||||
REF_ORGANIZATIONS_SCHEMA = """
|
||||
-- NHS organization codes and names
|
||||
-- Source: data/org_codes.csv
|
||||
CREATE TABLE IF NOT EXISTS ref_organizations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
org_code TEXT NOT NULL UNIQUE,
|
||||
org_name TEXT NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Index for fast lookups by organization code
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_organizations_code ON ref_organizations(org_code);
|
||||
"""
|
||||
|
||||
REF_DIRECTORIES_SCHEMA = """
|
||||
-- Medical directories/specialties
|
||||
-- Source: data/directory_list.csv
|
||||
CREATE TABLE IF NOT EXISTS ref_directories (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
directory_name TEXT NOT NULL UNIQUE,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
-- Index for fast lookups by directory name
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_directories_name ON ref_directories(directory_name);
|
||||
"""
|
||||
|
||||
REF_DRUG_DIRECTORY_MAP_SCHEMA = """
|
||||
-- Mapping from drug names to valid directories
|
||||
-- Source: data/drug_directory_list.csv
|
||||
-- A drug may map to multiple directories (one row per drug-directory pair)
|
||||
-- The is_single_valid flag indicates drugs with exactly ONE valid directory,
|
||||
-- which enables automatic directory assignment in department_identification()
|
||||
CREATE TABLE IF NOT EXISTS ref_drug_directory_map (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
drug_name TEXT NOT NULL,
|
||||
directory_name TEXT NOT NULL,
|
||||
is_single_valid BOOLEAN NOT NULL DEFAULT 0,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(drug_name, directory_name)
|
||||
);
|
||||
|
||||
-- Index for looking up directories by drug name (most common access pattern)
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_directory_map_drug ON ref_drug_directory_map(drug_name);
|
||||
|
||||
-- Index for reverse lookup (find drugs by directory)
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_directory_map_directory ON ref_drug_directory_map(directory_name);
|
||||
|
||||
-- Index for quick filtering of single-valid drugs
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_directory_map_single ON ref_drug_directory_map(is_single_valid);
|
||||
"""
|
||||
|
||||
REF_DRUG_INDICATION_CLUSTERS_SCHEMA = """
|
||||
-- Mapping from drugs to SNOMED clusters for indication validation
|
||||
-- Source: data/drug_indication_clusters.csv
|
||||
-- Used to validate that patients have appropriate GP diagnoses for their prescribed drugs
|
||||
-- A drug may map to multiple clusters (one row per drug-indication-cluster combination)
|
||||
CREATE TABLE IF NOT EXISTS ref_drug_indication_clusters (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
drug_name TEXT NOT NULL,
|
||||
indication TEXT NOT NULL,
|
||||
cluster_id TEXT NOT NULL,
|
||||
cluster_description TEXT,
|
||||
nice_ta_reference TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(drug_name, indication, cluster_id)
|
||||
);
|
||||
|
||||
-- Index for looking up clusters by drug name (most common access pattern)
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_indication_clusters_drug ON ref_drug_indication_clusters(drug_name);
|
||||
|
||||
-- Index for looking up drugs by cluster (for finding all drugs treating a condition)
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_indication_clusters_cluster ON ref_drug_indication_clusters(cluster_id);
|
||||
|
||||
-- Index for looking up by indication text
|
||||
CREATE INDEX IF NOT EXISTS idx_ref_drug_indication_clusters_indication ON ref_drug_indication_clusters(indication);
|
||||
"""
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Fact Table Schemas
|
||||
# =============================================================================
|
||||
|
||||
FACT_INTERVENTIONS_SCHEMA = """
|
||||
-- Patient intervention records (fact table)
|
||||
-- Source: HCD activity data (CSV/Parquet files or Snowflake)
|
||||
-- This is the main fact table storing all patient intervention events
|
||||
CREATE TABLE IF NOT EXISTS fact_interventions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
|
||||
-- Patient identification
|
||||
upid TEXT NOT NULL, -- Unique Patient ID (Provider Code[:3] + PersonKey)
|
||||
provider_code TEXT NOT NULL, -- Original provider code (3-5 chars)
|
||||
person_key TEXT NOT NULL, -- Patient key from source system
|
||||
|
||||
-- Intervention details
|
||||
drug_name_raw TEXT, -- Original drug name from source
|
||||
drug_name_std TEXT NOT NULL, -- Standardized drug name (via ref_drug_names)
|
||||
intervention_date DATE NOT NULL, -- Date of intervention
|
||||
price_actual REAL NOT NULL DEFAULT 0, -- Cost of intervention in GBP
|
||||
|
||||
-- Organization and directory
|
||||
org_name TEXT, -- Organization name (cleaned, no commas)
|
||||
directory TEXT, -- Medical directory/specialty (may be "Undefined")
|
||||
|
||||
-- Source tracking
|
||||
source_file TEXT, -- Original file this record came from
|
||||
loaded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
-- Additional clinical fields (optional, used in directory fallback logic)
|
||||
treatment_function_code INTEGER,
|
||||
additional_detail_1 TEXT,
|
||||
additional_detail_2 TEXT,
|
||||
additional_detail_3 TEXT,
|
||||
additional_detail_4 TEXT,
|
||||
additional_detail_5 TEXT
|
||||
);
|
||||
|
||||
-- Primary indexes for common filter patterns used in generate_graph()
|
||||
-- UPID: Used for patient grouping, pathway analysis
|
||||
CREATE INDEX IF NOT EXISTS idx_fact_interventions_upid ON fact_interventions(upid);
|
||||
|
||||
-- Drug name (standardized): Used for drug filtering
|
||||
CREATE INDEX IF NOT EXISTS idx_fact_interventions_drug ON fact_interventions(drug_name_std);
|
||||
|
||||
-- Intervention date: Used for date range filtering (start_date, end_date, last_seen)
|
||||
CREATE INDEX IF NOT EXISTS idx_fact_interventions_date ON fact_interventions(intervention_date);
|
||||
|
||||
-- Directory: Used for directory/specialty filtering
|
||||
CREATE INDEX IF NOT EXISTS idx_fact_interventions_directory ON fact_interventions(directory);
|
||||
|
||||
-- Organization: Used for trust filtering (Provider Code maps to org_name)
|
||||
CREATE INDEX IF NOT EXISTS idx_fact_interventions_org ON fact_interventions(org_name);
|
||||
|
||||
-- Composite index for common filter combination (trust + drug + directory)
|
||||
CREATE INDEX IF NOT EXISTS idx_fact_interventions_composite
|
||||
ON fact_interventions(org_name, drug_name_std, directory);
|
||||
|
||||
-- Composite index for date-based patient analysis
|
||||
CREATE INDEX IF NOT EXISTS idx_fact_interventions_upid_date
|
||||
ON fact_interventions(upid, intervention_date);
|
||||
"""
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Materialized View Schemas (Cached Aggregations)
|
||||
# =============================================================================
|
||||
|
||||
MV_PATIENT_TREATMENT_SUMMARY_SCHEMA = """
|
||||
-- Materialized view of patient treatment summaries
|
||||
-- Pre-computed aggregations per patient for faster pathway analysis
|
||||
-- Refreshed when fact_interventions data changes
|
||||
CREATE TABLE IF NOT EXISTS mv_patient_treatment_summary (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
|
||||
-- Patient identification
|
||||
upid TEXT NOT NULL UNIQUE, -- Unique Patient ID
|
||||
|
||||
-- Organization and directory (for filtering)
|
||||
org_name TEXT, -- Organization name (first org seen)
|
||||
directory TEXT, -- Primary directory (first directory assigned)
|
||||
|
||||
-- Date range
|
||||
first_seen_date DATE NOT NULL, -- First intervention date
|
||||
last_seen_date DATE NOT NULL, -- Last intervention date
|
||||
days_treated INTEGER NOT NULL DEFAULT 0, -- Duration: last_seen - first_seen
|
||||
|
||||
-- Cost aggregations
|
||||
total_cost REAL NOT NULL DEFAULT 0, -- Sum of all intervention costs
|
||||
avg_cost_per_intervention REAL, -- Average cost per intervention
|
||||
|
||||
-- Treatment summary
|
||||
intervention_count INTEGER NOT NULL DEFAULT 0, -- Total number of interventions
|
||||
unique_drug_count INTEGER NOT NULL DEFAULT 0, -- Number of distinct drugs
|
||||
|
||||
-- Drug sequence (pipe-separated standardized drug names in chronological order)
|
||||
-- Example: "ADALIMUMAB|ETANERCEPT|INFLIXIMAB"
|
||||
drug_sequence TEXT,
|
||||
|
||||
-- Drug frequency counts (JSON: {"ADALIMUMAB": 5, "ETANERCEPT": 3})
|
||||
-- Stores count of each drug for this patient
|
||||
drug_counts_json TEXT,
|
||||
|
||||
-- Drug cost totals (JSON: {"ADALIMUMAB": 15000.00, "ETANERCEPT": 8000.00})
|
||||
-- Stores total cost per drug for this patient
|
||||
drug_costs_json TEXT,
|
||||
|
||||
-- Per-drug date ranges (JSON: {"ADALIMUMAB": {"first": "2023-01-01", "last": "2023-06-15"}, ...})
|
||||
-- Stores first/last date for each drug
|
||||
drug_date_ranges_json TEXT,
|
||||
|
||||
-- Metadata
|
||||
computed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
source_row_count INTEGER -- Number of fact_interventions rows used
|
||||
);
|
||||
|
||||
-- Index for fast patient lookup
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_upid ON mv_patient_treatment_summary(upid);
|
||||
|
||||
-- Indexes for common filter patterns
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_org ON mv_patient_treatment_summary(org_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_directory ON mv_patient_treatment_summary(directory);
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_first_seen ON mv_patient_treatment_summary(first_seen_date);
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_last_seen ON mv_patient_treatment_summary(last_seen_date);
|
||||
|
||||
-- Composite index for date range filtering (common in generate_graph)
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_date_range
|
||||
ON mv_patient_treatment_summary(first_seen_date, last_seen_date);
|
||||
|
||||
-- Composite index for org + directory + dates (full filter pattern)
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_filter_composite
|
||||
ON mv_patient_treatment_summary(org_name, directory, first_seen_date, last_seen_date);
|
||||
|
||||
-- Index for drug sequence pattern matching
|
||||
CREATE INDEX IF NOT EXISTS idx_mv_patient_summary_drug_seq ON mv_patient_treatment_summary(drug_sequence);
|
||||
"""
|
||||
|
||||
MATERIALIZED_VIEWS_SCHEMA = f"""
|
||||
-- Materialized Views Schema
|
||||
-- Pre-computed aggregations for performance
|
||||
|
||||
{MV_PATIENT_TREATMENT_SUMMARY_SCHEMA}
|
||||
"""
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# File Tracking Schemas (Incremental Updates)
|
||||
# =============================================================================
|
||||
|
||||
PROCESSED_FILES_SCHEMA = """
|
||||
-- Tracks processed data files for incremental updates
|
||||
-- Enables detecting changed files by comparing hashes
|
||||
-- Stores processing status and statistics
|
||||
CREATE TABLE IF NOT EXISTS processed_files (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
|
||||
-- File identification
|
||||
file_path TEXT NOT NULL, -- Full path to the file
|
||||
file_name TEXT NOT NULL, -- Just the filename (for display)
|
||||
file_hash TEXT NOT NULL, -- SHA256 hash of file contents
|
||||
|
||||
-- File metadata
|
||||
file_size_bytes INTEGER, -- Size of file in bytes
|
||||
file_modified_at TIMESTAMP, -- File's last modification timestamp
|
||||
|
||||
-- Processing results
|
||||
row_count INTEGER DEFAULT 0, -- Number of rows processed from this file
|
||||
status TEXT NOT NULL DEFAULT 'pending', -- pending, processing, success, error
|
||||
error_message TEXT, -- Error details if status='error'
|
||||
|
||||
-- Timestamps
|
||||
first_processed_at TIMESTAMP, -- When first processed
|
||||
last_processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
processing_duration_seconds REAL, -- How long processing took
|
||||
|
||||
-- Uniqueness: only one record per file path
|
||||
-- Hash changes indicate file content changed (needs reprocessing)
|
||||
UNIQUE(file_path)
|
||||
);
|
||||
|
||||
-- Index for fast lookup by file path
|
||||
CREATE INDEX IF NOT EXISTS idx_processed_files_path ON processed_files(file_path);
|
||||
|
||||
-- Index for finding files by status (e.g., find all pending or errored files)
|
||||
CREATE INDEX IF NOT EXISTS idx_processed_files_status ON processed_files(status);
|
||||
|
||||
-- Index for finding files by hash (detect if same file appears at different paths)
|
||||
CREATE INDEX IF NOT EXISTS idx_processed_files_hash ON processed_files(file_hash);
|
||||
|
||||
-- Index for finding recently processed files
|
||||
CREATE INDEX IF NOT EXISTS idx_processed_files_last_processed ON processed_files(last_processed_at);
|
||||
"""
|
||||
|
||||
FILE_TRACKING_SCHEMA = f"""
|
||||
-- File Tracking Schema
|
||||
-- Supports incremental data loading
|
||||
|
||||
{PROCESSED_FILES_SCHEMA}
|
||||
"""
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Combined Schemas
|
||||
# =============================================================================
|
||||
|
||||
REFERENCE_TABLES_SCHEMA = f"""
|
||||
-- Reference Tables Schema
|
||||
-- Contains lookup data migrated from CSV files
|
||||
|
||||
{REF_DRUG_NAMES_SCHEMA}
|
||||
|
||||
{REF_ORGANIZATIONS_SCHEMA}
|
||||
|
||||
{REF_DIRECTORIES_SCHEMA}
|
||||
|
||||
{REF_DRUG_DIRECTORY_MAP_SCHEMA}
|
||||
|
||||
{REF_DRUG_INDICATION_CLUSTERS_SCHEMA}
|
||||
"""
|
||||
|
||||
FACT_TABLES_SCHEMA = f"""
|
||||
-- Fact Tables Schema
|
||||
-- Contains patient intervention data
|
||||
|
||||
{FACT_INTERVENTIONS_SCHEMA}
|
||||
"""
|
||||
|
||||
ALL_TABLES_SCHEMA = f"""
|
||||
-- Complete Database Schema
|
||||
-- Reference tables + Fact tables + Materialized views + File tracking
|
||||
|
||||
{REFERENCE_TABLES_SCHEMA}
|
||||
|
||||
{FACT_TABLES_SCHEMA}
|
||||
|
||||
{MATERIALIZED_VIEWS_SCHEMA}
|
||||
|
||||
{FILE_TRACKING_SCHEMA}
|
||||
"""
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Schema Helper Functions
|
||||
# =============================================================================
|
||||
|
||||
def create_reference_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Create all reference tables in the database.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
"""
|
||||
logger.info("Creating reference tables...")
|
||||
conn.executescript(REFERENCE_TABLES_SCHEMA)
|
||||
logger.info("Reference tables created successfully")
|
||||
|
||||
|
||||
def drop_reference_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Drop all reference tables from the database.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Warning:
|
||||
This will delete all reference data. Use with caution.
|
||||
"""
|
||||
logger.warning("Dropping reference tables...")
|
||||
conn.executescript("""
|
||||
DROP TABLE IF EXISTS ref_drug_names;
|
||||
DROP TABLE IF EXISTS ref_organizations;
|
||||
DROP TABLE IF EXISTS ref_directories;
|
||||
DROP TABLE IF EXISTS ref_drug_directory_map;
|
||||
DROP TABLE IF EXISTS ref_drug_indication_clusters;
|
||||
""")
|
||||
logger.info("Reference tables dropped")
|
||||
|
||||
|
||||
def get_reference_table_counts(conn: sqlite3.Connection) -> dict[str, int]:
|
||||
"""
|
||||
Get row counts for all reference tables.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping table name to row count.
|
||||
"""
|
||||
tables = ["ref_drug_names", "ref_organizations", "ref_directories", "ref_drug_directory_map", "ref_drug_indication_clusters"]
|
||||
counts = {}
|
||||
|
||||
for table in tables:
|
||||
cursor = conn.execute(f"SELECT COUNT(*) FROM {table}")
|
||||
result = cursor.fetchone()
|
||||
counts[table] = result[0] if result else 0
|
||||
|
||||
return counts
|
||||
|
||||
|
||||
def verify_reference_tables_exist(conn: sqlite3.Connection) -> list[str]:
|
||||
"""
|
||||
Verify that all reference tables exist.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
List of missing table names. Empty list means all tables exist.
|
||||
"""
|
||||
required_tables = ["ref_drug_names", "ref_organizations", "ref_directories", "ref_drug_directory_map", "ref_drug_indication_clusters"]
|
||||
missing = []
|
||||
|
||||
for table in required_tables:
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?",
|
||||
(table,)
|
||||
)
|
||||
if cursor.fetchone() is None:
|
||||
missing.append(table)
|
||||
|
||||
return missing
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Fact Table Helper Functions
|
||||
# =============================================================================
|
||||
|
||||
def create_fact_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Create all fact tables in the database (including materialized views).
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
"""
|
||||
logger.info("Creating fact tables...")
|
||||
conn.executescript(FACT_TABLES_SCHEMA)
|
||||
conn.executescript(MATERIALIZED_VIEWS_SCHEMA)
|
||||
logger.info("Fact tables created successfully")
|
||||
|
||||
|
||||
def drop_fact_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Drop all fact tables from the database.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Warning:
|
||||
This will delete all patient intervention data. Use with caution.
|
||||
"""
|
||||
logger.warning("Dropping fact tables...")
|
||||
conn.executescript("""
|
||||
DROP TABLE IF EXISTS fact_interventions;
|
||||
DROP TABLE IF EXISTS mv_patient_treatment_summary;
|
||||
""")
|
||||
logger.info("Fact tables dropped")
|
||||
|
||||
|
||||
def get_fact_table_counts(conn: sqlite3.Connection) -> dict[str, int]:
|
||||
"""
|
||||
Get row counts for all fact tables (including materialized views).
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping table name to row count.
|
||||
"""
|
||||
tables = ["fact_interventions", "mv_patient_treatment_summary"]
|
||||
counts = {}
|
||||
|
||||
for table in tables:
|
||||
cursor = conn.execute(f"SELECT COUNT(*) FROM {table}")
|
||||
result = cursor.fetchone()
|
||||
counts[table] = result[0] if result else 0
|
||||
|
||||
return counts
|
||||
|
||||
|
||||
def verify_fact_tables_exist(conn: sqlite3.Connection) -> list[str]:
|
||||
"""
|
||||
Verify that all fact tables exist (including materialized views).
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
List of missing table names. Empty list means all tables exist.
|
||||
"""
|
||||
required_tables = ["fact_interventions", "mv_patient_treatment_summary"]
|
||||
missing = []
|
||||
|
||||
for table in required_tables:
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?",
|
||||
(table,)
|
||||
)
|
||||
if cursor.fetchone() is None:
|
||||
missing.append(table)
|
||||
|
||||
return missing
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# File Tracking Helper Functions
|
||||
# =============================================================================
|
||||
|
||||
def create_file_tracking_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Create file tracking tables in the database.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
"""
|
||||
logger.info("Creating file tracking tables...")
|
||||
conn.executescript(FILE_TRACKING_SCHEMA)
|
||||
logger.info("File tracking tables created successfully")
|
||||
|
||||
|
||||
def drop_file_tracking_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Drop file tracking tables from the database.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Warning:
|
||||
This will delete all file tracking history.
|
||||
"""
|
||||
logger.warning("Dropping file tracking tables...")
|
||||
conn.executescript("""
|
||||
DROP TABLE IF EXISTS processed_files;
|
||||
""")
|
||||
logger.info("File tracking tables dropped")
|
||||
|
||||
|
||||
def get_file_tracking_counts(conn: sqlite3.Connection) -> dict[str, int]:
|
||||
"""
|
||||
Get row counts for file tracking tables.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping table name to row count.
|
||||
"""
|
||||
tables = ["processed_files"]
|
||||
counts = {}
|
||||
|
||||
for table in tables:
|
||||
cursor = conn.execute(f"SELECT COUNT(*) FROM {table}")
|
||||
result = cursor.fetchone()
|
||||
counts[table] = result[0] if result else 0
|
||||
|
||||
return counts
|
||||
|
||||
|
||||
def verify_file_tracking_tables_exist(conn: sqlite3.Connection) -> list[str]:
|
||||
"""
|
||||
Verify that file tracking tables exist.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
List of missing table names. Empty list means all tables exist.
|
||||
"""
|
||||
required_tables = ["processed_files"]
|
||||
missing = []
|
||||
|
||||
for table in required_tables:
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?",
|
||||
(table,)
|
||||
)
|
||||
if cursor.fetchone() is None:
|
||||
missing.append(table)
|
||||
|
||||
return missing
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Combined Helper Functions
|
||||
# =============================================================================
|
||||
|
||||
def create_all_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Create all tables (reference + fact) in the database.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
"""
|
||||
logger.info("Creating all database tables...")
|
||||
conn.executescript(ALL_TABLES_SCHEMA)
|
||||
logger.info("All tables created successfully")
|
||||
|
||||
|
||||
def drop_all_tables(conn: sqlite3.Connection) -> None:
|
||||
"""
|
||||
Drop all tables from the database.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Warning:
|
||||
This will delete all data. Use with extreme caution.
|
||||
"""
|
||||
logger.warning("Dropping all tables...")
|
||||
drop_file_tracking_tables(conn)
|
||||
drop_fact_tables(conn)
|
||||
drop_reference_tables(conn)
|
||||
logger.info("All tables dropped")
|
||||
|
||||
|
||||
def get_all_table_counts(conn: sqlite3.Connection) -> dict[str, int]:
|
||||
"""
|
||||
Get row counts for all tables.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping table name to row count.
|
||||
"""
|
||||
counts = {}
|
||||
counts.update(get_reference_table_counts(conn))
|
||||
counts.update(get_fact_table_counts(conn))
|
||||
counts.update(get_file_tracking_counts(conn))
|
||||
return counts
|
||||
|
||||
|
||||
def verify_all_tables_exist(conn: sqlite3.Connection) -> list[str]:
|
||||
"""
|
||||
Verify that all tables exist.
|
||||
|
||||
Args:
|
||||
conn: SQLite database connection.
|
||||
|
||||
Returns:
|
||||
List of missing table names. Empty list means all tables exist.
|
||||
"""
|
||||
missing = []
|
||||
missing.extend(verify_reference_tables_exist(conn))
|
||||
missing.extend(verify_fact_tables_exist(conn))
|
||||
missing.extend(verify_file_tracking_tables_exist(conn))
|
||||
return missing
|
||||
@@ -0,0 +1,797 @@
|
||||
"""
|
||||
Snowflake connector module for NHS Patient Pathway Analysis.
|
||||
|
||||
Provides connection handling with SSO browser authentication for NHS environments.
|
||||
Uses the externalbrowser authenticator which opens a browser window for NHS identity
|
||||
management authentication.
|
||||
|
||||
Usage:
|
||||
from data_processing.snowflake_connector import SnowflakeConnector, get_connector
|
||||
|
||||
# Using context manager (recommended)
|
||||
with get_connector() as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT * FROM table LIMIT 10")
|
||||
results = cursor.fetchall()
|
||||
|
||||
# Manual connection management
|
||||
connector = SnowflakeConnector()
|
||||
try:
|
||||
conn = connector.connect()
|
||||
cursor = conn.cursor()
|
||||
# ... use cursor ...
|
||||
finally:
|
||||
connector.close()
|
||||
"""
|
||||
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Generator, Optional, TYPE_CHECKING
|
||||
import time
|
||||
|
||||
# Snowflake connector is an optional dependency
|
||||
SNOWFLAKE_AVAILABLE = False
|
||||
try:
|
||||
import snowflake.connector
|
||||
from snowflake.connector import SnowflakeConnection
|
||||
from snowflake.connector.cursor import SnowflakeCursor
|
||||
SNOWFLAKE_AVAILABLE = True
|
||||
except ImportError:
|
||||
snowflake = None # type: ignore[assignment]
|
||||
|
||||
# Type hints for when snowflake is not available
|
||||
if TYPE_CHECKING:
|
||||
from snowflake.connector import SnowflakeConnection
|
||||
from snowflake.connector.cursor import SnowflakeCursor
|
||||
|
||||
from config import get_snowflake_config, SnowflakeConfig
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class SnowflakeConnectionError(Exception):
|
||||
"""Raised when Snowflake connection fails."""
|
||||
pass
|
||||
|
||||
|
||||
class SnowflakeNotConfiguredError(Exception):
|
||||
"""Raised when Snowflake is not configured (no account)."""
|
||||
pass
|
||||
|
||||
|
||||
class SnowflakeNotAvailableError(Exception):
|
||||
"""Raised when snowflake-connector-python is not installed."""
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConnectionInfo:
|
||||
"""Information about the current connection state."""
|
||||
connected: bool = False
|
||||
account: str = ""
|
||||
warehouse: str = ""
|
||||
database: str = ""
|
||||
schema: str = ""
|
||||
user: str = ""
|
||||
role: str = ""
|
||||
connected_at: Optional[datetime] = None
|
||||
last_query_at: Optional[datetime] = None
|
||||
query_count: int = 0
|
||||
|
||||
|
||||
class SnowflakeConnector:
|
||||
"""
|
||||
Manages Snowflake connections with SSO browser authentication.
|
||||
|
||||
This class provides connection management for NHS Snowflake access using
|
||||
the externalbrowser authenticator which triggers NHS SSO login via browser.
|
||||
|
||||
Attributes:
|
||||
config: SnowflakeConfig with connection settings
|
||||
connection_info: ConnectionInfo tracking current state
|
||||
|
||||
Example:
|
||||
connector = SnowflakeConnector()
|
||||
with connector.get_connection() as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT CURRENT_USER()")
|
||||
print(cursor.fetchone()[0])
|
||||
"""
|
||||
|
||||
def __init__(self, config: Optional[SnowflakeConfig] = None):
|
||||
"""
|
||||
Initialize the connector with configuration.
|
||||
|
||||
Args:
|
||||
config: Optional SnowflakeConfig. If not provided, loads from
|
||||
config/snowflake.toml using get_snowflake_config().
|
||||
"""
|
||||
self._config = config or get_snowflake_config()
|
||||
self._connection: Optional[SnowflakeConnection] = None
|
||||
self._connection_info = ConnectionInfo()
|
||||
|
||||
@property
|
||||
def config(self) -> SnowflakeConfig:
|
||||
"""Return the Snowflake configuration."""
|
||||
return self._config
|
||||
|
||||
@property
|
||||
def connection_info(self) -> ConnectionInfo:
|
||||
"""Return information about the current connection state."""
|
||||
return self._connection_info
|
||||
|
||||
@property
|
||||
def is_connected(self) -> bool:
|
||||
"""Return True if currently connected to Snowflake."""
|
||||
return self._connection is not None and not self._connection.is_closed()
|
||||
|
||||
def _check_availability(self) -> None:
|
||||
"""Check that snowflake-connector-python is installed."""
|
||||
if not SNOWFLAKE_AVAILABLE:
|
||||
raise SnowflakeNotAvailableError(
|
||||
"snowflake-connector-python is not installed. "
|
||||
"Install it with: pip install snowflake-connector-python"
|
||||
)
|
||||
|
||||
def _check_configured(self) -> None:
|
||||
"""Check that Snowflake is configured."""
|
||||
if not self._config.is_configured:
|
||||
raise SnowflakeNotConfiguredError(
|
||||
"Snowflake account is not configured. "
|
||||
"Edit config/snowflake.toml and set connection.account"
|
||||
)
|
||||
|
||||
def connect(self) -> SnowflakeConnection:
|
||||
"""
|
||||
Establish a connection to Snowflake.
|
||||
|
||||
Uses the externalbrowser authenticator which opens a browser window
|
||||
for NHS SSO authentication. The browser popup is expected and normal.
|
||||
|
||||
Returns:
|
||||
Active SnowflakeConnection
|
||||
|
||||
Raises:
|
||||
SnowflakeNotAvailableError: If snowflake-connector-python not installed
|
||||
SnowflakeNotConfiguredError: If account is not configured
|
||||
SnowflakeConnectionError: If connection fails
|
||||
"""
|
||||
self._check_availability()
|
||||
self._check_configured()
|
||||
|
||||
# Close existing connection if any
|
||||
if self._connection is not None:
|
||||
self.close()
|
||||
|
||||
conn_cfg = self._config.connection
|
||||
timeout_cfg = self._config.timeouts
|
||||
|
||||
logger.info(f"Connecting to Snowflake account: {conn_cfg.account}")
|
||||
logger.info(f"Using warehouse: {conn_cfg.warehouse}, database: {conn_cfg.database}")
|
||||
logger.info(f"Authenticator: {conn_cfg.authenticator}")
|
||||
if conn_cfg.authenticator == "externalbrowser":
|
||||
logger.info("Browser window will open for NHS SSO authentication")
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Build connection parameters
|
||||
connect_params = {
|
||||
"account": conn_cfg.account,
|
||||
"warehouse": conn_cfg.warehouse,
|
||||
"database": conn_cfg.database,
|
||||
"schema": conn_cfg.schema,
|
||||
"authenticator": conn_cfg.authenticator,
|
||||
"login_timeout": timeout_cfg.login_timeout,
|
||||
"network_timeout": timeout_cfg.connection_timeout,
|
||||
}
|
||||
|
||||
# Optional parameters (only add if set)
|
||||
if conn_cfg.user:
|
||||
connect_params["user"] = conn_cfg.user
|
||||
if conn_cfg.role:
|
||||
connect_params["role"] = conn_cfg.role
|
||||
|
||||
self._connection = snowflake.connector.connect(**connect_params)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"Connected to Snowflake successfully in {elapsed:.1f}s")
|
||||
|
||||
# Update connection info
|
||||
self._connection_info = ConnectionInfo(
|
||||
connected=True,
|
||||
account=conn_cfg.account,
|
||||
warehouse=conn_cfg.warehouse,
|
||||
database=conn_cfg.database,
|
||||
schema=conn_cfg.schema,
|
||||
user=self._get_current_user(),
|
||||
role=self._get_current_role(),
|
||||
connected_at=datetime.now(),
|
||||
query_count=0,
|
||||
)
|
||||
|
||||
return self._connection
|
||||
|
||||
except Exception as e:
|
||||
elapsed = time.time() - start_time
|
||||
logger.error(f"Failed to connect to Snowflake after {elapsed:.1f}s: {e}")
|
||||
self._connection_info = ConnectionInfo(connected=False)
|
||||
raise SnowflakeConnectionError(f"Failed to connect to Snowflake: {e}") from e
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the Snowflake connection if open."""
|
||||
if self._connection is not None:
|
||||
try:
|
||||
self._connection.close()
|
||||
logger.info("Snowflake connection closed")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error closing Snowflake connection: {e}")
|
||||
finally:
|
||||
self._connection = None
|
||||
self._connection_info = ConnectionInfo(connected=False)
|
||||
|
||||
def _get_current_user(self) -> str:
|
||||
"""Get the current authenticated user."""
|
||||
if self._connection is None:
|
||||
return ""
|
||||
try:
|
||||
cursor = self._connection.cursor()
|
||||
cursor.execute("SELECT CURRENT_USER()")
|
||||
result = cursor.fetchone()
|
||||
return result[0] if result else ""
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
def _get_current_role(self) -> str:
|
||||
"""Get the current active role."""
|
||||
if self._connection is None:
|
||||
return ""
|
||||
try:
|
||||
cursor = self._connection.cursor()
|
||||
cursor.execute("SELECT CURRENT_ROLE()")
|
||||
result = cursor.fetchone()
|
||||
return result[0] if result else ""
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
@contextmanager
|
||||
def get_connection(self) -> Generator[SnowflakeConnection, None, None]:
|
||||
"""
|
||||
Context manager for connection handling.
|
||||
|
||||
Creates a new connection if not already connected, yields the connection,
|
||||
and ensures proper cleanup on exit.
|
||||
|
||||
Yields:
|
||||
Active SnowflakeConnection
|
||||
|
||||
Example:
|
||||
connector = SnowflakeConnector()
|
||||
with connector.get_connection() as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT 1")
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
assert self._connection is not None, "Connection should be established"
|
||||
try:
|
||||
yield self._connection
|
||||
finally:
|
||||
# Keep connection open for reuse
|
||||
pass
|
||||
|
||||
@contextmanager
|
||||
def get_cursor(
|
||||
self,
|
||||
dict_cursor: bool = False
|
||||
) -> Generator[SnowflakeCursor, None, None]:
|
||||
"""
|
||||
Context manager that provides a cursor.
|
||||
|
||||
Args:
|
||||
dict_cursor: If True, returns cursor that yields dict-like rows
|
||||
|
||||
Yields:
|
||||
SnowflakeCursor for executing queries
|
||||
|
||||
Example:
|
||||
connector = SnowflakeConnector()
|
||||
with connector.get_cursor() as cursor:
|
||||
cursor.execute("SELECT * FROM table LIMIT 10")
|
||||
for row in cursor:
|
||||
print(row)
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
assert self._connection is not None, "Connection should be established"
|
||||
cursor: Any = None
|
||||
try:
|
||||
if dict_cursor:
|
||||
cursor = self._connection.cursor(snowflake.connector.DictCursor) # type: ignore[union-attr]
|
||||
else:
|
||||
cursor = self._connection.cursor()
|
||||
yield cursor # type: ignore[misc]
|
||||
self._connection_info.last_query_at = datetime.now()
|
||||
self._connection_info.query_count += 1
|
||||
finally:
|
||||
if cursor is not None:
|
||||
cursor.close()
|
||||
|
||||
def execute(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None,
|
||||
timeout: Optional[int] = None
|
||||
) -> list[tuple]:
|
||||
"""
|
||||
Execute a query and return all results.
|
||||
|
||||
Args:
|
||||
query: SQL query to execute
|
||||
params: Optional query parameters for parameterized queries
|
||||
timeout: Optional query timeout in seconds (overrides config)
|
||||
|
||||
Returns:
|
||||
List of result rows as tuples
|
||||
|
||||
Raises:
|
||||
SnowflakeConnectionError: If not connected
|
||||
Various snowflake errors for query issues
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
effective_timeout = timeout or self._config.timeouts.query_timeout
|
||||
|
||||
with self.get_cursor() as cursor:
|
||||
logger.info(f"Executing query (timeout={effective_timeout}s)")
|
||||
logger.debug(f"Query: {query[:200]}...")
|
||||
|
||||
if effective_timeout > 0:
|
||||
cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
|
||||
|
||||
start_time = time.time()
|
||||
cursor.execute(query, params)
|
||||
results = cursor.fetchall()
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
logger.info(f"Query returned {len(results)} rows in {elapsed:.2f}s")
|
||||
return results
|
||||
|
||||
def execute_dict(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None,
|
||||
timeout: Optional[int] = None
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Execute a query and return results as list of dictionaries.
|
||||
|
||||
Args:
|
||||
query: SQL query to execute
|
||||
params: Optional query parameters
|
||||
timeout: Optional query timeout in seconds
|
||||
|
||||
Returns:
|
||||
List of result rows as dictionaries
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
effective_timeout = timeout or self._config.timeouts.query_timeout
|
||||
|
||||
with self.get_cursor(dict_cursor=True) as cursor:
|
||||
logger.info(f"Executing query (timeout={effective_timeout}s)")
|
||||
logger.debug(f"Query: {query[:200]}...")
|
||||
|
||||
if effective_timeout > 0:
|
||||
cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
|
||||
|
||||
start_time = time.time()
|
||||
cursor.execute(query, params)
|
||||
results = cursor.fetchall()
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
logger.info(f"Query returned {len(results)} rows in {elapsed:.2f}s")
|
||||
return results # type: ignore[return-value]
|
||||
|
||||
def execute_chunked(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None,
|
||||
chunk_size: Optional[int] = None,
|
||||
timeout: Optional[int] = None,
|
||||
max_rows: Optional[int] = None,
|
||||
) -> Generator[list[tuple], None, None]:
|
||||
"""
|
||||
Execute a query and yield results in chunks for memory efficiency.
|
||||
|
||||
This method is useful for large result sets that would exceed memory
|
||||
if loaded all at once. Results are yielded as chunks of rows.
|
||||
|
||||
Args:
|
||||
query: SQL query to execute
|
||||
params: Optional query parameters for parameterized queries
|
||||
chunk_size: Number of rows per chunk (default from config)
|
||||
timeout: Optional query timeout in seconds (overrides config)
|
||||
max_rows: Maximum total rows to return (default from config, 0 for no limit)
|
||||
|
||||
Yields:
|
||||
List of result rows as tuples for each chunk
|
||||
|
||||
Example:
|
||||
for chunk in connector.execute_chunked("SELECT * FROM large_table"):
|
||||
process_chunk(chunk)
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
effective_timeout = timeout or self._config.timeouts.query_timeout
|
||||
effective_chunk_size = chunk_size or self._config.query.chunk_size
|
||||
effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
|
||||
|
||||
with self.get_cursor() as cursor:
|
||||
logger.info(f"Executing chunked query (chunk_size={effective_chunk_size}, timeout={effective_timeout}s)")
|
||||
logger.debug(f"Query: {query[:200]}...")
|
||||
|
||||
if effective_timeout > 0:
|
||||
cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
|
||||
|
||||
start_time = time.time()
|
||||
cursor.execute(query, params)
|
||||
|
||||
total_rows = 0
|
||||
chunk_num = 0
|
||||
|
||||
while True:
|
||||
# Determine how many rows to fetch this chunk
|
||||
if effective_max_rows > 0:
|
||||
remaining = effective_max_rows - total_rows
|
||||
if remaining <= 0:
|
||||
break
|
||||
fetch_size = min(effective_chunk_size, remaining)
|
||||
else:
|
||||
fetch_size = effective_chunk_size
|
||||
|
||||
chunk = cursor.fetchmany(fetch_size)
|
||||
if not chunk:
|
||||
break
|
||||
|
||||
chunk_num += 1
|
||||
total_rows += len(chunk)
|
||||
logger.debug(f"Chunk {chunk_num}: {len(chunk)} rows (total: {total_rows})")
|
||||
yield chunk
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"Chunked query returned {total_rows} rows in {chunk_num} chunks ({elapsed:.2f}s)")
|
||||
|
||||
def execute_chunked_dict(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None,
|
||||
chunk_size: Optional[int] = None,
|
||||
timeout: Optional[int] = None,
|
||||
max_rows: Optional[int] = None,
|
||||
) -> Generator[list[dict], None, None]:
|
||||
"""
|
||||
Execute a query and yield dict results in chunks for memory efficiency.
|
||||
|
||||
Same as execute_chunked but returns rows as dictionaries.
|
||||
|
||||
Args:
|
||||
query: SQL query to execute
|
||||
params: Optional query parameters
|
||||
chunk_size: Number of rows per chunk (default from config)
|
||||
timeout: Optional query timeout in seconds
|
||||
max_rows: Maximum total rows to return (default from config, 0 for no limit)
|
||||
|
||||
Yields:
|
||||
List of result rows as dictionaries for each chunk
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
effective_timeout = timeout or self._config.timeouts.query_timeout
|
||||
effective_chunk_size = chunk_size or self._config.query.chunk_size
|
||||
effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
|
||||
|
||||
with self.get_cursor(dict_cursor=True) as cursor:
|
||||
logger.info(f"Executing chunked dict query (chunk_size={effective_chunk_size}, timeout={effective_timeout}s)")
|
||||
logger.debug(f"Query: {query[:200]}...")
|
||||
|
||||
if effective_timeout > 0:
|
||||
cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
|
||||
|
||||
start_time = time.time()
|
||||
cursor.execute(query, params)
|
||||
|
||||
total_rows = 0
|
||||
chunk_num = 0
|
||||
|
||||
while True:
|
||||
# Determine how many rows to fetch this chunk
|
||||
if effective_max_rows > 0:
|
||||
remaining = effective_max_rows - total_rows
|
||||
if remaining <= 0:
|
||||
break
|
||||
fetch_size = min(effective_chunk_size, remaining)
|
||||
else:
|
||||
fetch_size = effective_chunk_size
|
||||
|
||||
chunk = cursor.fetchmany(fetch_size)
|
||||
if not chunk:
|
||||
break
|
||||
|
||||
chunk_num += 1
|
||||
total_rows += len(chunk)
|
||||
logger.debug(f"Chunk {chunk_num}: {len(chunk)} rows (total: {total_rows})")
|
||||
yield chunk # type: ignore[misc]
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"Chunked dict query returned {total_rows} rows in {chunk_num} chunks ({elapsed:.2f}s)")
|
||||
|
||||
def execute_with_row_limit(
|
||||
self,
|
||||
query: str,
|
||||
params: Optional[tuple] = None,
|
||||
max_rows: Optional[int] = None,
|
||||
timeout: Optional[int] = None
|
||||
) -> tuple[list[dict], bool]:
|
||||
"""
|
||||
Execute a query with a row limit and indicate if more rows were available.
|
||||
|
||||
This is useful for pagination or previewing large result sets.
|
||||
|
||||
Args:
|
||||
query: SQL query to execute
|
||||
params: Optional query parameters
|
||||
max_rows: Maximum rows to return (default from config)
|
||||
timeout: Optional query timeout in seconds
|
||||
|
||||
Returns:
|
||||
Tuple of (results list, has_more bool)
|
||||
- results: List of result rows as dictionaries (up to max_rows)
|
||||
- has_more: True if there were more rows than max_rows
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
effective_timeout = timeout or self._config.timeouts.query_timeout
|
||||
effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
|
||||
|
||||
with self.get_cursor(dict_cursor=True) as cursor:
|
||||
logger.info(f"Executing query with limit (max_rows={effective_max_rows}, timeout={effective_timeout}s)")
|
||||
logger.debug(f"Query: {query[:200]}...")
|
||||
|
||||
if effective_timeout > 0:
|
||||
cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
|
||||
|
||||
start_time = time.time()
|
||||
cursor.execute(query, params)
|
||||
|
||||
# Fetch one more than max to detect if there are more rows
|
||||
results = cursor.fetchmany(effective_max_rows + 1)
|
||||
elapsed = time.time() - start_time
|
||||
|
||||
has_more = len(results) > effective_max_rows
|
||||
if has_more:
|
||||
results = results[:effective_max_rows]
|
||||
|
||||
logger.info(f"Query returned {len(results)} rows (has_more={has_more}) in {elapsed:.2f}s")
|
||||
return results, has_more # type: ignore[return-value]
|
||||
|
||||
def fetch_activity_data(
|
||||
self,
|
||||
start_date: Optional[date] = None,
|
||||
end_date: Optional[date] = None,
|
||||
provider_codes: Optional[list[str]] = None,
|
||||
max_rows: Optional[int] = None,
|
||||
timeout: Optional[int] = None,
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Fetch high-cost drug activity data from Snowflake.
|
||||
|
||||
Queries the CDM.Acute__Conmon__PatientLevelDrugs table and returns
|
||||
data in a format compatible with the existing analysis pipeline.
|
||||
|
||||
Args:
|
||||
start_date: Optional start date for filtering (inclusive)
|
||||
end_date: Optional end date for filtering (inclusive)
|
||||
provider_codes: Optional list of provider codes to filter by
|
||||
max_rows: Maximum rows to return (default from config)
|
||||
timeout: Query timeout in seconds (default from config)
|
||||
|
||||
Returns:
|
||||
List of dictionaries with keys matching expected DataFrame columns:
|
||||
- PseudoNHSNoLinked: Pseudonymised NHS number (for UPID creation)
|
||||
- Provider Code: NHS provider code
|
||||
- PersonKey: Local patient identifier
|
||||
- Drug Name: Raw drug name
|
||||
- Intervention Date: Date of intervention
|
||||
- Price Actual: Cost of intervention
|
||||
- OrganisationName: Provider organisation name
|
||||
- Treatment Function Code: NHS treatment function code
|
||||
- Additional Detail 1-5: Additional details for directory identification
|
||||
|
||||
Raises:
|
||||
SnowflakeConnectionError: If not connected or query fails
|
||||
"""
|
||||
if not self.is_connected:
|
||||
self.connect()
|
||||
|
||||
# Build the query
|
||||
table_name = 'DATA_HUB.CDM."Acute__Conmon__PatientLevelDrugs"'
|
||||
|
||||
query = f'''
|
||||
SELECT
|
||||
"PseudoNHSNoLinked",
|
||||
"ProviderCode" AS "Provider Code",
|
||||
"LocalPatientID" AS "PersonKey",
|
||||
"DrugName" AS "Drug Name",
|
||||
"InterventionDate" AS "Intervention Date",
|
||||
"PriceActual" AS "Price Actual",
|
||||
"ProviderName" AS "OrganisationName",
|
||||
"TreatmentFunctionCode" AS "Treatment Function Code",
|
||||
"TreatmentFunctionDesc" AS "Treatment Function Desc",
|
||||
"AdditionalDetail1" AS "Additional Detail 1",
|
||||
"AdditionalDescription1" AS "Additional Description 1",
|
||||
"AdditionalDetail2" AS "Additional Detail 2",
|
||||
"AdditionalDescription2" AS "Additional Description 2",
|
||||
"AdditionalDetail3" AS "Additional Detail 3",
|
||||
"AdditionalDescription3" AS "Additional Description 3",
|
||||
"AdditionalDetail4" AS "Additional Detail 4",
|
||||
"AdditionalDescription4" AS "Additional Description 4",
|
||||
"AdditionalDetail5" AS "Additional Detail 5",
|
||||
"AdditionalDescription5" AS "Additional Description 5"
|
||||
FROM {table_name}
|
||||
WHERE 1=1
|
||||
'''
|
||||
|
||||
params = []
|
||||
|
||||
# Add date filters
|
||||
if start_date:
|
||||
query += ' AND "InterventionDate" >= %s'
|
||||
params.append(start_date.isoformat())
|
||||
if end_date:
|
||||
query += ' AND "InterventionDate" <= %s'
|
||||
params.append(end_date.isoformat())
|
||||
|
||||
# Add provider filter
|
||||
if provider_codes:
|
||||
placeholders = ", ".join(["%s"] * len(provider_codes))
|
||||
query += f' AND "ProviderCode" IN ({placeholders})'
|
||||
params.extend(provider_codes)
|
||||
|
||||
# Add ordering for consistent results
|
||||
query += ' ORDER BY "InterventionDate", "ProviderCode", "PseudoNHSNoLinked"'
|
||||
|
||||
logger.info(f"Fetching activity data from Snowflake")
|
||||
if start_date:
|
||||
logger.info(f" Date range: {start_date} to {end_date or 'now'}")
|
||||
if provider_codes:
|
||||
logger.info(f" Providers: {provider_codes}")
|
||||
|
||||
effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
|
||||
effective_timeout = timeout or self._config.timeouts.query_timeout
|
||||
|
||||
# Execute with chunked results for large datasets
|
||||
all_results = []
|
||||
total_rows = 0
|
||||
|
||||
for chunk in self.execute_chunked_dict(
|
||||
query,
|
||||
params=tuple(params) if params else None,
|
||||
timeout=effective_timeout,
|
||||
max_rows=effective_max_rows,
|
||||
):
|
||||
all_results.extend(chunk)
|
||||
total_rows += len(chunk)
|
||||
logger.debug(f"Fetched {total_rows} rows so far...")
|
||||
|
||||
logger.info(f"Fetched {len(all_results)} activity records from Snowflake")
|
||||
return all_results
|
||||
|
||||
def test_connection(self) -> tuple[bool, str]:
|
||||
"""
|
||||
Test the Snowflake connection.
|
||||
|
||||
Returns:
|
||||
Tuple of (success: bool, message: str)
|
||||
"""
|
||||
try:
|
||||
self._check_availability()
|
||||
except SnowflakeNotAvailableError as e:
|
||||
return False, str(e)
|
||||
|
||||
try:
|
||||
self._check_configured()
|
||||
except SnowflakeNotConfiguredError as e:
|
||||
return False, str(e)
|
||||
|
||||
try:
|
||||
self.connect()
|
||||
user = self._get_current_user()
|
||||
role = self._get_current_role()
|
||||
return True, f"Connected as {user} with role {role}"
|
||||
except Exception as e:
|
||||
return False, f"Connection failed: {e}"
|
||||
|
||||
def __enter__(self) -> "SnowflakeConnector":
|
||||
"""Context manager entry."""
|
||||
self.connect()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb) -> None:
|
||||
"""Context manager exit."""
|
||||
self.close()
|
||||
|
||||
|
||||
# Module-level singleton for convenience
|
||||
_default_connector: Optional[SnowflakeConnector] = None
|
||||
|
||||
|
||||
def get_connector(config: Optional[SnowflakeConfig] = None) -> SnowflakeConnector:
|
||||
"""
|
||||
Get a Snowflake connector (creates singleton on first call).
|
||||
|
||||
Args:
|
||||
config: Optional configuration. If provided, creates new connector
|
||||
with this config. If None, uses/creates default connector.
|
||||
|
||||
Returns:
|
||||
SnowflakeConnector instance
|
||||
"""
|
||||
global _default_connector
|
||||
|
||||
if config is not None:
|
||||
# Custom config requested, create new connector
|
||||
return SnowflakeConnector(config)
|
||||
|
||||
if _default_connector is None:
|
||||
_default_connector = SnowflakeConnector()
|
||||
|
||||
return _default_connector
|
||||
|
||||
|
||||
def reset_connector() -> None:
|
||||
"""Reset the default connector (closes connection and clears singleton)."""
|
||||
global _default_connector
|
||||
|
||||
if _default_connector is not None:
|
||||
_default_connector.close()
|
||||
_default_connector = None
|
||||
|
||||
|
||||
def is_snowflake_available() -> bool:
|
||||
"""Return True if snowflake-connector-python is installed."""
|
||||
return SNOWFLAKE_AVAILABLE
|
||||
|
||||
|
||||
def is_snowflake_configured() -> bool:
|
||||
"""Return True if Snowflake account is configured."""
|
||||
try:
|
||||
config = get_snowflake_config()
|
||||
return config.is_configured
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# Export public API
|
||||
__all__ = [
|
||||
"SnowflakeConnector",
|
||||
"SnowflakeConnectionError",
|
||||
"SnowflakeNotConfiguredError",
|
||||
"SnowflakeNotAvailableError",
|
||||
"ConnectionInfo",
|
||||
"get_connector",
|
||||
"reset_connector",
|
||||
"is_snowflake_available",
|
||||
"is_snowflake_configured",
|
||||
"SNOWFLAKE_AVAILABLE",
|
||||
]
|
||||
@@ -0,0 +1,496 @@
|
||||
# Reflex Deployment Guide
|
||||
|
||||
This guide covers deployment options for the Patient Pathway Analysis web application built with Reflex.
|
||||
|
||||
## Overview
|
||||
|
||||
Reflex applications compile to a FastAPI backend and Next.js frontend. This creates two deployment artifacts that can be deployed together or separately depending on your infrastructure requirements.
|
||||
|
||||
## Development Mode
|
||||
|
||||
For local development:
|
||||
|
||||
```bash
|
||||
# Start development server with hot reload
|
||||
reflex run
|
||||
|
||||
# Access the application at http://localhost:3000
|
||||
```
|
||||
|
||||
## Production Deployment Options
|
||||
|
||||
### Option 1: Simple Production (Single Server)
|
||||
|
||||
The simplest approach for internal deployments:
|
||||
|
||||
```bash
|
||||
# Run in production mode (optimized build)
|
||||
reflex run --env prod
|
||||
```
|
||||
|
||||
This starts:
|
||||
- FastAPI backend on port 8000
|
||||
- Next.js frontend on port 3000
|
||||
|
||||
For background execution:
|
||||
|
||||
```bash
|
||||
# Using nohup (Linux/macOS)
|
||||
nohup reflex run --env prod > reflex.log 2>&1 &
|
||||
|
||||
# Using PowerShell (Windows)
|
||||
Start-Process -NoNewWindow -FilePath "reflex" -ArgumentList "run --env prod"
|
||||
```
|
||||
|
||||
### Option 2: Separate Backend and Frontend
|
||||
|
||||
For more control, run backend and frontend separately:
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start backend only
|
||||
reflex run --env prod --backend-only
|
||||
|
||||
# Terminal 2: Start frontend only
|
||||
reflex run --env prod --frontend-only
|
||||
```
|
||||
|
||||
### Option 3: Static Export
|
||||
|
||||
Export the frontend as static files for deployment on static hosting or CDN:
|
||||
|
||||
```bash
|
||||
# Export application
|
||||
reflex export
|
||||
|
||||
# This creates:
|
||||
# - frontend.zip (static Next.js build)
|
||||
# - backend.zip (Python application source)
|
||||
```
|
||||
|
||||
Then:
|
||||
1. Unzip `frontend.zip` and serve via nginx, Apache, or any static file server
|
||||
2. Run the backend separately using uvicorn/gunicorn
|
||||
|
||||
### Option 4: Docker Deployment
|
||||
|
||||
Create a `Dockerfile` for containerized deployment:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install Node.js for Reflex frontend build
|
||||
RUN apt-get update && apt-get install -y curl && \
|
||||
curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
|
||||
apt-get install -y nodejs && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy requirements and install dependencies
|
||||
COPY requirements.txt pyproject.toml ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy application code
|
||||
COPY . .
|
||||
|
||||
# Initialize Reflex (downloads frontend dependencies)
|
||||
RUN reflex init --loglevel debug
|
||||
|
||||
# Expose ports
|
||||
EXPOSE 3000 8000
|
||||
|
||||
# Start in production mode
|
||||
CMD ["reflex", "run", "--env", "prod"]
|
||||
```
|
||||
|
||||
Build and run:
|
||||
|
||||
```bash
|
||||
# Build the image
|
||||
docker build -t pathway-analysis .
|
||||
|
||||
# Run the container
|
||||
docker run -p 3000:3000 -p 8000:8000 \
|
||||
-v $(pwd)/data:/app/data \
|
||||
-v $(pwd)/config:/app/config \
|
||||
pathway-analysis
|
||||
```
|
||||
|
||||
### Option 5: Docker Compose (Recommended for Production)
|
||||
|
||||
Create `docker-compose.yml` for multi-container deployment:
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
backend:
|
||||
build: .
|
||||
command: reflex run --env prod --backend-only
|
||||
ports:
|
||||
- "8000:8000"
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
- ./config:/app/config
|
||||
environment:
|
||||
- REFLEX_ENV=prod
|
||||
restart: unless-stopped
|
||||
|
||||
frontend:
|
||||
build: .
|
||||
command: reflex run --env prod --frontend-only
|
||||
ports:
|
||||
- "3000:3000"
|
||||
depends_on:
|
||||
- backend
|
||||
environment:
|
||||
- REFLEX_ENV=prod
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
Run with:
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
## Reverse Proxy Configuration
|
||||
|
||||
### Nginx
|
||||
|
||||
For production deployments behind nginx:
|
||||
|
||||
```nginx
|
||||
# /etc/nginx/sites-available/pathway-analysis
|
||||
server {
|
||||
listen 80;
|
||||
server_name your-server.nhs.uk;
|
||||
|
||||
# Backend API endpoints
|
||||
location /admin {
|
||||
proxy_pass http://localhost:8000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location /ping {
|
||||
proxy_pass http://localhost:8000;
|
||||
}
|
||||
|
||||
location /upload {
|
||||
proxy_pass http://localhost:8000;
|
||||
client_max_body_size 100M; # For large data file uploads
|
||||
}
|
||||
|
||||
# WebSocket connections (required for Reflex state sync)
|
||||
location /_event/ {
|
||||
proxy_pass http://localhost:8000;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 86400; # 24 hours for long-running connections
|
||||
}
|
||||
|
||||
# Frontend (all other requests)
|
||||
location / {
|
||||
proxy_pass http://localhost:3000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Enable the site:
|
||||
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/pathway-analysis /etc/nginx/sites-enabled/
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
### Caddy (Alternative)
|
||||
|
||||
Caddy provides automatic HTTPS:
|
||||
|
||||
```caddyfile
|
||||
# Caddyfile
|
||||
your-server.nhs.uk {
|
||||
# Backend API
|
||||
handle /admin/* {
|
||||
reverse_proxy localhost:8000
|
||||
}
|
||||
handle /ping {
|
||||
reverse_proxy localhost:8000
|
||||
}
|
||||
handle /upload {
|
||||
reverse_proxy localhost:8000
|
||||
}
|
||||
handle /_event/* {
|
||||
reverse_proxy localhost:8000
|
||||
}
|
||||
|
||||
# Frontend
|
||||
handle {
|
||||
reverse_proxy localhost:3000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Process Management
|
||||
|
||||
### Systemd (Linux)
|
||||
|
||||
Create service files for automatic startup:
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/pathway-backend.service
|
||||
[Unit]
|
||||
Description=Pathway Analysis Backend
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=www-data
|
||||
WorkingDirectory=/opt/pathway-analysis
|
||||
ExecStart=/usr/bin/reflex run --env prod --backend-only
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/pathway-frontend.service
|
||||
[Unit]
|
||||
Description=Pathway Analysis Frontend
|
||||
After=network.target pathway-backend.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=www-data
|
||||
WorkingDirectory=/opt/pathway-analysis
|
||||
ExecStart=/usr/bin/reflex run --env prod --frontend-only
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Enable and start:
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable pathway-backend pathway-frontend
|
||||
sudo systemctl start pathway-backend pathway-frontend
|
||||
```
|
||||
|
||||
### Windows Service
|
||||
|
||||
Use NSSM (Non-Sucking Service Manager) on Windows:
|
||||
|
||||
```powershell
|
||||
# Install NSSM
|
||||
choco install nssm
|
||||
|
||||
# Create service
|
||||
nssm install PathwayAnalysis "C:\Path\To\reflex.exe" "run --env prod"
|
||||
nssm set PathwayAnalysis AppDirectory "C:\Path\To\Patient pathway analysis"
|
||||
nssm start PathwayAnalysis
|
||||
```
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
### Production Environment Variables
|
||||
|
||||
Set these environment variables for production:
|
||||
|
||||
```bash
|
||||
# Reflex configuration
|
||||
export REFLEX_ENV=prod
|
||||
|
||||
# Database paths (if using custom locations)
|
||||
export PATHWAY_DB_PATH=/var/data/pathways.db
|
||||
export PATHWAY_CACHE_DIR=/var/cache/pathway-analysis
|
||||
|
||||
# Snowflake (if using)
|
||||
export SNOWFLAKE_ACCOUNT=your-account
|
||||
export SNOWFLAKE_WAREHOUSE=your-warehouse
|
||||
```
|
||||
|
||||
### Snowflake Configuration
|
||||
|
||||
Ensure `config/snowflake.toml` is properly configured for production:
|
||||
|
||||
```toml
|
||||
[connection]
|
||||
account = "your-production-account"
|
||||
warehouse = "ANALYTICS_WH"
|
||||
database = "DATA_HUB"
|
||||
schema = "CDM"
|
||||
authenticator = "externalbrowser" # or "oauth" for service accounts
|
||||
|
||||
[cache]
|
||||
enabled = true
|
||||
directory = "/var/cache/pathway-analysis"
|
||||
ttl_seconds = 86400 # 24 hours
|
||||
```
|
||||
|
||||
## Reflex Cloud
|
||||
|
||||
For managed hosting, consider [Reflex Cloud](https://reflex.dev/cloud/):
|
||||
|
||||
```bash
|
||||
# Deploy to Reflex Cloud
|
||||
reflex deploy
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- Zero configuration deployment
|
||||
- Automatic scaling
|
||||
- Built-in SSL certificates
|
||||
- Managed state management with Redis
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Network Security
|
||||
|
||||
1. **Firewall Rules**: Only expose necessary ports (typically just 80/443)
|
||||
2. **HTTPS**: Use TLS certificates (Let's Encrypt or organizational certs)
|
||||
3. **VPN**: Consider restricting access to NHS network only
|
||||
|
||||
### Data Security
|
||||
|
||||
1. **Database Access**: Ensure SQLite database permissions are restricted
|
||||
2. **File Uploads**: Validate file types and scan for malware
|
||||
3. **Snowflake**: Use least-privilege service accounts
|
||||
|
||||
### Authentication
|
||||
|
||||
For NHS deployments, consider adding authentication:
|
||||
|
||||
```python
|
||||
# Example: Add basic auth middleware
|
||||
import reflex as rx
|
||||
from starlette.middleware import Middleware
|
||||
from starlette.middleware.authentication import AuthenticationMiddleware
|
||||
|
||||
# In rxconfig.py
|
||||
config = rx.Config(
|
||||
app_name="pathways_app",
|
||||
# Add authentication middleware
|
||||
)
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
The application provides endpoints for monitoring:
|
||||
|
||||
- `/ping` - Basic health check
|
||||
- Backend port 8000 - FastAPI health
|
||||
|
||||
### Logging
|
||||
|
||||
Configure logging for production:
|
||||
|
||||
```python
|
||||
# In pathways_app/pathways_app.py
|
||||
import logging
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
|
||||
handlers=[
|
||||
logging.FileHandler('/var/log/pathway-analysis/app.log'),
|
||||
logging.StreamHandler()
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Port already in use:**
|
||||
```bash
|
||||
# Find and kill process using port 3000
|
||||
lsof -i :3000
|
||||
kill -9 <PID>
|
||||
```
|
||||
|
||||
**Build cache issues:**
|
||||
```bash
|
||||
# Clear Reflex build cache
|
||||
rm -rf .web
|
||||
reflex run --env prod
|
||||
```
|
||||
|
||||
**Database connection errors:**
|
||||
```bash
|
||||
# Verify database exists and has correct permissions
|
||||
ls -la data/pathways.db
|
||||
sqlite3 data/pathways.db ".tables"
|
||||
```
|
||||
|
||||
**Snowflake authentication:**
|
||||
- Ensure browser is available for SSO popup
|
||||
- Check firewall allows connections to Snowflake endpoints
|
||||
- Verify account identifier is correct
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Backend (FastAPI/Uvicorn)
|
||||
|
||||
For high-traffic deployments:
|
||||
|
||||
```bash
|
||||
# Run with multiple workers
|
||||
uvicorn pathways_app:app --workers 4 --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### State Management
|
||||
|
||||
For multi-instance deployments, configure Redis for state management:
|
||||
|
||||
```python
|
||||
# rxconfig.py
|
||||
config = rx.Config(
|
||||
app_name="pathways_app",
|
||||
state_manager_mode="redis",
|
||||
redis_url="redis://localhost:6379/0",
|
||||
)
|
||||
```
|
||||
|
||||
### Caching
|
||||
|
||||
Enable aggressive caching for Snowflake queries in `config/snowflake.toml`:
|
||||
|
||||
```toml
|
||||
[cache]
|
||||
enabled = true
|
||||
ttl_seconds = 86400 # 24 hours for historical data
|
||||
ttl_current_data_seconds = 3600 # 1 hour for recent data
|
||||
max_size_mb = 1000 # 1GB cache
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Environment | Command | Ports |
|
||||
|-------------|---------|-------|
|
||||
| Development | `reflex run` | 3000, 8000 |
|
||||
| Production | `reflex run --env prod` | 3000, 8000 |
|
||||
| Backend only | `reflex run --backend-only` | 8000 |
|
||||
| Frontend only | `reflex run --frontend-only` | 3000 |
|
||||
| Export | `reflex export` | Static files |
|
||||
| Cloud | `reflex deploy` | Managed |
|
||||
|
||||
For more information, see:
|
||||
- [Reflex Documentation](https://reflex.dev/docs/)
|
||||
- [Reflex Cloud](https://reflex.dev/cloud/)
|
||||
- [FastAPI Deployment](https://fastapi.tiangolo.com/deployment/)
|
||||
@@ -0,0 +1,403 @@
|
||||
# User Guide - NHS Patient Pathway Analysis Tool
|
||||
|
||||
This guide explains how to use the NHS High-Cost Drug Patient Pathway Analysis Tool to analyze treatment pathways for secondary care patients.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Getting Started](#getting-started)
|
||||
2. [Interface Overview](#interface-overview)
|
||||
3. [Selecting Your Data Source](#selecting-your-data-source)
|
||||
4. [Configuring Analysis Filters](#configuring-analysis-filters)
|
||||
5. [Selecting Drugs, Trusts, and Directories](#selecting-drugs-trusts-and-directories)
|
||||
6. [Running the Analysis](#running-the-analysis)
|
||||
7. [Understanding the Pathway Chart](#understanding-the-pathway-chart)
|
||||
8. [Exporting Results](#exporting-results)
|
||||
9. [GP Indication Validation](#gp-indication-validation)
|
||||
10. [Keyboard Navigation and Accessibility](#keyboard-navigation-and-accessibility)
|
||||
11. [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Accessing the Application
|
||||
|
||||
Start the application by running:
|
||||
|
||||
```bash
|
||||
reflex run
|
||||
```
|
||||
|
||||
Then open your browser to **http://localhost:3000**
|
||||
|
||||
The application will automatically load reference data (drugs, trusts, directories) when you first access it.
|
||||
|
||||
### First-Time Setup
|
||||
|
||||
1. Click **Load Reference Data** on the Home page to populate the filter options
|
||||
2. Select your preferred data source (SQLite, File Upload, or Snowflake)
|
||||
3. Configure your date range and other filters
|
||||
4. Click **Run Analysis** to generate your first pathway chart
|
||||
|
||||
---
|
||||
|
||||
## Interface Overview
|
||||
|
||||
The application has four main pages, accessible from the sidebar navigation:
|
||||
|
||||
| Page | Purpose |
|
||||
|------|---------|
|
||||
| **Home** | Main analysis dashboard with data source selection, filters, and chart display |
|
||||
| **Drug Selection** | Select which high-cost drugs to include in the analysis |
|
||||
| **Trust Selection** | Filter by specific NHS trusts |
|
||||
| **Directory Selection** | Filter by medical directories/specialties |
|
||||
|
||||
### Navigation
|
||||
|
||||
- **Desktop**: Use the sidebar on the left to switch between pages
|
||||
- **Mobile**: Use the top navigation bar
|
||||
- **Keyboard**: Press Tab to navigate, Enter to select
|
||||
|
||||
---
|
||||
|
||||
## Selecting Your Data Source
|
||||
|
||||
The application supports three data sources:
|
||||
|
||||
### 1. SQLite Database (Recommended)
|
||||
|
||||
Pre-loaded patient data stored locally for fast performance.
|
||||
|
||||
**Advantages:**
|
||||
- Fastest analysis performance
|
||||
- Works offline
|
||||
- No authentication required
|
||||
|
||||
**To use:** Click "Use SQLite" in the Data Source section
|
||||
|
||||
### 2. File Upload
|
||||
|
||||
Upload CSV or Parquet files directly.
|
||||
|
||||
**Supported formats:**
|
||||
- CSV files (.csv)
|
||||
- Apache Parquet files (.parquet, .pq)
|
||||
|
||||
**To use:**
|
||||
1. Drag and drop a file, or click the upload area
|
||||
2. Wait for the file to process
|
||||
3. Click "Use File" to select it as your data source
|
||||
|
||||
### 3. Snowflake
|
||||
|
||||
Query live data from the NHS data warehouse.
|
||||
|
||||
**Requirements:**
|
||||
- Snowflake must be configured (see `config/snowflake.toml`)
|
||||
- Browser-based NHS SSO authentication
|
||||
|
||||
**To use:** Click "Use Snowflake" - you'll be prompted to authenticate via your browser
|
||||
|
||||
---
|
||||
|
||||
## Configuring Analysis Filters
|
||||
|
||||
The Home page provides several filter options:
|
||||
|
||||
### Date Range
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| **Start Date** | Include patients initiated from this date onwards |
|
||||
| **End Date** | Include patients initiated until this date |
|
||||
| **Last Seen After** | Only include patients with activity after this date (excludes patients who haven't been seen recently) |
|
||||
|
||||
**Tip:** The default range is the last 12 months.
|
||||
|
||||
### Minimum Patients
|
||||
|
||||
Filter out pathways with fewer patients than the threshold you set.
|
||||
|
||||
- Use the slider for quick adjustment (0-100)
|
||||
- Or type a specific number in the text field
|
||||
- Set to 0 to show all pathways regardless of patient count
|
||||
|
||||
### Custom Title
|
||||
|
||||
Override the automatically generated chart title with your own text.
|
||||
|
||||
- Leave empty to use the default title: "Patients initiated [start date] to [end date]"
|
||||
- Useful for specific reports or presentations
|
||||
|
||||
---
|
||||
|
||||
## Selecting Drugs, Trusts, and Directories
|
||||
|
||||
Each selection page works the same way:
|
||||
|
||||
### Navigation
|
||||
|
||||
1. Click "Drug Selection", "Trust Selection", or "Directory Selection" in the sidebar
|
||||
2. The page shows all available options with checkboxes
|
||||
|
||||
### Search
|
||||
|
||||
Type in the search box to filter the list. The list updates as you type.
|
||||
|
||||
### Selection Actions
|
||||
|
||||
| Button | Action |
|
||||
|--------|--------|
|
||||
| **Select All** | Check all visible items |
|
||||
| **Clear All** | Uncheck all items |
|
||||
| **Select Defaults** | (Drugs only) Select pre-configured default drugs (Include=1 in include.csv) |
|
||||
|
||||
### Selection Behavior
|
||||
|
||||
- **No items selected** = Include ALL items in analysis
|
||||
- **Some items selected** = Include ONLY the selected items
|
||||
|
||||
This means leaving a filter empty is equivalent to "select all".
|
||||
|
||||
---
|
||||
|
||||
## Running the Analysis
|
||||
|
||||
### Steps
|
||||
|
||||
1. Ensure your data source is selected and configured
|
||||
2. Set your date range and other filters
|
||||
3. Select desired drugs, trusts, and directories (or leave empty for all)
|
||||
4. Click the green **Run Analysis** button
|
||||
|
||||
### During Analysis
|
||||
|
||||
- The button shows a spinner while analysis is running
|
||||
- Status messages appear below the button
|
||||
- The interface remains responsive - you can review settings
|
||||
|
||||
### After Analysis
|
||||
|
||||
- The pathway chart appears in the chart section
|
||||
- Export buttons become available
|
||||
- GP indication validation results appear (if Snowflake is connected)
|
||||
|
||||
---
|
||||
|
||||
## Understanding the Pathway Chart
|
||||
|
||||
The analysis generates an interactive **icicle chart** showing patient treatment pathways.
|
||||
|
||||
### Hierarchy Structure
|
||||
|
||||
The chart displays a hierarchical structure:
|
||||
|
||||
```
|
||||
N&WICS (Regional Total)
|
||||
└─ Trust Name (e.g., "Norfolk and Norwich University Hospitals")
|
||||
└─ Directory (e.g., "Rheumatology", "Gastroenterology")
|
||||
└─ Drug Name (e.g., "ADALIMUMAB", "INFLIXIMAB")
|
||||
```
|
||||
|
||||
### Reading the Chart
|
||||
|
||||
- **Width** of each section indicates relative patient count
|
||||
- **Color intensity** indicates proportion of patients at that level
|
||||
- **Labels** show the category name and patient count
|
||||
|
||||
### Interacting with the Chart
|
||||
|
||||
| Action | Effect |
|
||||
|--------|--------|
|
||||
| **Click** a section | Zoom in to show details for that branch |
|
||||
| **Click** the root | Zoom out to show full hierarchy |
|
||||
| **Hover** over a section | See tooltip with patient count |
|
||||
| Use the **toolbar** | Reset, download image, pan, zoom |
|
||||
|
||||
### Plotly Toolbar
|
||||
|
||||
The chart includes a Plotly toolbar (top right) with:
|
||||
|
||||
- **Download as PNG** - Save static image
|
||||
- **Zoom controls** - Zoom in/out
|
||||
- **Pan** - Click and drag to move
|
||||
- **Reset** - Return to original view
|
||||
|
||||
---
|
||||
|
||||
## Exporting Results
|
||||
|
||||
Two export options are available after running an analysis:
|
||||
|
||||
### Export HTML
|
||||
|
||||
Creates an interactive HTML file that can be opened in any browser.
|
||||
|
||||
- **Output**: `data/exports/pathway_chart_[timestamp].html`
|
||||
- **Use case**: Sharing interactive charts via email or file share
|
||||
- **Features**: Full interactivity, no software required to view
|
||||
|
||||
### Export CSV
|
||||
|
||||
Exports the underlying data as a spreadsheet.
|
||||
|
||||
- **Output**: `data/exports/pathway_data_[timestamp].csv`
|
||||
- **Use case**: Further analysis in Excel, importing to other tools
|
||||
- **Includes**: Patient IDs, drugs, dates, costs, directories, indication validation status
|
||||
|
||||
### Export Location
|
||||
|
||||
All exports are saved to the `data/exports/` directory with timestamped filenames to prevent overwriting.
|
||||
|
||||
---
|
||||
|
||||
## GP Indication Validation
|
||||
|
||||
When connected to Snowflake, the application validates whether patients have appropriate GP diagnoses for their prescribed drugs.
|
||||
|
||||
### What It Does
|
||||
|
||||
1. Looks up the drug's licensed indications (e.g., ADALIMUMAB for rheumatoid arthritis)
|
||||
2. Finds corresponding SNOMED codes for those indications
|
||||
3. Checks each patient's GP records for matching diagnoses
|
||||
4. Reports the match rate per drug
|
||||
|
||||
### Understanding Results
|
||||
|
||||
After analysis, a table shows:
|
||||
|
||||
| Column | Meaning |
|
||||
|--------|---------|
|
||||
| **Drug Name** | The high-cost drug |
|
||||
| **Total Patients** | Number of patients prescribed this drug |
|
||||
| **With GP Indication** | Patients with matching GP diagnosis |
|
||||
| **Match Rate** | Percentage with valid indication |
|
||||
|
||||
### Match Rate Interpretation
|
||||
|
||||
| Rate | Meaning | Color |
|
||||
|------|---------|-------|
|
||||
| **80%+** | Good coverage - most patients have GP diagnoses | Green |
|
||||
| **50-79%** | Moderate coverage - investigate missing cases | Orange |
|
||||
| **<50%** | Low coverage - may indicate data quality issues or off-label use | Red |
|
||||
|
||||
### Why Rates May Be Low
|
||||
|
||||
Low match rates don't necessarily indicate problems:
|
||||
|
||||
- **Cross-provider treatment**: Patient's GP is outside the data coverage
|
||||
- **Recent diagnoses**: Diagnosis not yet recorded in GP system
|
||||
- **Specialist-only conditions**: Some conditions are only managed in secondary care
|
||||
- **Off-label prescribing**: Legitimate use for indications not in the mapping
|
||||
|
||||
### Enabling/Disabling
|
||||
|
||||
Indication validation is enabled by default when Snowflake is connected. It requires:
|
||||
- Active Snowflake connection
|
||||
- Drug-to-cluster mappings in the database
|
||||
|
||||
---
|
||||
|
||||
## Keyboard Navigation and Accessibility
|
||||
|
||||
The application is designed to be accessible:
|
||||
|
||||
### Skip Link
|
||||
|
||||
Press **Tab** when the page loads to reveal a "Skip to main content" link that bypasses navigation.
|
||||
|
||||
### Keyboard Navigation
|
||||
|
||||
| Key | Action |
|
||||
|-----|--------|
|
||||
| **Tab** | Move to next interactive element |
|
||||
| **Shift+Tab** | Move to previous element |
|
||||
| **Enter** | Activate buttons, links, checkboxes |
|
||||
| **Space** | Toggle checkboxes |
|
||||
| **Arrow keys** | Adjust sliders |
|
||||
|
||||
### Screen Reader Support
|
||||
|
||||
- All buttons and inputs have descriptive labels
|
||||
- Status messages announce via ARIA live regions
|
||||
- Charts include figure descriptions
|
||||
|
||||
### Theme Toggle
|
||||
|
||||
A dark/light mode toggle is available at the bottom of the sidebar for visual preference.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No data available" Error
|
||||
|
||||
**Cause**: No data matches your current filter settings
|
||||
|
||||
**Solutions:**
|
||||
1. Check your date range - is it too narrow?
|
||||
2. Verify your data source has data loaded
|
||||
3. Check if selected trusts/drugs have any matching records
|
||||
4. Try clearing all selections (to include everything)
|
||||
|
||||
### Chart Not Displaying
|
||||
|
||||
**Cause**: Analysis completed but no data met the minimum patients threshold
|
||||
|
||||
**Solutions:**
|
||||
1. Lower the minimum patients threshold
|
||||
2. Expand your date range
|
||||
3. Select more drugs or trusts
|
||||
|
||||
### Snowflake Connection Failed
|
||||
|
||||
**Cause**: Unable to connect to Snowflake
|
||||
|
||||
**Solutions:**
|
||||
1. Check that `config/snowflake.toml` exists and is configured
|
||||
2. Complete browser authentication when prompted
|
||||
3. Verify your network allows Snowflake connections
|
||||
4. Try using SQLite as an alternative data source
|
||||
|
||||
### File Upload Failed
|
||||
|
||||
**Cause**: File format or content issue
|
||||
|
||||
**Solutions:**
|
||||
1. Ensure file is CSV or Parquet format
|
||||
2. Check file isn't corrupted or empty
|
||||
3. Verify file contains required columns
|
||||
4. Try a smaller file to test
|
||||
|
||||
### Slow Performance
|
||||
|
||||
**Cause**: Large data volume or complex filtering
|
||||
|
||||
**Solutions:**
|
||||
1. Use SQLite instead of file upload for large datasets
|
||||
2. Narrow your date range
|
||||
3. Select fewer drugs/trusts to analyze
|
||||
4. Increase minimum patients threshold to reduce chart complexity
|
||||
|
||||
### Reference Data Not Loading
|
||||
|
||||
**Cause**: Missing or corrupted reference files
|
||||
|
||||
**Solutions:**
|
||||
1. Click "Load Reference Data" to retry
|
||||
2. Check that `data/` directory contains required CSV files:
|
||||
- `include.csv`
|
||||
- `defaultTrusts.csv`
|
||||
- `directory_list.csv`
|
||||
3. Verify files aren't empty or malformed
|
||||
|
||||
---
|
||||
|
||||
## Getting Help
|
||||
|
||||
If you encounter issues not covered in this guide:
|
||||
|
||||
1. Check the [README](../README.md) for installation and setup information
|
||||
2. Review [DEPLOYMENT.md](./DEPLOYMENT.md) for server configuration
|
||||
3. Consult [CLAUDE.md](../CLAUDE.md) for technical architecture details
|
||||
4. Contact your local support team for NHS-specific questions
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
# Guardrails
|
||||
|
||||
Known failure patterns. Read EVERY iteration. Follow ALL of these rules.
|
||||
If you discover a new failure pattern during your work, add it to this file.
|
||||
|
||||
---
|
||||
|
||||
## Reflex Guardrails
|
||||
|
||||
### Use .to() methods for Var operations in rx.foreach
|
||||
- **When**: Working with items inside `rx.foreach` render functions
|
||||
- **Rule**: Use `item.to(int)` for numeric comparisons, `item.to_string()` for text operations
|
||||
- **Why**: Items from rx.foreach are `ObjectItemOperation` Vars, not plain Python values. Using `>=` or f-strings directly causes TypeError.
|
||||
|
||||
**Bad:**
|
||||
```python
|
||||
def render_row(item):
|
||||
color = rx.cond(item["value"] >= 50, "green", "red") # TypeError!
|
||||
return rx.text(f"{item['name']}: {item['value']}") # Won't interpolate!
|
||||
```
|
||||
|
||||
**Good:**
|
||||
```python
|
||||
def render_row(item):
|
||||
color = rx.cond(item["value"].to(int) >= 50, "green", "red")
|
||||
return rx.text(item["name"].to_string() + ": " + item["value"].to_string())
|
||||
```
|
||||
|
||||
### Use rx.cond for conditional rendering, not Python if
|
||||
- **When**: Conditionally showing/hiding components or changing styles based on state
|
||||
- **Rule**: Use `rx.cond(condition, true_component, false_component)` — not Python `if`
|
||||
- **Why**: Python `if` evaluates at definition time; `rx.cond` evaluates reactively at render time
|
||||
|
||||
### State variables must have default values
|
||||
- **When**: Defining state variables in the State class
|
||||
- **Rule**: Always provide a default: `my_var: str = ""` not just `my_var: str`
|
||||
- **Why**: Reflex requires defaults for state initialization
|
||||
|
||||
### Computed vars use @rx.var decorator
|
||||
- **When**: Creating derived/computed values from state
|
||||
- **Rule**: Use `@rx.var` decorator, return a value, and include return type annotation
|
||||
- **Why**: Without the decorator, the method won't be reactive
|
||||
|
||||
```python
|
||||
@rx.var
|
||||
def filtered_count(self) -> int:
|
||||
return len(self.filtered_data)
|
||||
```
|
||||
|
||||
### Event handlers don't return values to components
|
||||
- **When**: Creating methods that handle user interactions
|
||||
- **Rule**: Event handlers modify state; they don't return values directly to UI
|
||||
- **Why**: Use state variables and computed vars to communicate between handlers and UI
|
||||
|
||||
---
|
||||
|
||||
## Design System Guardrails
|
||||
|
||||
### Never hardcode colors
|
||||
- **When**: Any styling that involves color
|
||||
- **Rule**: Import from `pathways_app.styles` and use `Colors.PRIMARY`, `Colors.SLATE_700`, etc.
|
||||
- **Why**: Hardcoded colors break consistency and make theming impossible
|
||||
|
||||
### Never hardcode spacing
|
||||
- **When**: Any padding, margin, gap values
|
||||
- **Rule**: Use `Spacing.SM`, `Spacing.LG`, etc. from the styles module
|
||||
- **Why**: Consistent spacing is fundamental to visual cohesion
|
||||
|
||||
### Use design system typography
|
||||
- **When**: Any text styling
|
||||
- **Rule**: Use the typography classes/helpers from styles.py
|
||||
- **Why**: Typography hierarchy creates visual structure
|
||||
|
||||
---
|
||||
|
||||
## Code Quality Guardrails
|
||||
|
||||
### Verify compilation before committing
|
||||
- **When**: After ANY code changes
|
||||
- **Rule**: Run `python -m py_compile <file>` AND `reflex run` (briefly) to check
|
||||
- **Why**: Committing broken code wastes the next iteration fixing preventable errors
|
||||
|
||||
### One component per function
|
||||
- **When**: Creating UI components
|
||||
- **Rule**: Each logical component should be its own function returning `rx.Component`
|
||||
- **Why**: Smaller functions are easier to debug and reuse
|
||||
|
||||
### Keep state minimal
|
||||
- **When**: Designing state structure
|
||||
- **Rule**: Only store what's necessary; derive everything else with computed vars
|
||||
- **Why**: Duplicate state leads to sync bugs
|
||||
|
||||
---
|
||||
|
||||
## Process Guardrails
|
||||
|
||||
### One task per iteration
|
||||
- **When**: Temptation to do additional tasks after completing the current one
|
||||
- **Rule**: Complete ONE task, validate it, commit it, update progress, then stop
|
||||
- **Why**: Multiple tasks increase error risk and make failures harder to diagnose
|
||||
|
||||
### Never mark complete without validation
|
||||
- **When**: Task feels "done" but hasn't been tested
|
||||
- **Rule**: All validation tiers must pass before marking `[x]`
|
||||
- **Why**: "Feels done" is not "is done"
|
||||
|
||||
### Write explicit handoff notes
|
||||
- **When**: Every iteration, before stopping
|
||||
- **Rule**: The "Next iteration should" section must contain specific, actionable guidance
|
||||
- **Why**: The next iteration has zero memory. If you don't write it down, it's lost.
|
||||
|
||||
### Check existing code for patterns
|
||||
- **When**: Unsure how to implement something in Reflex
|
||||
- **Rule**: Look at `pathways_app.py` for working examples before inventing new patterns
|
||||
- **Why**: The existing codebase has solved many Reflex quirks already
|
||||
|
||||
---
|
||||
|
||||
<!--
|
||||
ADD NEW GUARDRAILS BELOW as failures are observed during the loop.
|
||||
|
||||
Format:
|
||||
### [Short descriptive name]
|
||||
- **When**: What situation triggers this guardrail?
|
||||
- **Rule**: What must you do (or not do)?
|
||||
- **Why**: What failure prompted adding this guardrail?
|
||||
-->
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 885 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.7 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 9.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 5.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 5.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.5 KiB |
@@ -0,0 +1,17 @@
|
||||
"""
|
||||
UI components for the Patient Pathway Analysis Reflex application.
|
||||
|
||||
This module exports reusable layout and navigation components.
|
||||
"""
|
||||
|
||||
from .layout import sidebar, navbar, content_area, main_layout
|
||||
from .navigation import nav_item, nav_section
|
||||
|
||||
__all__ = [
|
||||
"sidebar",
|
||||
"navbar",
|
||||
"content_area",
|
||||
"main_layout",
|
||||
"nav_item",
|
||||
"nav_section",
|
||||
]
|
||||
@@ -0,0 +1,262 @@
|
||||
"""
|
||||
Layout components for the Patient Pathway Analysis tool.
|
||||
|
||||
Provides the main application layout with sidebar navigation and content area.
|
||||
Includes accessibility features: skip links, ARIA landmarks, keyboard navigation.
|
||||
"""
|
||||
|
||||
import reflex as rx
|
||||
from .navigation import nav_item
|
||||
|
||||
|
||||
# NHS Color scheme
|
||||
NHS_BLUE = "rgb(0, 94, 184)"
|
||||
NHS_DARK_BLUE = "rgb(0, 48, 135)"
|
||||
NHS_LIGHT_BLUE = "rgb(65, 182, 230)"
|
||||
NHS_WHITE = "white"
|
||||
NHS_GREY = "rgb(231, 231, 231)"
|
||||
|
||||
|
||||
def skip_link() -> rx.Component:
|
||||
"""
|
||||
Skip link for keyboard users to bypass navigation.
|
||||
|
||||
Visually hidden until focused, allowing keyboard users to skip
|
||||
directly to main content.
|
||||
"""
|
||||
return rx.link(
|
||||
"Skip to main content",
|
||||
href="#main-content",
|
||||
position="absolute",
|
||||
top="-40px",
|
||||
left="0",
|
||||
background=NHS_BLUE,
|
||||
color="white",
|
||||
padding="8px 16px",
|
||||
z_index="1000",
|
||||
text_decoration="none",
|
||||
font_weight="bold",
|
||||
_focus={
|
||||
"top": "0",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def logo_section() -> rx.Component:
|
||||
"""NHS branding logo section at top of sidebar."""
|
||||
return rx.hstack(
|
||||
rx.image(
|
||||
src="/logo.png",
|
||||
height="32px",
|
||||
alt="NHS Norfolk and Waveney Logo",
|
||||
),
|
||||
rx.text(
|
||||
"HCD Analysis",
|
||||
size="5",
|
||||
weight="bold",
|
||||
color=NHS_BLUE,
|
||||
),
|
||||
padding="16px",
|
||||
spacing="3",
|
||||
align="center",
|
||||
width="100%",
|
||||
border_bottom=f"1px solid {NHS_GREY}",
|
||||
)
|
||||
|
||||
|
||||
def sidebar(current_page: str = "home") -> rx.Component:
|
||||
"""
|
||||
Create the sidebar navigation panel.
|
||||
|
||||
Args:
|
||||
current_page: The current active page name for highlighting
|
||||
|
||||
Returns:
|
||||
A sidebar component with navigation items and ARIA landmark
|
||||
"""
|
||||
return rx.el.nav(
|
||||
rx.vstack(
|
||||
# Logo section
|
||||
logo_section(),
|
||||
# Navigation items
|
||||
rx.vstack(
|
||||
nav_item(
|
||||
"Home",
|
||||
"/",
|
||||
"home",
|
||||
is_active=(current_page == "home"),
|
||||
),
|
||||
nav_item(
|
||||
"Drug Selection",
|
||||
"/drugs",
|
||||
"pill",
|
||||
is_active=(current_page == "drugs"),
|
||||
),
|
||||
nav_item(
|
||||
"Trust Selection",
|
||||
"/trusts",
|
||||
"building",
|
||||
is_active=(current_page == "trusts"),
|
||||
),
|
||||
nav_item(
|
||||
"Directory Selection",
|
||||
"/directories",
|
||||
"folder",
|
||||
is_active=(current_page == "directories"),
|
||||
),
|
||||
padding="8px",
|
||||
spacing="1",
|
||||
width="100%",
|
||||
align="start",
|
||||
),
|
||||
# Spacer to push theme toggle to bottom
|
||||
rx.spacer(),
|
||||
# Theme toggle at bottom
|
||||
rx.box(
|
||||
rx.hstack(
|
||||
rx.el.label(
|
||||
"Theme:",
|
||||
html_for="theme-toggle",
|
||||
font_size="14px",
|
||||
color="gray",
|
||||
),
|
||||
rx.color_mode.switch(id="theme-toggle"),
|
||||
spacing="2",
|
||||
align="center",
|
||||
),
|
||||
padding="16px",
|
||||
border_top=f"1px solid {NHS_GREY}",
|
||||
width="100%",
|
||||
),
|
||||
height="100vh",
|
||||
width="100%",
|
||||
spacing="0",
|
||||
align="start",
|
||||
),
|
||||
aria_label="Main navigation",
|
||||
width="240px",
|
||||
min_width="240px",
|
||||
background="white",
|
||||
border_right=f"1px solid {NHS_GREY}",
|
||||
position="fixed",
|
||||
left="0",
|
||||
top="0",
|
||||
height="100vh",
|
||||
overflow_y="auto",
|
||||
z_index="100",
|
||||
)
|
||||
|
||||
|
||||
def navbar() -> rx.Component:
|
||||
"""
|
||||
Create a top navigation bar for mobile/smaller screens.
|
||||
|
||||
Returns:
|
||||
A horizontal navbar component (collapsed sidebar for mobile) with ARIA support
|
||||
"""
|
||||
return rx.el.header(
|
||||
rx.hstack(
|
||||
rx.image(src="/logo.png", height="28px", alt="NHS Norfolk and Waveney Logo"),
|
||||
rx.text("HCD Analysis", size="4", weight="bold"),
|
||||
rx.spacer(),
|
||||
rx.el.label(
|
||||
rx.color_mode.switch(id="theme-toggle-mobile"),
|
||||
html_for="theme-toggle-mobile",
|
||||
aria_label="Toggle dark mode",
|
||||
),
|
||||
width="100%",
|
||||
padding="12px 16px",
|
||||
align="center",
|
||||
justify="between",
|
||||
),
|
||||
background="white",
|
||||
border_bottom=f"1px solid {NHS_GREY}",
|
||||
display=["flex", "flex", "none"], # Show on mobile, hide on desktop
|
||||
width="100%",
|
||||
position="fixed",
|
||||
top="0",
|
||||
left="0",
|
||||
z_index="100",
|
||||
role="banner",
|
||||
)
|
||||
|
||||
|
||||
def content_area(*children, page_title: str = "") -> rx.Component:
|
||||
"""
|
||||
Create the main content area.
|
||||
|
||||
Args:
|
||||
*children: Child components to render in the content area
|
||||
page_title: Optional title to display at top of content
|
||||
|
||||
Returns:
|
||||
A styled content area component with ARIA main landmark
|
||||
"""
|
||||
content_children = list(children)
|
||||
|
||||
if page_title:
|
||||
content_children.insert(
|
||||
0,
|
||||
rx.heading(
|
||||
page_title,
|
||||
size="6",
|
||||
weight="bold",
|
||||
color=NHS_DARK_BLUE,
|
||||
margin_bottom="16px",
|
||||
),
|
||||
)
|
||||
|
||||
return rx.el.main(
|
||||
rx.vstack(
|
||||
*content_children,
|
||||
width="100%",
|
||||
max_width="1200px",
|
||||
padding="24px",
|
||||
spacing="4",
|
||||
align="start",
|
||||
),
|
||||
id="main-content",
|
||||
tabindex="-1", # Allow focus for skip link
|
||||
# Offset for sidebar on desktop
|
||||
margin_left=["0", "0", "240px"],
|
||||
# Offset for navbar on mobile
|
||||
margin_top=["60px", "60px", "0"],
|
||||
min_height="100vh",
|
||||
background=rx.color_mode_cond(
|
||||
light="rgb(249, 250, 251)", # Light gray background
|
||||
dark="rgb(17, 24, 39)", # Dark background
|
||||
),
|
||||
width="100%",
|
||||
_focus={
|
||||
"outline": "none", # Hide focus ring on main (only accessible via skip link)
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def main_layout(
|
||||
content: rx.Component,
|
||||
current_page: str = "home",
|
||||
) -> rx.Component:
|
||||
"""
|
||||
Create the complete page layout with sidebar and content.
|
||||
|
||||
Args:
|
||||
content: The main content to display
|
||||
current_page: The current page name for navigation highlighting
|
||||
|
||||
Returns:
|
||||
A complete page layout component with accessibility features
|
||||
"""
|
||||
return rx.fragment(
|
||||
# Skip link for keyboard users
|
||||
skip_link(),
|
||||
# Sidebar (visible on desktop)
|
||||
rx.box(
|
||||
sidebar(current_page=current_page),
|
||||
display=["none", "none", "block"], # Hide on mobile
|
||||
),
|
||||
# Navbar (visible on mobile)
|
||||
navbar(),
|
||||
# Main content
|
||||
content,
|
||||
)
|
||||
@@ -0,0 +1,86 @@
|
||||
"""
|
||||
Navigation components for the Patient Pathway Analysis tool.
|
||||
|
||||
Provides sidebar navigation items with icons, matching the CustomTkinter design.
|
||||
Includes accessibility features: ARIA labels, keyboard navigation, focus indicators.
|
||||
"""
|
||||
|
||||
import reflex as rx
|
||||
from typing import Callable
|
||||
|
||||
|
||||
def nav_item(
|
||||
text: str,
|
||||
href: str,
|
||||
icon: str,
|
||||
is_active: bool = False,
|
||||
) -> rx.Component:
|
||||
"""
|
||||
Create a navigation item with icon.
|
||||
|
||||
Args:
|
||||
text: The display text for the nav item
|
||||
href: The route to navigate to
|
||||
icon: The Lucide icon name (e.g., "home", "pill", "building", "folder")
|
||||
is_active: Whether this item is currently active
|
||||
|
||||
Returns:
|
||||
A styled navigation button component with accessibility support
|
||||
"""
|
||||
# NHS colors - use blue for active state
|
||||
active_bg = "rgb(0, 94, 184)" # NHS Blue
|
||||
hover_bg = "rgb(0, 48, 135)" # NHS Dark Blue
|
||||
|
||||
return rx.link(
|
||||
rx.hstack(
|
||||
rx.icon(icon, size=20, aria_hidden="true"), # Hide decorative icon from screen readers
|
||||
rx.text(text, size="3", weight="medium"),
|
||||
width="100%",
|
||||
padding="12px 16px",
|
||||
spacing="3",
|
||||
align="center",
|
||||
border_radius="8px",
|
||||
bg=rx.cond(is_active, active_bg, "transparent"),
|
||||
color=rx.cond(is_active, "white", "inherit"),
|
||||
_hover={
|
||||
"background": rx.cond(is_active, active_bg, "rgba(0, 94, 184, 0.1)"),
|
||||
},
|
||||
_focus_visible={
|
||||
"outline": "2px solid rgb(0, 94, 184)",
|
||||
"outline_offset": "2px",
|
||||
},
|
||||
transition="background 0.2s ease",
|
||||
),
|
||||
href=href,
|
||||
text_decoration="none",
|
||||
width="100%",
|
||||
aria_current=rx.cond(is_active, "page", ""),
|
||||
)
|
||||
|
||||
|
||||
def nav_section(title: str, children: list[rx.Component]) -> rx.Component:
|
||||
"""
|
||||
Create a labeled section of navigation items.
|
||||
|
||||
Args:
|
||||
title: Section header text
|
||||
children: List of nav_item components
|
||||
|
||||
Returns:
|
||||
A styled section with header and items
|
||||
"""
|
||||
return rx.vstack(
|
||||
rx.text(
|
||||
title,
|
||||
size="1",
|
||||
weight="bold",
|
||||
color="gray",
|
||||
padding_x="16px",
|
||||
padding_top="16px",
|
||||
padding_bottom="8px",
|
||||
),
|
||||
*children,
|
||||
width="100%",
|
||||
spacing="1",
|
||||
align="start",
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
@@ -0,0 +1,64 @@
|
||||
# Progress Log
|
||||
|
||||
## Design Context
|
||||
|
||||
### Project Vision
|
||||
Complete UI redesign of HCD Analysis tool. Modern, bold design with NHS color scheme inspiration (not constrained by it). Single-page dashboard replacing multi-page sidebar layout. Light mode only.
|
||||
|
||||
### Key Design Decisions
|
||||
1. **No sidebar** — all filters in a prominent filter bar
|
||||
2. **No user auth UI** — local app, no login needed
|
||||
3. **Chart navigation via tabs** — top bar has chart type selection (Icicle now, more later)
|
||||
4. **Instant filtering** — debounced (300ms), not "Apply" button
|
||||
5. **Two date ranges**:
|
||||
- "Initiated" filter (default: OFF, include all patients)
|
||||
- "Last Seen" filter (default: ON, last 6 months)
|
||||
- "To" date always = latest date in dataset
|
||||
6. **Searchable dropdowns** — Drugs, Indications, Directorates with search + counts
|
||||
7. **Data source hidden** — SQLite only, refresh via CLI, show freshness indicator
|
||||
8. **KPIs reactive** — update when filters change
|
||||
|
||||
### Color Palette (from DESIGN_SYSTEM.md)
|
||||
- Heritage Blue: #003087 (deep, authoritative)
|
||||
- Primary Blue: #0066CC (main actions)
|
||||
- Vibrant Blue: #1E88E5 (highlights, hovers)
|
||||
- Sky Blue: #4FC3F7 (accents)
|
||||
- Pale Blue: #E3F2FD (backgrounds)
|
||||
- Neutrals: Slate family (#1E293B → #F1F5F9)
|
||||
|
||||
### Typography
|
||||
- Font: Inter (Google Fonts or system)
|
||||
- Display: 32px/700, Heading1: 24px/600, Body: 14px/400, Caption: 12px/500
|
||||
|
||||
## Reflex Patterns
|
||||
|
||||
### Var operations in rx.foreach
|
||||
When using `rx.foreach`, items are Reflex Vars. Use:
|
||||
- `.to(int)` for numeric comparisons
|
||||
- `.to_string()` for text operations
|
||||
- Never use f-strings or Python operators directly
|
||||
|
||||
### Conditional rendering
|
||||
Use `rx.cond(condition, true_value, false_value)` not Python `if`.
|
||||
|
||||
### State structure
|
||||
- Event handlers modify state
|
||||
- `@rx.var` decorated methods for computed/derived values
|
||||
- All state vars need defaults
|
||||
|
||||
## Existing Codebase Reference
|
||||
|
||||
### Key files to reference
|
||||
- `pathways_app/pathways_app.py` — existing Reflex app (2100+ lines)
|
||||
- `analysis/pathway_analyzer.py` — chart data preparation logic
|
||||
- `data_processing/loader.py` — SQLite data loading
|
||||
- `core/models.py` — AnalysisFilters dataclass
|
||||
|
||||
### Patterns that work in existing code
|
||||
- `State` class with filter variables
|
||||
- `rx.plotly()` for chart rendering
|
||||
- Multi-select with `rx.checkbox` groups
|
||||
- Theme configuration via `rx.theme()`
|
||||
|
||||
## Iteration Log
|
||||
<!-- Each iteration appends a structured entry below. See RALPH_PROMPT.md for format. -->
|
||||
@@ -0,0 +1,69 @@
|
||||
[tool.setuptools]
|
||||
py-modules = []
|
||||
packages = []
|
||||
[project]
|
||||
name = "patient-pathway-analysis"
|
||||
version = "0.1.0"
|
||||
description = "Add your description here"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"darkdetect==0.8.0",
|
||||
"decorator==5.1.1",
|
||||
"et-xmlfile==1.1.0",
|
||||
"executing==1.2.0",
|
||||
"fastparquet>=2024.11.0",
|
||||
"idna==3.4",
|
||||
"itsdangerous==2.1.2",
|
||||
"jedi==0.18.2",
|
||||
"jinja2==3.1.2",
|
||||
"jupyter-core==5.3.1",
|
||||
"numpy==1.25.0",
|
||||
"packaging==23.1",
|
||||
"pandas==2.0.3",
|
||||
"pillow==10.0.0",
|
||||
"plotly==5.15.0",
|
||||
"pyarrow>=20.0.0",
|
||||
"python-dateutil==2.8.2",
|
||||
"reflex>=0.6.0",
|
||||
"tenacity==8.2.2",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
test = [
|
||||
"pytest>=8.0.0",
|
||||
"pytest-cov>=4.0.0",
|
||||
]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
addopts = [
|
||||
"-v",
|
||||
"--tb=short",
|
||||
"--strict-markers",
|
||||
]
|
||||
markers = [
|
||||
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
|
||||
"integration: marks tests as integration tests (require external resources)",
|
||||
"largedata: marks tests that require large datasets (deselect with '-m \"not largedata\"')",
|
||||
]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["core", "data_processing", "analysis", "visualization", "tools"]
|
||||
branch = true
|
||||
omit = [
|
||||
"*/tests/*",
|
||||
"*/__pycache__/*",
|
||||
]
|
||||
|
||||
[tool.coverage.report]
|
||||
exclude_lines = [
|
||||
"pragma: no cover",
|
||||
"def __repr__",
|
||||
"raise NotImplementedError",
|
||||
"if TYPE_CHECKING:",
|
||||
]
|
||||
show_missing = true
|
||||
@@ -0,0 +1,346 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Ralph Wiggum Loop - Reflex UI Redesign variant.
|
||||
|
||||
.DESCRIPTION
|
||||
Outer loop for iterative Reflex frontend development.
|
||||
Each iteration spawns a fresh `claude --print` invocation.
|
||||
Memory persists via filesystem only: git commits, progress.txt, IMPLEMENTATION_PLAN.md, guardrails.md.
|
||||
Completion detected via <promise>COMPLETE</promise> in output.
|
||||
|
||||
Circuit breakers prevent runaway costs:
|
||||
- No git changes for N consecutive iterations (stalled)
|
||||
- Same error repeated N consecutive iterations (stuck)
|
||||
- Maximum iteration count reached
|
||||
|
||||
.PARAMETER MaxIterations
|
||||
Maximum number of loop iterations before stopping. Default: 15.
|
||||
|
||||
.PARAMETER Model
|
||||
Claude model to use. Default: "sonnet".
|
||||
|
||||
.PARAMETER BranchName
|
||||
Optional git branch name. If provided, creates/checks out the branch before starting.
|
||||
|
||||
.PARAMETER MaxNoProgress
|
||||
Number of consecutive iterations with no git changes before circuit breaker trips. Default: 3.
|
||||
|
||||
.PARAMETER MaxSameError
|
||||
Number of consecutive iterations with the same error before circuit breaker trips. Default: 3.
|
||||
|
||||
.EXAMPLE
|
||||
.\ralph.ps1 -MaxIterations 15 -Model "sonnet" -BranchName "feature/ui-redesign"
|
||||
|
||||
.EXAMPLE
|
||||
.\ralph.ps1 -Model "opus" -MaxNoProgress 2
|
||||
#>
|
||||
|
||||
param(
|
||||
[int]$MaxIterations = 15,
|
||||
[string]$Model = "sonnet",
|
||||
[string]$BranchName,
|
||||
[int]$MaxNoProgress = 3,
|
||||
[int]$MaxSameError = 3
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
$promptFile = Join-Path $scriptDir "RALPH_PROMPT.md"
|
||||
$planFile = Join-Path $scriptDir "IMPLEMENTATION_PLAN.md"
|
||||
$designFile = Join-Path $scriptDir "DESIGN_SYSTEM.md"
|
||||
$guardrailsFile = Join-Path $scriptDir "guardrails.md"
|
||||
$progressFile = Join-Path $scriptDir "progress.txt"
|
||||
$logDir = Join-Path $scriptDir "logs"
|
||||
|
||||
# --- Validation ---
|
||||
|
||||
if (-not (Test-Path $promptFile)) {
|
||||
Write-Error "RALPH_PROMPT.md not found at $promptFile"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if (-not (Test-Path $planFile)) {
|
||||
Write-Error "IMPLEMENTATION_PLAN.md not found at $planFile"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if (-not (Test-Path $designFile)) {
|
||||
Write-Error "DESIGN_SYSTEM.md not found at $designFile"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if (-not (Test-Path $guardrailsFile)) {
|
||||
Write-Warning "guardrails.md not found at $guardrailsFile - loop may miss known failure patterns"
|
||||
}
|
||||
|
||||
# Ensure progress.txt exists
|
||||
if (-not (Test-Path $progressFile)) {
|
||||
@"
|
||||
# Progress Log
|
||||
|
||||
## Design Context
|
||||
<!-- Design decisions and context go here -->
|
||||
|
||||
## Reflex Patterns
|
||||
<!-- Reusable Reflex patterns discovered during development -->
|
||||
|
||||
## Iteration Log
|
||||
<!-- Each iteration appends a structured entry below. See RALPH_PROMPT.md for format. -->
|
||||
"@ | Set-Content -Path $progressFile -Encoding UTF8
|
||||
Write-Host "Created progress.txt"
|
||||
}
|
||||
|
||||
# Ensure logs directory exists
|
||||
if (-not (Test-Path $logDir)) {
|
||||
New-Item -ItemType Directory -Path $logDir | Out-Null
|
||||
Write-Host "Created logs directory"
|
||||
}
|
||||
|
||||
# --- Git Setup ---
|
||||
|
||||
$gitInitialised = $false
|
||||
try {
|
||||
$result = git rev-parse --is-inside-work-tree 2>&1
|
||||
if ($LASTEXITCODE -eq 0 -and $result -eq "true") {
|
||||
$gitInitialised = $true
|
||||
}
|
||||
} catch {
|
||||
# Not a git repo — expected on first run
|
||||
}
|
||||
|
||||
if (-not $gitInitialised) {
|
||||
Write-Host "Initialising git repository..."
|
||||
git init
|
||||
git add -A
|
||||
git commit -m "Initial commit before Ralph loop"
|
||||
}
|
||||
|
||||
if ($BranchName) {
|
||||
$currentBranch = git branch --show-current
|
||||
if ($currentBranch -ne $BranchName) {
|
||||
$branchExists = git branch --list $BranchName
|
||||
if ($branchExists) {
|
||||
Write-Host "Switching to existing branch: $BranchName"
|
||||
git checkout $BranchName
|
||||
} else {
|
||||
Write-Host "Creating branch: $BranchName"
|
||||
git checkout -b $BranchName
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- Circuit Breaker State ---
|
||||
|
||||
$noProgressCount = 0
|
||||
$lastErrorSignature = ""
|
||||
$sameErrorCount = 0
|
||||
|
||||
# Capture the HEAD commit hash before the loop starts
|
||||
$preLoopHead = git rev-parse HEAD 2>$null
|
||||
|
||||
# --- Main Loop ---
|
||||
|
||||
$promptContent = Get-Content -Path $promptFile -Raw
|
||||
|
||||
# Count existing iterations from progress.txt to track total across runs
|
||||
$existingIterations = 0
|
||||
if (Test-Path $progressFile) {
|
||||
$existingIterations = (Select-String -Path $progressFile -Pattern "## Iteration" -AllMatches | Measure-Object).Count
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "===== Ralph Wiggum Loop (Reflex UI) =====" -ForegroundColor Cyan
|
||||
Write-Host "Model: $Model | Max iterations: $MaxIterations" -ForegroundColor Cyan
|
||||
Write-Host "Circuit breakers: no-progress=$MaxNoProgress, same-error=$MaxSameError" -ForegroundColor Cyan
|
||||
if ($BranchName) { Write-Host "Branch: $BranchName" -ForegroundColor Cyan }
|
||||
if ($existingIterations -gt 0) { Write-Host "Previous iterations: $existingIterations" -ForegroundColor Cyan }
|
||||
Write-Host "===========================================" -ForegroundColor Cyan
|
||||
Write-Host ""
|
||||
|
||||
for ($i = 1; $i -le $MaxIterations; $i++) {
|
||||
$totalIteration = $existingIterations + $i
|
||||
Write-Host ""
|
||||
Write-Host "--- Iteration $i of $MaxIterations (Total: $totalIteration) ---" -ForegroundColor Yellow
|
||||
|
||||
# Record HEAD before this iteration
|
||||
$headBefore = git rev-parse HEAD 2>$null
|
||||
|
||||
# Show start time and status
|
||||
$iterStart = Get-Date
|
||||
Write-Host " Started: $($iterStart.ToString('HH:mm:ss'))" -ForegroundColor DarkGray
|
||||
Write-Host " Spawning Claude ($Model)..." -ForegroundColor DarkGray
|
||||
Write-Host ""
|
||||
|
||||
# Spawn fresh Claude instance with stream-json for tool call visibility
|
||||
$logFile = Join-Path $logDir "iteration_$totalIteration.log"
|
||||
$rawLogFile = Join-Path $logDir "iteration_$totalIteration.raw.jsonl"
|
||||
$maxRetries = 10
|
||||
$retryCount = 0
|
||||
$outputString = ""
|
||||
$apiOverloaded = $false
|
||||
|
||||
do {
|
||||
$apiOverloaded = $false
|
||||
$textBuilder = [System.Text.StringBuilder]::new()
|
||||
$toolCount = 0
|
||||
|
||||
# Clear raw log file for this attempt
|
||||
if (Test-Path $rawLogFile) { Remove-Item $rawLogFile -Force }
|
||||
|
||||
if ($retryCount -gt 0) {
|
||||
$backoffSeconds = [Math]::Pow(2, $retryCount - 1)
|
||||
Write-Host " [Retry $retryCount/$maxRetries] API overloaded, waiting $backoffSeconds seconds..." -ForegroundColor DarkYellow
|
||||
Start-Sleep -Seconds $backoffSeconds
|
||||
Write-Host " Retrying Claude invocation..." -ForegroundColor DarkGray
|
||||
}
|
||||
|
||||
$promptContent | claude --print --verbose --dangerously-skip-permissions --model $Model --output-format stream-json 2>&1 | ForEach-Object {
|
||||
$line = $_.ToString().Trim()
|
||||
if (-not $line) { return }
|
||||
|
||||
# Save raw event for debugging
|
||||
Add-Content -Path $rawLogFile -Value $line -Encoding UTF8
|
||||
|
||||
try {
|
||||
$evt = $line | ConvertFrom-Json -ErrorAction Stop
|
||||
|
||||
# --- Tool use detection ---
|
||||
if ($evt.type -eq 'content_block_start' -and $evt.content_block.type -eq 'tool_use') {
|
||||
$toolCount++
|
||||
$toolName = $evt.content_block.name
|
||||
Write-Host " [$toolName]" -ForegroundColor DarkCyan
|
||||
}
|
||||
elseif ($evt.tool_name) {
|
||||
$toolCount++
|
||||
Write-Host " [$($evt.tool_name)]" -ForegroundColor DarkCyan
|
||||
}
|
||||
|
||||
# --- Text content ---
|
||||
elseif ($evt.type -eq 'content_block_delta' -and $evt.delta.type -eq 'text_delta' -and $evt.delta.text) {
|
||||
Write-Host -NoNewline $evt.delta.text
|
||||
[void]$textBuilder.Append($evt.delta.text)
|
||||
}
|
||||
|
||||
elseif ($evt.type -eq 'result') {
|
||||
if ($evt.result) {
|
||||
Write-Host $evt.result
|
||||
[void]$textBuilder.AppendLine($evt.result)
|
||||
}
|
||||
if ($evt.subtype -eq 'error_result' -and $evt.error) {
|
||||
Write-Host " [ERROR] $($evt.error)" -ForegroundColor Red
|
||||
[void]$textBuilder.AppendLine("ERROR: $($evt.error)")
|
||||
}
|
||||
}
|
||||
|
||||
elseif ($evt.message.content) {
|
||||
foreach ($block in $evt.message.content) {
|
||||
if ($block.type -eq 'text' -and $block.text) {
|
||||
Write-Host $block.text
|
||||
[void]$textBuilder.AppendLine($block.text)
|
||||
}
|
||||
elseif ($block.type -eq 'tool_use') {
|
||||
$toolCount++
|
||||
Write-Host " [$($block.name)]" -ForegroundColor DarkCyan
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
} catch {
|
||||
# Not valid JSON — likely stderr output
|
||||
if ($line) {
|
||||
Write-Host $line -ForegroundColor DarkYellow
|
||||
[void]$textBuilder.AppendLine($line)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$outputString = $textBuilder.ToString()
|
||||
|
||||
# Check for 529 overloaded error
|
||||
if ($outputString -match "529.*overloaded|overloaded_error") {
|
||||
$apiOverloaded = $true
|
||||
$retryCount++
|
||||
if ($retryCount -ge $maxRetries) {
|
||||
Write-Host " [ERROR] API overloaded after $maxRetries retries, giving up." -ForegroundColor Red
|
||||
}
|
||||
}
|
||||
} while ($apiOverloaded -and $retryCount -lt $maxRetries)
|
||||
|
||||
$outputString | Set-Content -Path $logFile -Encoding UTF8
|
||||
|
||||
# Show elapsed time and tool count
|
||||
$elapsed = (Get-Date) - $iterStart
|
||||
Write-Host ""
|
||||
Write-Host " Finished: $(Get-Date -Format 'HH:mm:ss') (elapsed: $($elapsed.ToString('mm\:ss')), tools: $toolCount)" -ForegroundColor DarkGray
|
||||
|
||||
# --- Circuit Breaker: No Progress ---
|
||||
$headAfter = git rev-parse HEAD 2>$null
|
||||
if ($headAfter -eq $headBefore) {
|
||||
$noProgressCount++
|
||||
Write-Host " [Circuit Breaker] No git commits this iteration ($noProgressCount/$MaxNoProgress)" -ForegroundColor DarkYellow
|
||||
if ($noProgressCount -ge $MaxNoProgress) {
|
||||
Write-Host ""
|
||||
Write-Host "===== CIRCUIT BREAKER: NO PROGRESS =====" -ForegroundColor Red
|
||||
Write-Host "No git commits for $MaxNoProgress consecutive iterations. The loop is stalled." -ForegroundColor Red
|
||||
Write-Host "Check progress.txt and logs/ for details on what went wrong." -ForegroundColor Red
|
||||
exit 1
|
||||
}
|
||||
} else {
|
||||
$noProgressCount = 0
|
||||
}
|
||||
|
||||
# --- Circuit Breaker: Repeated Error ---
|
||||
$errorLines = $outputString | Select-String -Pattern "(?i)(error|exception|failed|fatal)[:.].*" -AllMatches
|
||||
if ($errorLines) {
|
||||
$filteredErrors = $errorLines.Matches | Where-Object { $_.Value -notmatch "529|overloaded" } | Select-Object -First 3
|
||||
$currentErrorSignature = ($filteredErrors | ForEach-Object { $_.Value }) -join "|"
|
||||
if ($currentErrorSignature -and $currentErrorSignature -eq $lastErrorSignature) {
|
||||
$sameErrorCount++
|
||||
Write-Host " [Circuit Breaker] Same error pattern repeated ($sameErrorCount/$MaxSameError)" -ForegroundColor DarkYellow
|
||||
if ($sameErrorCount -ge $MaxSameError) {
|
||||
Write-Host ""
|
||||
Write-Host "===== CIRCUIT BREAKER: REPEATED ERROR =====" -ForegroundColor Red
|
||||
Write-Host "Same error pattern for $MaxSameError consecutive iterations:" -ForegroundColor Red
|
||||
Write-Host " $currentErrorSignature" -ForegroundColor Red
|
||||
Write-Host "Check progress.txt and logs/ for details." -ForegroundColor Red
|
||||
exit 1
|
||||
}
|
||||
} elseif ($currentErrorSignature) {
|
||||
$sameErrorCount = 0
|
||||
}
|
||||
$lastErrorSignature = $currentErrorSignature
|
||||
} else {
|
||||
$sameErrorCount = 0
|
||||
$lastErrorSignature = ""
|
||||
}
|
||||
|
||||
# --- Push to Remote ---
|
||||
$hasRemote = git remote 2>$null
|
||||
if ($hasRemote) {
|
||||
$currentBranch = git branch --show-current
|
||||
git push origin $currentBranch 2>$null
|
||||
if ($LASTEXITCODE -eq 0) {
|
||||
Write-Host " Pushed to remote." -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host " Push failed or no remote configured - continuing." -ForegroundColor DarkYellow
|
||||
}
|
||||
}
|
||||
|
||||
# --- Check for Completion ---
|
||||
if ($outputString -match "<promise>COMPLETE</promise>") {
|
||||
Write-Host ""
|
||||
Write-Host "===== COMPLETE =====" -ForegroundColor Green
|
||||
Write-Host "UI redesign finished after $i iteration(s) this run ($totalIteration total)." -ForegroundColor Green
|
||||
exit 0
|
||||
}
|
||||
|
||||
# Brief pause between iterations
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "===== MAX ITERATIONS REACHED =====" -ForegroundColor Red
|
||||
Write-Host "Completed $MaxIterations iterations without finishing all tasks." -ForegroundColor Red
|
||||
Write-Host "Check progress.txt for current state and what remains." -ForegroundColor Red
|
||||
exit 1
|
||||
Binary file not shown.
@@ -0,0 +1,9 @@
|
||||
import reflex as rx
|
||||
|
||||
config = rx.Config(
|
||||
app_name="pathways_app",
|
||||
plugins=[
|
||||
rx.plugins.SitemapPlugin(),
|
||||
rx.plugins.TailwindV4Plugin(),
|
||||
]
|
||||
)
|
||||
@@ -0,0 +1,9 @@
|
||||
"""
|
||||
Test suite for NHS High-Cost Drug Patient Pathway Analysis Tool.
|
||||
|
||||
This package contains unit tests and integration tests for:
|
||||
- Core configuration and models (config.py, models.py)
|
||||
- Data transformations (data.py, loader.py)
|
||||
- Analysis pipeline (pathway_analyzer.py, statistics.py)
|
||||
- Database operations (database.py, schema.py)
|
||||
"""
|
||||
@@ -0,0 +1,359 @@
|
||||
"""
|
||||
Performance benchmark for the Patient Pathway Analysis tool.
|
||||
|
||||
This script measures:
|
||||
1. Module import time
|
||||
2. Data loading time (SQLite)
|
||||
3. Analysis pipeline execution time
|
||||
4. Peak memory usage
|
||||
|
||||
Run with: python -m tests.benchmark_performance
|
||||
"""
|
||||
|
||||
import gc
|
||||
import sys
|
||||
import time
|
||||
import tracemalloc
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
# Store results for final report
|
||||
results: dict[str, Any] = {}
|
||||
|
||||
|
||||
def measure_time(func, *args, **kwargs):
|
||||
"""Measure execution time of a function."""
|
||||
gc.collect() # Clean up before timing
|
||||
start = time.perf_counter()
|
||||
result = func(*args, **kwargs)
|
||||
elapsed = time.perf_counter() - start
|
||||
return result, elapsed
|
||||
|
||||
|
||||
def measure_memory(func, *args, **kwargs):
|
||||
"""Measure peak memory usage of a function."""
|
||||
gc.collect() # Clean up before measuring
|
||||
tracemalloc.start()
|
||||
|
||||
result = func(*args, **kwargs)
|
||||
|
||||
current, peak = tracemalloc.get_traced_memory()
|
||||
tracemalloc.stop()
|
||||
|
||||
return result, peak
|
||||
|
||||
|
||||
def benchmark_imports():
|
||||
"""Benchmark module import times."""
|
||||
print("\n" + "=" * 60)
|
||||
print("1. MODULE IMPORT BENCHMARKS")
|
||||
print("=" * 60)
|
||||
|
||||
import_times = {}
|
||||
|
||||
# Benchmark core imports
|
||||
start = time.perf_counter()
|
||||
from core import PathConfig, AnalysisFilters, default_paths
|
||||
import_times['core'] = time.perf_counter() - start
|
||||
|
||||
# Benchmark data_processing imports
|
||||
start = time.perf_counter()
|
||||
from data_processing import DatabaseManager, get_loader
|
||||
import_times['data_processing'] = time.perf_counter() - start
|
||||
|
||||
# Benchmark analysis imports
|
||||
start = time.perf_counter()
|
||||
from analysis.pathway_analyzer import generate_icicle_chart
|
||||
import_times['analysis'] = time.perf_counter() - start
|
||||
|
||||
# Benchmark visualization imports
|
||||
start = time.perf_counter()
|
||||
from visualization.plotly_generator import create_icicle_figure
|
||||
import_times['visualization'] = time.perf_counter() - start
|
||||
|
||||
# Benchmark pandas/numpy
|
||||
start = time.perf_counter()
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import_times['pandas+numpy'] = time.perf_counter() - start
|
||||
|
||||
total_import_time = sum(import_times.values())
|
||||
|
||||
print(f"\n{'Module':<25} {'Time (ms)':<15}")
|
||||
print("-" * 40)
|
||||
for module, elapsed in import_times.items():
|
||||
print(f"{module:<25} {elapsed*1000:>10.1f} ms")
|
||||
print("-" * 40)
|
||||
print(f"{'TOTAL':<25} {total_import_time*1000:>10.1f} ms")
|
||||
|
||||
results['import_times'] = import_times
|
||||
results['total_import_time'] = total_import_time
|
||||
|
||||
return import_times
|
||||
|
||||
|
||||
def benchmark_data_loading():
|
||||
"""Benchmark data loading from different sources."""
|
||||
print("\n" + "=" * 60)
|
||||
print("2. DATA LOADING BENCHMARKS")
|
||||
print("=" * 60)
|
||||
|
||||
from data_processing import get_loader
|
||||
from core import default_paths
|
||||
import pandas as pd
|
||||
|
||||
load_times = {}
|
||||
row_counts = {}
|
||||
|
||||
# Check if SQLite database exists
|
||||
db_path = default_paths.data_dir / "pathways.db"
|
||||
if db_path.exists():
|
||||
print(f"\nLoading from SQLite: {db_path}")
|
||||
|
||||
# SQLite loading
|
||||
loader = get_loader('sqlite')
|
||||
result, elapsed = measure_time(loader.load)
|
||||
load_times['sqlite'] = elapsed
|
||||
row_counts['sqlite'] = result.row_count if result is not None else 0
|
||||
|
||||
print(f" Rows loaded: {row_counts['sqlite']:,}")
|
||||
print(f" Time: {elapsed*1000:.1f} ms ({elapsed:.2f} seconds)")
|
||||
print(f" Internal load time: {result.load_time_seconds*1000:.1f} ms")
|
||||
|
||||
# Store for later use
|
||||
results['loaded_df'] = result.df
|
||||
else:
|
||||
print(f"SQLite database not found at {db_path}")
|
||||
load_times['sqlite'] = None
|
||||
|
||||
results['load_times'] = load_times
|
||||
results['row_counts'] = row_counts
|
||||
|
||||
return load_times
|
||||
|
||||
|
||||
def benchmark_analysis_pipeline():
|
||||
"""Benchmark the full analysis pipeline."""
|
||||
print("\n" + "=" * 60)
|
||||
print("3. ANALYSIS PIPELINE BENCHMARKS")
|
||||
print("=" * 60)
|
||||
|
||||
from analysis.pathway_analyzer import (
|
||||
generate_icicle_chart,
|
||||
prepare_data,
|
||||
calculate_statistics,
|
||||
build_hierarchy,
|
||||
prepare_chart_data,
|
||||
)
|
||||
from core import default_paths
|
||||
import pandas as pd
|
||||
|
||||
# Get loaded data or load it
|
||||
df = results.get('loaded_df')
|
||||
if df is None or len(df) == 0:
|
||||
print("No data available for analysis benchmarks")
|
||||
return {}
|
||||
|
||||
analysis_times = {}
|
||||
|
||||
# Get available trusts, drugs, directories from data
|
||||
trusts = df['Provider Code'].unique().tolist()[:10] # Limit to 10 trusts
|
||||
drugs = ['ADALIMUMAB', 'ETANERCEPT', 'INFLIXIMAB', 'SECUKINUMAB', 'RITUXIMAB']
|
||||
directories = df['Directory'].dropna().unique().tolist()
|
||||
|
||||
# Filter to drugs that exist in data
|
||||
available_drugs = [d for d in drugs if d in df['Drug Name'].values]
|
||||
if not available_drugs:
|
||||
available_drugs = df['Drug Name'].unique().tolist()[:5]
|
||||
|
||||
print(f"\nAnalysis parameters:")
|
||||
print(f" Trusts: {len(trusts)}")
|
||||
print(f" Drugs: {available_drugs}")
|
||||
print(f" Directories: {len(directories)}")
|
||||
print(f" Data rows: {len(df):,}")
|
||||
|
||||
# Load org_codes for mapping trust codes to names
|
||||
org_codes = pd.read_csv(default_paths.org_codes_csv, index_col=1)
|
||||
trust_names = []
|
||||
for t in trusts:
|
||||
if t in org_codes.index:
|
||||
trust_names.append(org_codes.loc[t, 'Name'])
|
||||
|
||||
if not trust_names:
|
||||
trust_names = org_codes['Name'].tolist()[:10]
|
||||
|
||||
# Benchmark full pipeline
|
||||
print("\n Running full pipeline benchmark...")
|
||||
|
||||
# Use date range that should include data
|
||||
# Look at actual data dates
|
||||
if 'Intervention Date' in df.columns:
|
||||
min_date = df['Intervention Date'].min()
|
||||
max_date = df['Intervention Date'].max()
|
||||
print(f" Data date range: {min_date} to {max_date}")
|
||||
|
||||
# Use a reasonable analysis window
|
||||
start_date = "2020-01-01"
|
||||
end_date = "2025-01-01"
|
||||
last_seen_date = "2020-01-01"
|
||||
else:
|
||||
start_date = "2020-01-01"
|
||||
end_date = "2025-01-01"
|
||||
last_seen_date = "2020-01-01"
|
||||
|
||||
print(f" Analysis window: {start_date} to {end_date}")
|
||||
print(f" Last seen filter: > {last_seen_date}")
|
||||
|
||||
# Full pipeline with memory tracking
|
||||
gc.collect()
|
||||
tracemalloc.start()
|
||||
start_time = time.perf_counter()
|
||||
|
||||
try:
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=df,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
last_seen_date=last_seen_date,
|
||||
trust_filter=trust_names,
|
||||
drug_filter=available_drugs,
|
||||
directory_filter=directories,
|
||||
minimum_num_patients=1,
|
||||
title="Performance Benchmark",
|
||||
paths=default_paths,
|
||||
)
|
||||
|
||||
elapsed = time.perf_counter() - start_time
|
||||
current, peak = tracemalloc.get_traced_memory()
|
||||
tracemalloc.stop()
|
||||
|
||||
analysis_times['full_pipeline'] = elapsed
|
||||
results['analysis_memory_peak'] = peak
|
||||
|
||||
if ice_df is not None:
|
||||
print(f"\n Pipeline completed:")
|
||||
print(f" Execution time: {elapsed*1000:.1f} ms ({elapsed:.2f} seconds)")
|
||||
print(f" Peak memory: {peak / 1024 / 1024:.1f} MB")
|
||||
print(f" Result rows: {len(ice_df)}")
|
||||
print(f" Chart title: {title}")
|
||||
else:
|
||||
print("\n Pipeline returned no data (likely date filtering)")
|
||||
print(f" Execution time: {elapsed*1000:.1f} ms")
|
||||
|
||||
except Exception as e:
|
||||
tracemalloc.stop()
|
||||
print(f"\n Pipeline error: {e}")
|
||||
traceback_str = ''.join(tracemalloc.format_exc() if hasattr(tracemalloc, 'format_exc') else [])
|
||||
print(f" {str(e)}")
|
||||
analysis_times['full_pipeline'] = None
|
||||
|
||||
results['analysis_times'] = analysis_times
|
||||
return analysis_times
|
||||
|
||||
|
||||
def benchmark_visualization():
|
||||
"""Benchmark chart generation."""
|
||||
print("\n" + "=" * 60)
|
||||
print("4. VISUALIZATION BENCHMARKS")
|
||||
print("=" * 60)
|
||||
|
||||
from visualization.plotly_generator import create_icicle_figure
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
|
||||
viz_times = {}
|
||||
|
||||
# Create sample data for visualization benchmark
|
||||
n_rows = 1000
|
||||
sample_data = {
|
||||
'parents': ['N&WICS'] * n_rows,
|
||||
'ids': [f'N&WICS - Test{i}' for i in range(n_rows)],
|
||||
'labels': [f'Test{i}' for i in range(n_rows)],
|
||||
'value': np.random.randint(1, 100, n_rows),
|
||||
'colour': np.random.random(n_rows),
|
||||
'cost': np.random.randint(1000, 100000, n_rows),
|
||||
'costpp': np.random.randint(100, 10000, n_rows),
|
||||
'cost_pp_pa': [str(np.random.randint(100, 10000)) for _ in range(n_rows)],
|
||||
'First seen': pd.to_datetime(['2024-01-01'] * n_rows),
|
||||
'Last seen': pd.to_datetime(['2024-12-31'] * n_rows),
|
||||
'First seen (Parent)': ['2024-01-01'] * n_rows,
|
||||
'Last seen (Parent)': ['2024-12-31'] * n_rows,
|
||||
'average_spacing': ['Test spacing'] * n_rows,
|
||||
'avg_days': pd.to_timedelta([100] * n_rows, unit='D'),
|
||||
}
|
||||
sample_df = pd.DataFrame(sample_data)
|
||||
|
||||
print(f"\n Sample data: {n_rows} rows")
|
||||
|
||||
# Benchmark figure creation
|
||||
fig, elapsed = measure_time(create_icicle_figure, sample_df, "Benchmark Test")
|
||||
viz_times['figure_creation'] = elapsed
|
||||
|
||||
print(f" Figure creation: {elapsed*1000:.1f} ms")
|
||||
|
||||
results['viz_times'] = viz_times
|
||||
return viz_times
|
||||
|
||||
|
||||
def print_summary():
|
||||
"""Print final summary report."""
|
||||
print("\n" + "=" * 60)
|
||||
print("PERFORMANCE SUMMARY")
|
||||
print("=" * 60)
|
||||
|
||||
print("\nRESULTS:")
|
||||
|
||||
# Import times
|
||||
if 'total_import_time' in results:
|
||||
print(f"\n Import time (all modules): {results['total_import_time']*1000:.1f} ms")
|
||||
|
||||
# Data loading
|
||||
if 'load_times' in results and results['load_times'].get('sqlite'):
|
||||
print(f" SQLite load time: {results['load_times']['sqlite']*1000:.1f} ms")
|
||||
if 'row_counts' in results:
|
||||
print(f" Rows loaded: {results['row_counts'].get('sqlite', 0):,}")
|
||||
|
||||
# Analysis
|
||||
if 'analysis_times' in results and results['analysis_times'].get('full_pipeline'):
|
||||
print(f" Analysis pipeline: {results['analysis_times']['full_pipeline']*1000:.1f} ms")
|
||||
|
||||
# Memory
|
||||
if 'analysis_memory_peak' in results:
|
||||
print(f" Peak memory (analysis): {results['analysis_memory_peak'] / 1024 / 1024:.1f} MB")
|
||||
|
||||
# Visualization
|
||||
if 'viz_times' in results:
|
||||
print(f" Figure creation: {results['viz_times'].get('figure_creation', 0)*1000:.1f} ms")
|
||||
|
||||
# Calculate total startup time (imports + data loading)
|
||||
startup_time = results.get('total_import_time', 0)
|
||||
if results.get('load_times', {}).get('sqlite'):
|
||||
startup_time += results['load_times']['sqlite']
|
||||
print(f"\n Estimated startup time: {startup_time*1000:.1f} ms ({startup_time:.2f} seconds)")
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all benchmarks."""
|
||||
print("\n" + "=" * 60)
|
||||
print("PATIENT PATHWAY ANALYSIS - PERFORMANCE BENCHMARK")
|
||||
print("=" * 60)
|
||||
print(f"\nPython version: {sys.version}")
|
||||
print(f"Platform: {sys.platform}")
|
||||
|
||||
# Run benchmarks in order
|
||||
benchmark_imports()
|
||||
benchmark_data_loading()
|
||||
benchmark_analysis_pipeline()
|
||||
benchmark_visualization()
|
||||
|
||||
# Print summary
|
||||
print_summary()
|
||||
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,128 @@
|
||||
"""
|
||||
Pytest configuration and fixtures for the test suite.
|
||||
|
||||
This module provides shared fixtures used across multiple test modules.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Generator
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_dir() -> Generator[Path, None, None]:
|
||||
"""Create a temporary directory that is cleaned up after the test."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
yield Path(tmpdir)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_data_dir(temp_dir: Path) -> Path:
|
||||
"""
|
||||
Create a mock data directory with empty reference files.
|
||||
|
||||
Creates the expected directory structure and empty placeholder files
|
||||
so that PathConfig.validate() can pass file existence checks.
|
||||
"""
|
||||
data_dir = temp_dir / "data"
|
||||
data_dir.mkdir()
|
||||
|
||||
# Create empty reference files
|
||||
reference_files = [
|
||||
"drugnames.csv",
|
||||
"directory_list.csv",
|
||||
"treatment_function_codes.csv",
|
||||
"drug_directory_list.csv",
|
||||
"org_codes.csv",
|
||||
"include.csv",
|
||||
"defaultTrusts.csv",
|
||||
]
|
||||
|
||||
for filename in reference_files:
|
||||
(data_dir / filename).touch()
|
||||
|
||||
return data_dir
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_images_dir(temp_dir: Path) -> Path:
|
||||
"""
|
||||
Create a mock images directory with empty font files.
|
||||
|
||||
Creates the expected directory structure and empty placeholder files
|
||||
so that PathConfig.validate_fonts() can pass file existence checks.
|
||||
"""
|
||||
images_dir = temp_dir / "images"
|
||||
images_dir.mkdir()
|
||||
|
||||
# Create empty font files
|
||||
font_files = [
|
||||
"AvenirLTStd-Medium.ttf",
|
||||
"AvenirLTStd-Roman.ttf",
|
||||
"logo.ico",
|
||||
"logo.png",
|
||||
]
|
||||
|
||||
for filename in font_files:
|
||||
(images_dir / filename).touch()
|
||||
|
||||
return images_dir
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_project_dir(temp_dir: Path, mock_data_dir: Path, mock_images_dir: Path) -> Path:
|
||||
"""
|
||||
Create a complete mock project directory structure.
|
||||
|
||||
Combines data and images directories for full PathConfig validation.
|
||||
"""
|
||||
return temp_dir
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_date_range() -> tuple[date, date, date]:
|
||||
"""
|
||||
Return a sample valid date range for testing AnalysisFilters.
|
||||
|
||||
Returns:
|
||||
Tuple of (start_date, end_date, last_seen_date)
|
||||
"""
|
||||
return (
|
||||
date(2024, 1, 1), # start_date
|
||||
date(2024, 12, 31), # end_date
|
||||
date(2024, 6, 1), # last_seen_date
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_trusts() -> list[str]:
|
||||
"""Return a sample list of NHS trust names for testing."""
|
||||
return [
|
||||
"MANCHESTER UNIVERSITY NHS FOUNDATION TRUST",
|
||||
"LEEDS TEACHING HOSPITALS NHS TRUST",
|
||||
"SHEFFIELD TEACHING HOSPITALS NHS FOUNDATION TRUST",
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_drugs() -> list[str]:
|
||||
"""Return a sample list of drug names for testing."""
|
||||
return [
|
||||
"ADALIMUMAB",
|
||||
"ETANERCEPT",
|
||||
"INFLIXIMAB",
|
||||
"RITUXIMAB",
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_directories() -> list[str]:
|
||||
"""Return a sample list of medical directories for testing."""
|
||||
return [
|
||||
"RHEUMATOLOGY",
|
||||
"DERMATOLOGY",
|
||||
"GASTROENTEROLOGY",
|
||||
]
|
||||
@@ -0,0 +1,226 @@
|
||||
"""
|
||||
Tests for core/config.py - PathConfig dataclass.
|
||||
|
||||
Tests cover:
|
||||
- Default path construction
|
||||
- Custom path configuration
|
||||
- Path property access
|
||||
- validate() method for file existence checks
|
||||
- validate_fonts() method for font file checks
|
||||
- as_legacy_paths() method for backwards compatibility
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from core.config import PathConfig
|
||||
|
||||
|
||||
class TestPathConfigDefaults:
|
||||
"""Test default behavior of PathConfig."""
|
||||
|
||||
def test_default_base_dir_is_cwd(self):
|
||||
"""Default base_dir should be current working directory."""
|
||||
config = PathConfig()
|
||||
assert config.base_dir == Path.cwd()
|
||||
|
||||
def test_default_data_dir_is_under_base(self):
|
||||
"""Default data_dir should be 'data' under base_dir."""
|
||||
config = PathConfig()
|
||||
assert config.data_dir == config.base_dir / "data"
|
||||
|
||||
def test_default_images_dir_is_under_base(self):
|
||||
"""Default images_dir should be 'images' under base_dir."""
|
||||
config = PathConfig()
|
||||
assert config.images_dir == config.base_dir / "images"
|
||||
|
||||
|
||||
class TestPathConfigCustomPaths:
|
||||
"""Test custom path configuration."""
|
||||
|
||||
def test_custom_base_dir(self, temp_dir: Path):
|
||||
"""PathConfig should accept custom base_dir."""
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
assert config.base_dir == temp_dir
|
||||
assert config.data_dir == temp_dir / "data"
|
||||
assert config.images_dir == temp_dir / "images"
|
||||
|
||||
|
||||
class TestPathConfigProperties:
|
||||
"""Test path property accessors."""
|
||||
|
||||
def test_drugnames_csv_path(self):
|
||||
"""drugnames_csv should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.drugnames_csv == config.data_dir / "drugnames.csv"
|
||||
|
||||
def test_directory_list_csv_path(self):
|
||||
"""directory_list_csv should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.directory_list_csv == config.data_dir / "directory_list.csv"
|
||||
|
||||
def test_treatment_function_codes_csv_path(self):
|
||||
"""treatment_function_codes_csv should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.treatment_function_codes_csv == config.data_dir / "treatment_function_codes.csv"
|
||||
|
||||
def test_drug_directory_list_csv_path(self):
|
||||
"""drug_directory_list_csv should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.drug_directory_list_csv == config.data_dir / "drug_directory_list.csv"
|
||||
|
||||
def test_org_codes_csv_path(self):
|
||||
"""org_codes_csv should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.org_codes_csv == config.data_dir / "org_codes.csv"
|
||||
|
||||
def test_include_csv_path(self):
|
||||
"""include_csv should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.include_csv == config.data_dir / "include.csv"
|
||||
|
||||
def test_default_trusts_csv_path(self):
|
||||
"""default_trusts_csv should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.default_trusts_csv == config.data_dir / "defaultTrusts.csv"
|
||||
|
||||
def test_font_medium_path(self):
|
||||
"""font_medium should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.font_medium == config.images_dir / "AvenirLTStd-Medium.ttf"
|
||||
|
||||
def test_font_roman_path(self):
|
||||
"""font_roman should point to correct file."""
|
||||
config = PathConfig()
|
||||
assert config.font_roman == config.images_dir / "AvenirLTStd-Roman.ttf"
|
||||
|
||||
|
||||
class TestPathConfigValidate:
|
||||
"""Test validate() method."""
|
||||
|
||||
def test_validate_passes_when_all_files_exist(self, mock_project_dir: Path):
|
||||
"""validate() should return empty list when all files exist."""
|
||||
config = PathConfig(base_dir=mock_project_dir)
|
||||
errors = config.validate()
|
||||
assert errors == []
|
||||
|
||||
def test_validate_fails_when_data_dir_missing(self, temp_dir: Path):
|
||||
"""validate() should report missing data directory."""
|
||||
# Create images dir but not data dir
|
||||
(temp_dir / "images").mkdir()
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
|
||||
errors = config.validate()
|
||||
|
||||
assert len(errors) >= 1
|
||||
assert any("Data directory not found" in e for e in errors)
|
||||
|
||||
def test_validate_fails_when_images_dir_missing(self, temp_dir: Path):
|
||||
"""validate() should report missing images directory."""
|
||||
# Create data dir but not images dir
|
||||
(temp_dir / "data").mkdir()
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
|
||||
errors = config.validate()
|
||||
|
||||
assert len(errors) >= 1
|
||||
assert any("Images directory not found" in e for e in errors)
|
||||
|
||||
def test_validate_fails_when_required_file_missing(self, temp_dir: Path):
|
||||
"""validate() should report missing required files."""
|
||||
# Create directories but only some files
|
||||
data_dir = temp_dir / "data"
|
||||
data_dir.mkdir()
|
||||
(temp_dir / "images").mkdir()
|
||||
|
||||
# Create only one file
|
||||
(data_dir / "drugnames.csv").touch()
|
||||
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
errors = config.validate()
|
||||
|
||||
# Should report 6 missing files (7 total - 1 created)
|
||||
# Exclude directory-related messages (data/images directory checks)
|
||||
# but include files that have "directory" in the filename
|
||||
missing_file_errors = [
|
||||
e for e in errors
|
||||
if "not found" in e
|
||||
and "Data directory not found" not in e
|
||||
and "Images directory not found" not in e
|
||||
]
|
||||
assert len(missing_file_errors) == 6
|
||||
|
||||
|
||||
class TestPathConfigValidateFonts:
|
||||
"""Test validate_fonts() method."""
|
||||
|
||||
def test_validate_fonts_passes_when_fonts_exist(self, mock_project_dir: Path):
|
||||
"""validate_fonts() should return empty list when fonts exist."""
|
||||
config = PathConfig(base_dir=mock_project_dir)
|
||||
errors = config.validate_fonts()
|
||||
assert errors == []
|
||||
|
||||
def test_validate_fonts_fails_when_medium_font_missing(self, temp_dir: Path):
|
||||
"""validate_fonts() should report missing medium font."""
|
||||
images_dir = temp_dir / "images"
|
||||
images_dir.mkdir()
|
||||
# Create only roman font
|
||||
(images_dir / "AvenirLTStd-Roman.ttf").touch()
|
||||
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
errors = config.validate_fonts()
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "Medium font not found" in errors[0]
|
||||
|
||||
def test_validate_fonts_fails_when_roman_font_missing(self, temp_dir: Path):
|
||||
"""validate_fonts() should report missing roman font."""
|
||||
images_dir = temp_dir / "images"
|
||||
images_dir.mkdir()
|
||||
# Create only medium font
|
||||
(images_dir / "AvenirLTStd-Medium.ttf").touch()
|
||||
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
errors = config.validate_fonts()
|
||||
|
||||
assert len(errors) == 1
|
||||
assert "Roman font not found" in errors[0]
|
||||
|
||||
|
||||
class TestPathConfigLegacyPaths:
|
||||
"""Test as_legacy_paths() method for backwards compatibility."""
|
||||
|
||||
def test_legacy_paths_returns_dict(self, temp_dir: Path):
|
||||
"""as_legacy_paths() should return a dictionary."""
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
legacy = config.as_legacy_paths()
|
||||
assert isinstance(legacy, dict)
|
||||
|
||||
def test_legacy_paths_contains_expected_keys(self, temp_dir: Path):
|
||||
"""as_legacy_paths() should contain all expected keys."""
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
legacy = config.as_legacy_paths()
|
||||
|
||||
expected_keys = [
|
||||
"drugnames_csv",
|
||||
"directory_list_csv",
|
||||
"treatment_function_codes_csv",
|
||||
"drug_directory_list_csv",
|
||||
"org_codes_csv",
|
||||
"include_csv",
|
||||
"default_trusts_csv",
|
||||
"na_directory_rows_csv",
|
||||
"ta_recommendations_xlsx",
|
||||
]
|
||||
|
||||
for key in expected_keys:
|
||||
assert key in legacy
|
||||
|
||||
def test_legacy_paths_have_dot_slash_prefix(self, temp_dir: Path):
|
||||
"""as_legacy_paths() values should start with './'."""
|
||||
config = PathConfig(base_dir=temp_dir)
|
||||
legacy = config.as_legacy_paths()
|
||||
|
||||
for key, value in legacy.items():
|
||||
assert value.startswith("./"), f"{key} should start with ./ but got {value}"
|
||||
@@ -0,0 +1,924 @@
|
||||
"""
|
||||
Tests for tools/data.py - Data transformation functions.
|
||||
|
||||
Tests cover:
|
||||
- patient_id(): UPID generation from Provider Code and PersonKey
|
||||
- drug_names(): Drug name standardization via CSV mapping
|
||||
- department_identification(): Directory assignment with 5-level fallback chain
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Generator
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pytest
|
||||
|
||||
from core.config import PathConfig
|
||||
from tools.data import patient_id, drug_names, department_identification
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Fixtures for data transformation tests
|
||||
# ============================================================================
|
||||
|
||||
@pytest.fixture
|
||||
def sample_patient_df() -> pd.DataFrame:
|
||||
"""Create a sample DataFrame with patient data for UPID generation."""
|
||||
return pd.DataFrame({
|
||||
"Provider Code": ["RXA123", "RXB456", "RXC789", "RXA123"],
|
||||
"PersonKey": [1001, 2002, 3003, 1001],
|
||||
"Drug Name": ["Test Drug", "Another Drug", "Test Drug", "Test Drug"],
|
||||
"Price Actual": [100.0, 200.0, 150.0, 100.0],
|
||||
})
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_drug_df() -> pd.DataFrame:
|
||||
"""Create a sample DataFrame with drug names for standardization."""
|
||||
return pd.DataFrame({
|
||||
"Drug Name": [
|
||||
"ABATACEPT 250MG POWDER",
|
||||
"adalimumab (homecare)",
|
||||
"ETANERCEPT (LEFT EYE)",
|
||||
"infliximab (RIGHT EYE)",
|
||||
"Unknown Drug",
|
||||
],
|
||||
"Provider Code": ["RXA", "RXB", "RXC", "RXD", "RXE"],
|
||||
"PersonKey": [1, 2, 3, 4, 5],
|
||||
})
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_data_for_transforms(temp_dir: Path) -> Path:
|
||||
"""
|
||||
Create mock data directory with reference files for transformation tests.
|
||||
|
||||
Creates:
|
||||
- drugnames.csv: Drug name mapping
|
||||
- directory_list.csv: Valid directories
|
||||
- drug_directory_list.csv: Drug-to-directory mappings
|
||||
- treatment_function_codes.csv: Treatment function codes
|
||||
"""
|
||||
data_dir = temp_dir / "data"
|
||||
data_dir.mkdir()
|
||||
|
||||
# Create drugnames.csv (no header, raw_name,standard_name)
|
||||
drugnames_content = """ABATACEPT,ABATACEPT
|
||||
ABATACEPT 250MG POWDER,ABATACEPT
|
||||
ABATACEPT (HOMECARE),ABATACEPT
|
||||
ADALIMUMAB,ADALIMUMAB
|
||||
ADALIMUMAB (HOMECARE),ADALIMUMAB
|
||||
ETANERCEPT,ETANERCEPT
|
||||
ETANERCEPT (LEFT EYE),ETANERCEPT
|
||||
ETANERCEPT (RIGHT EYE),ETANERCEPT
|
||||
INFLIXIMAB,INFLIXIMAB
|
||||
INFLIXIMAB (RIGHT EYE),INFLIXIMAB
|
||||
"""
|
||||
(data_dir / "drugnames.csv").write_text(drugnames_content)
|
||||
|
||||
# Create directory_list.csv (has header)
|
||||
directory_list_content = """directory
|
||||
RHEUMATOLOGY
|
||||
DERMATOLOGY
|
||||
GASTROENTEROLOGY
|
||||
OPHTHALMOLOGY
|
||||
NEUROLOGY
|
||||
CLINICAL HAEMATOLOGY
|
||||
PAEDIATRICS
|
||||
"""
|
||||
(data_dir / "directory_list.csv").write_text(directory_list_content)
|
||||
|
||||
# Create drug_directory_list.csv (has header, drug|directories)
|
||||
drug_directory_content = """DRUG,DIRECTORIES
|
||||
ABATACEPT,RHEUMATOLOGY|PAEDIATRICS
|
||||
ADALIMUMAB,RHEUMATOLOGY|GASTROENTEROLOGY|DERMATOLOGY|OPHTHALMOLOGY
|
||||
ETANERCEPT,RHEUMATOLOGY|DERMATOLOGY
|
||||
INFLIXIMAB,RHEUMATOLOGY|GASTROENTEROLOGY|DERMATOLOGY
|
||||
RITUXIMAB,CLINICAL HAEMATOLOGY
|
||||
"""
|
||||
(data_dir / "drug_directory_list.csv").write_text(drug_directory_content)
|
||||
|
||||
# Create treatment_function_codes.csv
|
||||
treatment_function_codes_content = """Code,Service
|
||||
100,GENERAL SURGERY
|
||||
410,RHEUMATOLOGY
|
||||
330,DERMATOLOGY
|
||||
301,GASTROENTEROLOGY
|
||||
130,OPHTHALMOLOGY
|
||||
400,NEUROLOGY
|
||||
"""
|
||||
(data_dir / "treatment_function_codes.csv").write_text(treatment_function_codes_content)
|
||||
|
||||
# Create other required files (empty placeholders)
|
||||
(data_dir / "org_codes.csv").write_text("Name,Code\n")
|
||||
(data_dir / "include.csv").write_text("")
|
||||
(data_dir / "defaultTrusts.csv").write_text("")
|
||||
|
||||
return data_dir
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def test_paths(mock_data_for_transforms: Path, temp_dir: Path) -> PathConfig:
|
||||
"""Create PathConfig pointing to mock data directory."""
|
||||
return PathConfig(base_dir=temp_dir)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Tests for patient_id()
|
||||
# ============================================================================
|
||||
|
||||
class TestPatientId:
|
||||
"""Test UPID generation from Provider Code and PersonKey."""
|
||||
|
||||
def test_upid_created(self, sample_patient_df: pd.DataFrame):
|
||||
"""UPID column should be created."""
|
||||
result = patient_id(sample_patient_df)
|
||||
assert "UPID" in result.columns
|
||||
|
||||
def test_upid_format(self, sample_patient_df: pd.DataFrame):
|
||||
"""UPID should be Provider Code (first 3 chars) + PersonKey."""
|
||||
result = patient_id(sample_patient_df)
|
||||
expected_upids = ["RXA1001", "RXB2002", "RXC3003", "RXA1001"]
|
||||
assert result["UPID"].tolist() == expected_upids
|
||||
|
||||
def test_upid_handles_short_provider_codes(self):
|
||||
"""UPID should work with provider codes shorter than 3 chars."""
|
||||
df = pd.DataFrame({
|
||||
"Provider Code": ["AB", "X"],
|
||||
"PersonKey": [100, 200],
|
||||
})
|
||||
result = patient_id(df)
|
||||
assert result["UPID"].tolist() == ["AB100", "X200"]
|
||||
|
||||
def test_upid_preserves_other_columns(self, sample_patient_df: pd.DataFrame):
|
||||
"""Other columns should be preserved after UPID generation."""
|
||||
original_columns = sample_patient_df.columns.tolist()
|
||||
result = patient_id(sample_patient_df)
|
||||
|
||||
for col in original_columns:
|
||||
assert col in result.columns
|
||||
|
||||
def test_upid_same_patient_same_upid(self, sample_patient_df: pd.DataFrame):
|
||||
"""Same patient should have same UPID across rows."""
|
||||
result = patient_id(sample_patient_df)
|
||||
# First and last rows have same Provider Code and PersonKey
|
||||
assert result.iloc[0]["UPID"] == result.iloc[3]["UPID"]
|
||||
|
||||
def test_upid_different_patients_different_upids(self, sample_patient_df: pd.DataFrame):
|
||||
"""Different patients should have different UPIDs."""
|
||||
result = patient_id(sample_patient_df)
|
||||
unique_upids = result["UPID"].nunique()
|
||||
# We have 3 unique patients (rows 0 and 3 are same patient)
|
||||
assert unique_upids == 3
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Tests for drug_names()
|
||||
# ============================================================================
|
||||
|
||||
class TestDrugNames:
|
||||
"""Test drug name standardization."""
|
||||
|
||||
def test_drug_names_mapped(self, sample_drug_df: pd.DataFrame, test_paths: PathConfig):
|
||||
"""Drug names should be mapped to standard names."""
|
||||
result = drug_names(sample_drug_df, paths=test_paths)
|
||||
|
||||
# First drug should map to ABATACEPT (note: '250MG POWDER' is in the mapping)
|
||||
assert result.iloc[0]["Drug Name"] == "ABATACEPT"
|
||||
|
||||
def test_drug_names_uppercase(self, sample_drug_df: pd.DataFrame, test_paths: PathConfig):
|
||||
"""Drug names should be converted to uppercase before mapping."""
|
||||
result = drug_names(sample_drug_df, paths=test_paths)
|
||||
|
||||
# 'adalimumab (homecare)' should become 'ADALIMUMAB'
|
||||
assert result.iloc[1]["Drug Name"] == "ADALIMUMAB"
|
||||
|
||||
def test_left_eye_removed(self, sample_drug_df: pd.DataFrame, test_paths: PathConfig):
|
||||
"""(LEFT EYE) suffix should be removed."""
|
||||
result = drug_names(sample_drug_df, paths=test_paths)
|
||||
|
||||
# 'ETANERCEPT (LEFT EYE)' should become 'ETANERCEPT'
|
||||
assert result.iloc[2]["Drug Name"] == "ETANERCEPT"
|
||||
assert "(LEFT EYE)" not in result.iloc[2]["Drug Name"]
|
||||
|
||||
def test_right_eye_removed(self, sample_drug_df: pd.DataFrame, test_paths: PathConfig):
|
||||
"""(RIGHT EYE) suffix should be removed."""
|
||||
result = drug_names(sample_drug_df, paths=test_paths)
|
||||
|
||||
# 'infliximab (RIGHT EYE)' should become 'INFLIXIMAB'
|
||||
assert result.iloc[3]["Drug Name"] == "INFLIXIMAB"
|
||||
assert "(RIGHT EYE)" not in result.iloc[3]["Drug Name"]
|
||||
|
||||
def test_unknown_drug_mapped_to_nan(self, sample_drug_df: pd.DataFrame, test_paths: PathConfig):
|
||||
"""Unknown drugs (not in mapping) should map to NaN."""
|
||||
result = drug_names(sample_drug_df, paths=test_paths)
|
||||
|
||||
# 'Unknown Drug' is not in drugnames.csv mapping
|
||||
assert pd.isna(result.iloc[4]["Drug Name"])
|
||||
|
||||
def test_preserves_other_columns(self, sample_drug_df: pd.DataFrame, test_paths: PathConfig):
|
||||
"""Other columns should be preserved."""
|
||||
original_columns = sample_drug_df.columns.tolist()
|
||||
result = drug_names(sample_drug_df, paths=test_paths)
|
||||
|
||||
for col in original_columns:
|
||||
assert col in result.columns
|
||||
|
||||
def test_drug_name_stripped(self, sample_drug_df: pd.DataFrame, test_paths: PathConfig):
|
||||
"""Drug names should be stripped of whitespace."""
|
||||
result = drug_names(sample_drug_df, paths=test_paths)
|
||||
|
||||
for name in result["Drug Name"].dropna():
|
||||
assert name == name.strip()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Tests for department_identification()
|
||||
# ============================================================================
|
||||
|
||||
class TestDepartmentIdentification:
|
||||
"""Test directory assignment with fallback chain."""
|
||||
|
||||
@pytest.fixture
|
||||
def department_test_df(self) -> pd.DataFrame:
|
||||
"""Create DataFrame for department identification tests."""
|
||||
return pd.DataFrame({
|
||||
"UPID": ["RXA1001", "RXA1001", "RXB2002", "RXC3003", "RXD4004"],
|
||||
"Drug Name": ["RITUXIMAB", "RITUXIMAB", "ADALIMUMAB", "ADALIMUMAB", "UNKNOWN"],
|
||||
"Provider Code": ["RXA", "RXA", "RXB", "RXC", "RXD"],
|
||||
"PersonKey": [1001, 1001, 2002, 3003, 4004],
|
||||
"Treatment Function Code": [410, 410, 330, np.nan, np.nan],
|
||||
"Additional Detail 1": ["RHEUMATOLOGY referral", np.nan, "DERMATOLOGY clinic", np.nan, np.nan],
|
||||
"Additional Description 1": [np.nan, np.nan, np.nan, "GASTRO ward", np.nan],
|
||||
"Additional Detail 2": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 2": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Detail 3": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 3": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Detail 4": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 4": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Detail 5": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 5": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
"Treatment Function Desc": [np.nan, np.nan, np.nan, np.nan, np.nan],
|
||||
})
|
||||
|
||||
def test_directory_column_created(
|
||||
self, department_test_df: pd.DataFrame, test_paths: PathConfig
|
||||
):
|
||||
"""Directory column should be created."""
|
||||
result = department_identification(department_test_df, paths=test_paths)
|
||||
assert "Directory" in result.columns
|
||||
|
||||
def test_directory_source_column_created(
|
||||
self, department_test_df: pd.DataFrame, test_paths: PathConfig
|
||||
):
|
||||
"""Directory_Source column should be created to track assignment method."""
|
||||
result = department_identification(department_test_df, paths=test_paths)
|
||||
assert "Directory_Source" in result.columns
|
||||
|
||||
def test_single_valid_directory_assigned(
|
||||
self, department_test_df: pd.DataFrame, test_paths: PathConfig
|
||||
):
|
||||
"""Drug with single valid directory should get that directory."""
|
||||
result = department_identification(department_test_df, paths=test_paths)
|
||||
|
||||
# RITUXIMAB has only one valid directory (CLINICAL HAEMATOLOGY)
|
||||
rituximab_rows = result[result["Drug Name"] == "RITUXIMAB"]
|
||||
for _, row in rituximab_rows.iterrows():
|
||||
assert row["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
assert row["Directory_Source"] == "SINGLE_VALID_DIR"
|
||||
|
||||
def test_undefined_for_unknown_drug(
|
||||
self, department_test_df: pd.DataFrame, test_paths: PathConfig
|
||||
):
|
||||
"""Unknown drug should get 'Undefined' directory."""
|
||||
result = department_identification(department_test_df, paths=test_paths)
|
||||
|
||||
# UNKNOWN drug is not in drug_directory_list
|
||||
unknown_rows = result[result["Drug Name"] == "UNKNOWN"]
|
||||
for _, row in unknown_rows.iterrows():
|
||||
assert row["Directory"] == "Undefined"
|
||||
assert row["Directory_Source"] == "UNDEFINED"
|
||||
|
||||
def test_no_duplicate_columns(
|
||||
self, department_test_df: pd.DataFrame, test_paths: PathConfig
|
||||
):
|
||||
"""No duplicate columns should be created."""
|
||||
result = department_identification(department_test_df, paths=test_paths)
|
||||
|
||||
column_counts = result.columns.value_counts()
|
||||
duplicates = column_counts[column_counts > 1]
|
||||
assert duplicates.empty, f"Duplicate columns found: {duplicates.index.tolist()}"
|
||||
|
||||
def test_handles_missing_upid(self, test_paths: PathConfig):
|
||||
"""Rows with missing UPID should be dropped."""
|
||||
df = pd.DataFrame({
|
||||
"UPID": ["RXA1001", "", np.nan, "RXB2002"],
|
||||
"Drug Name": ["RITUXIMAB", "RITUXIMAB", "RITUXIMAB", "RITUXIMAB"],
|
||||
"Provider Code": ["RXA", "RXA", "RXA", "RXB"],
|
||||
"PersonKey": [1001, 1002, 1003, 2002],
|
||||
"Treatment Function Code": [410, 410, 410, 410],
|
||||
"Additional Detail 1": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 1": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Detail 2": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 2": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Detail 3": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 3": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Detail 4": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 4": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Detail 5": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Additional Description 5": [np.nan, np.nan, np.nan, np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan, np.nan, np.nan, np.nan],
|
||||
"Treatment Function Desc": [np.nan, np.nan, np.nan, np.nan],
|
||||
})
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# Should only have 2 rows with valid UPIDs
|
||||
assert len(result) == 2
|
||||
assert "RXA1001" in result["UPID"].values
|
||||
assert "RXB2002" in result["UPID"].values
|
||||
|
||||
|
||||
class TestDepartmentIdentificationDirectorySources:
|
||||
"""Test that Directory_Source values are correctly assigned."""
|
||||
|
||||
@pytest.fixture
|
||||
def single_dir_df(self) -> pd.DataFrame:
|
||||
"""DataFrame for testing single valid directory assignment."""
|
||||
return pd.DataFrame({
|
||||
"UPID": ["RXA1001"],
|
||||
"Drug Name": ["RITUXIMAB"], # Has only CLINICAL HAEMATOLOGY
|
||||
"Provider Code": ["RXA"],
|
||||
"PersonKey": [1001],
|
||||
"Treatment Function Code": [np.nan],
|
||||
"Additional Detail 1": [np.nan],
|
||||
"Additional Description 1": [np.nan],
|
||||
"Additional Detail 2": [np.nan],
|
||||
"Additional Description 2": [np.nan],
|
||||
"Additional Detail 3": [np.nan],
|
||||
"Additional Description 3": [np.nan],
|
||||
"Additional Detail 4": [np.nan],
|
||||
"Additional Description 4": [np.nan],
|
||||
"Additional Detail 5": [np.nan],
|
||||
"Additional Description 5": [np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan],
|
||||
"Treatment Function Desc": [np.nan],
|
||||
})
|
||||
|
||||
def test_single_valid_dir_source(
|
||||
self, single_dir_df: pd.DataFrame, test_paths: PathConfig
|
||||
):
|
||||
"""SINGLE_VALID_DIR source should be assigned when drug has one directory."""
|
||||
result = department_identification(single_dir_df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
assert result.iloc[0]["Directory_Source"] == "SINGLE_VALID_DIR"
|
||||
|
||||
def test_undefined_source(self, test_paths: PathConfig):
|
||||
"""UNDEFINED source should be assigned when no directory can be determined."""
|
||||
df = pd.DataFrame({
|
||||
"UPID": ["RXA1001"],
|
||||
"Drug Name": ["NONEXISTENT"], # Not in drug_directory_list
|
||||
"Provider Code": ["RXA"],
|
||||
"PersonKey": [1001],
|
||||
"Treatment Function Code": [np.nan],
|
||||
"Additional Detail 1": [np.nan],
|
||||
"Additional Description 1": [np.nan],
|
||||
"Additional Detail 2": [np.nan],
|
||||
"Additional Description 2": [np.nan],
|
||||
"Additional Detail 3": [np.nan],
|
||||
"Additional Description 3": [np.nan],
|
||||
"Additional Detail 4": [np.nan],
|
||||
"Additional Description 4": [np.nan],
|
||||
"Additional Detail 5": [np.nan],
|
||||
"Additional Description 5": [np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan],
|
||||
"Treatment Function Desc": [np.nan],
|
||||
})
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "Undefined"
|
||||
assert result.iloc[0]["Directory_Source"] == "UNDEFINED"
|
||||
|
||||
|
||||
class TestDepartmentIdentificationEdgeCases:
|
||||
"""Test edge cases in department identification."""
|
||||
|
||||
def test_empty_dataframe(self, test_paths: PathConfig):
|
||||
"""Empty DataFrame should return empty DataFrame with required columns."""
|
||||
df = pd.DataFrame(columns=[
|
||||
"UPID", "Drug Name", "Provider Code", "PersonKey",
|
||||
"Treatment Function Code", "Additional Detail 1",
|
||||
"Additional Description 1", "Additional Detail 2",
|
||||
"Additional Description 2", "Additional Detail 3",
|
||||
"Additional Description 3", "Additional Detail 4",
|
||||
"Additional Description 4", "Additional Detail 5",
|
||||
"Additional Description 5", "NCDR Treatment Function Name",
|
||||
"Treatment Function Desc"
|
||||
])
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert len(result) == 0
|
||||
assert "Directory" in result.columns
|
||||
assert "Directory_Source" in result.columns
|
||||
|
||||
def test_all_same_patient_different_drugs(self, test_paths: PathConfig):
|
||||
"""Same patient with different drugs should get appropriate directories."""
|
||||
df = pd.DataFrame({
|
||||
"UPID": ["RXA1001", "RXA1001", "RXA1001"],
|
||||
"Drug Name": ["RITUXIMAB", "ADALIMUMAB", "ETANERCEPT"],
|
||||
"Provider Code": ["RXA", "RXA", "RXA"],
|
||||
"PersonKey": [1001, 1001, 1001],
|
||||
"Treatment Function Code": [np.nan, np.nan, np.nan],
|
||||
"Additional Detail 1": [np.nan, "DERMATOLOGY", np.nan],
|
||||
"Additional Description 1": [np.nan, np.nan, np.nan],
|
||||
"Additional Detail 2": [np.nan, np.nan, np.nan],
|
||||
"Additional Description 2": [np.nan, np.nan, np.nan],
|
||||
"Additional Detail 3": [np.nan, np.nan, np.nan],
|
||||
"Additional Description 3": [np.nan, np.nan, np.nan],
|
||||
"Additional Detail 4": [np.nan, np.nan, np.nan],
|
||||
"Additional Description 4": [np.nan, np.nan, np.nan],
|
||||
"Additional Detail 5": [np.nan, np.nan, np.nan],
|
||||
"Additional Description 5": [np.nan, np.nan, np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan, np.nan, np.nan],
|
||||
"Treatment Function Desc": [np.nan, np.nan, np.nan],
|
||||
})
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# RITUXIMAB should get CLINICAL HAEMATOLOGY (single valid dir)
|
||||
rituximab = result[result["Drug Name"] == "RITUXIMAB"]
|
||||
assert rituximab.iloc[0]["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
|
||||
# ADALIMUMAB has DERMATOLOGY extracted but DERMATOLOGY is a valid dir
|
||||
# The fallback chain uses CALCULATED_MOST_FREQ which picks the most frequent
|
||||
# valid directory from extracted sources. Since the extracted dir matches
|
||||
# a valid dir for ADALIMUMAB, it should use DERMATOLOGY.
|
||||
# However, UPID_INFERENCE may override this if another directory is more
|
||||
# frequent for this patient overall.
|
||||
adalimumab = result[result["Drug Name"] == "ADALIMUMAB"]
|
||||
# The directory should be valid for ADALIMUMAB
|
||||
valid_adalimumab_dirs = {"RHEUMATOLOGY", "GASTROENTEROLOGY", "DERMATOLOGY", "OPHTHALMOLOGY"}
|
||||
assert adalimumab.iloc[0]["Directory"] in valid_adalimumab_dirs or adalimumab.iloc[0]["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Tests for directory assignment fallback levels
|
||||
# ============================================================================
|
||||
|
||||
class TestDirectoryAssignmentFallbackLevels:
|
||||
"""
|
||||
Comprehensive tests for the 5-level fallback chain in department_identification().
|
||||
|
||||
Fallback levels:
|
||||
1. SINGLE_VALID_DIR: Drug has only one valid directory
|
||||
2. EXTRACTED_PRIMARY/EXTRACTED_FALLBACK: Extracted from Additional Detail columns
|
||||
3. CALCULATED_MOST_FREQ: Most frequent valid directory for UPID/Drug
|
||||
4. UPID_INFERENCE: Infer from most frequent directory for same UPID
|
||||
5. UNDEFINED: No directory could be determined
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def create_test_df(
|
||||
upids: list,
|
||||
drug_names: list,
|
||||
treatment_codes: list = None,
|
||||
additional_detail_1: list = None,
|
||||
) -> pd.DataFrame:
|
||||
"""Helper to create test DataFrames with required columns."""
|
||||
n = len(upids)
|
||||
df = pd.DataFrame({
|
||||
"UPID": upids,
|
||||
"Drug Name": drug_names,
|
||||
"Provider Code": ["RXA"] * n,
|
||||
"PersonKey": list(range(1001, 1001 + n)),
|
||||
"Treatment Function Code": treatment_codes if treatment_codes else [np.nan] * n,
|
||||
"Additional Detail 1": additional_detail_1 if additional_detail_1 else [np.nan] * n,
|
||||
"Additional Description 1": [np.nan] * n,
|
||||
"Additional Detail 2": [np.nan] * n,
|
||||
"Additional Description 2": [np.nan] * n,
|
||||
"Additional Detail 3": [np.nan] * n,
|
||||
"Additional Description 3": [np.nan] * n,
|
||||
"Additional Detail 4": [np.nan] * n,
|
||||
"Additional Description 4": [np.nan] * n,
|
||||
"Additional Detail 5": [np.nan] * n,
|
||||
"Additional Description 5": [np.nan] * n,
|
||||
"NCDR Treatment Function Name": [np.nan] * n,
|
||||
"Treatment Function Desc": [np.nan] * n,
|
||||
})
|
||||
return df
|
||||
|
||||
def test_level1_single_valid_dir_takes_precedence(self, test_paths: PathConfig):
|
||||
"""Level 1: Single valid directory should override all other sources."""
|
||||
# RITUXIMAB only has CLINICAL HAEMATOLOGY, even with DERMATOLOGY in Additional Detail
|
||||
df = self.create_test_df(
|
||||
upids=["RXA1001"],
|
||||
drug_names=["RITUXIMAB"],
|
||||
additional_detail_1=["DERMATOLOGY clinic"], # This should be ignored
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
assert result.iloc[0]["Directory_Source"] == "SINGLE_VALID_DIR"
|
||||
|
||||
def test_level2_extracted_from_additional_detail(self, test_paths: PathConfig):
|
||||
"""Level 2: Directory extracted from Additional Detail columns for multi-dir drugs."""
|
||||
# ADALIMUMAB has multiple valid dirs, so extraction should work
|
||||
df = self.create_test_df(
|
||||
upids=["RXA1001"],
|
||||
drug_names=["ADALIMUMAB"],
|
||||
additional_detail_1=["DERMATOLOGY referral"],
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# Should extract DERMATOLOGY from Additional Detail 1
|
||||
assert result.iloc[0]["Directory"] == "DERMATOLOGY"
|
||||
# Source should indicate calculated from most frequent (which uses the extracted value)
|
||||
assert result.iloc[0]["Directory_Source"] == "CALCULATED_MOST_FREQ"
|
||||
|
||||
def test_level2_extracted_from_treatment_function_code(self, test_paths: PathConfig):
|
||||
"""Level 2: Directory extracted from Treatment Function Code when no detail available."""
|
||||
# ADALIMUMAB with treatment function code 410 = RHEUMATOLOGY
|
||||
df = self.create_test_df(
|
||||
upids=["RXA1001"],
|
||||
drug_names=["ADALIMUMAB"],
|
||||
treatment_codes=[410], # Maps to RHEUMATOLOGY
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# Should get RHEUMATOLOGY from treatment function code
|
||||
assert result.iloc[0]["Directory"] == "RHEUMATOLOGY"
|
||||
assert result.iloc[0]["Directory_Source"] == "CALCULATED_MOST_FREQ"
|
||||
|
||||
def test_level3_calculated_most_freq_with_multiple_records(self, test_paths: PathConfig):
|
||||
"""Level 3: Most frequent valid directory wins when patient has multiple records."""
|
||||
# Same UPID, same drug, different extracted directories
|
||||
# ADALIMUMAB can be RHEUMATOLOGY, DERMATOLOGY, GASTROENTEROLOGY, OPHTHALMOLOGY
|
||||
df = self.create_test_df(
|
||||
upids=["RXA1001", "RXA1001", "RXA1001", "RXA1001", "RXA1001"],
|
||||
drug_names=["ADALIMUMAB"] * 5,
|
||||
additional_detail_1=[
|
||||
"RHEUMATOLOGY",
|
||||
"RHEUMATOLOGY",
|
||||
"RHEUMATOLOGY",
|
||||
"DERMATOLOGY",
|
||||
"GASTROENTEROLOGY",
|
||||
],
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# RHEUMATOLOGY appears 3 times, should win
|
||||
for _, row in result.iterrows():
|
||||
assert row["Directory"] == "RHEUMATOLOGY"
|
||||
assert row["Directory_Source"] == "CALCULATED_MOST_FREQ"
|
||||
|
||||
def test_level3_ignores_invalid_directories_in_frequency(self, test_paths: PathConfig):
|
||||
"""Level 3: Invalid directories should be ignored in frequency calculation."""
|
||||
# ETANERCEPT only valid for RHEUMATOLOGY and DERMATOLOGY
|
||||
# Even if GASTROENTEROLOGY appears more often, it should be ignored
|
||||
df = self.create_test_df(
|
||||
upids=["RXA1001", "RXA1001", "RXA1001", "RXA1001"],
|
||||
drug_names=["ETANERCEPT"] * 4,
|
||||
additional_detail_1=[
|
||||
"GASTROENTEROLOGY", # Invalid for ETANERCEPT
|
||||
"GASTROENTEROLOGY", # Invalid for ETANERCEPT
|
||||
"GASTROENTEROLOGY", # Invalid for ETANERCEPT
|
||||
"RHEUMATOLOGY", # Valid
|
||||
],
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# RHEUMATOLOGY should win as it's the only valid directory
|
||||
for _, row in result.iterrows():
|
||||
assert row["Directory"] == "RHEUMATOLOGY"
|
||||
|
||||
def test_level4_upid_inference(self, test_paths: PathConfig):
|
||||
"""Level 4: UPID inference when no valid directory found from extraction."""
|
||||
# Same UPID, one drug has directory (RITUXIMAB → CLINICAL HAEMATOLOGY)
|
||||
# Other drug (ADALIMUMAB) has no extractable directory
|
||||
# Note: ADALIMUMAB cannot use CLINICAL HAEMATOLOGY as it's not valid for it
|
||||
# So this tests the case where UPID_INFERENCE may not help if the inferred
|
||||
# directory isn't valid for the drug
|
||||
|
||||
# Better test: Two different patients, one has known directory
|
||||
# Actually, UPID_INFERENCE doesn't check validity - it just uses most frequent
|
||||
df = pd.DataFrame({
|
||||
"UPID": ["RXA1001", "RXA1001"],
|
||||
"Drug Name": ["RITUXIMAB", "UNKNOWN_DRUG"], # UNKNOWN has no mapping
|
||||
"Provider Code": ["RXA", "RXA"],
|
||||
"PersonKey": [1001, 1001],
|
||||
"Treatment Function Code": [np.nan, np.nan],
|
||||
"Additional Detail 1": [np.nan, np.nan],
|
||||
"Additional Description 1": [np.nan, np.nan],
|
||||
"Additional Detail 2": [np.nan, np.nan],
|
||||
"Additional Description 2": [np.nan, np.nan],
|
||||
"Additional Detail 3": [np.nan, np.nan],
|
||||
"Additional Description 3": [np.nan, np.nan],
|
||||
"Additional Detail 4": [np.nan, np.nan],
|
||||
"Additional Description 4": [np.nan, np.nan],
|
||||
"Additional Detail 5": [np.nan, np.nan],
|
||||
"Additional Description 5": [np.nan, np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan, np.nan],
|
||||
"Treatment Function Desc": [np.nan, np.nan],
|
||||
})
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# RITUXIMAB gets CLINICAL HAEMATOLOGY (single valid dir)
|
||||
rituximab = result[result["Drug Name"] == "RITUXIMAB"]
|
||||
assert rituximab.iloc[0]["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
assert rituximab.iloc[0]["Directory_Source"] == "SINGLE_VALID_DIR"
|
||||
|
||||
# UNKNOWN_DRUG should inherit CLINICAL HAEMATOLOGY via UPID_INFERENCE
|
||||
unknown = result[result["Drug Name"] == "UNKNOWN_DRUG"]
|
||||
assert unknown.iloc[0]["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
assert unknown.iloc[0]["Directory_Source"] == "UPID_INFERENCE"
|
||||
|
||||
def test_level5_undefined_when_no_fallback_available(self, test_paths: PathConfig):
|
||||
"""Level 5: UNDEFINED when all fallback levels fail."""
|
||||
# Unknown drug, no additional detail, alone in UPID
|
||||
df = self.create_test_df(
|
||||
upids=["RXZ9999"], # Unique UPID with no other records
|
||||
drug_names=["NONEXISTENT_DRUG"],
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "Undefined"
|
||||
assert result.iloc[0]["Directory_Source"] == "UNDEFINED"
|
||||
|
||||
|
||||
class TestDirectoryAssignmentTreatmentFunctionCode:
|
||||
"""Tests for Treatment Function Code extraction in directory assignment."""
|
||||
|
||||
@staticmethod
|
||||
def create_tfc_test_df(
|
||||
upids: list,
|
||||
drug_names: list,
|
||||
treatment_codes: list,
|
||||
) -> pd.DataFrame:
|
||||
"""Create test DataFrame with Treatment Function Codes."""
|
||||
n = len(upids)
|
||||
return pd.DataFrame({
|
||||
"UPID": upids,
|
||||
"Drug Name": drug_names,
|
||||
"Provider Code": ["RXA"] * n,
|
||||
"PersonKey": list(range(1001, 1001 + n)),
|
||||
"Treatment Function Code": treatment_codes,
|
||||
"Additional Detail 1": [np.nan] * n,
|
||||
"Additional Description 1": [np.nan] * n,
|
||||
"Additional Detail 2": [np.nan] * n,
|
||||
"Additional Description 2": [np.nan] * n,
|
||||
"Additional Detail 3": [np.nan] * n,
|
||||
"Additional Description 3": [np.nan] * n,
|
||||
"Additional Detail 4": [np.nan] * n,
|
||||
"Additional Description 4": [np.nan] * n,
|
||||
"Additional Detail 5": [np.nan] * n,
|
||||
"Additional Description 5": [np.nan] * n,
|
||||
"NCDR Treatment Function Name": [np.nan] * n,
|
||||
"Treatment Function Desc": [np.nan] * n,
|
||||
})
|
||||
|
||||
def test_tfc_410_maps_to_rheumatology(self, test_paths: PathConfig):
|
||||
"""Treatment Function Code 410 should map to RHEUMATOLOGY."""
|
||||
df = self.create_tfc_test_df(
|
||||
upids=["RXA1001"],
|
||||
drug_names=["ADALIMUMAB"], # Valid for RHEUMATOLOGY
|
||||
treatment_codes=[410],
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "RHEUMATOLOGY"
|
||||
|
||||
def test_tfc_330_maps_to_dermatology(self, test_paths: PathConfig):
|
||||
"""Treatment Function Code 330 should map to DERMATOLOGY."""
|
||||
df = self.create_tfc_test_df(
|
||||
upids=["RXA1001"],
|
||||
drug_names=["ADALIMUMAB"], # Valid for DERMATOLOGY
|
||||
treatment_codes=[330],
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "DERMATOLOGY"
|
||||
|
||||
def test_tfc_invalid_code_ignored(self, test_paths: PathConfig):
|
||||
"""Invalid Treatment Function Code should result in no extraction."""
|
||||
df = self.create_tfc_test_df(
|
||||
upids=["RXA1001"],
|
||||
drug_names=["ADALIMUMAB"],
|
||||
treatment_codes=[999], # Invalid code
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# Should fall through to UNDEFINED since code doesn't map to valid directory
|
||||
assert result.iloc[0]["Directory"] == "Undefined"
|
||||
assert result.iloc[0]["Directory_Source"] == "UNDEFINED"
|
||||
|
||||
def test_tfc_with_nan_treated_as_zero(self, test_paths: PathConfig):
|
||||
"""NaN Treatment Function Code should be treated as 0 (invalid)."""
|
||||
df = self.create_tfc_test_df(
|
||||
upids=["RXA1001"],
|
||||
drug_names=["UNKNOWN_DRUG"],
|
||||
treatment_codes=[np.nan],
|
||||
)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# Should fall through to UNDEFINED
|
||||
assert result.iloc[0]["Directory"] == "Undefined"
|
||||
|
||||
|
||||
class TestDirectoryAssignmentMultiplePatients:
|
||||
"""Tests for directory assignment with multiple patients."""
|
||||
|
||||
@staticmethod
|
||||
def create_multi_patient_df(
|
||||
data: list[tuple], # [(upid, drug, additional_detail)]
|
||||
) -> pd.DataFrame:
|
||||
"""Create test DataFrame for multiple patients."""
|
||||
n = len(data)
|
||||
return pd.DataFrame({
|
||||
"UPID": [d[0] for d in data],
|
||||
"Drug Name": [d[1] for d in data],
|
||||
"Provider Code": ["RXA"] * n,
|
||||
"PersonKey": list(range(1001, 1001 + n)),
|
||||
"Treatment Function Code": [np.nan] * n,
|
||||
"Additional Detail 1": [d[2] if len(d) > 2 else np.nan for d in data],
|
||||
"Additional Description 1": [np.nan] * n,
|
||||
"Additional Detail 2": [np.nan] * n,
|
||||
"Additional Description 2": [np.nan] * n,
|
||||
"Additional Detail 3": [np.nan] * n,
|
||||
"Additional Description 3": [np.nan] * n,
|
||||
"Additional Detail 4": [np.nan] * n,
|
||||
"Additional Description 4": [np.nan] * n,
|
||||
"Additional Detail 5": [np.nan] * n,
|
||||
"Additional Description 5": [np.nan] * n,
|
||||
"NCDR Treatment Function Name": [np.nan] * n,
|
||||
"Treatment Function Desc": [np.nan] * n,
|
||||
})
|
||||
|
||||
def test_different_patients_get_different_directories(self, test_paths: PathConfig):
|
||||
"""Different patients should get directories based on their own data."""
|
||||
data = [
|
||||
("RXA1001", "ADALIMUMAB", "DERMATOLOGY"),
|
||||
("RXA1002", "ADALIMUMAB", "RHEUMATOLOGY"),
|
||||
]
|
||||
df = self.create_multi_patient_df(data)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
patient1 = result[result["UPID"] == "RXA1001"]
|
||||
patient2 = result[result["UPID"] == "RXA1002"]
|
||||
|
||||
assert patient1.iloc[0]["Directory"] == "DERMATOLOGY"
|
||||
assert patient2.iloc[0]["Directory"] == "RHEUMATOLOGY"
|
||||
|
||||
def test_upid_inference_does_not_cross_patients(self, test_paths: PathConfig):
|
||||
"""UPID inference should not apply directories from other patients."""
|
||||
data = [
|
||||
("RXA1001", "RITUXIMAB", np.nan), # Gets CLINICAL HAEMATOLOGY (single dir)
|
||||
("RXA1002", "UNKNOWN_DRUG", np.nan), # Should NOT inherit from RXA1001
|
||||
]
|
||||
df = self.create_multi_patient_df(data)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
patient1 = result[result["UPID"] == "RXA1001"]
|
||||
patient2 = result[result["UPID"] == "RXA1002"]
|
||||
|
||||
assert patient1.iloc[0]["Directory"] == "CLINICAL HAEMATOLOGY"
|
||||
# Patient 2 should be UNDEFINED, not inherit from patient 1
|
||||
assert patient2.iloc[0]["Directory"] == "Undefined"
|
||||
assert patient2.iloc[0]["Directory_Source"] == "UNDEFINED"
|
||||
|
||||
def test_same_drug_different_patients_independent(self, test_paths: PathConfig):
|
||||
"""Same drug for different patients should be processed independently."""
|
||||
data = [
|
||||
("RXA1001", "ETANERCEPT", "DERMATOLOGY"),
|
||||
("RXA1001", "ETANERCEPT", "DERMATOLOGY"),
|
||||
("RXA1002", "ETANERCEPT", "RHEUMATOLOGY"),
|
||||
("RXA1002", "ETANERCEPT", "RHEUMATOLOGY"),
|
||||
]
|
||||
df = self.create_multi_patient_df(data)
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
patient1 = result[result["UPID"] == "RXA1001"]
|
||||
patient2 = result[result["UPID"] == "RXA1002"]
|
||||
|
||||
# Each patient should get their most frequent directory
|
||||
for _, row in patient1.iterrows():
|
||||
assert row["Directory"] == "DERMATOLOGY"
|
||||
for _, row in patient2.iterrows():
|
||||
assert row["Directory"] == "RHEUMATOLOGY"
|
||||
|
||||
|
||||
class TestDirectoryAssignmentExtractionPatterns:
|
||||
"""Tests for directory extraction patterns from text fields."""
|
||||
|
||||
@staticmethod
|
||||
def create_extraction_df(additional_detail: str, drug: str = "ADALIMUMAB") -> pd.DataFrame:
|
||||
"""Create a minimal DataFrame for testing extraction patterns."""
|
||||
return pd.DataFrame({
|
||||
"UPID": ["RXA1001"],
|
||||
"Drug Name": [drug],
|
||||
"Provider Code": ["RXA"],
|
||||
"PersonKey": [1001],
|
||||
"Treatment Function Code": [np.nan],
|
||||
"Additional Detail 1": [additional_detail],
|
||||
"Additional Description 1": [np.nan],
|
||||
"Additional Detail 2": [np.nan],
|
||||
"Additional Description 2": [np.nan],
|
||||
"Additional Detail 3": [np.nan],
|
||||
"Additional Description 3": [np.nan],
|
||||
"Additional Detail 4": [np.nan],
|
||||
"Additional Description 4": [np.nan],
|
||||
"Additional Detail 5": [np.nan],
|
||||
"Additional Description 5": [np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan],
|
||||
"Treatment Function Desc": [np.nan],
|
||||
})
|
||||
|
||||
def test_extraction_case_insensitive(self, test_paths: PathConfig):
|
||||
"""Directory extraction should be case insensitive."""
|
||||
df = self.create_extraction_df("dermatology clinic")
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "DERMATOLOGY"
|
||||
|
||||
def test_extraction_with_surrounding_text(self, test_paths: PathConfig):
|
||||
"""Directory should be extracted from surrounding text."""
|
||||
df = self.create_extraction_df("Referral to RHEUMATOLOGY department for assessment")
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
assert result.iloc[0]["Directory"] == "RHEUMATOLOGY"
|
||||
|
||||
def test_extraction_word_boundary(self, test_paths: PathConfig):
|
||||
"""Directory extraction should respect word boundaries."""
|
||||
# Test that partial matches don't occur - "RHEUM" should not match "RHEUMATOLOGY"
|
||||
# Using ADALIMUMAB which is valid for RHEUMATOLOGY
|
||||
df = self.create_extraction_df("RHEUMATOLOGY clinic")
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# RHEUMATOLOGY should be extracted correctly
|
||||
assert result.iloc[0]["Directory"] == "RHEUMATOLOGY"
|
||||
|
||||
def test_extraction_multiple_directories_first_wins(self, test_paths: PathConfig):
|
||||
"""When multiple directories present, first valid one should be used."""
|
||||
# Note: The actual behavior depends on the regex - typically first match
|
||||
df = self.create_extraction_df("RHEUMATOLOGY and DERMATOLOGY referral")
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# First directory in the text should be extracted
|
||||
assert result.iloc[0]["Directory"] in ["RHEUMATOLOGY", "DERMATOLOGY"]
|
||||
|
||||
def test_extraction_from_additional_description(self, test_paths: PathConfig):
|
||||
"""Directory can be extracted from Additional Description columns too."""
|
||||
df = pd.DataFrame({
|
||||
"UPID": ["RXA1001"],
|
||||
"Drug Name": ["ADALIMUMAB"],
|
||||
"Provider Code": ["RXA"],
|
||||
"PersonKey": [1001],
|
||||
"Treatment Function Code": [np.nan],
|
||||
"Additional Detail 1": [np.nan],
|
||||
"Additional Description 1": ["GASTROENTEROLOGY ward"],
|
||||
"Additional Detail 2": [np.nan],
|
||||
"Additional Description 2": [np.nan],
|
||||
"Additional Detail 3": [np.nan],
|
||||
"Additional Description 3": [np.nan],
|
||||
"Additional Detail 4": [np.nan],
|
||||
"Additional Description 4": [np.nan],
|
||||
"Additional Detail 5": [np.nan],
|
||||
"Additional Description 5": [np.nan],
|
||||
"NCDR Treatment Function Name": [np.nan],
|
||||
"Treatment Function Desc": [np.nan],
|
||||
})
|
||||
|
||||
result = department_identification(df, paths=test_paths)
|
||||
|
||||
# The function processes Additional Detail 1 first, then Description 1, etc.
|
||||
# But the final Primary_Directory comes from Additional Detail 1 specifically
|
||||
# So this test may not extract from Description 1 directly
|
||||
# Let's verify the actual behavior
|
||||
# In the code, additional_detail_columns includes both Detail and Description
|
||||
# but Primary_Source comes specifically from Additional Detail 1
|
||||
# The extraction happens on all columns but Primary_Source only from Detail 1
|
||||
# So with Detail 1 as NaN, Primary_Source will be NaN
|
||||
# This may result in UNDEFINED
|
||||
assert result.iloc[0]["Directory"] in ["GASTROENTEROLOGY", "Undefined"]
|
||||
@@ -0,0 +1,446 @@
|
||||
"""
|
||||
Large dataset performance tests for the Patient Pathway Analysis tool.
|
||||
|
||||
This module tests the system's ability to handle realistic workloads:
|
||||
1. Full dataset analysis (all drugs, trusts, directories)
|
||||
2. Memory usage under load
|
||||
3. Scalability characteristics
|
||||
|
||||
Run with: python -m pytest tests/test_large_dataset_performance.py -v
|
||||
"""
|
||||
|
||||
import gc
|
||||
import time
|
||||
import tracemalloc
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
# Mark all tests in this module as large dataset tests
|
||||
pytestmark = pytest.mark.largedata
|
||||
|
||||
|
||||
class TestLargeDatasetPerformance:
|
||||
"""Performance tests with full dataset."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_paths(self):
|
||||
"""Set up paths and verify data exists."""
|
||||
from core import default_paths
|
||||
from data_processing import get_loader
|
||||
|
||||
# Check if database exists
|
||||
db_path = default_paths.data_dir / "pathways.db"
|
||||
if not db_path.exists():
|
||||
pytest.skip("SQLite database not found")
|
||||
|
||||
self.paths = default_paths
|
||||
self.loader = get_loader('sqlite')
|
||||
|
||||
# Load data once
|
||||
result = self.loader.load()
|
||||
if result is None or result.df is None or len(result.df) == 0:
|
||||
pytest.skip("No data available in database")
|
||||
|
||||
self.df = result.df
|
||||
self.row_count = result.row_count
|
||||
|
||||
def test_data_load_time_acceptable(self):
|
||||
"""Data loading should complete in under 5 seconds."""
|
||||
from data_processing import get_loader
|
||||
|
||||
gc.collect()
|
||||
start = time.perf_counter()
|
||||
loader = get_loader('sqlite')
|
||||
result = loader.load()
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
assert result is not None, "Data loading failed"
|
||||
assert result.row_count > 0, "No data loaded"
|
||||
# Allow 5 seconds for data loading
|
||||
assert elapsed < 5.0, f"Data loading took {elapsed:.2f}s (target: <5s)"
|
||||
|
||||
def test_analysis_pipeline_completes(self):
|
||||
"""Full analysis pipeline should complete without error."""
|
||||
from analysis.pathway_analyzer import generate_icicle_chart
|
||||
import pandas as pd
|
||||
|
||||
# Get available filters from actual data
|
||||
trusts = self.df['Provider Code'].unique().tolist()[:20]
|
||||
drugs = self.df['Drug Name'].dropna().unique().tolist()[:10]
|
||||
directories = self.df['Directory'].dropna().unique().tolist()
|
||||
|
||||
# Load org codes for trust name mapping
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
trust_names = []
|
||||
for t in trusts:
|
||||
if t in org_codes.index:
|
||||
trust_names.append(org_codes.loc[t, 'Name'])
|
||||
if not trust_names:
|
||||
trust_names = org_codes['Name'].tolist()[:20]
|
||||
|
||||
# Run analysis with reasonable filter
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=self.df,
|
||||
start_date="2020-01-01",
|
||||
end_date="2025-01-01",
|
||||
last_seen_date="2020-01-01",
|
||||
trust_filter=trust_names,
|
||||
drug_filter=drugs,
|
||||
directory_filter=directories,
|
||||
minimum_num_patients=1,
|
||||
title="Large Dataset Test",
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
# Should produce some results
|
||||
assert ice_df is not None, "Analysis produced no results"
|
||||
assert len(ice_df) > 0, "Analysis produced empty results"
|
||||
|
||||
def test_analysis_pipeline_time_acceptable(self):
|
||||
"""Analysis pipeline should complete in under 60 seconds."""
|
||||
from analysis.pathway_analyzer import generate_icicle_chart
|
||||
import pandas as pd
|
||||
|
||||
# Get available filters from actual data
|
||||
trusts = self.df['Provider Code'].unique().tolist()[:20]
|
||||
drugs = self.df['Drug Name'].dropna().unique().tolist()[:10]
|
||||
directories = self.df['Directory'].dropna().unique().tolist()
|
||||
|
||||
# Load org codes for trust name mapping
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
trust_names = []
|
||||
for t in trusts:
|
||||
if t in org_codes.index:
|
||||
trust_names.append(org_codes.loc[t, 'Name'])
|
||||
if not trust_names:
|
||||
trust_names = org_codes['Name'].tolist()[:20]
|
||||
|
||||
gc.collect()
|
||||
start = time.perf_counter()
|
||||
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=self.df,
|
||||
start_date="2020-01-01",
|
||||
end_date="2025-01-01",
|
||||
last_seen_date="2020-01-01",
|
||||
trust_filter=trust_names,
|
||||
drug_filter=drugs,
|
||||
directory_filter=directories,
|
||||
minimum_num_patients=1,
|
||||
title="Performance Test",
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
# Allow 60 seconds for full analysis (observed ~19s with 440K rows)
|
||||
assert elapsed < 60.0, f"Analysis took {elapsed:.2f}s (target: <60s)"
|
||||
print(f"\n Analysis completed in {elapsed:.2f}s with {len(ice_df) if ice_df is not None else 0} result rows")
|
||||
|
||||
def test_memory_usage_acceptable(self):
|
||||
"""Memory usage should not exceed 500MB during analysis."""
|
||||
from analysis.pathway_analyzer import generate_icicle_chart
|
||||
import pandas as pd
|
||||
|
||||
# Get available filters from actual data
|
||||
trusts = self.df['Provider Code'].unique().tolist()[:15]
|
||||
drugs = self.df['Drug Name'].dropna().unique().tolist()[:5]
|
||||
directories = self.df['Directory'].dropna().unique().tolist()
|
||||
|
||||
# Load org codes for trust name mapping
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
trust_names = []
|
||||
for t in trusts:
|
||||
if t in org_codes.index:
|
||||
trust_names.append(org_codes.loc[t, 'Name'])
|
||||
if not trust_names:
|
||||
trust_names = org_codes['Name'].tolist()[:15]
|
||||
|
||||
gc.collect()
|
||||
tracemalloc.start()
|
||||
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=self.df,
|
||||
start_date="2020-01-01",
|
||||
end_date="2025-01-01",
|
||||
last_seen_date="2020-01-01",
|
||||
trust_filter=trust_names,
|
||||
drug_filter=drugs,
|
||||
directory_filter=directories,
|
||||
minimum_num_patients=1,
|
||||
title="Memory Test",
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
current, peak = tracemalloc.get_traced_memory()
|
||||
tracemalloc.stop()
|
||||
|
||||
peak_mb = peak / 1024 / 1024
|
||||
|
||||
# Allow 500MB peak memory
|
||||
assert peak_mb < 500, f"Peak memory {peak_mb:.1f}MB exceeds 500MB limit"
|
||||
print(f"\n Peak memory usage: {peak_mb:.1f}MB")
|
||||
|
||||
def test_figure_creation_scales(self):
|
||||
"""Figure creation time should scale linearly with result size."""
|
||||
from visualization.plotly_generator import create_icicle_figure
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
|
||||
# Test with different sizes
|
||||
sizes = [100, 500, 1000, 2000]
|
||||
times = []
|
||||
|
||||
for n_rows in sizes:
|
||||
sample_data = {
|
||||
'parents': ['N&WICS'] * n_rows,
|
||||
'ids': [f'N&WICS - Test{i}' for i in range(n_rows)],
|
||||
'labels': [f'Test{i}' for i in range(n_rows)],
|
||||
'value': np.random.randint(1, 100, n_rows),
|
||||
'colour': np.random.random(n_rows),
|
||||
'cost': np.random.randint(1000, 100000, n_rows),
|
||||
'costpp': np.random.randint(100, 10000, n_rows),
|
||||
'cost_pp_pa': [str(np.random.randint(100, 10000)) for _ in range(n_rows)],
|
||||
'First seen': pd.to_datetime(['2024-01-01'] * n_rows),
|
||||
'Last seen': pd.to_datetime(['2024-12-31'] * n_rows),
|
||||
'First seen (Parent)': ['2024-01-01'] * n_rows,
|
||||
'Last seen (Parent)': ['2024-12-31'] * n_rows,
|
||||
'average_spacing': ['Test spacing'] * n_rows,
|
||||
'avg_days': pd.to_timedelta([100] * n_rows, unit='D'),
|
||||
}
|
||||
sample_df = pd.DataFrame(sample_data)
|
||||
|
||||
gc.collect()
|
||||
start = time.perf_counter()
|
||||
fig = create_icicle_figure(sample_df, f"Scale Test {n_rows}")
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
times.append(elapsed)
|
||||
|
||||
# Check that time scaling is roughly linear (not exponential)
|
||||
# If time doubles when size doubles, it's linear
|
||||
# We allow some variance, so check that 10x data doesn't take more than 20x time
|
||||
time_ratio = times[-1] / times[0]
|
||||
size_ratio = sizes[-1] / sizes[0]
|
||||
|
||||
# Allow 3x the expected linear scaling
|
||||
max_allowed_ratio = size_ratio * 3
|
||||
|
||||
assert time_ratio < max_allowed_ratio, (
|
||||
f"Figure creation doesn't scale well: "
|
||||
f"{sizes[-1]} rows took {times[-1]:.3f}s vs {sizes[0]} rows at {times[0]:.3f}s "
|
||||
f"(ratio {time_ratio:.1f}x, expected <{max_allowed_ratio:.1f}x)"
|
||||
)
|
||||
|
||||
print(f"\n Figure scaling: {sizes[0]} rows: {times[0]*1000:.1f}ms, "
|
||||
f"{sizes[-1]} rows: {times[-1]*1000:.1f}ms (ratio: {time_ratio:.1f}x)")
|
||||
|
||||
|
||||
class TestDataVolumeStress:
|
||||
"""Stress tests to verify system handles various data volumes."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_paths(self):
|
||||
"""Set up paths and verify data exists."""
|
||||
from core import default_paths
|
||||
from data_processing import get_loader
|
||||
|
||||
# Check if database exists
|
||||
db_path = default_paths.data_dir / "pathways.db"
|
||||
if not db_path.exists():
|
||||
pytest.skip("SQLite database not found")
|
||||
|
||||
self.paths = default_paths
|
||||
self.loader = get_loader('sqlite')
|
||||
|
||||
# Load data once
|
||||
result = self.loader.load()
|
||||
if result is None or result.df is None or len(result.df) == 0:
|
||||
pytest.skip("No data available in database")
|
||||
|
||||
self.df = result.df
|
||||
|
||||
def test_handles_all_drugs(self):
|
||||
"""Analysis can handle filtering by all drugs."""
|
||||
from analysis.pathway_analyzer import prepare_data
|
||||
import pandas as pd
|
||||
|
||||
all_drugs = self.df['Drug Name'].dropna().unique().tolist()
|
||||
|
||||
# Load org codes
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
trust_names = org_codes['Name'].tolist()[:5]
|
||||
|
||||
result = prepare_data(
|
||||
df=self.df,
|
||||
trust_filter=trust_names,
|
||||
drug_filter=all_drugs,
|
||||
directory_filter=self.df['Directory'].dropna().unique().tolist(),
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
# Should complete without error (returns tuple)
|
||||
assert result is not None
|
||||
assert len(result) == 3 # (df, org_codes, directory_df)
|
||||
|
||||
def test_handles_all_trusts(self):
|
||||
"""Analysis can handle filtering by all trusts."""
|
||||
from analysis.pathway_analyzer import prepare_data
|
||||
import pandas as pd
|
||||
|
||||
# Load org codes
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
all_trust_names = org_codes['Name'].tolist()
|
||||
|
||||
result = prepare_data(
|
||||
df=self.df,
|
||||
trust_filter=all_trust_names,
|
||||
drug_filter=['ADALIMUMAB', 'ETANERCEPT'],
|
||||
directory_filter=self.df['Directory'].dropna().unique().tolist(),
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
# Should complete without error (returns tuple)
|
||||
assert result is not None
|
||||
assert len(result) == 3 # (df, org_codes, directory_df)
|
||||
|
||||
def test_handles_wide_date_range(self):
|
||||
"""Analysis can handle a wide date range via generate_icicle_chart."""
|
||||
from analysis.pathway_analyzer import generate_icicle_chart
|
||||
import pandas as pd
|
||||
|
||||
# Load org codes
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
trust_names = org_codes['Name'].tolist()[:10]
|
||||
|
||||
# Use very wide date range via full pipeline
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=self.df,
|
||||
start_date="2010-01-01",
|
||||
end_date="2030-01-01",
|
||||
last_seen_date="2010-01-01",
|
||||
trust_filter=trust_names,
|
||||
drug_filter=self.df['Drug Name'].dropna().unique().tolist()[:5],
|
||||
directory_filter=self.df['Directory'].dropna().unique().tolist(),
|
||||
minimum_num_patients=1,
|
||||
title="Wide Date Range Test",
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
# Should complete without error
|
||||
assert ice_df is not None or ice_df is None # Just verifying no exception
|
||||
|
||||
def test_handles_minimum_patient_threshold(self):
|
||||
"""Analysis correctly applies minimum patient threshold."""
|
||||
from analysis.pathway_analyzer import generate_icicle_chart
|
||||
import pandas as pd
|
||||
|
||||
# Load org codes
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
trust_names = org_codes['Name'].tolist()[:10]
|
||||
|
||||
# Run with minimum 50 patients
|
||||
ice_df_50, _ = generate_icicle_chart(
|
||||
df=self.df,
|
||||
start_date="2020-01-01",
|
||||
end_date="2025-01-01",
|
||||
last_seen_date="2020-01-01",
|
||||
trust_filter=trust_names,
|
||||
drug_filter=self.df['Drug Name'].dropna().unique().tolist()[:5],
|
||||
directory_filter=self.df['Directory'].dropna().unique().tolist(),
|
||||
minimum_num_patients=50,
|
||||
title="Threshold Test 50",
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
# Run with minimum 1 patient
|
||||
ice_df_1, _ = generate_icicle_chart(
|
||||
df=self.df,
|
||||
start_date="2020-01-01",
|
||||
end_date="2025-01-01",
|
||||
last_seen_date="2020-01-01",
|
||||
trust_filter=trust_names,
|
||||
drug_filter=self.df['Drug Name'].dropna().unique().tolist()[:5],
|
||||
directory_filter=self.df['Directory'].dropna().unique().tolist(),
|
||||
minimum_num_patients=1,
|
||||
title="Threshold Test 1",
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
# Higher threshold should produce fewer or equal results
|
||||
len_50 = len(ice_df_50) if ice_df_50 is not None else 0
|
||||
len_1 = len(ice_df_1) if ice_df_1 is not None else 0
|
||||
|
||||
assert len_50 <= len_1, (
|
||||
f"Higher minimum threshold should produce fewer results: "
|
||||
f"min=50 gave {len_50} rows, min=1 gave {len_1} rows"
|
||||
)
|
||||
|
||||
|
||||
class TestConcurrentOperations:
|
||||
"""Tests for handling multiple operations."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_paths(self):
|
||||
"""Set up paths and verify data exists."""
|
||||
from core import default_paths
|
||||
from data_processing import get_loader
|
||||
|
||||
# Check if database exists
|
||||
db_path = default_paths.data_dir / "pathways.db"
|
||||
if not db_path.exists():
|
||||
pytest.skip("SQLite database not found")
|
||||
|
||||
self.paths = default_paths
|
||||
|
||||
def test_multiple_data_loads(self):
|
||||
"""Multiple data loads should not cause issues."""
|
||||
from data_processing import get_loader
|
||||
|
||||
results = []
|
||||
for i in range(3):
|
||||
loader = get_loader('sqlite')
|
||||
result = loader.load()
|
||||
if result is not None:
|
||||
results.append(result.row_count)
|
||||
|
||||
# All loads should return same row count
|
||||
assert len(set(results)) == 1, f"Inconsistent row counts: {results}"
|
||||
|
||||
def test_sequential_analyses(self):
|
||||
"""Multiple sequential analyses should complete."""
|
||||
from analysis.pathway_analyzer import generate_icicle_chart
|
||||
from data_processing import get_loader
|
||||
import pandas as pd
|
||||
|
||||
# Load data
|
||||
loader = get_loader('sqlite')
|
||||
result = loader.load()
|
||||
if result is None or result.df is None:
|
||||
pytest.skip("No data available")
|
||||
|
||||
df = result.df
|
||||
|
||||
# Load org codes
|
||||
org_codes = pd.read_csv(self.paths.org_codes_csv, index_col=1)
|
||||
trust_names = org_codes['Name'].tolist()[:5]
|
||||
|
||||
# Run multiple analyses
|
||||
for i in range(3):
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=df,
|
||||
start_date="2020-01-01",
|
||||
end_date="2025-01-01",
|
||||
last_seen_date="2020-01-01",
|
||||
trust_filter=trust_names,
|
||||
drug_filter=['ADALIMUMAB'],
|
||||
directory_filter=df['Directory'].dropna().unique().tolist(),
|
||||
minimum_num_patients=1,
|
||||
title=f"Sequential Test {i+1}",
|
||||
paths=self.paths,
|
||||
)
|
||||
|
||||
# Each should complete
|
||||
assert ice_df is not None or ice_df is None # Just check no error
|
||||
@@ -0,0 +1,373 @@
|
||||
"""
|
||||
Tests for core/models.py - AnalysisFilters dataclass.
|
||||
|
||||
Tests cover:
|
||||
- Basic instantiation
|
||||
- validate() method for filter validation
|
||||
- Property accessors (has_trust_filter, etc.)
|
||||
- title property (custom vs auto-generated)
|
||||
- summary() method
|
||||
"""
|
||||
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from core.models import AnalysisFilters
|
||||
|
||||
|
||||
class TestAnalysisFiltersBasic:
|
||||
"""Test basic AnalysisFilters instantiation and access."""
|
||||
|
||||
def test_create_with_required_dates(self, sample_date_range):
|
||||
"""Should be able to create AnalysisFilters with just dates."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert filters.start_date == start
|
||||
assert filters.end_date == end
|
||||
assert filters.last_seen_date == last_seen
|
||||
|
||||
def test_default_lists_are_empty(self, sample_date_range):
|
||||
"""Default filter lists should be empty."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert filters.trusts == []
|
||||
assert filters.drugs == []
|
||||
assert filters.directories == []
|
||||
|
||||
def test_default_minimum_patients_is_zero(self, sample_date_range):
|
||||
"""Default minimum_patients should be 0."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert filters.minimum_patients == 0
|
||||
|
||||
def test_default_custom_title_is_empty(self, sample_date_range):
|
||||
"""Default custom_title should be empty string."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert filters.custom_title == ""
|
||||
|
||||
|
||||
class TestAnalysisFiltersValidate:
|
||||
"""Test validate() method."""
|
||||
|
||||
def test_validate_passes_valid_config(self, sample_date_range):
|
||||
"""validate() should return empty list for valid configuration."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
errors = filters.validate()
|
||||
assert errors == []
|
||||
|
||||
def test_validate_fails_when_end_before_start(self):
|
||||
"""validate() should fail when end_date is before start_date."""
|
||||
filters = AnalysisFilters(
|
||||
start_date=date(2024, 12, 31), # Later
|
||||
end_date=date(2024, 1, 1), # Earlier
|
||||
last_seen_date=date(2024, 6, 1),
|
||||
)
|
||||
|
||||
errors = filters.validate()
|
||||
|
||||
assert len(errors) >= 1
|
||||
assert any("cannot be before start date" in e for e in errors)
|
||||
|
||||
def test_validate_fails_when_last_seen_after_end(self):
|
||||
"""validate() should fail when last_seen_date is after end_date."""
|
||||
filters = AnalysisFilters(
|
||||
start_date=date(2024, 1, 1),
|
||||
end_date=date(2024, 6, 1),
|
||||
last_seen_date=date(2024, 12, 31), # After end_date
|
||||
)
|
||||
|
||||
errors = filters.validate()
|
||||
|
||||
assert len(errors) >= 1
|
||||
assert any("would exclude all patients" in e for e in errors)
|
||||
|
||||
def test_validate_fails_when_minimum_patients_negative(self, sample_date_range):
|
||||
"""validate() should fail when minimum_patients is negative."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
minimum_patients=-1,
|
||||
)
|
||||
|
||||
errors = filters.validate()
|
||||
|
||||
assert len(errors) >= 1
|
||||
assert any("cannot be negative" in e for e in errors)
|
||||
|
||||
def test_validate_fails_when_output_dir_missing(self, sample_date_range, temp_dir: Path):
|
||||
"""validate() should fail when output_dir doesn't exist."""
|
||||
start, end, last_seen = sample_date_range
|
||||
nonexistent_dir = temp_dir / "nonexistent"
|
||||
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
output_dir=nonexistent_dir,
|
||||
)
|
||||
|
||||
errors = filters.validate()
|
||||
|
||||
assert len(errors) >= 1
|
||||
assert any("does not exist" in e for e in errors)
|
||||
|
||||
def test_validate_passes_when_output_dir_exists(self, sample_date_range, temp_dir: Path):
|
||||
"""validate() should pass when output_dir exists."""
|
||||
start, end, last_seen = sample_date_range
|
||||
output_dir = temp_dir / "output"
|
||||
output_dir.mkdir()
|
||||
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
output_dir=output_dir,
|
||||
)
|
||||
|
||||
errors = filters.validate()
|
||||
assert errors == []
|
||||
|
||||
def test_validate_multiple_errors(self):
|
||||
"""validate() should report all errors, not just the first."""
|
||||
filters = AnalysisFilters(
|
||||
start_date=date(2024, 12, 31), # End before start
|
||||
end_date=date(2024, 1, 1),
|
||||
last_seen_date=date(2024, 6, 1),
|
||||
minimum_patients=-5, # Negative
|
||||
)
|
||||
|
||||
errors = filters.validate()
|
||||
|
||||
assert len(errors) >= 2
|
||||
|
||||
|
||||
class TestAnalysisFiltersHasFilters:
|
||||
"""Test has_*_filter properties."""
|
||||
|
||||
def test_has_trust_filter_false_when_empty(self, sample_date_range):
|
||||
"""has_trust_filter should be False when trusts list is empty."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert filters.has_trust_filter is False
|
||||
|
||||
def test_has_trust_filter_true_when_populated(self, sample_date_range, sample_trusts):
|
||||
"""has_trust_filter should be True when trusts list has items."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
trusts=sample_trusts,
|
||||
)
|
||||
|
||||
assert filters.has_trust_filter is True
|
||||
|
||||
def test_has_drug_filter_false_when_empty(self, sample_date_range):
|
||||
"""has_drug_filter should be False when drugs list is empty."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert filters.has_drug_filter is False
|
||||
|
||||
def test_has_drug_filter_true_when_populated(self, sample_date_range, sample_drugs):
|
||||
"""has_drug_filter should be True when drugs list has items."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
drugs=sample_drugs,
|
||||
)
|
||||
|
||||
assert filters.has_drug_filter is True
|
||||
|
||||
def test_has_directory_filter_false_when_empty(self, sample_date_range):
|
||||
"""has_directory_filter should be False when directories list is empty."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert filters.has_directory_filter is False
|
||||
|
||||
def test_has_directory_filter_true_when_populated(self, sample_date_range, sample_directories):
|
||||
"""has_directory_filter should be True when directories list has items."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
directories=sample_directories,
|
||||
)
|
||||
|
||||
assert filters.has_directory_filter is True
|
||||
|
||||
|
||||
class TestAnalysisFiltersTitle:
|
||||
"""Test title property."""
|
||||
|
||||
def test_title_returns_custom_when_set(self, sample_date_range):
|
||||
"""title should return custom_title when set."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
custom_title="My Custom Analysis",
|
||||
)
|
||||
|
||||
assert filters.title == "My Custom Analysis"
|
||||
|
||||
def test_title_auto_generates_when_not_set(self, sample_date_range):
|
||||
"""title should auto-generate from dates when custom_title is empty."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
assert "2024-01-01" in filters.title
|
||||
assert "2024-12-31" in filters.title
|
||||
|
||||
def test_title_auto_generated_includes_dates(self):
|
||||
"""Auto-generated title should include start and end dates."""
|
||||
filters = AnalysisFilters(
|
||||
start_date=date(2023, 6, 15),
|
||||
end_date=date(2024, 3, 20),
|
||||
last_seen_date=date(2024, 1, 1),
|
||||
)
|
||||
|
||||
assert "2023-06-15" in filters.title
|
||||
assert "2024-03-20" in filters.title
|
||||
|
||||
|
||||
class TestAnalysisFiltersSummary:
|
||||
"""Test summary() method."""
|
||||
|
||||
def test_summary_returns_string(self, sample_date_range):
|
||||
"""summary() should return a string."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
summary = filters.summary()
|
||||
assert isinstance(summary, str)
|
||||
|
||||
def test_summary_includes_date_range(self, sample_date_range):
|
||||
"""summary() should include date range information."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
summary = filters.summary()
|
||||
assert "Date range" in summary
|
||||
assert "2024-01-01" in summary or str(start) in summary
|
||||
|
||||
def test_summary_includes_minimum_patients(self, sample_date_range):
|
||||
"""summary() should include minimum patients value."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
minimum_patients=10,
|
||||
)
|
||||
|
||||
summary = filters.summary()
|
||||
assert "Minimum patients" in summary
|
||||
assert "10" in summary
|
||||
|
||||
def test_summary_shows_all_when_no_filters(self, sample_date_range):
|
||||
"""summary() should show 'All' when filter lists are empty."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
)
|
||||
|
||||
summary = filters.summary()
|
||||
assert "Trusts: All" in summary
|
||||
assert "Drugs: All" in summary
|
||||
assert "Directories: All" in summary
|
||||
|
||||
def test_summary_shows_count_when_filters_set(
|
||||
self, sample_date_range, sample_trusts, sample_drugs, sample_directories
|
||||
):
|
||||
"""summary() should show count when filter lists are populated."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
trusts=sample_trusts,
|
||||
drugs=sample_drugs,
|
||||
directories=sample_directories,
|
||||
)
|
||||
|
||||
summary = filters.summary()
|
||||
assert "3 selected" in summary # trusts count
|
||||
assert "4 selected" in summary # drugs count
|
||||
|
||||
def test_summary_includes_custom_title_when_set(self, sample_date_range):
|
||||
"""summary() should include custom title when set."""
|
||||
start, end, last_seen = sample_date_range
|
||||
filters = AnalysisFilters(
|
||||
start_date=start,
|
||||
end_date=end,
|
||||
last_seen_date=last_seen,
|
||||
custom_title="Special Analysis",
|
||||
)
|
||||
|
||||
summary = filters.summary()
|
||||
assert "Custom title" in summary
|
||||
assert "Special Analysis" in summary
|
||||
@@ -0,0 +1,351 @@
|
||||
"""
|
||||
Test to verify that the refactored analysis pipeline produces matching output.
|
||||
|
||||
This test compares the output of the refactored generate_icicle_chart() function
|
||||
from analysis/pathway_analyzer.py with expected output characteristics.
|
||||
|
||||
Since the original generate_graph() function calls figure() directly without
|
||||
returning data, we verify the refactored pipeline by:
|
||||
1. Running the pipeline with known test data
|
||||
2. Verifying the output DataFrame has correct structure
|
||||
3. Verifying statistical calculations are reasonable
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# Skip if we can't import the modules
|
||||
try:
|
||||
from analysis.pathway_analyzer import (
|
||||
generate_icicle_chart,
|
||||
prepare_data,
|
||||
calculate_statistics,
|
||||
build_hierarchy,
|
||||
prepare_chart_data,
|
||||
)
|
||||
from core import default_paths
|
||||
HAS_MODULES = True
|
||||
except ImportError:
|
||||
HAS_MODULES = False
|
||||
|
||||
|
||||
# Standard test filters (matching sample data)
|
||||
TEST_TRUST_FILTER = [
|
||||
'MANCHESTER UNIVERSITY NHS FOUNDATION TRUST', # R0A code
|
||||
'BARTS HEALTH NHS TRUST', # R1H code
|
||||
]
|
||||
TEST_DRUG_FILTER = ['ADALIMUMAB', 'ETANERCEPT', 'INFLIXIMAB']
|
||||
TEST_DIRECTORY_FILTER = ['Rheumatology', 'Dermatology', 'Gastroenterology']
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_intervention_data():
|
||||
"""
|
||||
Create sample intervention data similar to what comes from the data loader.
|
||||
|
||||
The data mimics the structure expected by generate_icicle_chart():
|
||||
- UPID: Unique patient identifier (Provider Code prefix + PersonKey)
|
||||
- Drug Name: Standardized drug name
|
||||
- Directory: Medical specialty
|
||||
- Intervention Date: Date of treatment
|
||||
- Price Actual: Cost of treatment
|
||||
- Provider Code: NHS Trust code (will be mapped to name via org_codes.csv)
|
||||
|
||||
Uses real trust codes from org_codes.csv:
|
||||
- R0A = MANCHESTER UNIVERSITY NHS FOUNDATION TRUST
|
||||
- R1H = BARTS HEALTH NHS TRUST
|
||||
"""
|
||||
# Create data for a small number of patients with varied pathways
|
||||
data = {
|
||||
'UPID': [
|
||||
# Patient 1: Trust1 (R0A), Rheumatology, Adalimumab only (5 treatments)
|
||||
'R0A12345', 'R0A12345', 'R0A12345', 'R0A12345', 'R0A12345',
|
||||
# Patient 2: Trust1 (R0A), Rheumatology, Adalimumab then Etanercept (4 treatments)
|
||||
'R0A67890', 'R0A67890', 'R0A67890', 'R0A67890',
|
||||
# Patient 3: Trust1 (R0A), Dermatology, Adalimumab only (3 treatments)
|
||||
'R0A11111', 'R0A11111', 'R0A11111',
|
||||
# Patient 4: Trust2 (R1H), Rheumatology, Etanercept only (6 treatments)
|
||||
'R1H22222', 'R1H22222', 'R1H22222', 'R1H22222', 'R1H22222', 'R1H22222',
|
||||
# Patient 5: Trust2 (R1H), Gastro, Infliximab only (4 treatments)
|
||||
'R1H33333', 'R1H33333', 'R1H33333', 'R1H33333',
|
||||
],
|
||||
'Drug Name': [
|
||||
'ADALIMUMAB', 'ADALIMUMAB', 'ADALIMUMAB', 'ADALIMUMAB', 'ADALIMUMAB',
|
||||
'ADALIMUMAB', 'ADALIMUMAB', 'ETANERCEPT', 'ETANERCEPT',
|
||||
'ADALIMUMAB', 'ADALIMUMAB', 'ADALIMUMAB',
|
||||
'ETANERCEPT', 'ETANERCEPT', 'ETANERCEPT', 'ETANERCEPT', 'ETANERCEPT', 'ETANERCEPT',
|
||||
'INFLIXIMAB', 'INFLIXIMAB', 'INFLIXIMAB', 'INFLIXIMAB',
|
||||
],
|
||||
'Directory': [
|
||||
'Rheumatology', 'Rheumatology', 'Rheumatology', 'Rheumatology', 'Rheumatology',
|
||||
'Rheumatology', 'Rheumatology', 'Rheumatology', 'Rheumatology',
|
||||
'Dermatology', 'Dermatology', 'Dermatology',
|
||||
'Rheumatology', 'Rheumatology', 'Rheumatology', 'Rheumatology', 'Rheumatology', 'Rheumatology',
|
||||
'Gastroenterology', 'Gastroenterology', 'Gastroenterology', 'Gastroenterology',
|
||||
],
|
||||
'Intervention Date': [
|
||||
# Patient 1 dates (every 2 weeks)
|
||||
datetime(2023, 1, 1), datetime(2023, 1, 15), datetime(2023, 1, 29), datetime(2023, 2, 12), datetime(2023, 2, 26),
|
||||
# Patient 2 dates (switch after 2 months)
|
||||
datetime(2023, 1, 5), datetime(2023, 2, 5), datetime(2023, 3, 5), datetime(2023, 4, 5),
|
||||
# Patient 3 dates
|
||||
datetime(2023, 2, 1), datetime(2023, 3, 1), datetime(2023, 4, 1),
|
||||
# Patient 4 dates (weekly for 6 weeks)
|
||||
datetime(2023, 1, 1), datetime(2023, 1, 8), datetime(2023, 1, 15), datetime(2023, 1, 22), datetime(2023, 1, 29), datetime(2023, 2, 5),
|
||||
# Patient 5 dates (every 4 weeks)
|
||||
datetime(2023, 1, 10), datetime(2023, 2, 7), datetime(2023, 3, 7), datetime(2023, 4, 4),
|
||||
],
|
||||
'Price Actual': [
|
||||
# Patient 1 costs
|
||||
500.0, 500.0, 500.0, 500.0, 500.0,
|
||||
# Patient 2 costs
|
||||
500.0, 500.0, 600.0, 600.0,
|
||||
# Patient 3 costs
|
||||
500.0, 500.0, 500.0,
|
||||
# Patient 4 costs
|
||||
400.0, 400.0, 400.0, 400.0, 400.0, 400.0,
|
||||
# Patient 5 costs
|
||||
800.0, 800.0, 800.0, 800.0,
|
||||
],
|
||||
'Provider Code': [
|
||||
# Trust codes (R0A = Manchester, R1H = Barts)
|
||||
'R0A', 'R0A', 'R0A', 'R0A', 'R0A',
|
||||
'R0A', 'R0A', 'R0A', 'R0A',
|
||||
'R0A', 'R0A', 'R0A',
|
||||
'R1H', 'R1H', 'R1H', 'R1H', 'R1H', 'R1H',
|
||||
'R1H', 'R1H', 'R1H', 'R1H',
|
||||
],
|
||||
}
|
||||
return pd.DataFrame(data)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAS_MODULES, reason="Required modules not available")
|
||||
class TestOutputStructure:
|
||||
"""Test that the refactored pipeline produces correct output structure."""
|
||||
|
||||
def test_ice_df_has_required_columns(self, sample_intervention_data):
|
||||
"""Verify ice_df has all required columns for Plotly icicle chart."""
|
||||
if default_paths.validate(): # Non-empty list means errors
|
||||
pytest.skip("Reference data files not available")
|
||||
|
||||
df = sample_intervention_data.copy()
|
||||
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=df,
|
||||
start_date='2022-01-01',
|
||||
end_date='2024-01-01',
|
||||
last_seen_date='2022-06-01',
|
||||
trust_filter=TEST_TRUST_FILTER,
|
||||
drug_filter=TEST_DRUG_FILTER,
|
||||
directory_filter=TEST_DIRECTORY_FILTER,
|
||||
minimum_num_patients=1,
|
||||
title="Test Output",
|
||||
paths=default_paths,
|
||||
)
|
||||
|
||||
if ice_df is None:
|
||||
pytest.skip("No data matched filters (trust code mapping may not match)")
|
||||
|
||||
# Required columns for Plotly icicle chart
|
||||
required_columns = ['parents', 'labels', 'ids', 'value', 'cost']
|
||||
for col in required_columns:
|
||||
assert col in ice_df.columns, f"Missing required column: {col}"
|
||||
|
||||
def test_ice_df_hierarchy_structure(self, sample_intervention_data):
|
||||
"""Verify the ice_df hierarchy is valid (parents reference existing ids)."""
|
||||
if default_paths.validate(): # Non-empty list means errors
|
||||
pytest.skip("Reference data files not available")
|
||||
|
||||
df = sample_intervention_data.copy()
|
||||
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=df,
|
||||
start_date='2022-01-01',
|
||||
end_date='2024-01-01',
|
||||
last_seen_date='2022-06-01',
|
||||
trust_filter=TEST_TRUST_FILTER,
|
||||
drug_filter=TEST_DRUG_FILTER,
|
||||
directory_filter=TEST_DIRECTORY_FILTER,
|
||||
minimum_num_patients=1,
|
||||
title="Test Output",
|
||||
)
|
||||
|
||||
if ice_df is None:
|
||||
pytest.skip("No data matched filters")
|
||||
|
||||
# Every parent should be in ids (except root which has empty parent)
|
||||
ids_set = set(ice_df['ids'].unique())
|
||||
for parent in ice_df['parents'].unique():
|
||||
if parent != '': # Root has empty parent
|
||||
assert parent in ids_set, f"Parent '{parent}' not found in ids"
|
||||
|
||||
def test_values_sum_correctly(self, sample_intervention_data):
|
||||
"""Verify that child values sum to parent values (with branchvalues='total')."""
|
||||
if default_paths.validate(): # Non-empty list means errors
|
||||
pytest.skip("Reference data files not available")
|
||||
|
||||
df = sample_intervention_data.copy()
|
||||
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=df,
|
||||
start_date='2022-01-01',
|
||||
end_date='2024-01-01',
|
||||
last_seen_date='2022-06-01',
|
||||
trust_filter=TEST_TRUST_FILTER,
|
||||
drug_filter=TEST_DRUG_FILTER,
|
||||
directory_filter=TEST_DIRECTORY_FILTER,
|
||||
minimum_num_patients=1,
|
||||
title="Test Output",
|
||||
)
|
||||
|
||||
if ice_df is None:
|
||||
pytest.skip("No data matched filters")
|
||||
|
||||
# Verify the structure is valid:
|
||||
# - Root (N&WICS) should have the highest value
|
||||
# - All child values should sum to at most their parent value
|
||||
root_row = ice_df[ice_df['ids'] == 'N&WICS']
|
||||
if len(root_row) > 0:
|
||||
root_value = root_row['value'].iloc[0]
|
||||
assert root_value > 0, "Root should have positive value"
|
||||
|
||||
# Check that children sum to parent value for nodes at same level
|
||||
# Note: The icicle chart uses branchvalues='total' so children should sum to parent
|
||||
# However, at pathway level, patients may appear in multiple pathway branches
|
||||
for parent_id in ice_df['ids'].unique():
|
||||
parent_row = ice_df[ice_df['ids'] == parent_id]
|
||||
if len(parent_row) == 0:
|
||||
continue
|
||||
parent_value = parent_row['value'].iloc[0]
|
||||
|
||||
children = ice_df[ice_df['parents'] == parent_id]
|
||||
if len(children) > 0:
|
||||
children_sum = children['value'].sum()
|
||||
# Children should sum to parent value in a properly constructed icicle chart
|
||||
# Allow for small differences due to filtering at minimum_num_patients
|
||||
assert children_sum <= parent_value, \
|
||||
f"Children of '{parent_id}' sum to {children_sum}, exceeds parent {parent_value}"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAS_MODULES, reason="Required modules not available")
|
||||
class TestPrepareData:
|
||||
"""Test the prepare_data() function independently."""
|
||||
|
||||
def test_prepare_data_filters_correctly(self, sample_intervention_data):
|
||||
"""Verify prepare_data applies filters correctly."""
|
||||
if default_paths.validate(): # Non-empty list means errors
|
||||
pytest.skip("Reference data files not available")
|
||||
|
||||
df = sample_intervention_data.copy()
|
||||
|
||||
# Filter to single drug
|
||||
result = prepare_data(
|
||||
df,
|
||||
TEST_TRUST_FILTER,
|
||||
['ADALIMUMAB'], # Only Adalimumab
|
||||
TEST_DIRECTORY_FILTER
|
||||
)
|
||||
|
||||
if result[0] is None:
|
||||
pytest.skip("No data matched filters")
|
||||
|
||||
filtered_df, org_codes, directory_df = result
|
||||
|
||||
# Should only have Adalimumab rows
|
||||
assert set(filtered_df['Drug Name'].unique()) == {'ADALIMUMAB'}
|
||||
|
||||
def test_prepare_data_creates_upid_treatment(self, sample_intervention_data):
|
||||
"""Verify prepare_data creates UPIDTreatment column."""
|
||||
if default_paths.validate(): # Non-empty list means errors
|
||||
pytest.skip("Reference data files not available")
|
||||
|
||||
df = sample_intervention_data.copy()
|
||||
|
||||
result = prepare_data(
|
||||
df,
|
||||
TEST_TRUST_FILTER,
|
||||
TEST_DRUG_FILTER,
|
||||
TEST_DIRECTORY_FILTER
|
||||
)
|
||||
|
||||
if result[0] is None:
|
||||
pytest.skip("No data matched filters")
|
||||
|
||||
filtered_df, org_codes, directory_df = result
|
||||
|
||||
# UPIDTreatment should be UPID + Drug Name
|
||||
assert 'UPIDTreatment' in filtered_df.columns
|
||||
# Check first row
|
||||
first_row = filtered_df.iloc[0]
|
||||
expected = first_row['UPID'] + first_row['Drug Name']
|
||||
assert first_row['UPIDTreatment'] == expected
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAS_MODULES, reason="Required modules not available")
|
||||
class TestCalculateStatistics:
|
||||
"""Test the calculate_statistics() function independently."""
|
||||
|
||||
def test_date_filtering(self, sample_intervention_data):
|
||||
"""Verify date filtering in calculate_statistics."""
|
||||
if default_paths.validate(): # Non-empty list means errors
|
||||
pytest.skip("Reference data files not available")
|
||||
|
||||
df = sample_intervention_data.copy()
|
||||
df['UPIDTreatment'] = df['UPID'] + df['Drug Name']
|
||||
|
||||
# These dates should include all our sample data
|
||||
start_date = '2022-01-01'
|
||||
end_date = '2024-01-01'
|
||||
last_seen_date = '2022-06-01'
|
||||
|
||||
result = calculate_statistics(df, start_date, end_date, last_seen_date, "Test")
|
||||
|
||||
if result[0] is None:
|
||||
pytest.skip("No data matched date filters")
|
||||
|
||||
patient_info, date_df, title = result
|
||||
|
||||
# Should have patient info DataFrame
|
||||
assert patient_info is not None
|
||||
assert len(patient_info) > 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAS_MODULES, reason="Required modules not available")
|
||||
class TestMinimumPatientFilter:
|
||||
"""Test that minimum_num_patients filter works correctly."""
|
||||
|
||||
def test_filters_small_pathways(self, sample_intervention_data):
|
||||
"""Verify pathways with fewer patients than threshold are excluded."""
|
||||
if default_paths.validate(): # Non-empty list means errors
|
||||
pytest.skip("Reference data files not available")
|
||||
|
||||
df = sample_intervention_data.copy()
|
||||
|
||||
# With minimum 10, nothing should pass (we only have 5 patients)
|
||||
ice_df, title = generate_icicle_chart(
|
||||
df=df,
|
||||
start_date='2022-01-01',
|
||||
end_date='2024-01-01',
|
||||
last_seen_date='2022-06-01',
|
||||
trust_filter=TEST_TRUST_FILTER,
|
||||
drug_filter=TEST_DRUG_FILTER,
|
||||
directory_filter=TEST_DIRECTORY_FILTER,
|
||||
minimum_num_patients=10, # Higher than our patient count
|
||||
title="Test Output",
|
||||
)
|
||||
|
||||
# Either None or empty DataFrame
|
||||
if ice_df is not None:
|
||||
# If filtered, should have very few or no patient pathways
|
||||
patient_rows = ice_df[ice_df['value'] < 10]
|
||||
# All remaining rows should have value >= 10
|
||||
remaining = ice_df[ice_df['value'] >= 10]
|
||||
# This may include aggregated rows
|
||||
pass # Test passes if no error
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
pytest.main([__file__, '-v'])
|
||||
@@ -0,0 +1,269 @@
|
||||
"""
|
||||
Test Plotly interactivity features in the visualization module.
|
||||
|
||||
Verifies that Plotly charts have the expected interactive capabilities:
|
||||
1. Hover templates are properly configured
|
||||
2. Icicle chart settings allow click-to-drill-down navigation
|
||||
3. Layout settings support proper display of interactive features
|
||||
|
||||
Phase 4.7.2: Verify Plotly interactivity (zoom, pan, hover)
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from datetime import datetime
|
||||
|
||||
import plotly.graph_objects as go
|
||||
|
||||
# Import the visualization module
|
||||
try:
|
||||
from visualization.plotly_generator import create_icicle_figure, save_figure_html
|
||||
HAS_VISUALIZATION = True
|
||||
except ImportError:
|
||||
HAS_VISUALIZATION = False
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_chart_data():
|
||||
"""
|
||||
Create sample chart data (ice_df) for testing visualization.
|
||||
|
||||
This mimics the output of prepare_chart_data() from analysis/pathway_analyzer.py
|
||||
"""
|
||||
# Sample hierarchy data: Root -> Trust -> Directory -> Drug
|
||||
data = {
|
||||
'parents': [
|
||||
'', # Root (N&WICS)
|
||||
'N&WICS', # Trust 1
|
||||
'N&WICS', # Trust 2
|
||||
'Trust1', # Directory in Trust1
|
||||
'Trust1', # Another Directory
|
||||
'Trust2', # Directory in Trust2
|
||||
'Trust1/Rheum', # Drug
|
||||
'Trust1/Derm', # Drug
|
||||
'Trust2/Rheum', # Drug
|
||||
],
|
||||
'ids': [
|
||||
'N&WICS',
|
||||
'Trust1',
|
||||
'Trust2',
|
||||
'Trust1/Rheum',
|
||||
'Trust1/Derm',
|
||||
'Trust2/Rheum',
|
||||
'Trust1/Rheum/Adalimumab',
|
||||
'Trust1/Derm/Adalimumab',
|
||||
'Trust2/Rheum/Etanercept',
|
||||
],
|
||||
'labels': [
|
||||
'Norfolk & Waveney ICS',
|
||||
'Manchester University Trust',
|
||||
'Barts Health Trust',
|
||||
'Rheumatology',
|
||||
'Dermatology',
|
||||
'Rheumatology',
|
||||
'Adalimumab',
|
||||
'Adalimumab',
|
||||
'Etanercept',
|
||||
],
|
||||
'value': [50, 30, 20, 20, 10, 20, 20, 10, 20],
|
||||
'colour': [1.0, 0.6, 0.4, 0.4, 0.2, 0.4, 0.4, 0.2, 0.4],
|
||||
'cost': [50000, 30000, 20000, 20000, 10000, 20000, 20000, 10000, 20000],
|
||||
'costpp': [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000],
|
||||
'cost_pp_pa': [2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000],
|
||||
'First seen': [
|
||||
pd.Timestamp('2023-01-01')] * 9,
|
||||
'Last seen': [
|
||||
pd.Timestamp('2023-12-31')] * 9,
|
||||
'First seen (Parent)': [
|
||||
pd.Timestamp('2023-01-01')] * 9,
|
||||
'Last seen (Parent)': [
|
||||
pd.Timestamp('2023-12-31')] * 9,
|
||||
'average_spacing': ['14 days'] * 9,
|
||||
'avg_days': [pd.Timedelta('180 days')] * 9,
|
||||
}
|
||||
return pd.DataFrame(data)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAS_VISUALIZATION, reason="Visualization module not available")
|
||||
class TestPlotlyFigureConfiguration:
|
||||
"""Test that Plotly figures have correct interactive configuration."""
|
||||
|
||||
def test_figure_has_hovertemplate(self, sample_chart_data):
|
||||
"""Verify the icicle chart has a hover template configured."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
# Get the icicle trace
|
||||
assert len(fig.data) > 0, "Figure should have at least one trace"
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
assert icicle_trace.type == 'icicle', "First trace should be an icicle chart"
|
||||
|
||||
# Verify hovertemplate is set and contains expected placeholders
|
||||
assert icicle_trace.hovertemplate is not None, "Hover template should be configured"
|
||||
assert '%{label}' in icicle_trace.hovertemplate, "Hover should include label"
|
||||
assert '%{customdata' in icicle_trace.hovertemplate, "Hover should include custom data"
|
||||
|
||||
def test_figure_has_texttemplate(self, sample_chart_data):
|
||||
"""Verify the icicle chart has a text template for in-chart text."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
|
||||
# Verify texttemplate is set
|
||||
assert icicle_trace.texttemplate is not None, "Text template should be configured"
|
||||
assert '%{label}' in icicle_trace.texttemplate, "Text should include label"
|
||||
|
||||
def test_figure_has_correct_branchvalues(self, sample_chart_data):
|
||||
"""Verify branchvalues is set to 'total' for proper hierarchy summing."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
|
||||
# branchvalues should be 'total' for proper hierarchy display
|
||||
assert icicle_trace.branchvalues == 'total', \
|
||||
"branchvalues should be 'total' for hierarchy summation"
|
||||
|
||||
def test_figure_has_maxdepth_for_drilldown(self, sample_chart_data):
|
||||
"""Verify maxdepth is set to allow drill-down navigation."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
|
||||
# maxdepth should be set to limit initial view depth
|
||||
# Users can then click to drill into deeper levels
|
||||
assert icicle_trace.maxdepth is not None, "maxdepth should be configured for drill-down"
|
||||
assert icicle_trace.maxdepth >= 2, "maxdepth should be at least 2 to show hierarchy"
|
||||
|
||||
def test_figure_layout_has_hoverlabel(self, sample_chart_data):
|
||||
"""Verify layout has hoverlabel configuration for readable tooltips."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
# Check hoverlabel configuration
|
||||
assert 'hoverlabel' in fig.layout, "Layout should have hoverlabel configuration"
|
||||
# Plotly uses 'font' as a dict with 'size' attribute
|
||||
assert fig.layout.hoverlabel.font is not None, "Hover label font should be configured"
|
||||
assert fig.layout.hoverlabel.font.size is not None, "Hover label font size should be set"
|
||||
assert fig.layout.hoverlabel.font.size >= 12, "Hover label should be readable (>=12px)"
|
||||
|
||||
def test_figure_has_proper_margins(self, sample_chart_data):
|
||||
"""Verify layout has margins configured for proper display."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
# Check margin configuration
|
||||
assert fig.layout.margin is not None, "Margins should be configured"
|
||||
assert fig.layout.margin.t >= 50, "Top margin should have room for title"
|
||||
|
||||
def test_figure_has_title(self, sample_chart_data):
|
||||
"""Verify the figure has a title configured."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Analysis")
|
||||
|
||||
assert fig.layout.title is not None, "Figure should have a title"
|
||||
assert "Test Analysis" in fig.layout.title.text, "Title should include custom text"
|
||||
|
||||
def test_figure_has_colorscale(self, sample_chart_data):
|
||||
"""Verify the icicle chart has a colorscale for visual differentiation."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
|
||||
# Check marker has colorscale
|
||||
assert icicle_trace.marker is not None, "Marker should be configured"
|
||||
assert icicle_trace.marker.colorscale is not None, "Colorscale should be set"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAS_VISUALIZATION, reason="Visualization module not available")
|
||||
class TestPlotlyInteractiveFeatures:
|
||||
"""Test that Plotly figures support expected interactive features."""
|
||||
|
||||
def test_figure_is_interactive_type(self, sample_chart_data):
|
||||
"""Verify the figure is a go.Figure which supports interactivity."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
assert isinstance(fig, go.Figure), "Should return a Plotly Figure object"
|
||||
|
||||
def test_figure_can_be_converted_to_html(self, sample_chart_data, tmp_path):
|
||||
"""Verify the figure can be saved as interactive HTML."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
# Save to temporary file
|
||||
html_path = save_figure_html(fig, str(tmp_path), "test_chart", open_browser=False)
|
||||
|
||||
assert html_path.endswith('.html'), "Should save as HTML file"
|
||||
|
||||
# Verify the HTML file exists and contains Plotly data
|
||||
with open(html_path, 'r', encoding='utf-8') as f:
|
||||
html_content = f.read()
|
||||
|
||||
assert 'plotly' in html_content.lower(), "HTML should contain Plotly"
|
||||
# Interactive HTML should include the plotly.js library
|
||||
assert 'cdn.plot.ly' in html_content or 'plotly-' in html_content, \
|
||||
"HTML should include Plotly.js for interactivity"
|
||||
|
||||
def test_figure_data_includes_ids_for_drilldown(self, sample_chart_data):
|
||||
"""Verify figure data includes ids necessary for click-to-drill navigation."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
|
||||
# ids are required for proper drill-down behavior in icicle charts
|
||||
assert icicle_trace.ids is not None, "ids should be provided for drill-down"
|
||||
assert len(icicle_trace.ids) > 0, "ids should not be empty"
|
||||
|
||||
def test_figure_data_includes_parents_for_hierarchy(self, sample_chart_data):
|
||||
"""Verify figure data includes parents for hierarchy navigation."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
|
||||
# parents are required for hierarchy structure
|
||||
assert icicle_trace.parents is not None, "parents should be provided"
|
||||
assert len(icicle_trace.parents) > 0, "parents should not be empty"
|
||||
|
||||
def test_figure_customdata_enables_rich_hover(self, sample_chart_data):
|
||||
"""Verify customdata is provided for rich hover information."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
icicle_trace = fig.data[0]
|
||||
|
||||
# customdata enables rich hover templates with additional info
|
||||
assert icicle_trace.customdata is not None, "customdata should be provided"
|
||||
|
||||
# customdata should be a 2D array with multiple columns of data
|
||||
assert len(icicle_trace.customdata) > 0, "customdata should have rows"
|
||||
# Each row should have multiple data points for hover display
|
||||
if hasattr(icicle_trace.customdata[0], '__len__'):
|
||||
assert len(icicle_trace.customdata[0]) >= 5, \
|
||||
"customdata should have multiple columns for rich hover"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAS_VISUALIZATION, reason="Visualization module not available")
|
||||
class TestReflexCompatibility:
|
||||
"""Test that figures are compatible with Reflex's rx.plotly() component."""
|
||||
|
||||
def test_figure_to_json_serializable(self, sample_chart_data):
|
||||
"""Verify figure can be serialized to JSON (required for Reflex)."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
# Reflex needs to serialize the figure to JSON for the frontend
|
||||
try:
|
||||
json_data = fig.to_json()
|
||||
assert json_data is not None
|
||||
assert len(json_data) > 0
|
||||
except Exception as e:
|
||||
pytest.fail(f"Figure should be JSON serializable: {e}")
|
||||
|
||||
def test_figure_to_dict(self, sample_chart_data):
|
||||
"""Verify figure can be converted to dict (used by Reflex internally)."""
|
||||
fig = create_icicle_figure(sample_chart_data, "Test Title")
|
||||
|
||||
# Reflex may use to_dict internally
|
||||
fig_dict = fig.to_dict()
|
||||
|
||||
assert 'data' in fig_dict, "Figure dict should have data"
|
||||
assert 'layout' in fig_dict, "Figure dict should have layout"
|
||||
assert len(fig_dict['data']) > 0, "Data should not be empty"
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
pytest.main([__file__, '-v'])
|
||||
@@ -0,0 +1,176 @@
|
||||
"""
|
||||
Test Phase 3.4.4: Measure directory assignment "Undefined" rate with real Snowflake data.
|
||||
|
||||
This test fetches HCD activity data from Snowflake, runs it through the directory
|
||||
assignment pipeline, and measures what percentage of records end up with "Undefined"
|
||||
directory vs. successfully assigned directories.
|
||||
"""
|
||||
|
||||
import json
|
||||
import pandas as pd
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add project root to path
|
||||
project_root = Path(__file__).parent.parent
|
||||
sys.path.insert(0, str(project_root))
|
||||
|
||||
from tools.data import patient_id, drug_names, department_identification
|
||||
from core import default_paths
|
||||
|
||||
|
||||
def load_snowflake_result(json_file: Path) -> pd.DataFrame:
|
||||
"""Load Snowflake query result from JSON file and convert to DataFrame."""
|
||||
with open(json_file, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
|
||||
# The result is in format: [{"type": "text", "text": "..."}]
|
||||
# where text contains JSON with {"columns": [...], "rows": [...]}
|
||||
if isinstance(data, list) and len(data) > 0 and 'text' in data[0]:
|
||||
records_text = data[0]['text']
|
||||
result_obj = json.loads(records_text)
|
||||
# Extract rows from the result object
|
||||
if isinstance(result_obj, dict) and 'rows' in result_obj:
|
||||
records = result_obj['rows']
|
||||
else:
|
||||
records = result_obj
|
||||
else:
|
||||
records = data
|
||||
|
||||
return pd.DataFrame(records)
|
||||
|
||||
|
||||
def analyze_directory_sources(df: pd.DataFrame) -> dict:
|
||||
"""Analyze the distribution of Directory_Source values."""
|
||||
if 'Directory_Source' not in df.columns:
|
||||
return {"error": "Directory_Source column not found"}
|
||||
|
||||
source_counts = df['Directory_Source'].value_counts()
|
||||
total = len(df)
|
||||
|
||||
result = {
|
||||
"total_records": total,
|
||||
"source_distribution": {},
|
||||
"undefined_rate": 0.0,
|
||||
"assigned_rate": 0.0
|
||||
}
|
||||
|
||||
for source, count in source_counts.items():
|
||||
pct = (count / total) * 100
|
||||
result["source_distribution"][source] = {
|
||||
"count": int(count),
|
||||
"percentage": round(pct, 2)
|
||||
}
|
||||
|
||||
# Calculate undefined vs assigned rates
|
||||
undefined_count = source_counts.get('UNDEFINED', 0)
|
||||
result["undefined_rate"] = round((undefined_count / total) * 100, 2) if total > 0 else 0
|
||||
result["assigned_rate"] = round(100 - result["undefined_rate"], 2)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def analyze_by_drug(df: pd.DataFrame) -> dict:
|
||||
"""Analyze undefined rate by drug."""
|
||||
if 'Drug Name' not in df.columns or 'Directory_Source' not in df.columns:
|
||||
return {"error": "Required columns not found"}
|
||||
|
||||
results = {}
|
||||
for drug in df['Drug Name'].dropna().unique():
|
||||
drug_df = df[df['Drug Name'] == drug]
|
||||
total = len(drug_df)
|
||||
undefined = len(drug_df[drug_df['Directory_Source'] == 'UNDEFINED'])
|
||||
results[drug] = {
|
||||
"total": total,
|
||||
"undefined": undefined,
|
||||
"undefined_rate": round((undefined / total) * 100, 2) if total > 0 else 0
|
||||
}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function to run the real data test."""
|
||||
# Path to the Snowflake result file (updated 2026-02-04)
|
||||
result_file = Path(r"C:\Users\charlwoodand\.claude\projects\C--Users-charlwoodand-Ralph-local-Tasks-Patient-pathway-analysis\2b846818-a586-47de-bfb9-a740bd07fc70\tool-results\mcp-snowflake-mcp-read_data-1770199331688.txt")
|
||||
|
||||
if not result_file.exists():
|
||||
print(f"ERROR: Result file not found: {result_file}")
|
||||
return
|
||||
|
||||
print("Loading Snowflake data...")
|
||||
df = load_snowflake_result(result_file)
|
||||
print(f"Loaded {len(df)} records")
|
||||
print(f"Columns: {list(df.columns)}")
|
||||
|
||||
# Rename columns to match expected format for tools/data.py functions
|
||||
column_mapping = {
|
||||
'ProviderCode': 'Provider Code',
|
||||
'PersonKey': 'PersonKey',
|
||||
'DrugName': 'Drug Name',
|
||||
'InterventionDate': 'Intervention Date',
|
||||
'TreatmentFunctionCode': 'Treatment Function Code',
|
||||
'AdditionalDetail1': 'Additional Detail 1',
|
||||
'AdditionalDescription1': 'Additional Description 1',
|
||||
'AdditionalDetail2': 'Additional Detail 2',
|
||||
'AdditionalDescription2': 'Additional Description 2',
|
||||
'PriceActual': 'Price Actual',
|
||||
'OrganisationName': 'OrganisationName'
|
||||
}
|
||||
|
||||
df = df.rename(columns=column_mapping)
|
||||
print(f"Renamed columns: {list(df.columns)}")
|
||||
|
||||
# Step 1: Generate UPID
|
||||
print("\nStep 1: Generating UPID...")
|
||||
df = patient_id(df)
|
||||
print(f"Sample UPIDs: {df['UPID'].head(5).tolist()}")
|
||||
|
||||
# Step 2: Standardize drug names
|
||||
print("\nStep 2: Standardizing drug names...")
|
||||
df = drug_names(df, default_paths)
|
||||
print(f"Unique drugs after standardization: {df['Drug Name'].dropna().unique().tolist()}")
|
||||
|
||||
# Step 3: Run directory assignment
|
||||
print("\nStep 3: Running directory assignment...")
|
||||
df = department_identification(df, default_paths)
|
||||
|
||||
# Step 4: Analyze results
|
||||
print("\n" + "="*60)
|
||||
print("DIRECTORY ASSIGNMENT RESULTS")
|
||||
print("="*60)
|
||||
|
||||
overall_stats = analyze_directory_sources(df)
|
||||
|
||||
print(f"\nTotal records processed: {overall_stats['total_records']}")
|
||||
print(f"\nDirectory Source Distribution:")
|
||||
for source, stats in sorted(overall_stats['source_distribution'].items(),
|
||||
key=lambda x: -x[1]['count']):
|
||||
print(f" {source}: {stats['count']:,} ({stats['percentage']:.1f}%)")
|
||||
|
||||
print(f"\n*** UNDEFINED RATE: {overall_stats['undefined_rate']:.1f}% ***")
|
||||
print(f"*** ASSIGNED RATE: {overall_stats['assigned_rate']:.1f}% ***")
|
||||
|
||||
# Analyze by drug
|
||||
print("\n" + "-"*60)
|
||||
print("UNDEFINED RATE BY DRUG")
|
||||
print("-"*60)
|
||||
|
||||
drug_stats = analyze_by_drug(df)
|
||||
for drug, stats in sorted(drug_stats.items(), key=lambda x: -x[1]['undefined_rate']):
|
||||
print(f" {drug}: {stats['undefined_rate']:.1f}% undefined ({stats['undefined']:,}/{stats['total']:,})")
|
||||
|
||||
# Show sample of directory assignments
|
||||
print("\n" + "-"*60)
|
||||
print("SAMPLE DIRECTORY ASSIGNMENTS")
|
||||
print("-"*60)
|
||||
|
||||
sample_cols = ['UPID', 'Drug Name', 'Directory', 'Directory_Source']
|
||||
available_cols = [c for c in sample_cols if c in df.columns]
|
||||
print(df[available_cols].head(20).to_string())
|
||||
|
||||
return overall_stats, drug_stats
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,647 @@
|
||||
import webbrowser
|
||||
from itertools import groupby
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import plotly.graph_objects as go
|
||||
|
||||
from core import AnalysisFilters, PathConfig, default_paths
|
||||
from core.logging_config import get_logger
|
||||
from tools import data
|
||||
|
||||
# Import refactored analysis functions
|
||||
from analysis.pathway_analyzer import (
|
||||
generate_icicle_chart as _generate_icicle_chart,
|
||||
prepare_data as _prepare_data,
|
||||
calculate_statistics as _calculate_statistics,
|
||||
build_hierarchy as _build_hierarchy,
|
||||
prepare_chart_data as _prepare_chart_data,
|
||||
)
|
||||
|
||||
# Import visualization functions
|
||||
from visualization.plotly_generator import (
|
||||
create_icicle_figure as _create_icicle_figure,
|
||||
save_figure_html as _save_figure_html,
|
||||
figure_legacy as _figure_legacy,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
pd.options.mode.chained_assignment = None # default='warn'
|
||||
def human_format(num):
|
||||
num = float('{:.3g}'.format(num))
|
||||
magnitude = 0
|
||||
while abs(num) >= 1000:
|
||||
magnitude += 1
|
||||
num /= 1000.0
|
||||
return '{}{}'.format('{:f}'.format(num).rstrip('0').rstrip('.'), ['', 'K', 'M', 'B', 'T'][magnitude])
|
||||
|
||||
def main(dir, paths: Optional[PathConfig] = None):
|
||||
"""
|
||||
Load and process patient intervention data from a file.
|
||||
|
||||
Uses the FileDataLoader abstraction to handle CSV/Parquet file loading
|
||||
with all necessary transformations (patient_id, drug_names, department_identification).
|
||||
|
||||
Args:
|
||||
dir: Path to CSV or Parquet file
|
||||
paths: PathConfig for reference data locations (uses default_paths if None)
|
||||
|
||||
Returns:
|
||||
DataFrame with processed patient intervention data
|
||||
"""
|
||||
from data_processing.loader import FileDataLoader
|
||||
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
loader = FileDataLoader(file_path=dir, paths=paths)
|
||||
result = loader.load()
|
||||
|
||||
logger.info("Initial data processing complete.")
|
||||
return result.df
|
||||
|
||||
|
||||
def drop_duplicate_treatments(df, ascending):
|
||||
df.sort_values(by=['Intervention Date'], ascending=ascending, inplace=True)
|
||||
df_treatment_steps = df.drop_duplicates(subset="UPIDTreatment", keep="first")
|
||||
if not ascending:
|
||||
df_treatment_steps.sort_values(by=['Intervention Date'], ascending=True, inplace=True)
|
||||
return df_treatment_steps
|
||||
|
||||
|
||||
def row_function(row):
|
||||
ids = ""
|
||||
parents = "N&WICS"
|
||||
count = row.count()
|
||||
for c in range(count):
|
||||
v = row[c]
|
||||
if type(v) != str:
|
||||
v = row[c + 1]
|
||||
if c == count - 1:
|
||||
ids = parents + " - " + v
|
||||
continue
|
||||
parents += " - " + v
|
||||
label = row[count - 1]
|
||||
value = parents + "," + label + "," + ids
|
||||
return value
|
||||
|
||||
|
||||
def count_list_values(x):
|
||||
return [len(list(group)) for key, group in groupby(sorted(x))]
|
||||
|
||||
|
||||
def sum_list_values(x):
|
||||
sum_list = []
|
||||
for count in range(len(x["Drug Name"])):
|
||||
if count == 0:
|
||||
sum_list.append(sum(x["Price Actual"][ : x["Drug Name"][count]]))
|
||||
else:
|
||||
sum_list.append(sum(x["Price Actual"][x["Drug Name"][count-1] : (x["Drug Name"][count-1] + x["Drug Name"][count])]))
|
||||
return sum_list
|
||||
|
||||
|
||||
def remove_nan_string(y):
|
||||
return [x for x in y if str(x) != 'nan']
|
||||
|
||||
|
||||
def min_max_treatment_dates(ice_df, row):
|
||||
ids = row[2]
|
||||
min_max = ice_df[ice_df["ids"].str.contains(ids)]
|
||||
min_date = str(min_max["First seen"].min().strftime('%Y-%m-%d'))
|
||||
max_date = str(min_max["Last seen"].max().strftime('%Y-%m-%d'))
|
||||
return min_date + ',' + max_date
|
||||
|
||||
|
||||
def start_date_drug(df, x):
|
||||
drug_count = x.notnull().sum()
|
||||
date_string = []
|
||||
for d in range(drug_count):
|
||||
UPID_date_var = str(x.name) + str(x[d])
|
||||
date = df.loc[UPID_date_var, "Intervention Date"]
|
||||
date_string.append(date)
|
||||
return date_string
|
||||
|
||||
|
||||
def end_date_drug(df, x):
|
||||
drug_count = x.notnull().sum()
|
||||
date_string = []
|
||||
# Need to -1 from drug count as start date gets counted from notnull above
|
||||
for d in range(drug_count - 1):
|
||||
UPID_date_var = str(x.name) + str(x[d])
|
||||
date = df.loc[UPID_date_var, "Intervention Date"]
|
||||
date_string.append(date)
|
||||
return date_string
|
||||
|
||||
|
||||
def list_to_string(x):
|
||||
list = x.ids.split(' - ')
|
||||
drug_list = list[len(list) - len(x.average_cost):]
|
||||
ret_string = ""
|
||||
for y in range(len(x.average_cost)):
|
||||
if (round(x.average_spacing[y], 0) > 1) and (round(x.average_administered[y], 0) > 2.5) and (int(x.value) > 0):
|
||||
string = "<br><b>" + str(drug_list[y]) + "</b><br>On average given " + str(
|
||||
round(x.average_administered[y], 1)) + \
|
||||
" times with a " + str(round(int(x.average_spacing[y]) / 7, 1)) + " weekly interval (" \
|
||||
+ str(round((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1),
|
||||
0)) + " weeks total treatment length)"
|
||||
#"<br>Average annual cost per annum:" + \
|
||||
#str(human_format(
|
||||
# (x.cost / x.value) / (((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1))/ 52)))
|
||||
else:
|
||||
string = "<br><b>" + str(drug_list[y]) + "</b><br>On average given " + str(
|
||||
round(x.average_administered[y], 1)) + \
|
||||
" times with a " + str(round(int(x.average_spacing[y]) / 7, 1)) + " weekly interval (" \
|
||||
+ str(round((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1),
|
||||
0)) + " weeks total treatment length)"
|
||||
#"<br>Average annual cost per annum unavailable"
|
||||
|
||||
ret_string += string
|
||||
|
||||
return ret_string
|
||||
|
||||
|
||||
def drug_frequency_average(x):
|
||||
drug_count = x.index.str.contains("drug_").sum()
|
||||
freq = []
|
||||
for d in range(drug_count):
|
||||
if x["freq_" + str(d)] > 1:
|
||||
duration = ((x["end_date_" + str(d)] - x["start_date_" + str(d)]) / np.timedelta64(1, 'D'))
|
||||
if duration > 0:
|
||||
freq_calc = duration / (x["freq_" + str(d)] - 1)
|
||||
else:
|
||||
freq_calc = 0
|
||||
else:
|
||||
freq_calc = 0
|
||||
freq.append(freq_calc)
|
||||
return freq
|
||||
|
||||
|
||||
def cost_pp_pa(x):
|
||||
if x["avg_days"]/ np.timedelta64(1, 'D') > 0:
|
||||
return str(round(x["costpp"] / ((x["avg_days"] / np.timedelta64(1, 'D')) / 365), 2))
|
||||
else:
|
||||
return "N/A"
|
||||
|
||||
|
||||
def generate_graph(
|
||||
df1,
|
||||
start_date=None,
|
||||
end_date=None,
|
||||
last_seen=None,
|
||||
save_dir=None,
|
||||
trustFilter=None,
|
||||
drugFilter=None,
|
||||
directorateFilter=None,
|
||||
title=None,
|
||||
minimum_num_patients=None,
|
||||
*,
|
||||
filters: Optional[AnalysisFilters] = None,
|
||||
paths: Optional[PathConfig] = None,
|
||||
):
|
||||
"""
|
||||
Generate patient pathway icicle chart.
|
||||
|
||||
This function can be called in two ways:
|
||||
1. New style: Pass filters=AnalysisFilters(...) with all parameters encapsulated
|
||||
2. Legacy style: Pass individual parameters (start_date, end_date, etc.)
|
||||
|
||||
If both are provided, the filters object takes precedence.
|
||||
|
||||
Args:
|
||||
df1: DataFrame with processed patient data
|
||||
filters: AnalysisFilters object with all filter parameters (preferred)
|
||||
paths: PathConfig object for file paths (optional, uses default_paths if not provided)
|
||||
|
||||
Legacy parameters (used if filters is None):
|
||||
start_date, end_date, last_seen, save_dir, trustFilter, drugFilter,
|
||||
directorateFilter, title, minimum_num_patients
|
||||
"""
|
||||
# Use PathConfig for file paths
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
# Extract parameters from AnalysisFilters if provided
|
||||
if filters is not None:
|
||||
start_date = filters.start_date
|
||||
end_date = filters.end_date
|
||||
last_seen = filters.last_seen_date
|
||||
save_dir = filters.output_dir
|
||||
trustFilter = filters.trusts
|
||||
drugFilter = filters.drugs
|
||||
directorateFilter = filters.directories
|
||||
title = filters.custom_title
|
||||
minimum_num_patients = filters.minimum_patients
|
||||
|
||||
df1["UPIDTreatment"] = df1["UPID"] + df1["Drug Name"]
|
||||
|
||||
# Get average number of doses count
|
||||
org_codes = pd.read_csv(paths.org_codes_csv, index_col=1)
|
||||
df1["Provider Code"] = df1["Provider Code"].map(org_codes["Name"])
|
||||
#df1.to_csv("./df1.csv", index=False)
|
||||
|
||||
df1 = df1[(df1["Provider Code"].isin(trustFilter)) & (df1["Drug Name"].isin(drugFilter)) & (df1["Directory"].isin(directorateFilter))]
|
||||
|
||||
if len(df1) == 0:
|
||||
logger.warning("No data found for selected filters.")
|
||||
return
|
||||
|
||||
# Find total cost for each patient - Total cost is ~£110Mil, about 30% is unattributable to a patient (no UPID)
|
||||
cost_df = df1[["UPID", "Price Actual"]]
|
||||
total_costs = pd.DataFrame(cost_df.groupby("UPID").sum())
|
||||
total_costs.rename(columns={"Price Actual": "Total cost"}, inplace=True)
|
||||
|
||||
# Series to map directory
|
||||
directory_df = df1[["UPID", "Directory"]]
|
||||
directory_df.drop_duplicates("UPID", inplace=True)
|
||||
directory_df.set_index("UPID", inplace=True)
|
||||
logger.info("Filtering unrelated interventions")
|
||||
|
||||
df_end_dates = drop_duplicate_treatments(df1, False)
|
||||
df1_unique = drop_duplicate_treatments(df1, True)
|
||||
logger.info("Identifying unique patients and interventions used")
|
||||
# Create list of total number of that drug for each patient
|
||||
df_drug_freq = df1.groupby("UPID").agg({"Drug Name": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_drug_cost = df1.groupby("UPID").agg({"Price Actual": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_drug_freq["Price Actual"] = df_drug_freq.index.map(df_drug_cost["Price Actual"])
|
||||
#df_drug_freq["Price Actual"] = df_drug_freq["Price Actual"].map(df_drug_cost)
|
||||
df_drug_freq["Drug Name"] = df_drug_freq["Drug Name"].apply(count_list_values)
|
||||
df_drug_freq["Drug cost total"] = df_drug_freq.apply(lambda x: sum_list_values(x), axis=1)
|
||||
|
||||
|
||||
# Aggregate interventions & dates of interventions into transposed list by UPID
|
||||
df_drugs = df1_unique.groupby("UPID").agg({"Drug Name": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_dates = df1_unique.groupby("UPID").agg({"Intervention Date": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_end_dates = df_end_dates.groupby("UPID").agg({"Intervention Date": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
|
||||
logger.info("Calculating each unique patient's intervention average frequency, cost and duration of each intervention")
|
||||
# The following sh*t show is to unwrap the lists into columns for different drugs, start/end dates, and average
|
||||
# frequency/average total injections of each one
|
||||
df_dates_unwrapped = pd.DataFrame(df_dates["Intervention Date"].values.tolist(), index=df_dates.index).add_prefix(
|
||||
'date_')
|
||||
df_end_dates_unwrapped = pd.DataFrame(df_end_dates["Intervention Date"].values.tolist(), index=df_end_dates.index).add_prefix(
|
||||
'date_end_')
|
||||
df_drugs_unwrapped = pd.DataFrame(df_drugs["Drug Name"].values.tolist(), index=df_drugs.index).add_prefix('drug_')
|
||||
|
||||
df_freq_unwrapped = pd.DataFrame(df_drug_freq["Drug Name"].values.tolist(), index=df_drug_freq.index).add_prefix(
|
||||
'freq_')
|
||||
start_dates = df1[["UPIDTreatment", "Intervention Date"]].sort_values(by=["Intervention Date"], ascending=True,
|
||||
inplace=False,
|
||||
ignore_index=True).drop_duplicates(
|
||||
subset="UPIDTreatment").set_index("UPIDTreatment")
|
||||
end_dates = df1[["UPIDTreatment", "Intervention Date"]].sort_values(by=["Intervention Date"], ascending=False,
|
||||
inplace=False,
|
||||
ignore_index=True).drop_duplicates(
|
||||
subset="UPIDTreatment").set_index("UPIDTreatment")
|
||||
|
||||
|
||||
|
||||
df_drugs_unwrapped["start_dates"] = df_drugs_unwrapped.apply(lambda x: start_date_drug(start_dates, x), axis=1)
|
||||
|
||||
df_ddrugs_unwrapped = pd.DataFrame(df_drugs_unwrapped["start_dates"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'start_date_')
|
||||
df_drugs_unwrapped.drop(["start_dates"], inplace=True, axis=1)
|
||||
df_drugs_unwrapped["end_dates"] = df_drugs_unwrapped.apply(lambda x: start_date_drug(end_dates, x), axis=1)
|
||||
df_dddrugs_unwrapped = pd.DataFrame(df_drugs_unwrapped["end_dates"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'end_date_')
|
||||
|
||||
df_drugs_unwrapped.drop(["end_dates"], inplace=True, axis=1)
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_ddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_dddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_dddddrugs_unwrapped = pd.DataFrame(df_drug_freq["Drug Name"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'freq_')
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_dddddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_drugs_unwrapped["frequency"] = df_drugs_unwrapped.apply(lambda x: drug_frequency_average(x), axis=1)
|
||||
|
||||
df_ddddddrugs_unwrapped = pd.DataFrame(df_drugs_unwrapped["frequency"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'spacing_')
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_ddddddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_dddddddrugs_unwrapped = pd.DataFrame(df_drug_freq["Drug cost total"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix('total_cost_drug_')
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_dddddddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_drugs_unwrapped.drop(["frequency"], inplace=True, axis=1)
|
||||
|
||||
# Insert first & last date seen into df (need to add last date seen)
|
||||
df_drugs_unwrapped.insert(0, "First seen", df_dates_unwrapped.min(axis=1))
|
||||
df_drugs_unwrapped.insert(1, "Last seen", df_end_dates_unwrapped.max(axis=1))
|
||||
|
||||
# Merge info from activity data with grouped info, and total cost info
|
||||
patient_info = df1.drop_duplicates(subset="UPID", keep="first").set_index("UPID")
|
||||
patient_info = pd.merge(patient_info, df_drugs_unwrapped, left_index=True, right_index=True)
|
||||
patient_info = pd.merge(patient_info, df_freq_unwrapped, left_index=True, right_index=True)
|
||||
patient_info = pd.merge(patient_info, total_costs, left_index=True, right_index=True)
|
||||
|
||||
#patient_info.to_csv("patient_info.csv", index=False)
|
||||
|
||||
# Filter initiation based on years provided
|
||||
patient_info = patient_info[(patient_info['First seen'] >= str(start_date)) & (
|
||||
patient_info['First seen'] < str(end_date))]
|
||||
if title == "":
|
||||
title = "Patients initiated from " + str(start_date) + " to " + str(end_date)
|
||||
|
||||
# Filter last seen based on date provided
|
||||
patient_info = patient_info[patient_info['Last seen'] > str(last_seen)]
|
||||
|
||||
# Remove patients with 0 drug, by filling blanks with NaN & dropping rows
|
||||
patient_info.drug_0.replace('N/A', np.nan, inplace=True)
|
||||
patient_info.dropna(subset=['drug_0'], inplace=True)
|
||||
|
||||
# Calculate duation of treatment
|
||||
patient_info['Days treated'] = patient_info["Last seen"] - patient_info["First seen"]
|
||||
date_df = patient_info[["First seen", "Last seen", 'Days treated']]
|
||||
|
||||
# Create df for ice chart with hierarchy of plot
|
||||
number_of_drugs = np.count_nonzero(patient_info.columns.str.startswith('drug_'))
|
||||
final_drug_index = patient_info.columns.to_list().index("drug_" + str(number_of_drugs - 1))
|
||||
|
||||
upid_drugs_df = patient_info.iloc[:, (final_drug_index - number_of_drugs + 1):final_drug_index + 1]
|
||||
|
||||
upid_drugs_df.insert(0, "Trust", upid_drugs_df.index.str[:3])
|
||||
upid_drugs_df.insert(1, "Directory", upid_drugs_df.index)
|
||||
|
||||
upid_drugs_df["Trust"] = upid_drugs_df["Trust"].map(org_codes["Name"])
|
||||
upid_drugs_df["Directory"] = upid_drugs_df["Directory"].map(directory_df["Directory"])
|
||||
|
||||
l_df = pd.DataFrame()
|
||||
ice_df2 = pd.DataFrame()
|
||||
ice_df = pd.DataFrame()
|
||||
|
||||
upid_drugs_df["value"] = upid_drugs_df.apply(lambda x: row_function(x), axis=1)
|
||||
# Merge in date info
|
||||
upid_drugs_df = pd.merge(upid_drugs_df, date_df, left_index=True, right_index=True)
|
||||
|
||||
upid_drugs_df["ids"] = upid_drugs_df["value"].str.split(',').str[2]
|
||||
avg_treatment_dfs = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)["Days treated"].mean()).set_index("ids")
|
||||
value_dfs = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False).size()).reset_index()
|
||||
first_seen_treatment_dfs = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)["First seen"].min()).set_index(
|
||||
"ids")
|
||||
last_seen_treatment_dfs = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)["Last seen"].max()).set_index(
|
||||
"ids")
|
||||
|
||||
# Calculate total cost for parents
|
||||
upid_drugs_df["Cost"] = upid_drugs_df.index.map(total_costs["Total cost"])
|
||||
cost_dfs = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)['Cost'].sum()).set_index("value", drop=True)
|
||||
|
||||
# Calculate average dosing for each drug
|
||||
upid_drugs_df = pd.merge(upid_drugs_df, df_drugs_unwrapped, left_index=True, right_index=True)
|
||||
# frequency_dfs = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)['Cost'].sum()).set_index("value", drop=True)
|
||||
|
||||
# Calculate average spacing between drugs
|
||||
spacing_average = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if 'spacing_' in col]].mean()).set_index(
|
||||
"value", drop=True)
|
||||
spacing_average = spacing_average.round()
|
||||
spacing_average['combined'] = spacing_average.values.tolist()
|
||||
spacing_average["ids"] = spacing_average.index
|
||||
spacing_average["ids"] = spacing_average["ids"].str.split(',').str[2]
|
||||
spacing_average.set_index("ids", inplace=True)
|
||||
|
||||
# Calculate average cost for each drug
|
||||
cost_average = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if 'total_cost_drug_' in col]].mean()).set_index(
|
||||
"value", drop=True)
|
||||
cost_average = cost_average.round(2)
|
||||
cost_average['combined'] = cost_average.values.tolist()
|
||||
cost_average["ids"] = cost_average.index
|
||||
cost_average["ids"] = cost_average["ids"].str.split(',').str[2]
|
||||
cost_average.set_index("ids", inplace=True)
|
||||
|
||||
|
||||
# Calculate average number of doses
|
||||
freq_average = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if 'freq_' in col]].mean()).set_index("ids",
|
||||
drop=True)
|
||||
# freq_average = freq_average.round()
|
||||
freq_average['combined'] = freq_average.values.tolist()
|
||||
|
||||
# Remove negative totals from "Cost" column
|
||||
num = cost_dfs._get_numeric_data()
|
||||
num[num < 0] = 0
|
||||
|
||||
value_dfs["Cost"] = value_dfs["value"].map(cost_dfs["Cost"])
|
||||
|
||||
ice_df[['parents', 'labels', 'ids']] = value_dfs["value"].str.split(',', expand=True)
|
||||
# ice_df["index"] = ice_df.ids
|
||||
# ice_df.set_index("index", inplace=True)
|
||||
|
||||
ice_df["average_administered"] = ice_df["ids"].map(freq_average["combined"])
|
||||
ice_df["cost"] = value_dfs["Cost"]
|
||||
ice_df["value"] = value_dfs["size"]
|
||||
|
||||
ice_df["average_cost"] = ice_df["ids"].map(cost_average["combined"])
|
||||
ice_df["average_cost"] = ice_df["average_cost"].apply(remove_nan_string)
|
||||
|
||||
ice_df["average_spacing"] = ice_df["ids"].map(spacing_average["combined"])
|
||||
ice_df["average_spacing"] = ice_df["average_spacing"].apply(remove_nan_string)
|
||||
ice_df["average_spacing"] = ice_df.apply(lambda x: list_to_string(x), axis=1)
|
||||
ice_df["average_spacing"] = ice_df["average_spacing"].str.replace("nan", "N/A")
|
||||
|
||||
|
||||
logger.info("Building graph dataframe structure.")
|
||||
# Add very top level of Trust
|
||||
new_row = pd.DataFrame({'parents': '', 'ids': "N&WICS", 'labels': 'N&WICS', 'value': 0, "cost": 0}, index=[0])
|
||||
ice_df = pd.concat(objs=[ice_df, new_row], ignore_index=True, axis=0)
|
||||
|
||||
# need to add parents as blocks...
|
||||
l3 = [x for x in ice_df.parents.unique() if x not in ice_df.ids]
|
||||
while len(l3) > 1:
|
||||
for l in l3:
|
||||
z = l.rfind("-")
|
||||
if z > 0:
|
||||
l_dict = {"parents": l[:z - 1], "ids": l, "value": 0, "labels": l[z + 2:], "cost": 0}
|
||||
l_df = pd.concat([l_df, pd.DataFrame(l_dict, index=[0])], ignore_index=True)
|
||||
ice_df2 = pd.concat([ice_df, l_df], ignore_index=True)
|
||||
l3 = [x for x in ice_df2.parents.unique() if x not in ice_df2.ids.unique()]
|
||||
ice_df = ice_df2.drop_duplicates("ids")
|
||||
|
||||
ice_df["level"] = ice_df["ids"].str.count('-')
|
||||
ice_df = ice_df[~ice_df['labels'].isin(["COST", "CHARGE", "N/A"])]
|
||||
ice_df.sort_values(by=["level"], ascending=False, inplace=True, ignore_index=True)
|
||||
|
||||
for index, row in ice_df.iterrows():
|
||||
lookup_index = ice_df.index[ice_df['ids'] == row['parents']]
|
||||
ice_df.loc[lookup_index, 'value'] = ice_df.loc[lookup_index, "value"] + ice_df.loc[index, "value"]
|
||||
ice_df.loc[lookup_index, 'cost'] = ice_df.loc[lookup_index, "cost"] + ice_df.loc[index, 'cost']
|
||||
|
||||
# Sum of parent values to create denominator for percentage - FOR PATIENT NUMBER COLOUR GRADING
|
||||
colour_df = pd.DataFrame(ice_df.groupby(["parents"])["value"].sum())
|
||||
ice_df['colour'] = ice_df["parents"].map(colour_df["value"])
|
||||
ice_df['colour'] = ice_df['value']/ice_df['colour']
|
||||
|
||||
# Sum of parent values to create denominator for percentage - FOR COST COLOUR GRADING
|
||||
#colour_df = pd.DataFrame(ice_df.groupby(["parents"])["cost"].sum())
|
||||
#ice_df['colour'] = ice_df["parents"].map(colour_df["cost"])
|
||||
#ice_df['colour'] = ice_df['cost'] / ice_df['colour']
|
||||
|
||||
|
||||
ice_df['costpp'] = ice_df['cost'] / ice_df['value']
|
||||
# Treatment length info
|
||||
ice_df['avg_days'] = ice_df["ids"].map(avg_treatment_dfs["Days treated"])
|
||||
ice_df['First seen'] = ice_df["ids"].map(first_seen_treatment_dfs["First seen"])
|
||||
ice_df['Last seen'] = ice_df["ids"].map(last_seen_treatment_dfs["Last seen"])
|
||||
|
||||
ice_df["dates"] = ice_df.apply(lambda x: min_max_treatment_dates(ice_df, x), axis=1)
|
||||
ice_df[['First seen (Parent)', 'Last seen (Parent)']] = ice_df["dates"].str.split(',', expand=True)
|
||||
|
||||
# Sort labels to be alphabetical
|
||||
# ice_df.sort_values(by=["labels"], ascending=True, inplace=True, ignore_index=True)
|
||||
ice_df['First seen'] = pd.to_datetime(ice_df['First seen'])
|
||||
ice_df['Last seen'] = pd.to_datetime(ice_df['Last seen'])
|
||||
ice_df["cost_pp_pa"] = ice_df.apply(lambda x: cost_pp_pa(x), axis=1)
|
||||
|
||||
# Filter out rows where value is less than minimum number of patients
|
||||
ice_df = ice_df[ice_df['value'] >= minimum_num_patients]
|
||||
|
||||
logger.info("Generating graph.")
|
||||
|
||||
figure(ice_df, title, save_dir)
|
||||
return
|
||||
|
||||
|
||||
def figure(ice_df4, dir_string, save_dir):
|
||||
"""
|
||||
Create and display icicle figure (legacy interface).
|
||||
|
||||
This function delegates to visualization.plotly_generator.figure_legacy()
|
||||
for backward compatibility.
|
||||
|
||||
Args:
|
||||
ice_df4: DataFrame with chart data
|
||||
dir_string: Title string (used for filename and chart title)
|
||||
save_dir: Directory to save the HTML file
|
||||
"""
|
||||
_figure_legacy(ice_df4, dir_string, save_dir)
|
||||
return
|
||||
|
||||
|
||||
# fig = go.Figure(go.Icicle(
|
||||
# labels=ice_df4.labels,
|
||||
# ids=ice_df4.ids,
|
||||
# # count="branches",
|
||||
# parents=ice_df4.parents,
|
||||
# customdata=np.stack((ice_df4.value, ice_df4.colour, ice_df4.cost, ice_df4.costpp, first_seen, last_seen,
|
||||
# first_seen_parent, last_seen_parent, average_spacing, ice_df4.cost_pp_pa), axis=1),
|
||||
# values=ice_df4.value,
|
||||
# branchvalues="total",
|
||||
# marker=dict(
|
||||
# colors=ice_df4.colour,
|
||||
# colorscale='Viridis'),
|
||||
# maxdepth=3,
|
||||
# texttemplate='<b>%{label}</b> '
|
||||
# '<br><b>Total patients:</b> %{customdata[0]} - %{customdata[1]:.3p} of patients in level'
|
||||
# '<br><b>Total cost:</b> £%{customdata[2]:.3~s}'
|
||||
# '<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}'
|
||||
# '<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}',
|
||||
# hovertemplate='<b>%{label}</b>'
|
||||
# '<br><b>Total patients:</b> %{customdata[0]} - %{customdata[1]:.3p} of patients in level'
|
||||
# '<br><b>Total cost:</b> £%{customdata[2]:.3~s}'
|
||||
# '<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}'
|
||||
# '<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}'
|
||||
# '<br><b>First seen:</b> %{customdata[4]}'
|
||||
# '<br><b>Last seen (including further treatments):</b> %{customdata[7]}'
|
||||
# '<br><b>Average treatment duration:</b>'
|
||||
# '%{customdata[8]}'
|
||||
# '<extra></extra>',
|
||||
# ))
|
||||
#
|
||||
#import os
|
||||
#def main():
|
||||
# input = "ice_df.csv"
|
||||
# save_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
# dir = "debugging"
|
||||
# ice_df4 = pd.read_csv(input)
|
||||
#
|
||||
# ice_df4['First seen'] = pd.to_datetime(ice_df4['First seen'])
|
||||
# ice_df4['avg_days'] = pd.to_timedelta(ice_df4['avg_days'])
|
||||
# ice_df4['Last seen'] = pd.to_datetime(ice_df4['Last seen'])
|
||||
# figure(ice_df4, dir, save_dir)
|
||||
#
|
||||
#if __name__ == "__main__":
|
||||
# main()
|
||||
|
||||
|
||||
def generate_graph_v2(
|
||||
df: pd.DataFrame,
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
last_seen_date: str,
|
||||
save_dir: str,
|
||||
trust_filter: list[str],
|
||||
drug_filter: list[str],
|
||||
directory_filter: list[str],
|
||||
minimum_num_patients: int = 0,
|
||||
title: str = "",
|
||||
paths: Optional[PathConfig] = None,
|
||||
) -> Optional[go.Figure]:
|
||||
"""
|
||||
Generate patient pathway icicle chart using refactored pipeline.
|
||||
|
||||
This is the modern API that uses the refactored analysis functions.
|
||||
It provides cleaner parameter names and returns the figure instead of
|
||||
automatically opening it in a browser.
|
||||
|
||||
Args:
|
||||
df: DataFrame with processed patient intervention data
|
||||
start_date: Start date for patient initiation filter (YYYY-MM-DD)
|
||||
end_date: End date for patient initiation filter (YYYY-MM-DD)
|
||||
last_seen_date: Filter for patients last seen after this date
|
||||
save_dir: Directory to save the HTML file
|
||||
trust_filter: List of trust names to include
|
||||
drug_filter: List of drug names to include
|
||||
directory_filter: List of directories to include
|
||||
minimum_num_patients: Minimum number of patients to include a pathway
|
||||
title: Chart title (auto-generated from dates if empty)
|
||||
paths: PathConfig for file paths (uses default if None)
|
||||
|
||||
Returns:
|
||||
Plotly Figure object, or None if no data
|
||||
"""
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
ice_df, final_title = _generate_icicle_chart(
|
||||
df=df,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
last_seen_date=last_seen_date,
|
||||
trust_filter=trust_filter,
|
||||
drug_filter=drug_filter,
|
||||
directory_filter=directory_filter,
|
||||
minimum_num_patients=minimum_num_patients,
|
||||
title=title,
|
||||
paths=paths,
|
||||
)
|
||||
|
||||
if ice_df is None or len(ice_df) == 0:
|
||||
return None
|
||||
|
||||
fig = create_icicle_figure(ice_df, final_title)
|
||||
|
||||
if save_dir:
|
||||
fig.write_html(f"{save_dir}/{final_title}.html")
|
||||
logger.info(f"Success! File saved to {save_dir}/{final_title}.html")
|
||||
|
||||
return fig
|
||||
|
||||
|
||||
def create_icicle_figure(ice_df: pd.DataFrame, title: str) -> go.Figure:
|
||||
"""
|
||||
Create Plotly icicle figure from prepared DataFrame.
|
||||
|
||||
This function delegates to visualization.plotly_generator.create_icicle_figure()
|
||||
for the actual figure generation.
|
||||
|
||||
Args:
|
||||
ice_df: DataFrame with parents, ids, labels, value, colour etc.
|
||||
title: Chart title
|
||||
|
||||
Returns:
|
||||
Plotly Figure object
|
||||
"""
|
||||
return _create_icicle_figure(ice_df, title)
|
||||
+331
@@ -0,0 +1,331 @@
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import csv
|
||||
import urllib.request
|
||||
import io # Added for StringIO
|
||||
import re # Added for regex escape and word boundaries
|
||||
from typing import Optional
|
||||
|
||||
from core import PathConfig, default_paths
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
def drug_names(df, paths: Optional[PathConfig] = None):
|
||||
# Generate dictionary to convert drug names from activity data to generic standardisation
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
d = {}
|
||||
with open(paths.drugnames_csv, 'r', newline='') as f:
|
||||
reader = csv.reader(f, delimiter=',')
|
||||
for drug_name, generic in reader:
|
||||
d[drug_name.upper()] = generic.upper()
|
||||
|
||||
# Map drug names with dictionary generated earlier
|
||||
df["Drug Name"] = df["Drug Name"].str.upper().map(d)
|
||||
|
||||
# Remove (Left eye) or (Right eye) from Drug Name, including whitespace
|
||||
df["Drug Name"] = df["Drug Name"].str.replace(r'\(LEFT EYE\)', '', regex=True) # Escaped parentheses
|
||||
df["Drug Name"] = df["Drug Name"].str.replace(r'\(RIGHT EYE\)', '', regex=True) # Escaped parentheses
|
||||
df["Drug Name"] = df["Drug Name"].str.strip()
|
||||
return df
|
||||
|
||||
|
||||
def patient_id(df):
|
||||
# Generate unique patient ID
|
||||
df["UPID"] = df["Provider Code"].str[:3] + df["PersonKey"].astype(str)
|
||||
return df
|
||||
|
||||
|
||||
def compress_csv(filepath):
|
||||
df = pd.read_csv(filepath)
|
||||
compressed_path = filepath.replace(".csv", "_bz2.csv")
|
||||
df.to_csv(compressed_path, compression="bz2", index=False)
|
||||
return compressed_path
|
||||
|
||||
|
||||
def department_identification(df, paths: Optional[PathConfig] = None):
|
||||
# --- Setup ---
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
# 1. Load directory_list.csv and prepare uppercase versions/pattern
|
||||
try:
|
||||
directory_df = pd.read_csv(paths.directory_list_csv)
|
||||
directory_list = directory_df["directory"].dropna().astype(str).tolist()
|
||||
if not directory_list:
|
||||
raise ValueError("directory_list.csv is empty or contains only NA values.")
|
||||
directory_list_upper = [d.upper() for d in directory_list]
|
||||
# Use word boundaries (\b) to avoid partial matches within words, escape special regex chars
|
||||
dir_pattern_upper = r'\b({})'.format('|'.join(map(re.escape, directory_list_upper)))
|
||||
except FileNotFoundError:
|
||||
logger.error(f"File not found: {paths.directory_list_csv}. Cannot extract directories.")
|
||||
return df
|
||||
except ValueError as e:
|
||||
logger.error(f"Error loading directory list: {e}")
|
||||
return df
|
||||
|
||||
# Simpler pattern for Primary_Source (no word boundaries)
|
||||
dir_pattern_primary_simple = r'({})'.format('|'.join(map(re.escape, directory_list_upper)))
|
||||
|
||||
# 2. Load treatment_function_codes.csv and prepare uppercase mapping
|
||||
treatment_codes = pd.read_csv(paths.treatment_function_codes_csv)
|
||||
mapping_treatment_codes = dict(treatment_codes[['Code', 'Service']].values)
|
||||
mapping_treatment_codes_upper = {k: str(v).upper() for k, v in mapping_treatment_codes.items()}
|
||||
|
||||
# 3. Load drug_directory_list.csv and parse into drug_to_valid_dirs
|
||||
drug_to_valid_dirs: dict[str, set[str]] = {}
|
||||
# Try pandas direct read - much simpler approach
|
||||
drug_dir_df = pd.read_csv(paths.drug_directory_list_csv, skipinitialspace=True)
|
||||
|
||||
# Identify the drug name column (first column) and directory column (second column)
|
||||
drug_col = drug_dir_df.columns[0]
|
||||
dir_col = drug_dir_df.columns[1]
|
||||
|
||||
# Process dataframe directly
|
||||
drug_to_valid_dirs = {}
|
||||
for _, row in drug_dir_df.iterrows():
|
||||
drug_name = str(row[drug_col]).strip().upper()
|
||||
try:
|
||||
# Directories are pipe-separated in the second column
|
||||
dirs_str = str(row[dir_col]) if not pd.isna(row[dir_col]) else ""
|
||||
dirs = {d.strip().upper() for d in dirs_str.split('|') if d.strip()}
|
||||
if drug_name and dirs and drug_name.lower() != 'nan':
|
||||
drug_to_valid_dirs[drug_name] = dirs
|
||||
except Exception:
|
||||
# Silently continue on row errors
|
||||
continue
|
||||
# 4. Create drug_to_single_dir map
|
||||
drug_to_single_dir = {
|
||||
drug: list(dirs)[0]
|
||||
for drug, dirs in drug_to_valid_dirs.items()
|
||||
if len(dirs) == 1
|
||||
}
|
||||
|
||||
# --- Data Preprocessing ---
|
||||
# Keep original extraction columns list
|
||||
additional_detail_columns = ["Additional Detail 1", "Additional Description 1", "Additional Detail 2", "Additional Description 2",
|
||||
"Additional Detail 3", "Additional Description 3", "Additional Detail 4", "Additional Description 4",
|
||||
"Additional Detail 5", "Additional Description 5", "NCDR Treatment Function Name", "Treatment Function Desc"]
|
||||
|
||||
# 6. Convert detail columns to uppercase BEFORE extraction
|
||||
for ad in additional_detail_columns:
|
||||
# Check if column exists and is object/string type before applying .str
|
||||
if ad in df.columns and pd.api.types.is_object_dtype(df[ad]):
|
||||
df[ad] = df[ad].str.upper()
|
||||
|
||||
# Original extraction loop (using original case list for extraction)
|
||||
# Extract directory from specified columns
|
||||
directory_df = pd.read_csv(paths.directory_list_csv)
|
||||
directory_list = directory_df["directory"].tolist() # Reload original case list
|
||||
|
||||
for ad in additional_detail_columns:
|
||||
try:
|
||||
# Ensure column is string type before cleaning
|
||||
if pd.api.types.is_string_dtype(df[ad]):
|
||||
# Extract directly from the uppercased string column
|
||||
extracted = df[ad].str.extract(dir_pattern_upper, expand=False)
|
||||
df.loc[extracted.index, ad] = extracted
|
||||
else:
|
||||
df[ad] = np.nan # Set non-string columns to NaN
|
||||
except AttributeError: # Skip columns that might not exist or are not string type
|
||||
df[ad] = np.nan # Ensure column exists but set to NaN if error
|
||||
except Exception as e: # Catch other potential errors during extract
|
||||
logger.error(f"Error processing column {ad}: {e}")
|
||||
df[ad] = np.nan
|
||||
|
||||
# 7. Process Treatment Function Code
|
||||
df["Treatment Function Code"].replace(np.nan, 0, inplace=True)
|
||||
# Ensure it's int type before mapping, handle potential errors
|
||||
try:
|
||||
df["Treatment Function Code"] = df["Treatment Function Code"].astype(int)
|
||||
except ValueError:
|
||||
# Handle cases where conversion to int fails (e.g., non-numeric values)
|
||||
# Try coercing errors to NaN, then fillna with 0
|
||||
df["Treatment Function Code"] = pd.to_numeric(df["Treatment Function Code"], errors='coerce').fillna(0).astype(int)
|
||||
|
||||
df["Treatment Function Code"] = df["Treatment Function Code"].map(mapping_treatment_codes_upper)
|
||||
df.rename(columns={'Treatment Function Code': 'Fallback_Source'}, inplace=True)
|
||||
|
||||
# Apply replacements before combining
|
||||
df.replace('MEDICAL OPHTHALMOLOGY', 'OPHTHALMOLOGY', inplace=True)
|
||||
|
||||
# --- Single Directory Assignment ---
|
||||
# 8. Apply single directory override
|
||||
# Ensure Drug Name is suitable for mapping (already done in drug_names func)
|
||||
df['Directory'] = df['Drug Name'].map(drug_to_single_dir)
|
||||
|
||||
# Initialize Directory_Source column - track which fallback level was used
|
||||
df['Directory_Source'] = pd.NA
|
||||
# Mark rows where single valid directory was assigned
|
||||
df.loc[df['Directory'].notna(), 'Directory_Source'] = 'SINGLE_VALID_DIR'
|
||||
|
||||
# --- Prepare Fallback Logic ---
|
||||
# 9. Create Primary source from Additional Detail 1
|
||||
if 'Additional Detail 1' in df.columns:
|
||||
df['Primary_Source'] = df['Additional Detail 1'].astype(pd.StringDtype())
|
||||
df['Primary_Source'] = df['Primary_Source'].str.upper() # Apply upper to strings
|
||||
else:
|
||||
df['Primary_Source'] = pd.NA # Use pd.NA for StringDtype
|
||||
|
||||
# Extract actual directory name using the pattern
|
||||
try:
|
||||
# Use simpler pattern for primary source
|
||||
df['Extracted_Primary_Dir'] = df['Primary_Source'].str.extract(dir_pattern_primary_simple, expand=False, flags=re.IGNORECASE)
|
||||
df['Extracted_Fallback_Dir'] = df['Fallback_Source'].str.extract(dir_pattern_upper, expand=False, flags=re.IGNORECASE)
|
||||
except Exception as e:
|
||||
logger.error(f"Error during directory extraction: {e}")
|
||||
# Assign NA columns if extraction fails
|
||||
df['Extracted_Primary_Dir'] = pd.NA
|
||||
df['Extracted_Fallback_Dir'] = pd.NA
|
||||
|
||||
# Strip potential whitespace from extracted directories
|
||||
if 'Extracted_Primary_Dir' in df.columns:
|
||||
df['Extracted_Primary_Dir'] = df['Extracted_Primary_Dir'].str.strip()
|
||||
if 'Extracted_Fallback_Dir' in df.columns:
|
||||
df['Extracted_Fallback_Dir'] = df['Extracted_Fallback_Dir'].str.strip()
|
||||
|
||||
# 10. Combine sources, prioritizing Primary_Source
|
||||
# Combine EXTRACTED directories
|
||||
df['Primary_Directory'] = df['Extracted_Primary_Dir'].fillna(df['Extracted_Fallback_Dir'])
|
||||
|
||||
# Track extraction source for Directory_Source column
|
||||
# Rows where we have Extracted_Primary_Dir will use EXTRACTED_PRIMARY
|
||||
# Rows where we only have Extracted_Fallback_Dir will use EXTRACTED_FALLBACK
|
||||
df['_extracted_source'] = pd.NA
|
||||
df.loc[df['Extracted_Primary_Dir'].notna(), '_extracted_source'] = 'EXTRACTED_PRIMARY'
|
||||
df.loc[(df['Extracted_Primary_Dir'].isna()) & (df['Extracted_Fallback_Dir'].notna()), '_extracted_source'] = 'EXTRACTED_FALLBACK'
|
||||
|
||||
# 11. Clean up intermediate columns
|
||||
df.drop(columns=['Primary_Source', 'Fallback_Source', 'Extracted_Primary_Dir', 'Extracted_Fallback_Dir'], inplace=True, errors='ignore')
|
||||
|
||||
# --- Identify Rows Needing Calculation ---
|
||||
# 12. Filter rows where Directory is not yet assigned
|
||||
df_to_process = df[df['Directory'].isnull()].copy()
|
||||
|
||||
# --- Calculate Most Frequent Valid Directory ---
|
||||
# 13. Drop rows without a potential primary directory
|
||||
df_to_process.dropna(subset=['Primary_Directory'], inplace=True)
|
||||
|
||||
# 14. Group and count potential directories
|
||||
if not df_to_process.empty:
|
||||
df_counts = df_to_process.groupby(['UPID', 'Drug Name', 'Primary_Directory'], observed=True)['Primary_Directory'].count().reset_index(name='count')
|
||||
|
||||
# 15. Sort by count descending
|
||||
df_counts.sort_values(['UPID', 'Drug Name', 'count'], ascending=[True, True, False], inplace=True)
|
||||
|
||||
# 16. Define helper function
|
||||
def find_first_valid_dir(group, drug_map):
|
||||
drug_name = group['Drug Name'].iloc[0]
|
||||
valid_dirs = drug_map.get(drug_name, set())
|
||||
|
||||
if not valid_dirs:
|
||||
return np.nan
|
||||
|
||||
for dir_candidate in group['Primary_Directory']:
|
||||
# Skip NA values
|
||||
if pd.isna(dir_candidate):
|
||||
continue
|
||||
|
||||
# Check if valid directory for this drug
|
||||
if isinstance(dir_candidate, str) and dir_candidate in valid_dirs:
|
||||
return dir_candidate
|
||||
|
||||
return np.nan # No valid directory found in the group
|
||||
|
||||
# 17. Group by UPID and Drug Name
|
||||
valid_groups = df_counts.groupby(['UPID', 'Drug Name'], observed=True, group_keys=False)
|
||||
|
||||
# 18. Apply helper function to find the best valid directory
|
||||
calculated_dirs = valid_groups.apply(lambda grp: find_first_valid_dir(grp, drug_to_valid_dirs))
|
||||
|
||||
# 19. Reset index to get UPID, Drug Name columns
|
||||
final_mapping = calculated_dirs.reset_index()
|
||||
|
||||
# 20. Rename the resulting column
|
||||
final_mapping.columns = ['UPID', 'Drug Name', 'Calculated_Directory']
|
||||
|
||||
# --- Merge Results and Finalize ---
|
||||
# 21. Merge calculated directories back to the main DataFrame
|
||||
df = pd.merge(df, final_mapping, on=['UPID', 'Drug Name'], how='left')
|
||||
|
||||
# 22. Fill NaN Directories with the calculated ones and track source
|
||||
# Find rows that will be filled from Calculated_Directory
|
||||
rows_to_fill = df['Directory'].isna() & df['Calculated_Directory'].notna()
|
||||
# For these rows, set Directory_Source based on _extracted_source (where the calculated dir came from)
|
||||
# The "calculated" directory is still derived from extraction, just via frequency analysis
|
||||
df.loc[rows_to_fill, 'Directory_Source'] = df.loc[rows_to_fill, '_extracted_source'].fillna('CALCULATED_MOST_FREQ')
|
||||
# Replace with the actual value of _extracted_source or fall back to CALCULATED_MOST_FREQ
|
||||
# Actually, let's simplify: if we're using the calculated most frequent directory, that's CALCULATED_MOST_FREQ
|
||||
df.loc[rows_to_fill, 'Directory_Source'] = 'CALCULATED_MOST_FREQ'
|
||||
|
||||
df['Directory'].fillna(df['Calculated_Directory'], inplace=True)
|
||||
|
||||
# 23. Drop temporary columns
|
||||
df.drop(columns=['Calculated_Directory', 'Primary_Directory', '_extracted_source'], inplace=True, errors='ignore')
|
||||
|
||||
else:
|
||||
# If df_to_process was empty, still need to drop temporary columns
|
||||
df.drop(columns=['Primary_Directory', '_extracted_source'], inplace=True, errors='ignore')
|
||||
|
||||
# 24. Drop rows with missing UPID (original logic)
|
||||
df['UPID'].replace('', np.nan, inplace=True) # Ensure empty strings are NaN
|
||||
df_orig = df.copy() # Save before dropna for future reference if needed
|
||||
df.dropna(subset=['UPID'], inplace=True)
|
||||
|
||||
# 25. Export rows with NA Directory to CSV for analysis (keep this for diagnostics)
|
||||
na_directory_rows = df[df['Directory'].isna()].copy()
|
||||
|
||||
# Export to CSV if there are any NA Directory rows
|
||||
if len(na_directory_rows) > 0:
|
||||
na_directory_rows.to_csv(paths.na_directory_rows_csv, index=False)
|
||||
|
||||
# 26. FALLBACK MECHANISM 1: Infer directory based on same UPID
|
||||
# Create a mapping of most frequent directory per UPID (only for UPIDs with a directory)
|
||||
if len(df[df['Directory'].isna()]) > 0:
|
||||
# First get valid directories per UPID
|
||||
valid_upid_dirs = df[df['Directory'].notna()].groupby('UPID')['Directory'].agg(
|
||||
lambda x: x.value_counts().index[0] if len(x.value_counts()) > 0 else None
|
||||
).to_dict()
|
||||
|
||||
# Apply UPID-based inference and track source
|
||||
for idx in df[df['Directory'].isna()].index:
|
||||
upid = df.loc[idx, 'UPID']
|
||||
if upid in valid_upid_dirs and valid_upid_dirs[upid] is not None:
|
||||
df.loc[idx, 'Directory'] = valid_upid_dirs[upid]
|
||||
df.loc[idx, 'Directory_Source'] = 'UPID_INFERENCE'
|
||||
|
||||
# 27. FALLBACK MECHANISM 2: Label remaining NA as "Undefined"
|
||||
# Track rows that will be marked as Undefined
|
||||
rows_undefined = df['Directory'].isna()
|
||||
df.loc[rows_undefined, 'Directory_Source'] = 'UNDEFINED'
|
||||
# Fill remaining NA directories with "Undefined"
|
||||
df['Directory'].fillna("Undefined", inplace=True)
|
||||
|
||||
# 28. Return the processed DataFrame
|
||||
return df
|
||||
|
||||
|
||||
|
||||
def ta_list_get(paths: Optional[PathConfig] = None):
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
link = "https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/TA%20recommendations.xlsx"
|
||||
urllib.request.urlretrieve(link, paths.ta_recommendations_xlsx)
|
||||
ta_db = pd.read_excel(paths.ta_recommendations_xlsx, index_col=0)
|
||||
|
||||
# Filter out TA's which are not Recommended or not Pharmaceutical
|
||||
ta_db = ta_db[ta_db["Categorisation (for specific recommendation)"].isin(["Recommended", "Optimised"])]
|
||||
ta_db = ta_db[ta_db["Technology type"] == "Pharmaceutical"]
|
||||
|
||||
# Amend TA001 strings to only the integer
|
||||
ta_db["TA ID"] = ta_db["TA ID"].str.replace(r'\D+', '', regex=True).astype(int)
|
||||
ta_db["TA ID"] = "NICE TA" + ta_db["TA ID"].astype(str)
|
||||
ta_series = ta_db[["TA ID", "Indication"]].drop_duplicates()
|
||||
return ta_series
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,712 @@
|
||||
version = 1
|
||||
requires-python = ">=3.10"
|
||||
resolution-markers = [
|
||||
"python_full_version >= '3.11'",
|
||||
"python_full_version < '3.11'",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "altgraph"
|
||||
version = "0.17.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/de/a8/7145824cf0b9e3c28046520480f207df47e927df83aa9555fb47f8505922/altgraph-0.17.4.tar.gz", hash = "sha256:1b5afbb98f6c4dcadb2e2ae6ab9fa994bbb8c1d75f4fa96d340f9437ae454406", size = 48418 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/4d/3f/3bc3f1d83f6e4a7fcb834d3720544ca597590425be5ba9db032b2bf322a2/altgraph-0.17.4-py2.py3-none-any.whl", hash = "sha256:642743b4750de17e655e6711601b077bc6598dbfa3ba5fa2b2a35ce12b508dff", size = 21212 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "babel"
|
||||
version = "2.12.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/ba/42/54426ba5d7aeebde9f4aaba9884596eb2fe02b413ad77d62ef0b0422e205/Babel-2.12.1.tar.gz", hash = "sha256:cc2d99999cd01d44420ae725a21c9e3711b3aadc7976d6147f622d8581963455", size = 9906735 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/df/c4/1088865e0246d7ecf56d819a233ab2b72f7d6ab043965ef327d0731b5434/Babel-2.12.1-py3-none-any.whl", hash = "sha256:b4246fb7677d3b98f501a39d43396d3cafdc8eadb045f4a31be01863f655c610", size = 10071794 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cramjam"
|
||||
version = "2.10.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/e9/dc/ccc87820b189e35323433e80de450bf2fb8826a5b64834c740e7d5e66ce2/cramjam-2.10.0.tar.gz", hash = "sha256:e821dd487384ae8004e977c3b13135ad6665ccf8c9874e68441cad1146e66d8a", size = 47801 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f0/83/3e5f558aebb0064b1d7b197869055118ee849ccc5d7a86520ba751a79cb9/cramjam-2.10.0-cp310-cp310-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:26c44f17938cf00a339899ce6ea7ba12af7b1210d707a80a7f14724fba39869b", size = 3514239 },
|
||||
{ url = "https://files.pythonhosted.org/packages/5d/34/de70de0a7e675d72d78b50f326451ea854f7f12608d3e093423bbe8fae1c/cramjam-2.10.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:ce208a3e4043b8ce89e5d90047da16882456ea395577b1ee07e8215dce7d7c91", size = 1841404 },
|
||||
{ url = "https://files.pythonhosted.org/packages/77/ae/5e12b524eb98c03a3c24c243c52894b633ee86c03c36c5e4b5d4738a6567/cramjam-2.10.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2c24907c972aca7b56c8326307e15d78f56199852dda1e67e4e54c2672afede4", size = 1678655 },
|
||||
{ url = "https://files.pythonhosted.org/packages/3a/d7/5adbd0b7bb55c5e40356949417e61ac4f950d656a49a8697a08a8b01d724/cramjam-2.10.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:f25db473667774725e4f34e738d644ffb205bf0bdc0e8146870a1104c5f42e4a", size = 2019539 },
|
||||
{ url = "https://files.pythonhosted.org/packages/db/c4/0cf4c9591b04a8e187df60defd920e3bb905b0db5a41d43e96213a0204d8/cramjam-2.10.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:51eb00c72d4a93e4a2ddcc751ba2a7a1318026247e80742866912ec82b39e5ce", size = 1752221 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f5/ca/0d06de89c531b4acf9782775a1527d1d498dc13f7abaa427c665a17ce86f/cramjam-2.10.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:def47645b1b970fd97f063da852b0ddc4f5bdee9af8d5b718d9682c7b828d89d", size = 1848859 },
|
||||
{ url = "https://files.pythonhosted.org/packages/b8/2e/f7f04638bd26808b9f4d03e988de12a06ca5db4551897c780a756ce44384/cramjam-2.10.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:42dcd7c83104edae70004a8dc494e4e57de4940e3019e5d2cbec2830d5908a85", size = 2003282 },
|
||||
{ url = "https://files.pythonhosted.org/packages/83/06/e2048df7a8e1b05a089c25ca0ac1b17c7aa4108c8d6328bf1f74314701b7/cramjam-2.10.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e0744e391ea8baf0ddea5a180b0aa71a6a302490c14d7a37add730bf0172c7c6", size = 2312472 },
|
||||
{ url = "https://files.pythonhosted.org/packages/aa/f5/5826951d6398d7f11baaef0ff15d510f7e90af2338af0a92d872adc51f70/cramjam-2.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5018c7414047f640b126df02e9286a8da7cc620798cea2b39bac79731c2ee336", size = 1964217 },
|
||||
{ url = "https://files.pythonhosted.org/packages/fd/4c/9a1282c4650a1aba666947214a1437973757463e9c60994c497fb9cb5cf5/cramjam-2.10.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:4b201aacc7a06079b063cfbcf5efe78b1e65c7279b2828d06ffaa90a8316579d", size = 2022270 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ac/e0/b78ab4ee7bcbd6116fdfe54cd771019bcc0d9039b81b070fe2780363c6f2/cramjam-2.10.0-cp310-cp310-musllinux_1_1_armv7l.whl", hash = "sha256:5264ac242697fbb1cfffa79d0153cbc4c088538bd99d60cfa374e8a8b83e2bb5", size = 2152240 },
|
||||
{ url = "https://files.pythonhosted.org/packages/94/0d/df2299892a7fa9b5d973111e81ee6772aaf27cc0489da41a34e66efe3cd5/cramjam-2.10.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:e193918c81139361f3f45db19696d31847601f2c0e79a38618f34d7bff6ee704", size = 2164031 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ee/39/67cc689fcba789076890c980472a40653749d91a8dc3165a8913a84f5670/cramjam-2.10.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:22a7ab05c62b0a71fcd6db4274af1508c5ea039a43fb143ac50a62f86e6f32f7", size = 2134442 },
|
||||
{ url = "https://files.pythonhosted.org/packages/85/4c/cd4bc9f05d76a127372b991e819b9eefd05a296adfc4f99ba0471033b528/cramjam-2.10.0-cp310-cp310-win32.whl", hash = "sha256:2464bdf0e2432e0f07a834f48c16022cd7f4648ed18badf52c32c13d6722518c", size = 1598011 },
|
||||
{ url = "https://files.pythonhosted.org/packages/4f/73/8ea115e1bcda57de7793211bd6b425bddffecd79a6b6d6a424ceaeed52bf/cramjam-2.10.0-cp310-cp310-win_amd64.whl", hash = "sha256:73b6ffc8ffe6546462ccc7e34ca3acd9eb3984e1232645f498544a7eab6b8aca", size = 1700050 },
|
||||
{ url = "https://files.pythonhosted.org/packages/15/a3/493dd4a4791ae14e4011d5fe7082a7aca8d31255f5cb50f930ede68561ce/cramjam-2.10.0-cp311-cp311-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:fb73ee9616e3efd2cf3857b019c66f9bf287bb47139ea48425850da2ae508670", size = 3514540 },
|
||||
{ url = "https://files.pythonhosted.org/packages/7a/26/22a5f8d408a0799b960ffcfa97f28c851e5800a904ef69988c3816819f79/cramjam-2.10.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:acef0e2c4d9f38428721a0ec878dee3fb73a35e640593d99c9803457dbb65214", size = 1841685 },
|
||||
{ url = "https://files.pythonhosted.org/packages/33/e8/76d0ae48c64007542b5563ae81712cf1c571f0bbbab45b778112e61c92b7/cramjam-2.10.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5b21b1672814ecce88f1da76635f0483d2d877d4cb8998db3692792f46279bf1", size = 1678629 },
|
||||
{ url = "https://files.pythonhosted.org/packages/61/a1/cf686e49740404b8a336e8134c5c22a0c2de64f918db0081b80d01682b5f/cramjam-2.10.0-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:7699d61c712bc77907c48fe63a21fffa03c4dd70401e1d14e368af031fde7c21", size = 2019846 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f1/f7/91b3bd99d903567ca2fd76fc600b4ce08a85e6c4800fc94f505ef9cf486e/cramjam-2.10.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3484f1595eef64cefed05804d7ec8a88695f89086c49b086634e44c16f3d4769", size = 1752196 },
|
||||
{ url = "https://files.pythonhosted.org/packages/0d/b4/3c9f9f32197c0ad7b33cc99bdf786c2bd4ccf97fdb82b07b6b211c896744/cramjam-2.10.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:38fba4594dd0e2b7423ef403039e63774086ebb0696d9060db20093f18a2f43e", size = 1849188 },
|
||||
{ url = "https://files.pythonhosted.org/packages/93/f6/9b35acb94bcab5e2089a1ff4268a3b40cd640b4200e82a4d5bf419e6a64e/cramjam-2.10.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b07fe3e48c881a75a11f722e1d5b052173b5e7c78b22518f659b8c9b4ac4c937", size = 2003528 },
|
||||
{ url = "https://files.pythonhosted.org/packages/13/4e/0c92d0c2ac978d1a95d6ff00095e5abbaeba766b5ff531d9700212db480e/cramjam-2.10.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3596b6ceaf85f872c1e56295c6ec80bb15fdd71e7ed9e0e5c3e654563dcc40a2", size = 2311664 },
|
||||
{ url = "https://files.pythonhosted.org/packages/84/ed/1db09adb133c569afd98b3f507ff372a39c3c7947cd0c42e161b5e6e13aa/cramjam-2.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e1c03360c1760f8608dc5ce1ddd7e5491180765360cae8104b428d5f86fbe1b9", size = 1964336 },
|
||||
{ url = "https://files.pythonhosted.org/packages/94/52/f7a45ba637a53bdde08fa98440341d04d7395de27a33dfd51b1211e35677/cramjam-2.10.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:3e0b70fe7796b63b87cb7ebfaad0ebaca7574fdf177311952f74b8bda6522fb8", size = 2022247 },
|
||||
{ url = "https://files.pythonhosted.org/packages/92/13/b2f101f98adbb1134d5f3a6ffd5859f88de705325e7eeeea8d57b0c106cd/cramjam-2.10.0-cp311-cp311-musllinux_1_1_armv7l.whl", hash = "sha256:d61a21e4153589bd53ffe71b553f93f2afbc8fb7baf63c91a83c933347473083", size = 2152365 },
|
||||
{ url = "https://files.pythonhosted.org/packages/19/62/85fe4091085a2d0cbe1c6271aad8f678434680fbedc9ab9fb694186c6551/cramjam-2.10.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:91ab85752a08dc875a05742cfda0234d7a70fadda07dd0b0582cfe991911f332", size = 2164416 },
|
||||
{ url = "https://files.pythonhosted.org/packages/63/3c/039bbde86826d13c6d328de70fed824cd7c2ab830d0c8b3fbdf4f61fc4e4/cramjam-2.10.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:c6afff7e9da53afb8d11eae27a20ee5709e2943b39af6c949b38424d0f271569", size = 2134635 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ee/69/77703decb6b354bed28adcf81b423e0085ce816a80102f1e395c81b68cf6/cramjam-2.10.0-cp311-cp311-win32.whl", hash = "sha256:adf484b06063134ae604d4fc826d942af7e751c9d0b2fcab5bf1058a8ebe242b", size = 1598155 },
|
||||
{ url = "https://files.pythonhosted.org/packages/00/ba/6e7ba6bbc6bde49b62ddcbc0a670ae099d99bf5c7c5bfc3b1134aa9e2de7/cramjam-2.10.0-cp311-cp311-win_amd64.whl", hash = "sha256:9e20ebea6ec77232cd12e4084c8be6d03534dc5f3d027d365b32766beafce6c3", size = 1700119 },
|
||||
{ url = "https://files.pythonhosted.org/packages/00/50/09b2cdeee0e757a902cb25559783b0d81aeea2b055034de55f57db64152f/cramjam-2.10.0-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:0acb17e3681138b48300b27d3409742c81d5734ec39c650a60a764c135197840", size = 3503057 },
|
||||
{ url = "https://files.pythonhosted.org/packages/66/53/6baa9ef73833bd609df07c4334dccb3f7d2d43c4750f5fffadc878dbc2c9/cramjam-2.10.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:647553c44cf6b5ce2d9b56e743cc1eab886940d776b36438183e807bb5a7a42b", size = 1836184 },
|
||||
{ url = "https://files.pythonhosted.org/packages/b9/53/514dbdda46c5ce2d32f7d92d2aa570c7b47f78d7cc6fd79ee3db4ac2dd2a/cramjam-2.10.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5c52805c7ccb533fe42d3d36c91d237c97c3b6551cd6b32f98b79eeb30d0f139", size = 1674041 },
|
||||
{ url = "https://files.pythonhosted.org/packages/fc/b8/07b88ee64f548ccd6d7f49589b8e5dffb5526e56572acee1a19fbd74cd5a/cramjam-2.10.0-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:337ceb50bde7708b2a4068f3000625c23ceb1b2497edce2e21fd08ef58549170", size = 2020058 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ab/bc/6ffdb375a7699751ea6341704b56050c8df428485e8363962cd6a87d3ab8/cramjam-2.10.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1c071765bdd5eefa3b2157a61e84d72e161b63f95eb702a0133fee293800a619", size = 1747828 },
|
||||
{ url = "https://files.pythonhosted.org/packages/4e/46/45e7eb96960fbbf30b280142488b61afd7092a2430414f2539c72adf292e/cramjam-2.10.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8b40d46d2aa566f8e3def953279cce0191e47364b453cda492db12a84dd97f78", size = 1850669 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ba/46/0ff7c54a9e649ad092bbbcaa21ae2535d8f53687c04836421bd4f930d780/cramjam-2.10.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4c7bab3703babb93c9dd4444ac9797d01ec46cf521e247d3319bfb292414d053", size = 1998309 },
|
||||
{ url = "https://files.pythonhosted.org/packages/1d/16/387beef4365f86ce3a45812d93e9ce230a2d7cd4ff0d81f7aad84a55d0d5/cramjam-2.10.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ba19308b8e19cdaadfbf47142f52b705d2cbfb8edd84a8271573e50fa7fa022d", size = 2361331 },
|
||||
{ url = "https://files.pythonhosted.org/packages/6f/5e/2d9fa4d310c9fa7b1db0ba9f27ea64f2975810bb18ba64f2c13e5e5728c9/cramjam-2.10.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:de3e4be5aa71b73c2640c9b86e435ec033592f7f79787937f8342259106a63ae", size = 1962253 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a7/e7/00debcc4589b6b4a2b6d7a1d523eb09683f7a3cfea9d0a1f67ab20e9f36e/cramjam-2.10.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:11c5ef0c70d6bdd8e1d8afed8b0430709b22decc3865eb6c0656aa00117a7b3d", size = 2016921 },
|
||||
{ url = "https://files.pythonhosted.org/packages/af/d1/c62de1b4630108fa4da62ec579d9925171013cad195b44e4b49e58ee1d38/cramjam-2.10.0-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:86b29e349064821ceeb14d60d01a11a0788f94e73ed4b3a5c3f9fac7aa4e2cd7", size = 2152996 },
|
||||
{ url = "https://files.pythonhosted.org/packages/1d/c2/429af269a0146f6fe54993e9cb41a35b1c231387307480ec84c641bd3629/cramjam-2.10.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:2c7008bb54bdc5d130c0e8581925dfcbdc6f0a4d2051de7a153bfced9a31910f", size = 2163476 },
|
||||
{ url = "https://files.pythonhosted.org/packages/2f/6d/0534780537175dd09aa4322119ab919acddfda404771b9e61b0bad00a955/cramjam-2.10.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:3a94fe7024137ed8bf200308000d106874afe52ff203f852f43b3547eddfa10e", size = 2132883 },
|
||||
{ url = "https://files.pythonhosted.org/packages/5d/2d/990b77c8257ff30ec5cf75fc110248f00a236dd8180410362ed6a32846ad/cramjam-2.10.0-cp312-cp312-win32.whl", hash = "sha256:ce11be5722c9d433c5e1eb3980f16eb7d80828b9614f089e28f4f1724fc8973f", size = 1597254 },
|
||||
{ url = "https://files.pythonhosted.org/packages/26/c7/baf6b960403313f9df3217f7b8039bb2e403559c95641e23a0b0056283c2/cramjam-2.10.0-cp312-cp312-win_amd64.whl", hash = "sha256:a01e89e99ba066dfa2df40fe99a2371565f4a3adc6811a73c8019d9929a312e8", size = 1699580 },
|
||||
{ url = "https://files.pythonhosted.org/packages/cc/9e/40ecf165dd9fd177c85d1d7b8614036865f15f39d116cf2c96dc84a3eb8a/cramjam-2.10.0-cp313-cp313-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:8bb0b6aaaa5f37091e05d756a3337faf0ddcffe8a68dbe8a710731b0d555ec8f", size = 3502800 },
|
||||
{ url = "https://files.pythonhosted.org/packages/af/63/83c7dbe9078ff7e9d8c449913a46a40ae8b9c260f2ec885a0249f00dd763/cramjam-2.10.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:27b2625c0840b9a5522eba30b165940084391762492e03b9d640fca5074016ae", size = 1835841 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d0/bd/d5f9bdd562d4387ca7e1dcfc5121297cba0623e696882bf7cfd343fae88d/cramjam-2.10.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:4ba90f7b8f986934f33aad8cc029cf7c74842d3ecd5eda71f7531330d38a8dc4", size = 1673882 },
|
||||
{ url = "https://files.pythonhosted.org/packages/30/ac/198378091434078efb9e25b69a142de1203bf2e54a674f15d6048221a13e/cramjam-2.10.0-cp313-cp313-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:6655d04942f7c02087a6bba4bdc8d88961aa8ddf3fb9a05b3bad06d2d1ca321b", size = 2019844 },
|
||||
{ url = "https://files.pythonhosted.org/packages/5c/63/ab625cd743cd1950e0b8a1922b5599ee9109085dcb55dad30a3d1751a8ab/cramjam-2.10.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7dda9be2caf067ac21c4aa63497833e0984908b66849c07aaa42b1cfa93f5e1c", size = 1747573 },
|
||||
{ url = "https://files.pythonhosted.org/packages/fe/c9/d17f6d5fc9e619298b98c86cfca2b728945b05135b0cc16be8e6305e00cb/cramjam-2.10.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:afa36aa006d7692718fce427ecb276211918447f806f80c19096a627f5122e3d", size = 1850318 },
|
||||
{ url = "https://files.pythonhosted.org/packages/60/83/9e35fcd2a373c30251088d4abfb87312a51bc39a0c15f5eda5099888f6fd/cramjam-2.10.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d46fd5a9e8eb5d56eccc6191a55e3e1e2b3ab24b19ab87563a2299a39c855fd7", size = 1997907 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e5/5d/c0999ebd3c829b50b93f57fbc478c6a31d7b785789d14221b5962631a610/cramjam-2.10.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e3012564760394dff89e7a10c5a244f8885cd155aec07bdbe2d6dc46be398614", size = 2361103 },
|
||||
{ url = "https://files.pythonhosted.org/packages/58/2c/866a73d33ea0950a3ea6e12d5d6f15abc8d5b5e2302c5e4aa9bd7c6d5179/cramjam-2.10.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e2d216ed4aca2090eabdd354204ae55ed3e13333d1a5b271981543696e634672", size = 1961830 },
|
||||
{ url = "https://files.pythonhosted.org/packages/70/2b/4f91b3d36d2b7288c8d180b0debce092357d41ca02bd3649f49354180613/cramjam-2.10.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:44c2660ee7c4c269646955e4e40c2693f803fbad12398bb31b2ad00cfc6027b8", size = 2016782 },
|
||||
{ url = "https://files.pythonhosted.org/packages/90/99/cff347c3279b99e3e9e1bc249319ec391c7cedb1bdc288929d4310bdd6f0/cramjam-2.10.0-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:636a48e2d01fe8d7955e9523efd2f8efce55a0221f3b5d5b4bdf37c7ff056bf1", size = 2152536 },
|
||||
{ url = "https://files.pythonhosted.org/packages/c3/36/2f4353217477d017300676545cfa7bef8e55a1fa818b4fb97c2ab6d7bfd4/cramjam-2.10.0-cp313-cp313-musllinux_1_1_i686.whl", hash = "sha256:44c15f6117031a84497433b5f55d30ee72d438fdcba9778fec0c5ca5d416aa96", size = 2162962 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/d2/808533ea5d8cccfa2bd272dc9900fa47d6cb93a6d0b2b18bcc23b0962a08/cramjam-2.10.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:76e4e42f2ecf1aca0a710adaa23000a192efb81a2aee3bcc16761f1777f08a74", size = 2132699 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f9/18/f8a96e4e2448196ce39be0684053e48b2920a2f6b8467b43cc8be62476aa/cramjam-2.10.0-cp313-cp313-win32.whl", hash = "sha256:5b34f4678d386c64d3be402fdf67f75e8f1869627ea2ec4decd43e828d3b6fba", size = 1597001 },
|
||||
{ url = "https://files.pythonhosted.org/packages/dc/4f/d90e9a8379452e3882e4d937ca566a5286eea98811571a7da0277959253e/cramjam-2.10.0-cp313-cp313-win_amd64.whl", hash = "sha256:88754dd516f0e2f4dd242880b8e760dc854e917315a17fe3fc626475bea9b252", size = 1699339 },
|
||||
{ url = "https://files.pythonhosted.org/packages/db/37/96e3b41fa2e2ca8924ec8ec53ed152c7cef1b6507ee676035a9d6e4da01c/cramjam-2.10.0-pp310-pypy310_pp73-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:77192bc1a9897ecd91cf977a5d5f990373e35a8d028c9141c8c3d3680a4a4cd7", size = 3539602 },
|
||||
{ url = "https://files.pythonhosted.org/packages/48/2e/5c102cda83b38f10e6021ede32915270bd2ae5c6b0f704d42b5cdef17802/cramjam-2.10.0-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:50b59e981f219d6840ac43cda8e885aff1457944ddbabaa16ac047690bfd6ad1", size = 1855894 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e5/be/21e0a88a28d8fbfdc7d33eb78ff7ef31e5f1a67f86538607b01a25017512/cramjam-2.10.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:d84581c869d279fab437182d5db2b590d44975084e8d50b164947f7aaa2c5f25", size = 1684764 },
|
||||
{ url = "https://files.pythonhosted.org/packages/aa/4e/cb3f28b36aa9391c31b66b5c47d3b47e469e337f7a660cabf72adc57c37d/cramjam-2.10.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:04f54bea9ce39c440d1ac6901fe4d647f9218dd5cd8fe903c6fe9c42bf5e1f3b", size = 1761657 },
|
||||
{ url = "https://files.pythonhosted.org/packages/1c/ba/0c7309f22708301ce617f1b24e7d74691909385ab5c34f72683c41f98414/cramjam-2.10.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cddd12ee5a2ef4100478db7f5563a9cdb8bc0a067fbd8ccd1ecdc446d2e6a41a", size = 1975717 },
|
||||
{ url = "https://files.pythonhosted.org/packages/02/2f/125ad8ba5482aca1704ac3510a4d8d7f9224b206060b974c4a1ac50962ec/cramjam-2.10.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:35bcecff38648908a4833928a892a1e7a32611171785bef27015107426bc1d9d", size = 1706860 },
|
||||
{ url = "https://files.pythonhosted.org/packages/5d/c9/03eae05fc36540ea92c1b136c727937bd82fd9a1f20986ac7c10191e9d40/cramjam-2.10.0-pp311-pypy311_pp73-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:1e826469cfbb6dcd5b967591e52855073267835229674cfa3d327088805855da", size = 3539823 },
|
||||
{ url = "https://files.pythonhosted.org/packages/de/34/e1066303c9dc9b6c9c8e5f820e277afa1c135ded170eb2190419af1e5df6/cramjam-2.10.0-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:1a200b74220dcd80c2bb99e3bfe1cdb1e4ed0f5c071959f4316abd65f9ef1e39", size = 1856103 },
|
||||
{ url = "https://files.pythonhosted.org/packages/81/dd/edc1207ebe09e2f1bb8a1e46dfba039bbc14f1875deed5f21f1002c3c51d/cramjam-2.10.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:2e419b65538786fc1f0cf776612262d4bf6c9449983d3fc0d0acfd86594fe551", size = 1684791 },
|
||||
{ url = "https://files.pythonhosted.org/packages/64/47/53dbc9070c54001f96972ddf7eba168340114593eb891fe89dfd816ffc73/cramjam-2.10.0-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bf1321a40da930edeff418d561dfb03e6d59d5b8ab5cbab1c4b03ff0aa4c6d21", size = 1761774 },
|
||||
{ url = "https://files.pythonhosted.org/packages/5e/23/ce7688d7fe92e870cf64001db5c396d778056d48b5384d387e0263e5133c/cramjam-2.10.0-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a04376601c8f9714fb3a6a0a1699b85aab665d9d952a2a31fb37cf70e1be1fba", size = 1975809 },
|
||||
{ url = "https://files.pythonhosted.org/packages/50/58/da5ada423f010318958db6de98c188afa915e31f5ad4ac072c2e73563a53/cramjam-2.10.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:2c1eb6e6c3d5c1cc3f7c7f8a52e034340a3c454641f019687fa94077c05da5c2", size = 1707057 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "customtkinter"
|
||||
version = "5.2.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "darkdetect" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/e3/85/2aea0f61e68c4896e0522bb1ff01badb7f40c83a550099156856037893ed/customtkinter-5.2.0.tar.gz", hash = "sha256:e93448a8d22121e20ec16e95960a8306e17cf7e0079766f5804b2e855e614937", size = 261634 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/82/23/00394404c38db474d31471e618abbbc0034483c0d4178ba6328647da1a32/customtkinter-5.2.0-py3-none-any.whl", hash = "sha256:f8b2db189959033539884d7faff99ebbb654c18097d761ed844180e32f0b5929", size = 295625 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "darkdetect"
|
||||
version = "0.8.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/45/77/7575be73bf12dee231d0c6e60ce7fb7a7be4fcd58823374fc59a6e48262e/darkdetect-0.8.0.tar.gz", hash = "sha256:b5428e1170263eb5dea44c25dc3895edd75e6f52300986353cd63533fe7df8b1", size = 7681 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl", hash = "sha256:a7509ccf517eaad92b31c214f593dbcf138ea8a43b2935406bbd565e15527a85", size = 8955 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "decorator"
|
||||
version = "5.1.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/66/0c/8d907af351aa16b42caae42f9d6aa37b900c67308052d10fdce809f8d952/decorator-5.1.1.tar.gz", hash = "sha256:637996211036b6385ef91435e4fae22989472f9d571faba8927ba8253acbc330", size = 35016 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl", hash = "sha256:b8c3f85900b9dc423225913c5aace94729fe1fa9763b38939a95226f02d37186", size = 9073 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "et-xmlfile"
|
||||
version = "1.1.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/3d/5d/0413a31d184a20c763ad741cc7852a659bf15094c24840c5bdd1754765cd/et_xmlfile-1.1.0.tar.gz", hash = "sha256:8eb9e2bc2f8c97e37a2dc85a09ecdcdec9d8a396530a6d5a33b30b9a92da0c5c", size = 3218 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/96/c2/3dd434b0108730014f1b96fd286040dc3bcb70066346f7e01ec2ac95865f/et_xmlfile-1.1.0-py3-none-any.whl", hash = "sha256:a2ba85d1d6a74ef63837eed693bcb89c3f752169b0e3e7ae5b16ca5e1b3deada", size = 4688 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "executing"
|
||||
version = "1.2.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/8f/ac/89ff37d8594b0eef176b7cec742ac868fef853b8e18df0309e3def9f480b/executing-1.2.0.tar.gz", hash = "sha256:19da64c18d2d851112f09c287f8d3dbbdf725ab0e569077efb6cdcbd3497c107", size = 654544 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/28/3c/bc3819dd8b1a1588c9215a87271b6178cc5498acaa83885211f5d4d9e693/executing-1.2.0-py2.py3-none-any.whl", hash = "sha256:0314a69e37426e3608aada02473b4161d4caf5a4b244d1d0c48072b8fee7bacc", size = 24360 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fastparquet"
|
||||
version = "2024.11.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "cramjam" },
|
||||
{ name = "fsspec" },
|
||||
{ name = "numpy" },
|
||||
{ name = "packaging" },
|
||||
{ name = "pandas" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b4/66/862da14f5fde4eff2cedc0f51a8dc34ba145088e5041b45b2d57ac54f922/fastparquet-2024.11.0.tar.gz", hash = "sha256:e3b1fc73fd3e1b70b0de254bae7feb890436cb67e99458b88cb9bd3cc44db419", size = 467192 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/3d/56/476f5b83476a256489879b78513bee737691a80905e246a2daa30ebcc362/fastparquet-2024.11.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:60ccf587410f0979105e17036df61bb60e1c2b81880dc91895cdb4ee65b71e7f", size = 910272 },
|
||||
{ url = "https://files.pythonhosted.org/packages/3b/ad/4ce73440df874479f7205fe5445090f71ed4e9bd77fdb3b740253ce82703/fastparquet-2024.11.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a5ad5fc14b0567e700bea3cd528a0bd45a6f9371370b49de8889fb3d10a6574a", size = 684095 },
|
||||
{ url = "https://files.pythonhosted.org/packages/20/37/c3164261d6183d529a59afef2749821b262c8581d837faa91043837c6f76/fastparquet-2024.11.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0b74333914f454344458dab9d1432fda9b70d62e28dc7acb1512d937ef1424ee", size = 1700355 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e6/95/cf4b175c22160ec21e4664830763bfaa80b2cf05133ef854c3f436d01c16/fastparquet-2024.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:41d1610130b5cb1ce36467766191c5418cba8631e2bfe3affffaf13f9be4e7a8", size = 1714663 },
|
||||
{ url = "https://files.pythonhosted.org/packages/2c/31/b6c8cdb6d5df964a192e4e8c8ecd979718afb9ca7e2dc9243a4368b370e9/fastparquet-2024.11.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d281edd625c33628ba028d3221180283d6161bc5ceb55eae1f0ca1678f864f26", size = 1666729 },
|
||||
{ url = "https://files.pythonhosted.org/packages/31/e5/8a0575c46a7973849f8f2a88af16618b9c7efe98f249f03e3e3de69c2b86/fastparquet-2024.11.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:fa56b19a29008c34cfe8831e810f770080debcbffc69aabd1df4d47572181f9c", size = 1741669 },
|
||||
{ url = "https://files.pythonhosted.org/packages/bb/6a/669f8c9cf2fc6e30c9353832f870e5a2e170b458d12c5080837f742d963d/fastparquet-2024.11.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:5914ecfa766b7763201b9f49d832a5e89c2dccad470ca4f9c9b228d9a8349756", size = 1782359 },
|
||||
{ url = "https://files.pythonhosted.org/packages/70/c0/1374cb43924739f4542e39d972481c1f4c7dd96808a1947450808e4e7df7/fastparquet-2024.11.0-cp310-cp310-win_amd64.whl", hash = "sha256:561202e8f0e859ccc1aa77c4aaad1d7901b2d50fd6f624ca018bae4c3c7a62ce", size = 670700 },
|
||||
{ url = "https://files.pythonhosted.org/packages/7c/51/e0d6e702523ac923ede6c05e240f4a02533ccf2cea9fec7a43491078e920/fastparquet-2024.11.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:374cdfa745aa7d5188430528d5841cf823eb9ad16df72ad6dadd898ccccce3be", size = 909934 },
|
||||
{ url = "https://files.pythonhosted.org/packages/0a/c8/5c0fb644c19a8d80b2ae4d8aa7d90c2d85d0bd4a948c5c700bea5c2802ea/fastparquet-2024.11.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4c8401bfd86cccaf0ab7c0ade58c91ae19317ff6092e1d4ad96c2178197d8124", size = 683844 },
|
||||
{ url = "https://files.pythonhosted.org/packages/33/4a/1e532fd1a0d4d8af7ffc7e3a8106c0bcd13ed914a93a61e299b3832dd3d2/fastparquet-2024.11.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f9cca4c6b5969df5561c13786f9d116300db1ec22c7941e237cfca4ce602f59b", size = 1791698 },
|
||||
{ url = "https://files.pythonhosted.org/packages/8d/e8/e1ede861bea68394a755d8be1aa2e2d60a3b9f6b551bfd56aeca74987e2e/fastparquet-2024.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9a9387e77ac608d8978774caaf1e19de67eaa1386806e514dcb19f741b19cfe5", size = 1804289 },
|
||||
{ url = "https://files.pythonhosted.org/packages/4f/1e/957090cccaede805583ca3f3e46e2762d0f9bf8860ecbce65197e47d84c1/fastparquet-2024.11.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6595d3771b3d587a31137e985f751b4d599d5c8e9af9c4858e373fdf5c3f8720", size = 1753638 },
|
||||
{ url = "https://files.pythonhosted.org/packages/85/72/344787c685fd1531f07ae712a855a7c34d13deaa26c3fd4a9231bea7dbab/fastparquet-2024.11.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:053695c2f730b78a2d3925df7cd5c6444d6c1560076af907993361cc7accf3e2", size = 1814407 },
|
||||
{ url = "https://files.pythonhosted.org/packages/6c/ec/ab9d5685f776a1965797eb68c4364c72edf57cd35beed2df49b34425d1df/fastparquet-2024.11.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:0a52eecc6270ae15f0d51347c3f762703dd667ca486f127dc0a21e7e59856ae5", size = 1874462 },
|
||||
{ url = "https://files.pythonhosted.org/packages/90/4f/7a4ea9a7ddf0a3409873f0787f355806f9e0b73f42f2acecacdd9a8eff0a/fastparquet-2024.11.0-cp311-cp311-win_amd64.whl", hash = "sha256:e29ff7a367fafa57c6896fb6abc84126e2466811aefd3e4ad4070b9e18820e54", size = 671023 },
|
||||
{ url = "https://files.pythonhosted.org/packages/08/76/068ac7ec9b4fc783be21a75a6a90b8c0654da4d46934d969e524ce287787/fastparquet-2024.11.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:dbad4b014782bd38b58b8e9f514fe958cfa7a6c4e187859232d29fd5c5ddd849", size = 915968 },
|
||||
{ url = "https://files.pythonhosted.org/packages/c7/9e/6d3b4188ad64ed51173263c07109a5f18f9c84a44fa39ab524fca7420cda/fastparquet-2024.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:403d31109d398b6be7ce84fa3483fc277c6a23f0b321348c0a505eb098a041cb", size = 685399 },
|
||||
{ url = "https://files.pythonhosted.org/packages/8f/6c/809220bc9fbe83d107df2d664c3fb62fb81867be8f5218ac66c2e6b6a358/fastparquet-2024.11.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cbbb9057a26acf0abad7adf58781ee357258b7708ee44a289e3bee97e2f55d42", size = 1758557 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e0/2c/b3b3e6ca2e531484289024138cd4709c22512b3fe68066d7f9849da4a76c/fastparquet-2024.11.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:63e0e416e25c15daa174aad8ba991c2e9e5b0dc347e5aed5562124261400f87b", size = 1781052 },
|
||||
{ url = "https://files.pythonhosted.org/packages/21/fe/97ed45092d0311c013996dae633122b7a51c5d9fe8dcbc2c840dc491201e/fastparquet-2024.11.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0e2d7f02f57231e6c86d26e9ea71953737202f20e948790e5d4db6d6a1a150dc", size = 1715797 },
|
||||
{ url = "https://files.pythonhosted.org/packages/24/df/02fa6aee6c0d53d1563b5bc22097076c609c4c5baa47056b0b4bed456fcf/fastparquet-2024.11.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:fbe4468146b633d8f09d7b196fea0547f213cb5ce5f76e9d1beb29eaa9593a93", size = 1795682 },
|
||||
{ url = "https://files.pythonhosted.org/packages/b0/25/f4f87557589e1923ee0e3bebbc84f08b7c56962bf90f51b116ddc54f2c9f/fastparquet-2024.11.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:29d5c718817bcd765fc519b17f759cad4945974421ecc1931d3bdc3e05e57fa9", size = 1857842 },
|
||||
{ url = "https://files.pythonhosted.org/packages/b1/f9/98cd0c39115879be1044d59c9b76e8292776e99bb93565bf990078fd11c4/fastparquet-2024.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:74a0b3c40ab373442c0fda96b75a36e88745d8b138fcc3a6143e04682cbbb8ca", size = 673269 },
|
||||
{ url = "https://files.pythonhosted.org/packages/47/e3/e7db38704be5db787270d43dde895eaa1a825ab25dc245e71df70860ec12/fastparquet-2024.11.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:59e5c5b51083d5b82572cdb7aed0346e3181e3ac9d2e45759da2e804bdafa7ee", size = 912523 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d3/66/e3387c99293dae441634e7724acaa425b27de19a00ee3d546775dace54a9/fastparquet-2024.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bdadf7b6bad789125b823bfc5b0a719ba5c4a2ef965f973702d3ea89cff057f6", size = 683779 },
|
||||
{ url = "https://files.pythonhosted.org/packages/0a/21/d112d0573d086b578bf04302a502e9a7605ea8f1244a7b8577cd945eec78/fastparquet-2024.11.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:46b2db02fc2a1507939d35441c8ab211d53afd75d82eec9767d1c3656402859b", size = 1751113 },
|
||||
{ url = "https://files.pythonhosted.org/packages/6b/a7/040507cee3a7798954e8fdbca21d2dbc532774b02b882d902b8a4a6849ef/fastparquet-2024.11.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a3afdef2895c9f459135a00a7ed3ceafebfbce918a9e7b5d550e4fae39c1b64d", size = 1780496 },
|
||||
{ url = "https://files.pythonhosted.org/packages/bc/75/d0d9f7533d780ec167eede16ad88073ee71696150511126c31940e7f73aa/fastparquet-2024.11.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:36b5c9bd2ffaaa26ff45d59a6cefe58503dd748e0c7fad80dd905749da0f2b9e", size = 1713608 },
|
||||
{ url = "https://files.pythonhosted.org/packages/30/fa/1d95bc86e45e80669c4f374b2ca26a9e5895a1011bb05d6341b4a7414693/fastparquet-2024.11.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:6b7df5d3b61a19d76e209fe8d3133759af1c139e04ebc6d43f3cc2d8045ef338", size = 1792779 },
|
||||
{ url = "https://files.pythonhosted.org/packages/13/3d/c076beeb926c79593374c04662a9422a76650eef17cd1c8e10951340764a/fastparquet-2024.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8b35823ac7a194134e5f82fa4a9659e42e8f9ad1f2d22a55fbb7b9e4053aabbb", size = 1851322 },
|
||||
{ url = "https://files.pythonhosted.org/packages/09/5a/1d0d47e64816002824d4a876644e8c65540fa23f91b701f0daa726931545/fastparquet-2024.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:d20632964e65530374ff7cddd42cc06aa0a1388934903693d6d22592a5ba827b", size = 673266 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fsspec"
|
||||
version = "2025.3.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/45/d8/8425e6ba5fcec61a1d16e41b1b71d2bf9344f1fe48012c2b48b9620feae5/fsspec-2025.3.2.tar.gz", hash = "sha256:e52c77ef398680bbd6a98c0e628fbc469491282981209907bbc8aea76a04fdc6", size = 299281 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/44/4b/e0cfc1a6f17e990f3e64b7d941ddc4acdc7b19d6edd51abf495f32b1a9e4/fsspec-2025.3.2-py3-none-any.whl", hash = "sha256:2daf8dc3d1dfa65b6aa37748d112773a7a08416f6c70d96b264c96476ecaf711", size = 194435 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "idna"
|
||||
version = "3.4"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/8b/e1/43beb3d38dba6cb420cefa297822eac205a277ab43e5ba5d5c46faf96438/idna-3.4.tar.gz", hash = "sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4", size = 183077 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl", hash = "sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2", size = 61538 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "itsdangerous"
|
||||
version = "2.1.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/7f/a1/d3fb83e7a61fa0c0d3d08ad0a94ddbeff3731c05212617dff3a94e097f08/itsdangerous-2.1.2.tar.gz", hash = "sha256:5dbbc68b317e5e42f327f9021763545dc3fc3bfe22e6deb96aaf1fc38874156a", size = 56143 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/68/5f/447e04e828f47465eeab35b5d408b7ebaaaee207f48b7136c5a7267a30ae/itsdangerous-2.1.2-py3-none-any.whl", hash = "sha256:2c2349112351b88699d8d4b6b075022c0808887cb7ad10069318a8b0bc88db44", size = 15749 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "jedi"
|
||||
version = "0.18.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "parso" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/15/02/afd43c5066de05f6b3188f3aa74136a3289e6c30e7a45f351546cab0928c/jedi-0.18.2.tar.gz", hash = "sha256:bae794c30d07f6d910d32a7048af09b5a39ed740918da923c6b780790ebac612", size = 1225011 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/6d/60/4acda63286ef6023515eb914543ba36496b8929cb7af49ecce63afde09c6/jedi-0.18.2-py2.py3-none-any.whl", hash = "sha256:203c1fd9d969ab8f2119ec0a3342e0b49910045abe6af0a3ae83a5764d54639e", size = 1568138 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "jinja2"
|
||||
version = "3.1.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "markupsafe" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/7a/ff/75c28576a1d900e87eb6335b063fab47a8ef3c8b4d88524c4bf78f670cce/Jinja2-3.1.2.tar.gz", hash = "sha256:31351a702a408a9e7595a8fc6150fc3f43bb6bf7e319770cbc0db9df9437e852", size = 268239 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl", hash = "sha256:6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61", size = 133101 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "jupyter-core"
|
||||
version = "5.3.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "platformdirs" },
|
||||
{ name = "pywin32", marker = "platform_python_implementation != 'PyPy' and sys_platform == 'win32'" },
|
||||
{ name = "traitlets" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/9e/53/f27bd74ceaa672a1ce17b4b2bee93c0742ca00cb9f540ec4fa60cf7319b5/jupyter_core-5.3.1.tar.gz", hash = "sha256:5ba5c7938a7f97a6b0481463f7ff0dbac7c15ba48cf46fa4035ca6e838aa1aba", size = 84448 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/8c/e0/3f9061c5e99a03612510f892647b15a91f910c5275b7b77c6c72edae1494/jupyter_core-5.3.1-py3-none-any.whl", hash = "sha256:ae9036db959a71ec1cac33081eeb040a79e681f08ab68b0883e9a676c7a90dce", size = 93670 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "macholib"
|
||||
version = "1.16.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "altgraph" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/95/ee/af1a3842bdd5902ce133bd246eb7ffd4375c38642aeb5dc0ae3a0329dfa2/macholib-1.16.3.tar.gz", hash = "sha256:07ae9e15e8e4cd9a788013d81f5908b3609aa76f9b1421bae9c4d7606ec86a30", size = 59309 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d1/5d/c059c180c84f7962db0aeae7c3b9303ed1d73d76f2bfbc32bc231c8be314/macholib-1.16.3-py2.py3-none-any.whl", hash = "sha256:0e315d7583d38b8c77e815b1ecbdbf504a8258d8b3e17b61165c6feb60d18f2c", size = 38094 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "markupsafe"
|
||||
version = "2.1.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/6d/7c/59a3248f411813f8ccba92a55feaac4bf360d29e2ff05ee7d8e1ef2d7dbf/MarkupSafe-2.1.3.tar.gz", hash = "sha256:af598ed32d6ae86f1b747b82783958b1a4ab8f617b06fe68795c7f026abbdcad", size = 19132 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/20/1d/713d443799d935f4d26a4f1510c9e61b1d288592fb869845e5cc92a1e055/MarkupSafe-2.1.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:cd0f502fe016460680cd20aaa5a76d241d6f35a1c3350c474bac1273803893fa", size = 17846 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f7/9c/86cbd8e0e1d81f0ba420f20539dd459c50537c7751e28102dbfee2b6f28c/MarkupSafe-2.1.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e09031c87a1e51556fdcb46e5bd4f59dfb743061cf93c4d6831bf894f125eb57", size = 13720 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a6/56/f1d4ee39e898a9e63470cbb7fae1c58cce6874f25f54220b89213a47f273/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:68e78619a61ecf91e76aa3e6e8e33fc4894a2bebe93410754bd28fce0a8a4f9f", size = 26498 },
|
||||
{ url = "https://files.pythonhosted.org/packages/12/b3/d9ed2c0971e1435b8a62354b18d3060b66c8cb1d368399ec0b9baa7c0ee5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:65c1a9bcdadc6c28eecee2c119465aebff8f7a584dd719facdd9e825ec61ab52", size = 25691 },
|
||||
{ url = "https://files.pythonhosted.org/packages/bf/b7/c5ba9b7ad9ad21fc4a60df226615cf43ead185d328b77b0327d603d00cc5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:525808b8019e36eb524b8c68acdd63a37e75714eac50e988180b169d64480a00", size = 25366 },
|
||||
{ url = "https://files.pythonhosted.org/packages/71/61/f5673d7aac2cf7f203859008bb3fc2b25187aa330067c5e9955e5c5ebbab/MarkupSafe-2.1.3-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:962f82a3086483f5e5f64dbad880d31038b698494799b097bc59c2edf392fce6", size = 30505 },
|
||||
{ url = "https://files.pythonhosted.org/packages/47/26/932140621773bfd4df3223fbdd9e78de3477f424f0d2987c313b1cb655ff/MarkupSafe-2.1.3-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:aa7bd130efab1c280bed0f45501b7c8795f9fdbeb02e965371bbef3523627779", size = 29616 },
|
||||
{ url = "https://files.pythonhosted.org/packages/3c/c8/74d13c999cbb49e3460bf769025659a37ef4a8e884de629720ab4e42dcdb/MarkupSafe-2.1.3-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:c9c804664ebe8f83a211cace637506669e7890fec1b4195b505c214e50dd4eb7", size = 29891 },
|
||||
{ url = "https://files.pythonhosted.org/packages/96/e4/4db3b1abc5a1fe7295aa0683eafd13832084509c3b8236f3faf8dd4eff75/MarkupSafe-2.1.3-cp310-cp310-win32.whl", hash = "sha256:10bbfe99883db80bdbaff2dcf681dfc6533a614f700da1287707e8a5d78a8431", size = 16525 },
|
||||
{ url = "https://files.pythonhosted.org/packages/84/a8/c4aebb8a14a1d39d5135eb8233a0b95831cdc42c4088358449c3ed657044/MarkupSafe-2.1.3-cp310-cp310-win_amd64.whl", hash = "sha256:1577735524cdad32f9f694208aa75e422adba74f1baee7551620e43a3141f559", size = 17083 },
|
||||
{ url = "https://files.pythonhosted.org/packages/fe/09/c31503cb8150cf688c1534a7135cc39bb9092f8e0e6369ec73494d16ee0e/MarkupSafe-2.1.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ad9e82fb8f09ade1c3e1b996a6337afac2b8b9e365f926f5a61aacc71adc5b3c", size = 17862 },
|
||||
{ url = "https://files.pythonhosted.org/packages/c0/c7/171f5ac6b065e1425e8fabf4a4dfbeca76fd8070072c6a41bd5c07d90d8b/MarkupSafe-2.1.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3c0fae6c3be832a0a0473ac912810b2877c8cb9d76ca48de1ed31e1c68386575", size = 13738 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a2/f7/9175ad1b8152092f7c3b78c513c1bdfe9287e0564447d1c2d3d1a2471540/MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b076b6226fb84157e3f7c971a47ff3a679d837cf338547532ab866c57930dbee", size = 28891 },
|
||||
{ url = "https://files.pythonhosted.org/packages/fe/21/2eff1de472ca6c99ec3993eab11308787b9879af9ca8bbceb4868cf4f2ca/MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bfce63a9e7834b12b87c64d6b155fdd9b3b96191b6bd334bf37db7ff1fe457f2", size = 28096 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f4/a0/103f94793c3bf829a18d2415117334ece115aeca56f2df1c47fa02c6dbd6/MarkupSafe-2.1.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:338ae27d6b8745585f87218a3f23f1512dbf52c26c28e322dbe54bcede54ccb9", size = 27631 },
|
||||
{ url = "https://files.pythonhosted.org/packages/43/70/f24470f33b2035b035ef0c0ffebf57006beb2272cf3df068fc5154e04ead/MarkupSafe-2.1.3-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:e4dd52d80b8c83fdce44e12478ad2e85c64ea965e75d66dbeafb0a3e77308fcc", size = 33863 },
|
||||
{ url = "https://files.pythonhosted.org/packages/32/d4/ce98c4ca713d91c4a17c1a184785cc00b9e9c25699d618956c2b9999500a/MarkupSafe-2.1.3-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:df0be2b576a7abbf737b1575f048c23fb1d769f267ec4358296f31c2479db8f9", size = 32591 },
|
||||
{ url = "https://files.pythonhosted.org/packages/bb/82/f88ccb3ca6204a4536cf7af5abdad7c3657adac06ab33699aa67279e0744/MarkupSafe-2.1.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:5bbe06f8eeafd38e5d0a4894ffec89378b6c6a625ff57e3028921f8ff59318ac", size = 33186 },
|
||||
{ url = "https://files.pythonhosted.org/packages/44/53/93405d37bb04a10c43b1bdd6f548097478d494d7eadb4b364e3e1337f0cc/MarkupSafe-2.1.3-cp311-cp311-win32.whl", hash = "sha256:dd15ff04ffd7e05ffcb7fe79f1b98041b8ea30ae9234aed2a9168b5797c3effb", size = 16537 },
|
||||
{ url = "https://files.pythonhosted.org/packages/be/bb/08b85bc194034efbf572e70c3951549c8eca0ada25363afc154386b5390a/MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl", hash = "sha256:134da1eca9ec0ae528110ccc9e48041e0828d79f24121a1a146161103c76e686", size = 17089 },
|
||||
{ url = "https://files.pythonhosted.org/packages/89/5a/ee546f2aa73a1d6fcfa24272f356fe06d29acca81e76b8d32ca53e429a2e/MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:f698de3fd0c4e6972b92290a45bd9b1536bffe8c6759c62471efaa8acb4c37bc", size = 17849 },
|
||||
{ url = "https://files.pythonhosted.org/packages/3a/72/9f683a059bde096776e8acf9aa34cbbba21ddc399861fe3953790d4f2cde/MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:aa57bd9cf8ae831a362185ee444e15a93ecb2e344c8e52e4d721ea3ab6ef1823", size = 13700 },
|
||||
{ url = "https://files.pythonhosted.org/packages/9d/78/92f15eb9b1e8f1668a9787ba103cf6f8d19a9efed8150245404836145c24/MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffcc3f7c66b5f5b7931a5aa68fc9cecc51e685ef90282f4a82f0f5e9b704ad11", size = 29319 },
|
||||
{ url = "https://files.pythonhosted.org/packages/51/94/9a04085114ff2c24f7424dbc890a281d73c5a74ea935dc2e69c66a3bd558/MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47d4f1c5f80fc62fdd7777d0d40a2e9dda0a05883ab11374334f6c4de38adffd", size = 28314 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ec/53/fcb3214bd370185e223b209ce6bb010fb887ea57173ca4f75bd211b24e10/MarkupSafe-2.1.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1f67c7038d560d92149c060157d623c542173016c4babc0c1913cca0564b9939", size = 27696 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/33/54d29854716725d7826079b8984dd235fac76dab1c32321e555d493e61f5/MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:9aad3c1755095ce347e26488214ef77e0485a3c34a50c5a5e2471dff60b9dd9c", size = 33746 },
|
||||
{ url = "https://files.pythonhosted.org/packages/11/40/ea7f85e2681d29bc9301c757257de561923924f24de1802d9c3baa396bb4/MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:14ff806850827afd6b07a5f32bd917fb7f45b046ba40c57abdb636674a8b559c", size = 32131 },
|
||||
{ url = "https://files.pythonhosted.org/packages/41/f1/bc770c37ecd58638c18f8ec85df205dacb818ccf933692082fd93010a4bc/MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8f9293864fe09b8149f0cc42ce56e3f0e54de883a9de90cd427f191c346eb2e1", size = 32878 },
|
||||
{ url = "https://files.pythonhosted.org/packages/49/74/bf95630aab0a9ed6a67556cd4e54f6aeb0e74f4cb0fd2f229154873a4be4/MarkupSafe-2.1.3-cp312-cp312-win32.whl", hash = "sha256:715d3562f79d540f251b99ebd6d8baa547118974341db04f5ad06d5ea3eb8007", size = 16426 },
|
||||
{ url = "https://files.pythonhosted.org/packages/44/44/dbaf65876e258facd65f586dde158387ab89963e7f2235551afc9c2e24c2/MarkupSafe-2.1.3-cp312-cp312-win_amd64.whl", hash = "sha256:1b8dd8c3fd14349433c79fa8abeb573a55fc0fdd769133baac1f5e07abf54aeb", size = 16979 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "numpy"
|
||||
version = "1.25.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d0/b2/fe774844d1857804cc884bba67bec38f649c99d0dc1ee7cbbf1da601357c/numpy-1.25.0.tar.gz", hash = "sha256:f1accae9a28dc3cda46a91de86acf69de0d1b5f4edd44a9b0c3ceb8036dfff19", size = 10426700 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a7/71/8cadc39a58fc18a91ad135c3a33b6a6a7c0ccf00adb4263d6f2aebf8124d/numpy-1.25.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:8aa130c3042052d656751df5e81f6d61edff3e289b5994edcf77f54118a8d9f4", size = 20055608 },
|
||||
{ url = "https://files.pythonhosted.org/packages/c8/7c/87cf5dc663803120901302db2494e625d762e19060b390d925e3e8666b18/numpy-1.25.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9e3f2b96e3b63c978bc29daaa3700c028fe3f049ea3031b58aa33fe2a5809d24", size = 13963319 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/f6/1ce8d0bdcf926a5d94ae2a793eee4364c76ba2d1a5b73ee9de9aebc3a0e0/numpy-1.25.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d6b267f349a99d3908b56645eebf340cb58f01bd1e773b4eea1a905b3f0e4208", size = 14132512 },
|
||||
{ url = "https://files.pythonhosted.org/packages/77/03/79b0bfc6e9dcd5eabbb17a714a2480ad3f932063eb8b39f6116ac207d5e3/numpy-1.25.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4aedd08f15d3045a4e9c648f1e04daca2ab1044256959f1f95aafeeb3d794c16", size = 17612667 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a8/a5/dded2b52d4a460f265973f2aaedc5ea82814d471241e5d17599506c4ee0e/numpy-1.25.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:6d183b5c58513f74225c376643234c369468e02947b47942eacbb23c1671f25d", size = 17449973 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a5/c7/586bc658351595f252dd6fa31a14ca28ca7de7d93171f933b1c193e7e32c/numpy-1.25.0-cp310-cp310-win32.whl", hash = "sha256:d76a84998c51b8b68b40448ddd02bd1081bb33abcdc28beee6cd284fe11036c6", size = 12607709 },
|
||||
{ url = "https://files.pythonhosted.org/packages/13/a0/bd219e125915e1d5706a5d00b87cd93932d6a204d976aea09fa0f36af5a1/numpy-1.25.0-cp310-cp310-win_amd64.whl", hash = "sha256:c0dc071017bc00abb7d7201bac06fa80333c6314477b3d10b52b58fa6a6e38f6", size = 15034656 },
|
||||
{ url = "https://files.pythonhosted.org/packages/bb/b9/0f7a1d48d5c65c7a2cc8d5de119318a254351a0146e696855ade26615455/numpy-1.25.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4c69fe5f05eea336b7a740e114dec995e2f927003c30702d896892403df6dbf0", size = 20041989 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e8/bd/937ffc7345985456c963089418c4c7efdb2ca3af36624c5ea60a07d99bcf/numpy-1.25.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:9c7211d7920b97aeca7b3773a6783492b5b93baba39e7c36054f6e749fc7490c", size = 13973163 },
|
||||
{ url = "https://files.pythonhosted.org/packages/8c/00/a65518f58b9bbba597cd757a765d7a34fea3d8fd089a8ecc7f6eb4e4f42d/numpy-1.25.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ecc68f11404930e9c7ecfc937aa423e1e50158317bf67ca91736a9864eae0232", size = 14123400 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f6/ae/546c18cad7525242d87def9ee1cba2e407028044f79c023ea8b2a11397d2/numpy-1.25.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e559c6afbca484072a98a51b6fa466aae785cfe89b69e8b856c3191bc8872a82", size = 17602714 },
|
||||
{ url = "https://files.pythonhosted.org/packages/fa/9f/9023a2135a86a80369c942670ef23c2c838aee3408f982e3b9bcaf9ffe61/numpy-1.25.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:6c284907e37f5e04d2412950960894b143a648dea3f79290757eb878b91acbd1", size = 17453872 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ef/29/a2503fed1bb38902e789f3e73259d760911fb7b51420896716502c727aa1/numpy-1.25.0-cp311-cp311-win32.whl", hash = "sha256:95367ccd88c07af21b379be1725b5322362bb83679d36691f124a16357390153", size = 12600664 },
|
||||
{ url = "https://files.pythonhosted.org/packages/de/8b/b2d73b913be92056b1f77b0b9d184d93f368353540adf91e699a10a2effb/numpy-1.25.0-cp311-cp311-win_amd64.whl", hash = "sha256:b76aa836a952059d70a2788a2d98cb2a533ccd46222558b6970348939e55fc24", size = 15026783 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "packaging"
|
||||
version = "23.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b9/6c/7c6658d258d7971c5eb0d9b69fa9265879ec9a9158031206d47800ae2213/packaging-23.1.tar.gz", hash = "sha256:a392980d2b6cffa644431898be54b0045151319d1e7ec34f0cfed48767dd334f", size = 134240 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/ab/c3/57f0601a2d4fe15de7a553c00adbc901425661bf048f2a22dfc500caf121/packaging-23.1-py3-none-any.whl", hash = "sha256:994793af429502c4ea2ebf6bf664629d07c1a9fe974af92966e4b8d2df7edc61", size = 48905 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pandas"
|
||||
version = "2.0.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "numpy" },
|
||||
{ name = "python-dateutil" },
|
||||
{ name = "pytz" },
|
||||
{ name = "tzdata" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/b1/a7/824332581e258b5aa4f3763ecb2a797e5f9a54269044ba2e50ac19936b32/pandas-2.0.3.tar.gz", hash = "sha256:c02f372a88e0d17f36d3093a644c73cfc1788e876a7c4bcb4020a77512e2043c", size = 5284455 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/3c/b2/0d4a5729ce1ce11630c4fc5d5522a33b967b3ca146c210f58efde7c40e99/pandas-2.0.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e4c7c9f27a4185304c7caf96dc7d91bc60bc162221152de697c98eb0b2648dd8", size = 11760908 },
|
||||
{ url = "https://files.pythonhosted.org/packages/4a/f6/f620ca62365d83e663a255a41b08d2fc2eaf304e0b8b21bb6d62a7390fe3/pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:f167beed68918d62bffb6ec64f2e1d8a7d297a038f86d4aed056b9493fca407f", size = 10823486 },
|
||||
{ url = "https://files.pythonhosted.org/packages/c2/59/cb4234bc9b968c57e81861b306b10cd8170272c57b098b724d3de5eda124/pandas-2.0.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ce0c6f76a0f1ba361551f3e6dceaff06bde7514a374aa43e33b588ec10420183", size = 11571897 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e3/59/35a2892bf09ded9c1bf3804461efe772836a5261ef5dfb4e264ce813ff99/pandas-2.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba619e410a21d8c387a1ea6e8a0e49bb42216474436245718d7f2e88a2f8d7c0", size = 12306421 },
|
||||
{ url = "https://files.pythonhosted.org/packages/94/71/3a0c25433c54bb29b48e3155b959ac78f4c4f2f06f94d8318aac612cb80f/pandas-2.0.3-cp310-cp310-win32.whl", hash = "sha256:3ef285093b4fe5058eefd756100a367f27029913760773c8bf1d2d8bebe5d210", size = 9540792 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ed/30/b97456e7063edac0e5a405128065f0cd2033adfe3716fb2256c186bd41d0/pandas-2.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:9ee1a69328d5c36c98d8e74db06f4ad518a1840e8ccb94a4ba86920986bb617e", size = 10664333 },
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/92/a5e5133421b49e901a12e02a6a7ef3a0130e10d13db8cb657fdd0cba3b90/pandas-2.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:b084b91d8d66ab19f5bb3256cbd5ea661848338301940e17f4492b2ce0801fe8", size = 11645672 },
|
||||
{ url = "https://files.pythonhosted.org/packages/8f/bb/aea1fbeed5b474cb8634364718abe9030d7cc7a30bf51f40bd494bbc89a2/pandas-2.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:37673e3bdf1551b95bf5d4ce372b37770f9529743d2498032439371fc7b7eb26", size = 10693229 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d6/90/e7d387f1a416b14e59290baa7a454a90d719baebbf77433ff1bdcc727800/pandas-2.0.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b9cb1e14fdb546396b7e1b923ffaeeac24e4cedd14266c3497216dd4448e4f2d", size = 11581591 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d0/28/88b81881c056376254618fad622a5e94b5126db8c61157ea1910cd1c040a/pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d9cd88488cceb7635aebb84809d087468eb33551097d600c6dad13602029c2df", size = 12219370 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e4/a5/212b9039e25bf8ebb97e417a96660e3dc925dacd3f8653d531b8f7fd9be4/pandas-2.0.3-cp311-cp311-win32.whl", hash = "sha256:694888a81198786f0e164ee3a581df7d505024fbb1f15202fc7db88a71d84ebd", size = 9482935 },
|
||||
{ url = "https://files.pythonhosted.org/packages/9e/71/756a1be6bee0209d8c0d8c5e3b9fc72c00373f384a4017095ec404aec3ad/pandas-2.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:6a21ab5c89dcbd57f78d0ae16630b090eec626360085a4148693def5452d8a6b", size = 10607692 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "parso"
|
||||
version = "0.8.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/a2/0e/41f0cca4b85a6ea74d66d2226a7cda8e41206a624f5b330b958ef48e2e52/parso-0.8.3.tar.gz", hash = "sha256:8c07be290bb59f03588915921e29e8a50002acaf2cdc5fa0e0114f91709fafa0", size = 400064 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/05/63/8011bd08a4111858f79d2b09aad86638490d62fbf881c44e434a6dfca87b/parso-0.8.3-py2.py3-none-any.whl", hash = "sha256:c001d4636cd3aecdaf33cbb40aebb59b094be2a74c556778ef5576c175e19e75", size = 100781 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "patient-pathway-analysis"
|
||||
version = "0.1.0"
|
||||
source = { virtual = "." }
|
||||
dependencies = [
|
||||
{ name = "customtkinter" },
|
||||
{ name = "darkdetect" },
|
||||
{ name = "decorator" },
|
||||
{ name = "et-xmlfile" },
|
||||
{ name = "executing" },
|
||||
{ name = "fastparquet" },
|
||||
{ name = "idna" },
|
||||
{ name = "itsdangerous" },
|
||||
{ name = "jedi" },
|
||||
{ name = "jinja2" },
|
||||
{ name = "jupyter-core" },
|
||||
{ name = "numpy" },
|
||||
{ name = "packaging" },
|
||||
{ name = "pandas" },
|
||||
{ name = "pillow" },
|
||||
{ name = "plotly" },
|
||||
{ name = "pyarrow" },
|
||||
{ name = "pyglet" },
|
||||
{ name = "pyinstaller" },
|
||||
{ name = "python-dateutil" },
|
||||
{ name = "tenacity" },
|
||||
{ name = "tkcalendar" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "customtkinter", specifier = "==5.2.0" },
|
||||
{ name = "darkdetect", specifier = "==0.8.0" },
|
||||
{ name = "decorator", specifier = "==5.1.1" },
|
||||
{ name = "et-xmlfile", specifier = "==1.1.0" },
|
||||
{ name = "executing", specifier = "==1.2.0" },
|
||||
{ name = "fastparquet", specifier = ">=2024.11.0" },
|
||||
{ name = "idna", specifier = "==3.4" },
|
||||
{ name = "itsdangerous", specifier = "==2.1.2" },
|
||||
{ name = "jedi", specifier = "==0.18.2" },
|
||||
{ name = "jinja2", specifier = "==3.1.2" },
|
||||
{ name = "jupyter-core", specifier = "==5.3.1" },
|
||||
{ name = "numpy", specifier = "==1.25.0" },
|
||||
{ name = "packaging", specifier = "==23.1" },
|
||||
{ name = "pandas", specifier = "==2.0.3" },
|
||||
{ name = "pillow", specifier = "==10.0.0" },
|
||||
{ name = "plotly", specifier = "==5.15.0" },
|
||||
{ name = "pyarrow", specifier = ">=20.0.0" },
|
||||
{ name = "pyglet", specifier = "==2.0.9" },
|
||||
{ name = "pyinstaller", specifier = ">=6.13.0" },
|
||||
{ name = "python-dateutil", specifier = "==2.8.2" },
|
||||
{ name = "tenacity", specifier = "==8.2.2" },
|
||||
{ name = "tkcalendar", specifier = "==1.6.1" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pefile"
|
||||
version = "2023.2.7"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/78/c5/3b3c62223f72e2360737fd2a57c30e5b2adecd85e70276879609a7403334/pefile-2023.2.7.tar.gz", hash = "sha256:82e6114004b3d6911c77c3953e3838654b04511b8b66e8583db70c65998017dc", size = 74854 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/55/26/d0ad8b448476d0a1e8d3ea5622dc77b916db84c6aa3cb1e1c0965af948fc/pefile-2023.2.7-py3-none-any.whl", hash = "sha256:da185cd2af68c08a6cd4481f7325ed600a88f6a813bad9dea07ab3ef73d8d8d6", size = 71791 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pillow"
|
||||
version = "10.0.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/0f/8b/2ebaf9adcf4260c00f842154865f8730cf745906aa5dd499141fb6063e26/Pillow-10.0.0.tar.gz", hash = "sha256:9c82b5b3e043c7af0d95792d0d20ccf68f61a1fec6b3530e718b688422727396", size = 50527522 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/73/26/75fd7c1adc40bbdcbebc1adc120388d581e1d98a106257369a9bf8c44865/Pillow-10.0.0-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:1f62406a884ae75fb2f818694469519fb685cc7eaff05d3451a9ebe55c646891", size = 3398696 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ef/53/024e161112beb11008d6c7529c954e2ec641ae17b99e03fe9a539e114ae6/Pillow-10.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:d5db32e2a6ccbb3d34d87c87b432959e0db29755727afb37290e10f6e8e62614", size = 3111904 },
|
||||
{ url = "https://files.pythonhosted.org/packages/23/08/bbd0a562bafe23b4c36d25072c89b8c31815f350a169016ede2644784ed6/Pillow-10.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:edf4392b77bdc81f36e92d3a07a5cd072f90253197f4a52a55a8cec48a12483b", size = 3117233 },
|
||||
{ url = "https://files.pythonhosted.org/packages/7b/c9/08de9a629ce7cdeaea0ddca716e9efcd1844b2650f5b9dd8ec5609e40ffe/Pillow-10.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:520f2a520dc040512699f20fa1c363eed506e94248d71f85412b625026f6142c", size = 3314487 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ac/0c/7eeab446ab3acfb1ef0150308b663fa6f886d02f1d0fe66e7f67ffd6a844/Pillow-10.0.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:8c11160913e3dd06c8ffdb5f233a4f254cb449f4dfc0f8f4549eda9e542c93d1", size = 3169197 },
|
||||
{ url = "https://files.pythonhosted.org/packages/3d/36/e78f09d510354977e10102dd811e928666021d9c451e05df962d56477772/Pillow-10.0.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:a74ba0c356aaa3bb8e3eb79606a87669e7ec6444be352870623025d75a14a2bf", size = 3421015 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f8/31/4cb552d54380f1d55a7c24db1c6fb8bb2370f57fc2fe31e11c1eb5f7e499/Pillow-10.0.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:d5d0dae4cfd56969d23d94dc8e89fb6a217be461c69090768227beb8ed28c0a3", size = 3355236 },
|
||||
{ url = "https://files.pythonhosted.org/packages/60/34/c90bacb4a72ead5c78e4d8291e0d3bb88cc3def3c76f059e9a8502fc421e/Pillow-10.0.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:22c10cc517668d44b211717fd9775799ccec4124b9a7f7b3635fc5386e584992", size = 3420276 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d0/4f/faebe1180e5e6ad6330c539dda7f6081182157393ba6816a438f759a0e59/Pillow-10.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:dffe31a7f47b603318c609f378ebcd57f1554a3a6a8effbc59c3c69f804296de", size = 2513088 },
|
||||
{ url = "https://files.pythonhosted.org/packages/7a/54/f6a14d95cba8ff082c550d836c9e5c23f1641d2ac291c23efe0494219b8c/Pillow-10.0.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:9fb218c8a12e51d7ead2a7c9e101a04982237d4855716af2e9499306728fb485", size = 3398781 },
|
||||
{ url = "https://files.pythonhosted.org/packages/b7/ad/71982d18fd28ed1f93c31b8648f980ebdbdbcf7d8c9c9b4af59290914ce9/Pillow-10.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d35e3c8d9b1268cbf5d3670285feb3528f6680420eafe35cccc686b73c1e330f", size = 3111873 },
|
||||
{ url = "https://files.pythonhosted.org/packages/45/5c/04224bf1a8247d6bbba375248d74668724a5a9879b4c42c23dfadd0c28ae/Pillow-10.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3ed64f9ca2f0a95411e88a4efbd7a29e5ce2cea36072c53dd9d26d9c76f753b3", size = 3117246 },
|
||||
{ url = "https://files.pythonhosted.org/packages/45/de/b07418f00cd78af292ceb4e2855c158ef8477dc1cbcdac3e1f32eb4e53b6/Pillow-10.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0b6eb5502f45a60a3f411c63187db83a3d3107887ad0d036c13ce836f8a36f1d", size = 3314475 },
|
||||
{ url = "https://files.pythonhosted.org/packages/79/53/3a7277ae95bfe86b8b4db0ed1d08c4924aa2dfbfe51b8fe0e310b160a9c6/Pillow-10.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:c1fbe7621c167ecaa38ad29643d77a9ce7311583761abf7836e1510c580bf3dd", size = 3169201 },
|
||||
{ url = "https://files.pythonhosted.org/packages/16/89/818fa238e37a47a29bb8495ca2cafdd514599a89f19ada7916348a74b5f9/Pillow-10.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:cd25d2a9d2b36fcb318882481367956d2cf91329f6892fe5d385c346c0649629", size = 3421012 },
|
||||
{ url = "https://files.pythonhosted.org/packages/72/17/6c1e6b0f78d21838844318057b7a939ab8a8d92deeb51d22563202b2db64/Pillow-10.0.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:3b08d4cc24f471b2c8ca24ec060abf4bebc6b144cb89cba638c720546b1cf538", size = 3355277 },
|
||||
{ url = "https://files.pythonhosted.org/packages/40/58/0a62422b3cf188dac72fe6c54b6f3f372ec2e84043eb4f8d2158626992b7/Pillow-10.0.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:d737a602fbd82afd892ca746392401b634e278cb65d55c4b7a8f48e9ef8d008d", size = 3420294 },
|
||||
{ url = "https://files.pythonhosted.org/packages/66/d4/054e491f0880bf0119ee79cdc03264e01d5732e06c454da8c69b83a7c8f2/Pillow-10.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:3a82c40d706d9aa9734289740ce26460a11aeec2d9c79b7af87bb35f0073c12f", size = 2513082 },
|
||||
{ url = "https://files.pythonhosted.org/packages/6a/33/c278084a811d7a7a17c8dd14cb261248fdd0265263760fb753a5a719241e/Pillow-10.0.0-cp311-cp311-win_arm64.whl", hash = "sha256:bc2ec7c7b5d66b8ec9ce9f720dbb5fa4bace0f545acd34870eff4a369b44bf37", size = 2501798 },
|
||||
{ url = "https://files.pythonhosted.org/packages/9c/e8/59271ada18cec229d4a79475a45a9e64367e54e5d1f488b030af63805960/Pillow-10.0.0-cp312-cp312-macosx_10_10_x86_64.whl", hash = "sha256:d80cf684b541685fccdd84c485b31ce73fc5c9b5d7523bf1394ce134a60c6883", size = 3398485 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f0/7f/ff6ce4360dccfacc3af3462cfcd2d7481a1cc8d6aa712927072016dd6755/Pillow-10.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:76de421f9c326da8f43d690110f0e79fe3ad1e54be811545d7d91898b4c8493e", size = 3111012 },
|
||||
{ url = "https://files.pythonhosted.org/packages/2e/a4/06f84d3fe7aa9558d2b80d8d4960fe07071a53e8d3ccac8b079905003048/Pillow-10.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:81ff539a12457809666fef6624684c008e00ff6bf455b4b89fd00a140eecd640", size = 3117406 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a8/7b/f8ed885d18096930991bbaac729024435e0343a3c81062811cf865205a79/Pillow-10.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ce543ed15570eedbb85df19b0a1a7314a9c8141a36ce089c0a894adbfccb4568", size = 3315095 },
|
||||
{ url = "https://files.pythonhosted.org/packages/54/2e/04bae205c5bf3ff7e58735b73a1d3943d0e33e0f7ca8637aa30a2acd06d0/Pillow-10.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:685ac03cc4ed5ebc15ad5c23bc555d68a87777586d970c2c3e216619a5476223", size = 3169235 },
|
||||
{ url = "https://files.pythonhosted.org/packages/5f/82/39a266a0626d2c0dd4ee341639fe7749268fc871429b90006eeb1583f24b/Pillow-10.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:d72e2ecc68a942e8cf9739619b7f408cc7b272b279b56b2c83c6123fcfa5cdff", size = 3421158 },
|
||||
{ url = "https://files.pythonhosted.org/packages/4d/61/eba2506ce68706ccb7d485cee968e35fa9ee797d77520760acf41a65f281/Pillow-10.0.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:d50b6aec14bc737742ca96e85d6d0a5f9bfbded018264b3b70ff9d8c33485551", size = 3355694 },
|
||||
{ url = "https://files.pythonhosted.org/packages/0f/0b/0f37aac8432fb91e9f7eec96a29afb354f172e593d2d6d8201e544f49b55/Pillow-10.0.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:00e65f5e822decd501e374b0650146063fbb30a7264b4d2744bdd7b913e0cab5", size = 3421380 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/af/06fa67e8c8c4ead837f6a4025b6605f4cb8ec0fcbff1e4c697712fabf9f9/Pillow-10.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:f31f9fdbfecb042d046f9d91270a0ba28368a723302786c0009ee9b9f1f60199", size = 2513485 },
|
||||
{ url = "https://files.pythonhosted.org/packages/83/c0/aaa4f7f9f0ed854d8b519739392ed17ee1aaaa352fd037646e97634a6bdb/Pillow-10.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:1ce91b6ec08d866b14413d3f0bbdea7e24dfdc8e59f562bb77bc3fe60b6144ca", size = 2502324 },
|
||||
{ url = "https://files.pythonhosted.org/packages/78/b9/e5bc84e6ed714c7f0ec0dfe3f82c050c16126294e3d078fe155f10bd5971/Pillow-10.0.0-pp310-pypy310_pp73-macosx_10_10_x86_64.whl", hash = "sha256:92be919bbc9f7d09f7ae343c38f5bb21c973d2576c1d45600fce4b74bafa7ac0", size = 3353092 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ef/0f/eea2ed37a53e816c8ed392a031468498687585c8d62ca89deeb687c0e89c/Pillow-10.0.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8f8182b523b2289f7c415f589118228d30ac8c355baa2f3194ced084dac2dbba", size = 3228084 },
|
||||
{ url = "https://files.pythonhosted.org/packages/12/2e/7f20311309d03ccfefc3df6c00524d996d15a18319b46953ac8ee158b5a9/Pillow-10.0.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:38250a349b6b390ee6047a62c086d3817ac69022c127f8a5dc058c31ccef17f3", size = 3303031 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a8/df/f52e3621148bb35d06c8f6a113ee949169388a2a3095550314fa6b6809f5/Pillow-10.0.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:88af2003543cc40c80f6fca01411892ec52b11021b3dc22ec3bc9d5afd1c5334", size = 2513263 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "platformdirs"
|
||||
version = "3.8.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/92/38/3dd18a282991c004851ea1f0953105a186cfc691eee2792778ac2ca060f8/platformdirs-3.8.1.tar.gz", hash = "sha256:f87ca4fcff7d2b0f81c6a748a77973d7af0f4d526f98f308477c3c436c74d528", size = 18533 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/9e/d8/563a9fc17153c588c8c2042d2f0f84a89057cdb1c30270f589c88b42d62c/platformdirs-3.8.1-py3-none-any.whl", hash = "sha256:cec7b889196b9144d088e4c57d9ceef7374f6c39694ad1577a0aab50d27ea28c", size = 16629 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "plotly"
|
||||
version = "5.15.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "packaging" },
|
||||
{ name = "tenacity" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/7b/1b/49b60763629f8b654798f78b800c8617b56a8fbb5d3ff93d610a96ebee4c/plotly-5.15.0.tar.gz", hash = "sha256:822eabe53997d5ebf23c77e1d1fcbf3bb6aa745eb05d532afd4b6f9a2e2ab02f", size = 7757675 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a5/07/5bef9376c975ce23306d9217ab69ca94c07f2a3c90b17c03e3ae4db87170/plotly-5.15.0-py2.py3-none-any.whl", hash = "sha256:3508876bbd6aefb8a692c21a7128ca87ce42498dd041efa5c933ee44b55aab24", size = 15519872 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyarrow"
|
||||
version = "20.0.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/a2/ee/a7810cb9f3d6e9238e61d312076a9859bf3668fd21c69744de9532383912/pyarrow-20.0.0.tar.gz", hash = "sha256:febc4a913592573c8d5805091a6c2b5064c8bd6e002131f01061797d91c783c1", size = 1125187 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/5b/23/77094eb8ee0dbe88441689cb6afc40ac312a1e15d3a7acc0586999518222/pyarrow-20.0.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:c7dd06fd7d7b410ca5dc839cc9d485d2bc4ae5240851bcd45d85105cc90a47d7", size = 30832591 },
|
||||
{ url = "https://files.pythonhosted.org/packages/c3/d5/48cc573aff00d62913701d9fac478518f693b30c25f2c157550b0b2565cb/pyarrow-20.0.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:d5382de8dc34c943249b01c19110783d0d64b207167c728461add1ecc2db88e4", size = 32273686 },
|
||||
{ url = "https://files.pythonhosted.org/packages/37/df/4099b69a432b5cb412dd18adc2629975544d656df3d7fda6d73c5dba935d/pyarrow-20.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6415a0d0174487456ddc9beaead703d0ded5966129fa4fd3114d76b5d1c5ceae", size = 41337051 },
|
||||
{ url = "https://files.pythonhosted.org/packages/4c/27/99922a9ac1c9226f346e3a1e15e63dee6f623ed757ff2893f9d6994a69d3/pyarrow-20.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:15aa1b3b2587e74328a730457068dc6c89e6dcbf438d4369f572af9d320a25ee", size = 42404659 },
|
||||
{ url = "https://files.pythonhosted.org/packages/21/d1/71d91b2791b829c9e98f1e0d85be66ed93aff399f80abb99678511847eaa/pyarrow-20.0.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:5605919fbe67a7948c1f03b9f3727d82846c053cd2ce9303ace791855923fd20", size = 40695446 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f1/ca/ae10fba419a6e94329707487835ec721f5a95f3ac9168500bcf7aa3813c7/pyarrow-20.0.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:a5704f29a74b81673d266e5ec1fe376f060627c2e42c5c7651288ed4b0db29e9", size = 42278528 },
|
||||
{ url = "https://files.pythonhosted.org/packages/7a/a6/aba40a2bf01b5d00cf9cd16d427a5da1fad0fb69b514ce8c8292ab80e968/pyarrow-20.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:00138f79ee1b5aca81e2bdedb91e3739b987245e11fa3c826f9e57c5d102fb75", size = 42918162 },
|
||||
{ url = "https://files.pythonhosted.org/packages/93/6b/98b39650cd64f32bf2ec6d627a9bd24fcb3e4e6ea1873c5e1ea8a83b1a18/pyarrow-20.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:f2d67ac28f57a362f1a2c1e6fa98bfe2f03230f7e15927aecd067433b1e70ce8", size = 44550319 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ab/32/340238be1eb5037e7b5de7e640ee22334417239bc347eadefaf8c373936d/pyarrow-20.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:4a8b029a07956b8d7bd742ffca25374dd3f634b35e46cc7a7c3fa4c75b297191", size = 25770759 },
|
||||
{ url = "https://files.pythonhosted.org/packages/47/a2/b7930824181ceadd0c63c1042d01fa4ef63eee233934826a7a2a9af6e463/pyarrow-20.0.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:24ca380585444cb2a31324c546a9a56abbe87e26069189e14bdba19c86c049f0", size = 30856035 },
|
||||
{ url = "https://files.pythonhosted.org/packages/9b/18/c765770227d7f5bdfa8a69f64b49194352325c66a5c3bb5e332dfd5867d9/pyarrow-20.0.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:95b330059ddfdc591a3225f2d272123be26c8fa76e8c9ee1a77aad507361cfdb", size = 32309552 },
|
||||
{ url = "https://files.pythonhosted.org/packages/44/fb/dfb2dfdd3e488bb14f822d7335653092dde150cffc2da97de6e7500681f9/pyarrow-20.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5f0fb1041267e9968c6d0d2ce3ff92e3928b243e2b6d11eeb84d9ac547308232", size = 41334704 },
|
||||
{ url = "https://files.pythonhosted.org/packages/58/0d/08a95878d38808051a953e887332d4a76bc06c6ee04351918ee1155407eb/pyarrow-20.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b8ff87cc837601532cc8242d2f7e09b4e02404de1b797aee747dd4ba4bd6313f", size = 42399836 },
|
||||
{ url = "https://files.pythonhosted.org/packages/f3/cd/efa271234dfe38f0271561086eedcad7bc0f2ddd1efba423916ff0883684/pyarrow-20.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:7a3a5dcf54286e6141d5114522cf31dd67a9e7c9133d150799f30ee302a7a1ab", size = 40711789 },
|
||||
{ url = "https://files.pythonhosted.org/packages/46/1f/7f02009bc7fc8955c391defee5348f510e589a020e4b40ca05edcb847854/pyarrow-20.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:a6ad3e7758ecf559900261a4df985662df54fb7fdb55e8e3b3aa99b23d526b62", size = 42301124 },
|
||||
{ url = "https://files.pythonhosted.org/packages/4f/92/692c562be4504c262089e86757a9048739fe1acb4024f92d39615e7bab3f/pyarrow-20.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6bb830757103a6cb300a04610e08d9636f0cd223d32f388418ea893a3e655f1c", size = 42916060 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a4/ec/9f5c7e7c828d8e0a3c7ef50ee62eca38a7de2fa6eb1b8fa43685c9414fef/pyarrow-20.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:96e37f0766ecb4514a899d9a3554fadda770fb57ddf42b63d80f14bc20aa7db3", size = 44547640 },
|
||||
{ url = "https://files.pythonhosted.org/packages/54/96/46613131b4727f10fd2ffa6d0d6f02efcc09a0e7374eff3b5771548aa95b/pyarrow-20.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:3346babb516f4b6fd790da99b98bed9708e3f02e734c84971faccb20736848dc", size = 25781491 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a1/d6/0c10e0d54f6c13eb464ee9b67a68b8c71bcf2f67760ef5b6fbcddd2ab05f/pyarrow-20.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:75a51a5b0eef32727a247707d4755322cb970be7e935172b6a3a9f9ae98404ba", size = 30815067 },
|
||||
{ url = "https://files.pythonhosted.org/packages/7e/e2/04e9874abe4094a06fd8b0cbb0f1312d8dd7d707f144c2ec1e5e8f452ffa/pyarrow-20.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:211d5e84cecc640c7a3ab900f930aaff5cd2702177e0d562d426fb7c4f737781", size = 32297128 },
|
||||
{ url = "https://files.pythonhosted.org/packages/31/fd/c565e5dcc906a3b471a83273039cb75cb79aad4a2d4a12f76cc5ae90a4b8/pyarrow-20.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4ba3cf4182828be7a896cbd232aa8dd6a31bd1f9e32776cc3796c012855e1199", size = 41334890 },
|
||||
{ url = "https://files.pythonhosted.org/packages/af/a9/3bdd799e2c9b20c1ea6dc6fa8e83f29480a97711cf806e823f808c2316ac/pyarrow-20.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:2c3a01f313ffe27ac4126f4c2e5ea0f36a5fc6ab51f8726cf41fee4b256680bd", size = 42421775 },
|
||||
{ url = "https://files.pythonhosted.org/packages/10/f7/da98ccd86354c332f593218101ae56568d5dcedb460e342000bd89c49cc1/pyarrow-20.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:a2791f69ad72addd33510fec7bb14ee06c2a448e06b649e264c094c5b5f7ce28", size = 40687231 },
|
||||
{ url = "https://files.pythonhosted.org/packages/bb/1b/2168d6050e52ff1e6cefc61d600723870bf569cbf41d13db939c8cf97a16/pyarrow-20.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:4250e28a22302ce8692d3a0e8ec9d9dde54ec00d237cff4dfa9c1fbf79e472a8", size = 42295639 },
|
||||
{ url = "https://files.pythonhosted.org/packages/b2/66/2d976c0c7158fd25591c8ca55aee026e6d5745a021915a1835578707feb3/pyarrow-20.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:89e030dc58fc760e4010148e6ff164d2f44441490280ef1e97a542375e41058e", size = 42908549 },
|
||||
{ url = "https://files.pythonhosted.org/packages/31/a9/dfb999c2fc6911201dcbf348247f9cc382a8990f9ab45c12eabfd7243a38/pyarrow-20.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6102b4864d77102dbbb72965618e204e550135a940c2534711d5ffa787df2a5a", size = 44557216 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a0/8e/9adee63dfa3911be2382fb4d92e4b2e7d82610f9d9f668493bebaa2af50f/pyarrow-20.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:96d6a0a37d9c98be08f5ed6a10831d88d52cac7b13f5287f1e0f625a0de8062b", size = 25660496 },
|
||||
{ url = "https://files.pythonhosted.org/packages/9b/aa/daa413b81446d20d4dad2944110dcf4cf4f4179ef7f685dd5a6d7570dc8e/pyarrow-20.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:a15532e77b94c61efadde86d10957950392999503b3616b2ffcef7621a002893", size = 30798501 },
|
||||
{ url = "https://files.pythonhosted.org/packages/ff/75/2303d1caa410925de902d32ac215dc80a7ce7dd8dfe95358c165f2adf107/pyarrow-20.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:dd43f58037443af715f34f1322c782ec463a3c8a94a85fdb2d987ceb5658e061", size = 32277895 },
|
||||
{ url = "https://files.pythonhosted.org/packages/92/41/fe18c7c0b38b20811b73d1bdd54b1fccba0dab0e51d2048878042d84afa8/pyarrow-20.0.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:aa0d288143a8585806e3cc7c39566407aab646fb9ece164609dac1cfff45f6ae", size = 41327322 },
|
||||
{ url = "https://files.pythonhosted.org/packages/da/ab/7dbf3d11db67c72dbf36ae63dcbc9f30b866c153b3a22ef728523943eee6/pyarrow-20.0.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b6953f0114f8d6f3d905d98e987d0924dabce59c3cda380bdfaa25a6201563b4", size = 42411441 },
|
||||
{ url = "https://files.pythonhosted.org/packages/90/c3/0c7da7b6dac863af75b64e2f827e4742161128c350bfe7955b426484e226/pyarrow-20.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:991f85b48a8a5e839b2128590ce07611fae48a904cae6cab1f089c5955b57eb5", size = 40677027 },
|
||||
{ url = "https://files.pythonhosted.org/packages/be/27/43a47fa0ff9053ab5203bb3faeec435d43c0d8bfa40179bfd076cdbd4e1c/pyarrow-20.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:97c8dc984ed09cb07d618d57d8d4b67a5100a30c3818c2fb0b04599f0da2de7b", size = 42281473 },
|
||||
{ url = "https://files.pythonhosted.org/packages/bc/0b/d56c63b078876da81bbb9ba695a596eabee9b085555ed12bf6eb3b7cab0e/pyarrow-20.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9b71daf534f4745818f96c214dbc1e6124d7daf059167330b610fc69b6f3d3e3", size = 42893897 },
|
||||
{ url = "https://files.pythonhosted.org/packages/92/ac/7d4bd020ba9145f354012838692d48300c1b8fe5634bfda886abcada67ed/pyarrow-20.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e8b88758f9303fa5a83d6c90e176714b2fd3852e776fc2d7e42a22dd6c2fb368", size = 44543847 },
|
||||
{ url = "https://files.pythonhosted.org/packages/9d/07/290f4abf9ca702c5df7b47739c1b2c83588641ddfa2cc75e34a301d42e55/pyarrow-20.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:30b3051b7975801c1e1d387e17c588d8ab05ced9b1e14eec57915f79869b5031", size = 25653219 },
|
||||
{ url = "https://files.pythonhosted.org/packages/95/df/720bb17704b10bd69dde086e1400b8eefb8f58df3f8ac9cff6c425bf57f1/pyarrow-20.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:ca151afa4f9b7bc45bcc791eb9a89e90a9eb2772767d0b1e5389609c7d03db63", size = 30853957 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d9/72/0d5f875efc31baef742ba55a00a25213a19ea64d7176e0fe001c5d8b6e9a/pyarrow-20.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:4680f01ecd86e0dd63e39eb5cd59ef9ff24a9d166db328679e36c108dc993d4c", size = 32247972 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d5/bc/e48b4fa544d2eea72f7844180eb77f83f2030b84c8dad860f199f94307ed/pyarrow-20.0.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7f4c8534e2ff059765647aa69b75d6543f9fef59e2cd4c6d18015192565d2b70", size = 41256434 },
|
||||
{ url = "https://files.pythonhosted.org/packages/c3/01/974043a29874aa2cf4f87fb07fd108828fc7362300265a2a64a94965e35b/pyarrow-20.0.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3e1f8a47f4b4ae4c69c4d702cfbdfe4d41e18e5c7ef6f1bb1c50918c1e81c57b", size = 42353648 },
|
||||
{ url = "https://files.pythonhosted.org/packages/68/95/cc0d3634cde9ca69b0e51cbe830d8915ea32dda2157560dda27ff3b3337b/pyarrow-20.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:a1f60dc14658efaa927f8214734f6a01a806d7690be4b3232ba526836d216122", size = 40619853 },
|
||||
{ url = "https://files.pythonhosted.org/packages/29/c2/3ad40e07e96a3e74e7ed7cc8285aadfa84eb848a798c98ec0ad009eb6bcc/pyarrow-20.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:204a846dca751428991346976b914d6d2a82ae5b8316a6ed99789ebf976551e6", size = 42241743 },
|
||||
{ url = "https://files.pythonhosted.org/packages/eb/cb/65fa110b483339add6a9bc7b6373614166b14e20375d4daa73483755f830/pyarrow-20.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:f3b117b922af5e4c6b9a9115825726cac7d8b1421c37c2b5e24fbacc8930612c", size = 42839441 },
|
||||
{ url = "https://files.pythonhosted.org/packages/98/7b/f30b1954589243207d7a0fbc9997401044bf9a033eec78f6cb50da3f304a/pyarrow-20.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:e724a3fd23ae5b9c010e7be857f4405ed5e679db5c93e66204db1a69f733936a", size = 44503279 },
|
||||
{ url = "https://files.pythonhosted.org/packages/37/40/ad395740cd641869a13bcf60851296c89624662575621968dcfafabaa7f6/pyarrow-20.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:82f1ee5133bd8f49d31be1299dc07f585136679666b502540db854968576faf9", size = 25944982 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyglet"
|
||||
version = "2.0.9"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c8/6d/6f21100a8a60d16049dd4d187b36e643619f694c9803ae3d92fcbac366a8/pyglet-2.0.9.zip", hash = "sha256:a0922e42f2d258505678e2f4a355c5476c1a6352c3f3a37754042ddb7e7cf72f", size = 6525060 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/94/a1/475458ccf34d2996abdb6ef29fa8d3fed2e62f72df5f2a7f4b4b076915c7/pyglet-2.0.9-py3-none-any.whl", hash = "sha256:8520b22dde75f47167e1fedeed58ac0bb0c890c0dca17d8528427d6b318cd9cc", size = 854706 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyinstaller"
|
||||
version = "6.13.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "altgraph" },
|
||||
{ name = "macholib", marker = "sys_platform == 'darwin'" },
|
||||
{ name = "packaging" },
|
||||
{ name = "pefile", marker = "sys_platform == 'win32'" },
|
||||
{ name = "pyinstaller-hooks-contrib" },
|
||||
{ name = "pywin32-ctypes", marker = "sys_platform == 'win32'" },
|
||||
{ name = "setuptools" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/a8/b1/2949fe6d3874e961898ca5cfc1bf2cf13bdeea488b302e74a745bc28c8ba/pyinstaller-6.13.0.tar.gz", hash = "sha256:38911feec2c5e215e5159a7e66fdb12400168bd116143b54a8a7a37f08733456", size = 4276427 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b4/02/d1a347d35b1b627da1e148159e617576555619ac3bb8bbd5fed661fc7bb5/pyinstaller-6.13.0-py3-none-macosx_10_13_universal2.whl", hash = "sha256:aa404f0b02cd57948098055e76ee190b8e65ccf7a2a3f048e5000f668317069f", size = 1001923 },
|
||||
{ url = "https://files.pythonhosted.org/packages/6b/80/6da39f7aeac65c9ca5afad0fac37887d75fdfd480178a7077c9d30b0704c/pyinstaller-6.13.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:92efcf2f09e78f07b568c5cb7ed48c9940f5dad627af4b49bede6320fab2a06e", size = 718135 },
|
||||
{ url = "https://files.pythonhosted.org/packages/05/2c/d21d31f780a489609e7bf6385c0f7635238dc98b37cba8645b53322b7450/pyinstaller-6.13.0-py3-none-manylinux2014_i686.whl", hash = "sha256:9f82f113c463f012faa0e323d952ca30a6f922685d9636e754bd3a256c7ed200", size = 728543 },
|
||||
{ url = "https://files.pythonhosted.org/packages/e1/20/e6ca87bbed6c0163533195707f820f05e10b8da1223fc6972cfe3c3c50c7/pyinstaller-6.13.0-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:db0e7945ebe276f604eb7c36e536479556ab32853412095e19172a5ec8fca1c5", size = 726868 },
|
||||
{ url = "https://files.pythonhosted.org/packages/20/d5/53b19285f8817ab6c4b07c570208d62606bab0e5a049d50c93710a1d9dc6/pyinstaller-6.13.0-py3-none-manylinux2014_s390x.whl", hash = "sha256:92fe7337c5aa08d42b38d7a79614492cb571489f2cb0a8f91dc9ef9ccbe01ed3", size = 725037 },
|
||||
{ url = "https://files.pythonhosted.org/packages/84/5b/08e0b305ba71e6d7cb247e27d714da7536895b0283132d74d249bf662366/pyinstaller-6.13.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:bc09795f5954135dd4486c1535650958c8218acb954f43860e4b05fb515a21c0", size = 721027 },
|
||||
{ url = "https://files.pythonhosted.org/packages/1f/9c/d8d0a7120103471be8dbe1c5419542aa794b9b9ec2ef628b542f9e6f9ef0/pyinstaller-6.13.0-py3-none-musllinux_1_1_aarch64.whl", hash = "sha256:589937548d34978c568cfdc39f31cf386f45202bc27fdb8facb989c79dfb4c02", size = 723443 },
|
||||
{ url = "https://files.pythonhosted.org/packages/52/c7/8a9d81569dda2352068ecc6ee779d5feff6729569dd1b4ffd1236ecd38fe/pyinstaller-6.13.0-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:b7260832f7501ba1d2ce1834d4cddc0f2b94315282bc89c59333433715015447", size = 719915 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d5/e6/cccadb02b90198c7ed4ffb8bc34d420efb72b996f47cbd4738067a602d65/pyinstaller-6.13.0-py3-none-win32.whl", hash = "sha256:80c568848529635aa7ca46d8d525f68486d53e03f68b7bb5eba2c88d742e302c", size = 1294997 },
|
||||
{ url = "https://files.pythonhosted.org/packages/1a/06/15cbe0e25d1e73d5b981fa41ff0bb02b15e924e30b8c61256f4a28c4c837/pyinstaller-6.13.0-py3-none-win_amd64.whl", hash = "sha256:8d4296236b85aae570379488c2da833b28828b17c57c2cc21fccd7e3811fe372", size = 1352714 },
|
||||
{ url = "https://files.pythonhosted.org/packages/83/ef/74379298d46e7caa6aa7ceccc865106d3d4b15ac487ffdda2a35bfb6fe79/pyinstaller-6.13.0-py3-none-win_arm64.whl", hash = "sha256:d9f21d56ca2443aa6a1e255e7ad285c76453893a454105abe1b4d45e92bb9a20", size = 1293589 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyinstaller-hooks-contrib"
|
||||
version = "2025.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "packaging" },
|
||||
{ name = "setuptools" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/18/46/195324574e44e52c1ba7f7b0607bc9d488b057d93e253918f1a2759d6a98/pyinstaller_hooks_contrib-2025.3.tar.gz", hash = "sha256:af129da5cd6219669fbda360e295cc822abac55b7647d03fec63a8fcf0a608cf", size = 162501 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/9a/98/0273ffc4f85a4038c8d316a75ef5ac1f10f1bbe5ba50c27871b73da2e3d2/pyinstaller_hooks_contrib-2025.3-py3-none-any.whl", hash = "sha256:70cba46b1a6b82ae9104f074c25926e31f3dde50ff217434d1d660355b949683", size = 434307 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "python-dateutil"
|
||||
version = "2.8.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "six" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/4c/c4/13b4776ea2d76c115c1d1b84579f3764ee6d57204f6be27119f13a61d0a9/python-dateutil-2.8.2.tar.gz", hash = "sha256:0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86", size = 357324 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl", hash = "sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9", size = 247702 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pytz"
|
||||
version = "2023.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/5e/32/12032aa8c673ee16707a9b6cdda2b09c0089131f35af55d443b6a9c69c1d/pytz-2023.3.tar.gz", hash = "sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588", size = 317095 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/7f/99/ad6bd37e748257dd70d6f85d916cafe79c0b0f5e2e95b11f7fbc82bf3110/pytz-2023.3-py2.py3-none-any.whl", hash = "sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb", size = 502345 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pywin32"
|
||||
version = "306"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/08/dc/28c668097edfaf4eac4617ef7adf081b9cf50d254672fcf399a70f5efc41/pywin32-306-cp310-cp310-win32.whl", hash = "sha256:06d3420a5155ba65f0b72f2699b5bacf3109f36acbe8923765c22938a69dfc8d", size = 8506422 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d3/d6/891894edec688e72c2e308b3243fad98b4066e1839fd2fe78f04129a9d31/pywin32-306-cp310-cp310-win_amd64.whl", hash = "sha256:84f4471dbca1887ea3803d8848a1616429ac94a4a8d05f4bc9c5dcfd42ca99c8", size = 9226392 },
|
||||
{ url = "https://files.pythonhosted.org/packages/8b/1e/fc18ad83ca553e01b97aa8393ff10e33c1fb57801db05488b83282ee9913/pywin32-306-cp311-cp311-win32.whl", hash = "sha256:e65028133d15b64d2ed8f06dd9fbc268352478d4f9289e69c190ecd6818b6407", size = 8507689 },
|
||||
{ url = "https://files.pythonhosted.org/packages/7e/9e/ad6b1ae2a5ad1066dc509350e0fbf74d8d50251a51e420a2a8feaa0cecbd/pywin32-306-cp311-cp311-win_amd64.whl", hash = "sha256:a7639f51c184c0272e93f244eb24dafca9b1855707d94c192d4a0b4c01e1100e", size = 9227547 },
|
||||
{ url = "https://files.pythonhosted.org/packages/91/20/f744bff1da8f43388498503634378dbbefbe493e65675f2cc52f7185c2c2/pywin32-306-cp311-cp311-win_arm64.whl", hash = "sha256:70dba0c913d19f942a2db25217d9a1b726c278f483a919f1abfed79c9cf64d3a", size = 10388324 },
|
||||
{ url = "https://files.pythonhosted.org/packages/14/91/17e016d5923e178346aabda3dfec6629d1a26efe587d19667542105cf0a6/pywin32-306-cp312-cp312-win32.whl", hash = "sha256:383229d515657f4e3ed1343da8be101000562bf514591ff383ae940cad65458b", size = 8507705 },
|
||||
{ url = "https://files.pythonhosted.org/packages/83/1c/25b79fc3ec99b19b0a0730cc47356f7e2959863bf9f3cd314332bddb4f68/pywin32-306-cp312-cp312-win_amd64.whl", hash = "sha256:37257794c1ad39ee9be652da0462dc2e394c8159dfd913a8a4e8eb6fd346da0e", size = 9227429 },
|
||||
{ url = "https://files.pythonhosted.org/packages/1c/43/e3444dc9a12f8365d9603c2145d16bf0a2f8180f343cf87be47f5579e547/pywin32-306-cp312-cp312-win_arm64.whl", hash = "sha256:5821ec52f6d321aa59e2db7e0a35b997de60c201943557d108af9d4ae1ec7040", size = 10388145 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pywin32-ctypes"
|
||||
version = "0.2.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/85/9f/01a1a99704853cb63f253eea009390c88e7131c67e66a0a02099a8c917cb/pywin32-ctypes-0.2.3.tar.gz", hash = "sha256:d162dc04946d704503b2edc4d55f3dba5c1d539ead017afa00142c38b9885755", size = 29471 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/de/3d/8161f7711c017e01ac9f008dfddd9410dff3674334c233bde66e7ba65bbf/pywin32_ctypes-0.2.3-py3-none-any.whl", hash = "sha256:8a1513379d709975552d202d942d9837758905c8d01eb82b8bcc30918929e7b8", size = 30756 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "setuptools"
|
||||
version = "80.0.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/26/da/7a7021c150030617f90aa4a90a5b23f7b49af877f70ca46967e991645117/setuptools-80.0.1.tar.gz", hash = "sha256:20fe373a22ef9f3925512650d1db90b1b8de01cdb6df91ab1788263139cbf9a2", size = 1354165 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/2a/8e/2ee81652472f3c11503d1780c41844a9a9656989b69c29811a4631e4aeb9/setuptools-80.0.1-py3-none-any.whl", hash = "sha256:f4b49d457765b3aae7cbbeb1c71f6633a61b729408c2d1a837dae064cca82ef2", size = 1240915 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "six"
|
||||
version = "1.16.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/71/39/171f1c67cd00715f190ba0b100d606d440a28c93c7714febeca8b79af85e/six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", size = 34041 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254", size = 11053 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tenacity"
|
||||
version = "8.2.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d3/f0/6ccd8854f4421ce1f227caf3421d9be2979aa046939268c9300030c0d250/tenacity-8.2.2.tar.gz", hash = "sha256:43af037822bd0029025877f3b2d97cc4d7bb0c2991000a3d59d71517c5c969e0", size = 40186 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/b0/c23bd61e1b32c9b96fbca996c87784e196a812da8d621d8d04851f6c8181/tenacity-8.2.2-py3-none-any.whl", hash = "sha256:2f277afb21b851637e8f52e6a613ff08734c347dc19ade928e519d7d2d8569b0", size = 24390 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tkcalendar"
|
||||
version = "1.6.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "babel" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/65/3d/3406cf7963661ed890082bff17ed4c5e26b5a564306639303d4fbb2a047f/tkcalendar-1.6.1.tar.gz", hash = "sha256:5edf958c0a59429e90309e9b805b2e229192bbcab952460247204d7030eea5cf", size = 32916 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/e9/d4/9528ea6ecb5d4394f425df651957da6f6a715b41c5b12d43d41888c14394/tkcalendar-1.6.1-py3-none-any.whl", hash = "sha256:9d3a80816a7b32d64fab696fa3d2a007fb23c87953267d5e343a38ff4cd7c15c", size = 40912 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "traitlets"
|
||||
version = "5.9.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/39/c3/205e88f02959712b62008502952707313640369144a7fded4cbc61f48321/traitlets-5.9.0.tar.gz", hash = "sha256:f6cde21a9c68cf756af02035f72d5a723bf607e862e7be33ece505abf4a3bad9", size = 150207 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/77/75/c28e9ef7abec2b7e9ff35aea3e0be6c1aceaf7873c26c95ae1f0d594de71/traitlets-5.9.0-py3-none-any.whl", hash = "sha256:9e6ec080259b9a5940c797d58b613b5e31441c2257b87c2e795c5228ae80d2d8", size = 117376 },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tzdata"
|
||||
version = "2023.3"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/70/e5/81f99b9fced59624562ab62a33df639a11b26c582be78864b339dafa420d/tzdata-2023.3.tar.gz", hash = "sha256:11ef1e08e54acb0d4f95bdb1be05da659673de4acbd21bf9c69e94cc5e907a3a", size = 187483 }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d5/fb/a79efcab32b8a1f1ddca7f35109a50e4a80d42ac1c9187ab46522b2407d7/tzdata-2023.3-py2.py3-none-any.whl", hash = "sha256:7e65763eef3120314099b6939b5546db7adce1e7d6f2e179e3df563c70511eda", size = 341835 },
|
||||
]
|
||||
@@ -0,0 +1,18 @@
|
||||
"""
|
||||
Visualization package for patient pathway charts.
|
||||
|
||||
This package contains functions for generating interactive Plotly visualizations:
|
||||
- plotly_generator: Create icicle charts for patient pathway analysis
|
||||
"""
|
||||
|
||||
from visualization.plotly_generator import (
|
||||
create_icicle_figure,
|
||||
save_figure_html,
|
||||
open_figure_in_browser,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"create_icicle_figure",
|
||||
"save_figure_html",
|
||||
"open_figure_in_browser",
|
||||
]
|
||||
@@ -0,0 +1,231 @@
|
||||
"""
|
||||
Plotly chart generation for patient pathway analysis.
|
||||
|
||||
This module contains functions for creating interactive icicle charts
|
||||
that visualize patient treatment pathways. The charts display hierarchical
|
||||
data: Trust → Directory → Drug → Pathway.
|
||||
"""
|
||||
|
||||
import webbrowser
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import plotly.graph_objects as go
|
||||
|
||||
from core.logging_config import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def create_icicle_figure(ice_df: pd.DataFrame, title: str) -> go.Figure:
|
||||
"""
|
||||
Create Plotly icicle figure from prepared DataFrame.
|
||||
|
||||
This function generates an interactive icicle chart showing patient pathway
|
||||
hierarchies with custom data including costs, dates, and treatment durations.
|
||||
|
||||
Args:
|
||||
ice_df: DataFrame with columns:
|
||||
- parents: Parent node in hierarchy
|
||||
- ids: Unique identifier for each node
|
||||
- labels: Display label for each node
|
||||
- value: Number of patients
|
||||
- colour: Color value for visualization
|
||||
- cost: Total cost
|
||||
- costpp: Cost per patient
|
||||
- cost_pp_pa: Cost per patient per annum
|
||||
- First seen: First intervention date
|
||||
- Last seen: Last intervention date
|
||||
- First seen (Parent): Earliest date in parent group
|
||||
- Last seen (Parent): Latest date in parent group
|
||||
- average_spacing: Formatted string with dosing information
|
||||
- avg_days: Average treatment duration
|
||||
title: Chart title
|
||||
|
||||
Returns:
|
||||
Plotly Figure object ready for display or export
|
||||
"""
|
||||
ice_df = ice_df.copy()
|
||||
ice_df.sort_values(by=["labels"], ascending=True, inplace=True, ignore_index=True)
|
||||
|
||||
first_seen = ice_df["First seen"].astype(str).replace("NaT", "N/A").to_list()
|
||||
last_seen = ice_df["Last seen"].astype(str).replace("NaT", "N/A").to_list()
|
||||
first_seen_parent = ice_df["First seen (Parent)"].astype(str).to_list()
|
||||
last_seen_parent = ice_df["Last seen (Parent)"].astype(str).to_list()
|
||||
average_spacing = ice_df.average_spacing.astype(str).to_list()
|
||||
|
||||
fig = go.Figure(
|
||||
go.Icicle(
|
||||
labels=ice_df.labels,
|
||||
ids=ice_df.ids,
|
||||
parents=ice_df.parents,
|
||||
customdata=np.stack(
|
||||
(
|
||||
ice_df.value,
|
||||
ice_df.colour,
|
||||
ice_df.cost,
|
||||
ice_df.costpp,
|
||||
first_seen,
|
||||
last_seen,
|
||||
first_seen_parent,
|
||||
last_seen_parent,
|
||||
average_spacing,
|
||||
ice_df.cost_pp_pa,
|
||||
),
|
||||
axis=1,
|
||||
),
|
||||
values=ice_df.value,
|
||||
branchvalues="total",
|
||||
marker=dict(colors=ice_df.colour, colorscale="Viridis"),
|
||||
maxdepth=3,
|
||||
texttemplate="<b>%{label}</b> "
|
||||
"<br><b>Total patients:</b> %{customdata[0]} (including children/further treatments)"
|
||||
"<br><b>First seen:</b> %{customdata[4]}"
|
||||
"<br><b>Last seen (including further treatments):</b> %{customdata[7]}"
|
||||
"<br><b>Average treatment duration:</b> %{customdata[8]}"
|
||||
"<br><b>Total cost:</b> £%{customdata[2]:.3~s}"
|
||||
"<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}"
|
||||
"<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}",
|
||||
hovertemplate="<b>%{label}</b>"
|
||||
"<br><b>Total patients:</b> %{customdata[0]} - %{customdata[1]:.3p} of patients in level"
|
||||
"<br><b>Total cost:</b> £%{customdata[2]:.3~s}"
|
||||
"<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}"
|
||||
"<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}"
|
||||
"<br><b>First seen:</b> %{customdata[4]}"
|
||||
"<br><b>Last seen (including further treatments):</b> %{customdata[7]}"
|
||||
"<br><b>Average treatment duration:</b>"
|
||||
"%{customdata[8]}"
|
||||
"<extra></extra>",
|
||||
)
|
||||
)
|
||||
fig.update_traces(sort=False)
|
||||
fig.update_layout(
|
||||
margin=dict(t=60, l=1, r=1, b=60),
|
||||
title=f"Norfolk & Waveney ICS high-cost drug patient pathways - {title}",
|
||||
title_x=0.5,
|
||||
hoverlabel=dict(font_size=16),
|
||||
)
|
||||
|
||||
return fig
|
||||
|
||||
|
||||
def save_figure_html(
|
||||
fig: go.Figure, save_dir: str, title: str, open_browser: bool = False
|
||||
) -> str:
|
||||
"""
|
||||
Save Plotly figure to HTML file.
|
||||
|
||||
Args:
|
||||
fig: Plotly Figure object
|
||||
save_dir: Directory to save the HTML file
|
||||
title: Title used for filename
|
||||
open_browser: If True, open the file in the default browser
|
||||
|
||||
Returns:
|
||||
Path to the saved HTML file
|
||||
"""
|
||||
filepath = f"{save_dir}/{title}.html"
|
||||
fig.write_html(filepath)
|
||||
logger.info(f"Success! File saved to {filepath}")
|
||||
|
||||
if open_browser:
|
||||
open_figure_in_browser(filepath)
|
||||
|
||||
return filepath
|
||||
|
||||
|
||||
def open_figure_in_browser(filepath: str) -> None:
|
||||
"""
|
||||
Open an HTML file in the default browser.
|
||||
|
||||
Args:
|
||||
filepath: Path to the HTML file
|
||||
"""
|
||||
webbrowser.open_new_tab("file:///" + filepath)
|
||||
|
||||
|
||||
def figure_legacy(ice_df: pd.DataFrame, dir_string: str, save_dir: str) -> None:
|
||||
"""
|
||||
Create and display icicle figure (legacy interface).
|
||||
|
||||
This function maintains backward compatibility with the original figure()
|
||||
function signature. It creates the figure, saves it to HTML, and opens
|
||||
it in the browser.
|
||||
|
||||
Args:
|
||||
ice_df: DataFrame with chart data
|
||||
dir_string: Title string (used for filename and chart title)
|
||||
save_dir: Directory to save the HTML file
|
||||
|
||||
Note:
|
||||
This function is provided for backward compatibility.
|
||||
New code should use create_icicle_figure() + save_figure_html() instead.
|
||||
"""
|
||||
# Handle avg_days column for display
|
||||
ice_df = ice_df.copy()
|
||||
ice_df.sort_values(by=["labels"], ascending=True, inplace=True, ignore_index=True)
|
||||
|
||||
first_seen = ice_df["First seen"].astype(str).replace("NaT", "N/A").to_list()
|
||||
last_seen = ice_df["Last seen"].astype(str).replace("NaT", "N/A").to_list()
|
||||
first_seen_parent = ice_df["First seen (Parent)"].astype(str).to_list()
|
||||
last_seen_parent = ice_df["Last seen (Parent)"].astype(str).to_list()
|
||||
average_spacing = ice_df.average_spacing.astype(str).to_list()
|
||||
avg_seen = ice_df["avg_days"].dt.round("D").astype(str).replace("0 days", "N/A").to_list()
|
||||
|
||||
fig = go.Figure(
|
||||
go.Icicle(
|
||||
labels=ice_df.labels,
|
||||
ids=ice_df.ids,
|
||||
parents=ice_df.parents,
|
||||
customdata=np.stack(
|
||||
(
|
||||
ice_df.value,
|
||||
ice_df.colour,
|
||||
ice_df.cost,
|
||||
ice_df.costpp,
|
||||
first_seen,
|
||||
last_seen,
|
||||
first_seen_parent,
|
||||
last_seen_parent,
|
||||
average_spacing,
|
||||
ice_df.cost_pp_pa,
|
||||
),
|
||||
axis=1,
|
||||
),
|
||||
values=ice_df.value,
|
||||
branchvalues="total",
|
||||
marker=dict(colors=ice_df.colour, colorscale="Viridis"),
|
||||
maxdepth=3,
|
||||
texttemplate="<b>%{label}</b> "
|
||||
"<br><b>Total patients:</b> %{customdata[0]} (including children/further treatments)"
|
||||
"<br><b>First seen:</b> %{customdata[4]}"
|
||||
"<br><b>Last seen (including further treatments):</b> %{customdata[7]}"
|
||||
"<br><b>Average treatment duration:</b> %{customdata[8]}"
|
||||
"<br><b>Total cost:</b> £%{customdata[2]:.3~s}"
|
||||
"<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}"
|
||||
"<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}",
|
||||
hovertemplate="<b>%{label}</b>"
|
||||
"<br><b>Total patients:</b> %{customdata[0]} - %{customdata[1]:.3p} of patients in level"
|
||||
"<br><b>Total cost:</b> £%{customdata[2]:.3~s}"
|
||||
"<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}"
|
||||
"<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}"
|
||||
"<br><b>First seen:</b> %{customdata[4]}"
|
||||
"<br><b>Last seen (including further treatments):</b> %{customdata[7]}"
|
||||
"<br><b>Average treatment duration:</b>"
|
||||
"%{customdata[8]}"
|
||||
"<extra></extra>",
|
||||
)
|
||||
)
|
||||
fig.update_traces(sort=False)
|
||||
fig.update_layout(
|
||||
margin=dict(t=60, l=1, r=1, b=60),
|
||||
title=f"Norfolk & Waveney ICS high-cost drug patient pathways - {dir_string}",
|
||||
title_x=0.5,
|
||||
hoverlabel=dict(font_size=16),
|
||||
)
|
||||
|
||||
filepath = f"{save_dir}/{dir_string}.html"
|
||||
fig.write_html(filepath)
|
||||
logger.info(f"Success! File saved to {filepath}")
|
||||
webbrowser.open_new_tab("file:///" + filepath)
|
||||
Reference in New Issue
Block a user