chore: cleanup archive, update ralph loop files, add analytics spec
- Remove old iteration logs and deprecated files from archive/can_delete/ - Update RALPH_PROMPT.md and guardrails.md for Phase 10+ work - Update ralph.ps1 banner text - Add AdditionalAnalytics.md chart specification - Add run.bat convenience script
This commit is contained in:
Andrew Charlwood
@@ -7,7 +7,16 @@
|
||||
"WebFetch(domain:github.com)",
|
||||
"WebFetch(domain:docs.flet.dev)",
|
||||
"Bash(python -c:*)",
|
||||
"Bash(Remove-Item -Path \"C:\\\\Users\\\\charlwoodand\\\\Ralph local\\\\Tasks\\\\UI Redesign\\\\logs\\\\*\" -Force)"
|
||||
"Bash(Remove-Item -Path \"C:\\\\Users\\\\charlwoodand\\\\Ralph local\\\\Tasks\\\\UI Redesign\\\\logs\\\\*\" -Force)",
|
||||
"Bash(git commit -m \"$\\(cat <<''EOF''\ndocs: update CLAUDE.md to reflect slimmed database architecture\n\nRemove references to deleted tables \\(fact_interventions,\nmv_patient_treatment_summary, ref_drug_snomed_mapping, processed_files\\),\ndeleted files \\(patient_data.py, load_snomed_mapping.py\\), and removed\nclasses \\(SQLiteDataLoader\\). Update package structure, data loaders,\ndatabase schema, fallback chain, and AppState descriptions.\nEOF\n\\)\")",
|
||||
"Bash(Get-ChildItem -Recurse -Force)",
|
||||
"Bash(Select-Object FullName)",
|
||||
"Bash(uv run python:*)",
|
||||
"Bash(uv run:*)",
|
||||
"Bash(PYTHONPATH=src uv run python:*)",
|
||||
"Bash(uv pip install:*)",
|
||||
"Bash(uv lock:*)",
|
||||
"Bash(git commit:*)"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
# Additional Analytics Charts — Implementation Plan
|
||||
|
||||
## UI Approach: Tabbed Chart Area
|
||||
Extend existing `chart_card.py` tab bar. Currently has Icicle (active), Sankey (disabled), Timeline (disabled). Replace/extend with new tabs.
|
||||
|
||||
## Charts to Build (Priority Order)
|
||||
|
||||
### Tab 1: Icicle (existing — no change)
|
||||
|
||||
### Tab 2: First-Line Market Share — Horizontal Bar Chart
|
||||
**What**: % of patients starting on each first-line drug, grouped by directorate or indication
|
||||
**Data source**: `pathway_nodes WHERE level = 3` (drug level). The `colour` column already holds proportion of parent. `value` = patient count.
|
||||
**Query**: Filter by `chart_type`, `date_filter_id`, optionally `directory` or `trust_name`. Group by `directory`, then show drugs as bars.
|
||||
**Viz**: Horizontal grouped bar chart. One cluster per directorate/indication (top N), bars within = drugs, length = % of patients. Sorted by total patients desc. NHS blue palette.
|
||||
**Interaction**: Responds to all existing filters (date, chart type, trust, drug, directorate). Clicking a directorate cluster could filter the icicle.
|
||||
|
||||
### Tab 3: Pathway Cost Effectiveness — Lollipop/Dot Plot
|
||||
**What**: Compare annualized cost per patient across complete treatment pathways within a directorate/indication. Highlights most vs least cost-effective pathways.
|
||||
**Data source**: `pathway_nodes WHERE level >= 4` (pathway nodes). Fields: `cost_pp_pa` (annualized), `value` (patient count), `ids` (parse to get pathway sequence), `directory`.
|
||||
**Calculation**: `cost_pp_pa` is already computed as `(total_cost / patients) * (365 / avg_days)` — this IS the "total cost over N years / N years" the user described.
|
||||
**Query**: Filter to a specific directorate/indication, then show all pathway variants ranked by `cost_pp_pa`.
|
||||
**Viz**: Horizontal lollipop chart (dot on stick). Y-axis = pathway label (e.g., "Adalimumab → Secukinumab → Rituximab"), X-axis = £ per patient per annum. Dot size = patient count. Colour gradient: green (cheap) → amber → red (expensive).
|
||||
**Interaction**: Directorate/indication selector drives which pathways are shown. Could also compare across directorates at the drug level (level 3).
|
||||
|
||||
**Bonus metric — "Pathway Retention" (fewest switches)**:
|
||||
- For each 2nd-line pathway (e.g., "Drug A → Drug B"), calculate what % of patients escalate to a 3rd line
|
||||
- Derivation: `value("Drug A → Drug B") - SUM(value("Drug A → Drug B → *"))` = patients who stayed on 2nd line
|
||||
- Show as a secondary annotation or companion chart: "Drug B retains 72% of patients (no 3rd-line switch needed)"
|
||||
- This identifies the most effective 2nd-line choices
|
||||
|
||||
### Tab 4: Cost Waterfall — Waterfall Chart
|
||||
**What**: Break down £ per patient per annum by directorate, showing relative cost contribution
|
||||
**Data source**: `pathway_nodes WHERE level = 2` (directorate/indication level). Field: `cost_pp_pa`, `value`.
|
||||
**Viz**: Plotly waterfall chart. Each bar = one directorate's average cost_pp_pa. Sorted highest to lowest. Running reference line optional. Use NHS colours.
|
||||
**Note**: User specifically wants cost_pp_pa (annualized), not total cost.
|
||||
**Interaction**: Responds to chart_type toggle, date filter, trust filter.
|
||||
|
||||
### Tab 5: Drug Switching Sankey — Sankey Diagram
|
||||
**What**: Flow of patients from 1st-line → 2nd-line → 3rd-line drugs
|
||||
**Data source**: `pathway_nodes WHERE level >= 3`. Parse `ids` to extract drug transition sequences.
|
||||
**Parsing**: `ids` format at level 4+: `"TRUST - DIRECTORY - DRUG_A - DRUG_A|DRUG_B"`. Split by " - ", take segments from level 3 onwards, split by "|" to get ordered drug list.
|
||||
**Viz**: Plotly Sankey. Left nodes = 1st-line drugs, middle = 2nd-line, right = 3rd-line. Link width = patient count. Colour by drug or by directorate.
|
||||
**Interaction**: Filter by directorate/indication to see switching within a specialty. Filter by trust to compare switching patterns.
|
||||
|
||||
### Tab 6: Dosing Interval Comparison — Grouped Bar Chart
|
||||
**What**: Compare average dosing frequency/weekly interval for a drug across trusts or directorates
|
||||
**Data source**: Level 3+ nodes, `average_spacing` (HTML string), `average_administered` (JSON array)
|
||||
**Parsing needed**:
|
||||
- `average_spacing`: regex to extract weekly interval number from `"given X times with Y weekly interval"`
|
||||
- `average_administered`: `json.loads()` to get dose counts
|
||||
**Viz**: Horizontal grouped bars. Y-axis = trust or directorate, X-axis = weekly interval (or total administrations). One colour per drug if comparing multiple.
|
||||
**Interaction**: Drug selector to pick which drug to compare. Group-by selector (trust vs directorate).
|
||||
|
||||
### Tab 7: Directorate × Drug Heatmap
|
||||
**What**: Matrix showing which drugs are used in which directorates, cells coloured by patient count or cost_pp_pa
|
||||
**Data source**: Level 3 nodes, pivot `directory` × drug (parsed from `labels` or `ids`)
|
||||
**Viz**: Plotly heatmap. Rows = directorates (sorted by total patients), columns = drugs (sorted by frequency). Cell colour = patient count or cost. Hover shows details.
|
||||
**Interaction**: Toggle between patient count / cost / cost_pp_pa colouring.
|
||||
|
||||
### Tab 8: Treatment Duration Bars
|
||||
**What**: Compare average treatment durations across drugs within a directorate
|
||||
**Data source**: Level 3 nodes, `avg_days` field
|
||||
**Viz**: Horizontal bar chart. Y-axis = drug, X-axis = average days. Colour intensity by patient count.
|
||||
**Interaction**: Directorate filter drives which drugs are shown.
|
||||
|
||||
---
|
||||
|
||||
## Data Layer Changes
|
||||
|
||||
### New query functions needed (in `src/data_processing/pathway_queries.py`):
|
||||
|
||||
```python
|
||||
def get_drug_market_share(db_path, date_filter_id, chart_type, directory=None, trust=None):
|
||||
"""Level 3 nodes grouped by directory, returning drug, value, colour."""
|
||||
|
||||
def get_pathway_costs(db_path, date_filter_id, chart_type, directory=None):
|
||||
"""Level 4+ nodes with cost_pp_pa, parsed pathway labels, patient counts."""
|
||||
|
||||
def get_cost_waterfall(db_path, date_filter_id, chart_type, trust=None):
|
||||
"""Level 2 nodes with cost_pp_pa per directorate/indication."""
|
||||
|
||||
def get_drug_transitions(db_path, date_filter_id, chart_type, directory=None):
|
||||
"""Level 3+ nodes parsed into source→target drug transitions with patient counts."""
|
||||
|
||||
def get_dosing_intervals(db_path, date_filter_id, chart_type, drug=None):
|
||||
"""Level 3 nodes for a specific drug, parsed average_spacing by trust/directory."""
|
||||
|
||||
def get_drug_directory_matrix(db_path, date_filter_id, chart_type):
|
||||
"""Level 3 nodes pivoted as directory × drug with value/cost metrics."""
|
||||
|
||||
def get_treatment_durations(db_path, date_filter_id, chart_type, directory=None):
|
||||
"""Level 3 nodes with avg_days by drug within a directorate."""
|
||||
```
|
||||
|
||||
### Parsing utilities needed:
|
||||
|
||||
```python
|
||||
def parse_average_spacing(spacing_html: str) -> dict:
|
||||
"""Extract drug_name, dose_count, weekly_interval, total_weeks from HTML string."""
|
||||
|
||||
def parse_pathway_drugs(ids: str, level: int) -> list[str]:
|
||||
"""Extract ordered drug list from ids column at level 4+."""
|
||||
|
||||
def calculate_retention_rate(nodes: list[dict]) -> dict:
|
||||
"""For each N-drug pathway, calculate % not escalating to N+1 drugs."""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Callback Architecture
|
||||
|
||||
Each tab gets its own callback triggered by `chart-data` store + `active-tab` state:
|
||||
|
||||
```
|
||||
active-tab change → render selected chart
|
||||
chart-data change → re-render active chart
|
||||
```
|
||||
|
||||
Only the active tab's chart is computed (lazy rendering). Store the active tab in `app-state`.
|
||||
|
||||
New callback per chart type in `dash_app/callbacks/`:
|
||||
- `market_share.py` — builds bar chart from level 3 data
|
||||
- `pathway_costs.py` — builds lollipop + retention annotations
|
||||
- `cost_waterfall.py` — builds waterfall from level 2 data
|
||||
- `sankey.py` — builds Sankey from parsed transitions
|
||||
- `dosing.py` — builds grouped bars from parsed spacing
|
||||
- `heatmap.py` — builds heatmap from pivoted matrix
|
||||
- `duration.py` — builds bar chart from avg_days
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Data parsing utilities** — shared parsing for spacing, pathway drugs, retention
|
||||
2. **Query functions** — one per chart type in pathway_queries.py
|
||||
3. **Tab infrastructure** — extend chart_card.py with all tab labels, lazy rendering
|
||||
4. **Charts one at a time** (in priority order):
|
||||
- First-Line Market Share (simplest, validates the tab pattern)
|
||||
- Pathway Cost Effectiveness + Retention (user's key insight)
|
||||
- Cost Waterfall
|
||||
- Drug Switching Sankey
|
||||
- Dosing Interval
|
||||
- Heatmap
|
||||
- Treatment Duration
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
- Run `python run_dash.py` after each chart addition
|
||||
- Verify each chart responds to filter changes (date, chart type, trust, directorate, drug)
|
||||
- Test with both "directory" and "indication" chart types
|
||||
- Verify icicle chart still works correctly (no regressions)
|
||||
- Check tab switching is smooth with no unnecessary recomputation
|
||||
+18
-6
@@ -1,8 +1,8 @@
|
||||
# Ralph Wiggum Loop — Dash Application Maintenance
|
||||
# Ralph Wiggum Loop — Dash Application: Additional Analytics Charts
|
||||
|
||||
You are operating inside an automated loop maintaining an NHS patient pathway analysis tool built with Dash (Plotly) + Dash Mantine Components. Each iteration you receive fresh context — you have NO memory of previous iterations. Your only memory is the filesystem.
|
||||
You are operating inside an automated loop adding analytics charts to an NHS patient pathway analysis tool built with Dash (Plotly) + Dash Mantine Components. Each iteration you receive fresh context — you have NO memory of previous iterations. Your only memory is the filesystem.
|
||||
|
||||
**Current Focus**: Maintain and enhance the Dash application in `dash_app/`. The backend (`src/`) provides shared data access and visualization functions. The design target is `01_nhs_classic.html`.
|
||||
**Current Focus**: Phase 9 — Add 7 new analytics chart tabs alongside the existing icicle chart. Tab bar in chart_card.py, lazy rendering, shared query/figure functions in `src/`. See IMPLEMENTATION_PLAN.md Phase 9 for full task list.
|
||||
|
||||
## First Actions Every Iteration
|
||||
|
||||
@@ -24,11 +24,16 @@ Then run `git log --oneline -5` to see recent commits.
|
||||
- Match the design as closely as possible — `className` in Dash = `class` in HTML
|
||||
|
||||
**When building data loading or chart callbacks**, reference the shared functions in `src/`:
|
||||
- `src/data_processing/pathway_queries.py`: `load_initial_data()` and `load_pathway_nodes()` — shared query functions
|
||||
- `src/visualization/plotly_generator.py`: `create_icicle_from_nodes()` — icicle chart from list-of-dicts
|
||||
- `src/data_processing/pathway_queries.py`: `load_initial_data()`, `load_pathway_nodes()`, and 7 new chart-specific query functions (Phase 9)
|
||||
- `src/visualization/plotly_generator.py`: `create_icicle_from_nodes()` — icicle chart from list-of-dicts. Add new figure functions here for each chart type.
|
||||
- `dash_app/data/queries.py`: Thin wrapper calling shared functions with correct DB path
|
||||
- The original logic is archived in `archive/pathways_app/pathways_app.py` for reference.
|
||||
|
||||
**When building new analytics charts (Phase 9)**, also read:
|
||||
- `AdditionalAnalytics.md` — Full specification for each chart: data source, visualization type, interaction, parsing requirements
|
||||
- `src/data_processing/pathway_queries.py` — Existing query patterns to follow. All new queries go here.
|
||||
- Key data columns: `level` (0=root, 1=trust, 2=directory, 3=drug, 4+=pathway), `ids` (hierarchy path), `cost_pp_pa`, `avg_days`, `average_spacing`, `average_administered`
|
||||
|
||||
## Narration
|
||||
|
||||
Narrate your work as you go. Your output is the only visibility the operator has into what's happening. For every significant action, explain what you're doing and why:
|
||||
@@ -80,8 +85,9 @@ html.Div(className="top-header", children=[...])
|
||||
|
||||
# Mantine components for rich UI
|
||||
import dash_mantine_components as dmc
|
||||
dmc.Drawer(id="drug-drawer", position="right", size="480px", children=[...])
|
||||
dmc.Modal(id="drug-modal", opened=False, centered=True, size="lg", children=[...])
|
||||
dmc.Accordion(children=[dmc.AccordionItem(...)])
|
||||
dmc.ChipGroup(id="all-drugs-chips", multiple=True, children=[dmc.Chip(...)])
|
||||
|
||||
# State management
|
||||
dcc.Store(id="app-state", storage_type="session", data={})
|
||||
@@ -95,6 +101,9 @@ def load_pathway_data(app_state):
|
||||
...
|
||||
```
|
||||
|
||||
### Important: Use frontend-developer agent for UX decisions
|
||||
When building modals, filter bar layout, or other UX-sensitive components, spawn the `frontend-developer` agent to review data shapes and recommend optimal patterns. Data shapes: 42 drugs, 7 trusts, 19 directorates × 163 indications.
|
||||
|
||||
### Database Access Pattern
|
||||
|
||||
```python
|
||||
@@ -224,6 +233,9 @@ DO NOT output it if any task is still `[ ]` or `[B]` or `[~]`.
|
||||
- **dcc.Store for state** — no server-side globals
|
||||
- **Unidirectional callbacks** — app-state → chart-data → UI
|
||||
- **Port icicle_figure exactly** — same customdata, colorscale, templates
|
||||
- **Lazy tab rendering** — only compute the active tab's chart, not all 8
|
||||
- **New figure functions** go in `src/visualization/`, not in `dash_app/callbacks/`
|
||||
- **New query functions** go in `src/data_processing/pathway_queries.py` with thin wrappers in `dash_app/data/queries.py`
|
||||
- Keep commits atomic and well-described
|
||||
- If stuck for 2+ attempts, document in progress.txt and move on
|
||||
- `python run_dash.py` must work after every task
|
||||
|
||||
@@ -1,647 +0,0 @@
|
||||
import webbrowser
|
||||
from itertools import groupby
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import plotly.graph_objects as go
|
||||
|
||||
from core import AnalysisFilters, PathConfig, default_paths
|
||||
from core.logging_config import get_logger
|
||||
from tools import data
|
||||
|
||||
# Import refactored analysis functions
|
||||
from analysis.pathway_analyzer import (
|
||||
generate_icicle_chart as _generate_icicle_chart,
|
||||
prepare_data as _prepare_data,
|
||||
calculate_statistics as _calculate_statistics,
|
||||
build_hierarchy as _build_hierarchy,
|
||||
prepare_chart_data as _prepare_chart_data,
|
||||
)
|
||||
|
||||
# Import visualization functions
|
||||
from visualization.plotly_generator import (
|
||||
create_icicle_figure as _create_icicle_figure,
|
||||
save_figure_html as _save_figure_html,
|
||||
figure_legacy as _figure_legacy,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
pd.options.mode.chained_assignment = None # default='warn'
|
||||
def human_format(num):
|
||||
num = float('{:.3g}'.format(num))
|
||||
magnitude = 0
|
||||
while abs(num) >= 1000:
|
||||
magnitude += 1
|
||||
num /= 1000.0
|
||||
return '{}{}'.format('{:f}'.format(num).rstrip('0').rstrip('.'), ['', 'K', 'M', 'B', 'T'][magnitude])
|
||||
|
||||
def main(dir, paths: Optional[PathConfig] = None):
|
||||
"""
|
||||
Load and process patient intervention data from a file.
|
||||
|
||||
Uses the FileDataLoader abstraction to handle CSV/Parquet file loading
|
||||
with all necessary transformations (patient_id, drug_names, department_identification).
|
||||
|
||||
Args:
|
||||
dir: Path to CSV or Parquet file
|
||||
paths: PathConfig for reference data locations (uses default_paths if None)
|
||||
|
||||
Returns:
|
||||
DataFrame with processed patient intervention data
|
||||
"""
|
||||
from data_processing.loader import FileDataLoader
|
||||
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
loader = FileDataLoader(file_path=dir, paths=paths)
|
||||
result = loader.load()
|
||||
|
||||
logger.info("Initial data processing complete.")
|
||||
return result.df
|
||||
|
||||
|
||||
def drop_duplicate_treatments(df, ascending):
|
||||
df.sort_values(by=['Intervention Date'], ascending=ascending, inplace=True)
|
||||
df_treatment_steps = df.drop_duplicates(subset="UPIDTreatment", keep="first")
|
||||
if not ascending:
|
||||
df_treatment_steps.sort_values(by=['Intervention Date'], ascending=True, inplace=True)
|
||||
return df_treatment_steps
|
||||
|
||||
|
||||
def row_function(row):
|
||||
ids = ""
|
||||
parents = "N&WICS"
|
||||
count = row.count()
|
||||
for c in range(count):
|
||||
v = row[c]
|
||||
if type(v) != str:
|
||||
v = row[c + 1]
|
||||
if c == count - 1:
|
||||
ids = parents + " - " + v
|
||||
continue
|
||||
parents += " - " + v
|
||||
label = row[count - 1]
|
||||
value = parents + "," + label + "," + ids
|
||||
return value
|
||||
|
||||
|
||||
def count_list_values(x):
|
||||
return [len(list(group)) for key, group in groupby(sorted(x))]
|
||||
|
||||
|
||||
def sum_list_values(x):
|
||||
sum_list = []
|
||||
for count in range(len(x["Drug Name"])):
|
||||
if count == 0:
|
||||
sum_list.append(sum(x["Price Actual"][ : x["Drug Name"][count]]))
|
||||
else:
|
||||
sum_list.append(sum(x["Price Actual"][x["Drug Name"][count-1] : (x["Drug Name"][count-1] + x["Drug Name"][count])]))
|
||||
return sum_list
|
||||
|
||||
|
||||
def remove_nan_string(y):
|
||||
return [x for x in y if str(x) != 'nan']
|
||||
|
||||
|
||||
def min_max_treatment_dates(ice_df, row):
|
||||
ids = row[2]
|
||||
min_max = ice_df[ice_df["ids"].str.contains(ids)]
|
||||
min_date = str(min_max["First seen"].min().strftime('%Y-%m-%d'))
|
||||
max_date = str(min_max["Last seen"].max().strftime('%Y-%m-%d'))
|
||||
return min_date + ',' + max_date
|
||||
|
||||
|
||||
def start_date_drug(df, x):
|
||||
drug_count = x.notnull().sum()
|
||||
date_string = []
|
||||
for d in range(drug_count):
|
||||
UPID_date_var = str(x.name) + str(x[d])
|
||||
date = df.loc[UPID_date_var, "Intervention Date"]
|
||||
date_string.append(date)
|
||||
return date_string
|
||||
|
||||
|
||||
def end_date_drug(df, x):
|
||||
drug_count = x.notnull().sum()
|
||||
date_string = []
|
||||
# Need to -1 from drug count as start date gets counted from notnull above
|
||||
for d in range(drug_count - 1):
|
||||
UPID_date_var = str(x.name) + str(x[d])
|
||||
date = df.loc[UPID_date_var, "Intervention Date"]
|
||||
date_string.append(date)
|
||||
return date_string
|
||||
|
||||
|
||||
def list_to_string(x):
|
||||
list = x.ids.split(' - ')
|
||||
drug_list = list[len(list) - len(x.average_cost):]
|
||||
ret_string = ""
|
||||
for y in range(len(x.average_cost)):
|
||||
if (round(x.average_spacing[y], 0) > 1) and (round(x.average_administered[y], 0) > 2.5) and (int(x.value) > 0):
|
||||
string = "<br><b>" + str(drug_list[y]) + "</b><br>On average given " + str(
|
||||
round(x.average_administered[y], 1)) + \
|
||||
" times with a " + str(round(int(x.average_spacing[y]) / 7, 1)) + " weekly interval (" \
|
||||
+ str(round((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1),
|
||||
0)) + " weeks total treatment length)"
|
||||
#"<br>Average annual cost per annum:" + \
|
||||
#str(human_format(
|
||||
# (x.cost / x.value) / (((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1))/ 52)))
|
||||
else:
|
||||
string = "<br><b>" + str(drug_list[y]) + "</b><br>On average given " + str(
|
||||
round(x.average_administered[y], 1)) + \
|
||||
" times with a " + str(round(int(x.average_spacing[y]) / 7, 1)) + " weekly interval (" \
|
||||
+ str(round((int(x.average_spacing[y]) / 7) * round(x.average_administered[y], 1),
|
||||
0)) + " weeks total treatment length)"
|
||||
#"<br>Average annual cost per annum unavailable"
|
||||
|
||||
ret_string += string
|
||||
|
||||
return ret_string
|
||||
|
||||
|
||||
def drug_frequency_average(x):
|
||||
drug_count = x.index.str.contains("drug_").sum()
|
||||
freq = []
|
||||
for d in range(drug_count):
|
||||
if x["freq_" + str(d)] > 1:
|
||||
duration = ((x["end_date_" + str(d)] - x["start_date_" + str(d)]) / np.timedelta64(1, 'D'))
|
||||
if duration > 0:
|
||||
freq_calc = duration / (x["freq_" + str(d)] - 1)
|
||||
else:
|
||||
freq_calc = 0
|
||||
else:
|
||||
freq_calc = 0
|
||||
freq.append(freq_calc)
|
||||
return freq
|
||||
|
||||
|
||||
def cost_pp_pa(x):
|
||||
if x["avg_days"]/ np.timedelta64(1, 'D') > 0:
|
||||
return str(round(x["costpp"] / ((x["avg_days"] / np.timedelta64(1, 'D')) / 365), 2))
|
||||
else:
|
||||
return "N/A"
|
||||
|
||||
|
||||
def generate_graph(
|
||||
df1,
|
||||
start_date=None,
|
||||
end_date=None,
|
||||
last_seen=None,
|
||||
save_dir=None,
|
||||
trustFilter=None,
|
||||
drugFilter=None,
|
||||
directorateFilter=None,
|
||||
title=None,
|
||||
minimum_num_patients=None,
|
||||
*,
|
||||
filters: Optional[AnalysisFilters] = None,
|
||||
paths: Optional[PathConfig] = None,
|
||||
):
|
||||
"""
|
||||
Generate patient pathway icicle chart.
|
||||
|
||||
This function can be called in two ways:
|
||||
1. New style: Pass filters=AnalysisFilters(...) with all parameters encapsulated
|
||||
2. Legacy style: Pass individual parameters (start_date, end_date, etc.)
|
||||
|
||||
If both are provided, the filters object takes precedence.
|
||||
|
||||
Args:
|
||||
df1: DataFrame with processed patient data
|
||||
filters: AnalysisFilters object with all filter parameters (preferred)
|
||||
paths: PathConfig object for file paths (optional, uses default_paths if not provided)
|
||||
|
||||
Legacy parameters (used if filters is None):
|
||||
start_date, end_date, last_seen, save_dir, trustFilter, drugFilter,
|
||||
directorateFilter, title, minimum_num_patients
|
||||
"""
|
||||
# Use PathConfig for file paths
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
# Extract parameters from AnalysisFilters if provided
|
||||
if filters is not None:
|
||||
start_date = filters.start_date
|
||||
end_date = filters.end_date
|
||||
last_seen = filters.last_seen_date
|
||||
save_dir = filters.output_dir
|
||||
trustFilter = filters.trusts
|
||||
drugFilter = filters.drugs
|
||||
directorateFilter = filters.directories
|
||||
title = filters.custom_title
|
||||
minimum_num_patients = filters.minimum_patients
|
||||
|
||||
df1["UPIDTreatment"] = df1["UPID"] + df1["Drug Name"]
|
||||
|
||||
# Get average number of doses count
|
||||
org_codes = pd.read_csv(paths.org_codes_csv, index_col=1)
|
||||
df1["Provider Code"] = df1["Provider Code"].map(org_codes["Name"])
|
||||
#df1.to_csv("./df1.csv", index=False)
|
||||
|
||||
df1 = df1[(df1["Provider Code"].isin(trustFilter)) & (df1["Drug Name"].isin(drugFilter)) & (df1["Directory"].isin(directorateFilter))]
|
||||
|
||||
if len(df1) == 0:
|
||||
logger.warning("No data found for selected filters.")
|
||||
return
|
||||
|
||||
# Find total cost for each patient - Total cost is ~£110Mil, about 30% is unattributable to a patient (no UPID)
|
||||
cost_df = df1[["UPID", "Price Actual"]]
|
||||
total_costs = pd.DataFrame(cost_df.groupby("UPID").sum())
|
||||
total_costs.rename(columns={"Price Actual": "Total cost"}, inplace=True)
|
||||
|
||||
# Series to map directory
|
||||
directory_df = df1[["UPID", "Directory"]]
|
||||
directory_df.drop_duplicates("UPID", inplace=True)
|
||||
directory_df.set_index("UPID", inplace=True)
|
||||
logger.info("Filtering unrelated interventions")
|
||||
|
||||
df_end_dates = drop_duplicate_treatments(df1, False)
|
||||
df1_unique = drop_duplicate_treatments(df1, True)
|
||||
logger.info("Identifying unique patients and interventions used")
|
||||
# Create list of total number of that drug for each patient
|
||||
df_drug_freq = df1.groupby("UPID").agg({"Drug Name": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_drug_cost = df1.groupby("UPID").agg({"Price Actual": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_drug_freq["Price Actual"] = df_drug_freq.index.map(df_drug_cost["Price Actual"])
|
||||
#df_drug_freq["Price Actual"] = df_drug_freq["Price Actual"].map(df_drug_cost)
|
||||
df_drug_freq["Drug Name"] = df_drug_freq["Drug Name"].apply(count_list_values)
|
||||
df_drug_freq["Drug cost total"] = df_drug_freq.apply(lambda x: sum_list_values(x), axis=1)
|
||||
|
||||
|
||||
# Aggregate interventions & dates of interventions into transposed list by UPID
|
||||
df_drugs = df1_unique.groupby("UPID").agg({"Drug Name": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_dates = df1_unique.groupby("UPID").agg({"Intervention Date": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
df_end_dates = df_end_dates.groupby("UPID").agg({"Intervention Date": lambda x: list(x)}).reset_index().set_index("UPID")
|
||||
|
||||
logger.info("Calculating each unique patient's intervention average frequency, cost and duration of each intervention")
|
||||
# The following sh*t show is to unwrap the lists into columns for different drugs, start/end dates, and average
|
||||
# frequency/average total injections of each one
|
||||
df_dates_unwrapped = pd.DataFrame(df_dates["Intervention Date"].values.tolist(), index=df_dates.index).add_prefix(
|
||||
'date_')
|
||||
df_end_dates_unwrapped = pd.DataFrame(df_end_dates["Intervention Date"].values.tolist(), index=df_end_dates.index).add_prefix(
|
||||
'date_end_')
|
||||
df_drugs_unwrapped = pd.DataFrame(df_drugs["Drug Name"].values.tolist(), index=df_drugs.index).add_prefix('drug_')
|
||||
|
||||
df_freq_unwrapped = pd.DataFrame(df_drug_freq["Drug Name"].values.tolist(), index=df_drug_freq.index).add_prefix(
|
||||
'freq_')
|
||||
start_dates = df1[["UPIDTreatment", "Intervention Date"]].sort_values(by=["Intervention Date"], ascending=True,
|
||||
inplace=False,
|
||||
ignore_index=True).drop_duplicates(
|
||||
subset="UPIDTreatment").set_index("UPIDTreatment")
|
||||
end_dates = df1[["UPIDTreatment", "Intervention Date"]].sort_values(by=["Intervention Date"], ascending=False,
|
||||
inplace=False,
|
||||
ignore_index=True).drop_duplicates(
|
||||
subset="UPIDTreatment").set_index("UPIDTreatment")
|
||||
|
||||
|
||||
|
||||
df_drugs_unwrapped["start_dates"] = df_drugs_unwrapped.apply(lambda x: start_date_drug(start_dates, x), axis=1)
|
||||
|
||||
df_ddrugs_unwrapped = pd.DataFrame(df_drugs_unwrapped["start_dates"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'start_date_')
|
||||
df_drugs_unwrapped.drop(["start_dates"], inplace=True, axis=1)
|
||||
df_drugs_unwrapped["end_dates"] = df_drugs_unwrapped.apply(lambda x: start_date_drug(end_dates, x), axis=1)
|
||||
df_dddrugs_unwrapped = pd.DataFrame(df_drugs_unwrapped["end_dates"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'end_date_')
|
||||
|
||||
df_drugs_unwrapped.drop(["end_dates"], inplace=True, axis=1)
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_ddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_dddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_dddddrugs_unwrapped = pd.DataFrame(df_drug_freq["Drug Name"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'freq_')
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_dddddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_drugs_unwrapped["frequency"] = df_drugs_unwrapped.apply(lambda x: drug_frequency_average(x), axis=1)
|
||||
|
||||
df_ddddddrugs_unwrapped = pd.DataFrame(df_drugs_unwrapped["frequency"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix(
|
||||
'spacing_')
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_ddddddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_dddddddrugs_unwrapped = pd.DataFrame(df_drug_freq["Drug cost total"].values.tolist(),
|
||||
index=df_drugs_unwrapped.index).add_prefix('total_cost_drug_')
|
||||
df_drugs_unwrapped = pd.merge(df_drugs_unwrapped, df_dddddddrugs_unwrapped, left_index=True, right_index=True)
|
||||
df_drugs_unwrapped.drop(["frequency"], inplace=True, axis=1)
|
||||
|
||||
# Insert first & last date seen into df (need to add last date seen)
|
||||
df_drugs_unwrapped.insert(0, "First seen", df_dates_unwrapped.min(axis=1))
|
||||
df_drugs_unwrapped.insert(1, "Last seen", df_end_dates_unwrapped.max(axis=1))
|
||||
|
||||
# Merge info from activity data with grouped info, and total cost info
|
||||
patient_info = df1.drop_duplicates(subset="UPID", keep="first").set_index("UPID")
|
||||
patient_info = pd.merge(patient_info, df_drugs_unwrapped, left_index=True, right_index=True)
|
||||
patient_info = pd.merge(patient_info, df_freq_unwrapped, left_index=True, right_index=True)
|
||||
patient_info = pd.merge(patient_info, total_costs, left_index=True, right_index=True)
|
||||
|
||||
#patient_info.to_csv("patient_info.csv", index=False)
|
||||
|
||||
# Filter initiation based on years provided
|
||||
patient_info = patient_info[(patient_info['First seen'] >= str(start_date)) & (
|
||||
patient_info['First seen'] < str(end_date))]
|
||||
if title == "":
|
||||
title = "Patients initiated from " + str(start_date) + " to " + str(end_date)
|
||||
|
||||
# Filter last seen based on date provided
|
||||
patient_info = patient_info[patient_info['Last seen'] > str(last_seen)]
|
||||
|
||||
# Remove patients with 0 drug, by filling blanks with NaN & dropping rows
|
||||
patient_info.drug_0.replace('N/A', np.nan, inplace=True)
|
||||
patient_info.dropna(subset=['drug_0'], inplace=True)
|
||||
|
||||
# Calculate duation of treatment
|
||||
patient_info['Days treated'] = patient_info["Last seen"] - patient_info["First seen"]
|
||||
date_df = patient_info[["First seen", "Last seen", 'Days treated']]
|
||||
|
||||
# Create df for ice chart with hierarchy of plot
|
||||
number_of_drugs = np.count_nonzero(patient_info.columns.str.startswith('drug_'))
|
||||
final_drug_index = patient_info.columns.to_list().index("drug_" + str(number_of_drugs - 1))
|
||||
|
||||
upid_drugs_df = patient_info.iloc[:, (final_drug_index - number_of_drugs + 1):final_drug_index + 1]
|
||||
|
||||
upid_drugs_df.insert(0, "Trust", upid_drugs_df.index.str[:3])
|
||||
upid_drugs_df.insert(1, "Directory", upid_drugs_df.index)
|
||||
|
||||
upid_drugs_df["Trust"] = upid_drugs_df["Trust"].map(org_codes["Name"])
|
||||
upid_drugs_df["Directory"] = upid_drugs_df["Directory"].map(directory_df["Directory"])
|
||||
|
||||
l_df = pd.DataFrame()
|
||||
ice_df2 = pd.DataFrame()
|
||||
ice_df = pd.DataFrame()
|
||||
|
||||
upid_drugs_df["value"] = upid_drugs_df.apply(lambda x: row_function(x), axis=1)
|
||||
# Merge in date info
|
||||
upid_drugs_df = pd.merge(upid_drugs_df, date_df, left_index=True, right_index=True)
|
||||
|
||||
upid_drugs_df["ids"] = upid_drugs_df["value"].str.split(',').str[2]
|
||||
avg_treatment_dfs = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)["Days treated"].mean()).set_index("ids")
|
||||
value_dfs = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False).size()).reset_index()
|
||||
first_seen_treatment_dfs = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)["First seen"].min()).set_index(
|
||||
"ids")
|
||||
last_seen_treatment_dfs = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)["Last seen"].max()).set_index(
|
||||
"ids")
|
||||
|
||||
# Calculate total cost for parents
|
||||
upid_drugs_df["Cost"] = upid_drugs_df.index.map(total_costs["Total cost"])
|
||||
cost_dfs = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)['Cost'].sum()).set_index("value", drop=True)
|
||||
|
||||
# Calculate average dosing for each drug
|
||||
upid_drugs_df = pd.merge(upid_drugs_df, df_drugs_unwrapped, left_index=True, right_index=True)
|
||||
# frequency_dfs = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)['Cost'].sum()).set_index("value", drop=True)
|
||||
|
||||
# Calculate average spacing between drugs
|
||||
spacing_average = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if 'spacing_' in col]].mean()).set_index(
|
||||
"value", drop=True)
|
||||
spacing_average = spacing_average.round()
|
||||
spacing_average['combined'] = spacing_average.values.tolist()
|
||||
spacing_average["ids"] = spacing_average.index
|
||||
spacing_average["ids"] = spacing_average["ids"].str.split(',').str[2]
|
||||
spacing_average.set_index("ids", inplace=True)
|
||||
|
||||
# Calculate average cost for each drug
|
||||
cost_average = pd.DataFrame(upid_drugs_df.groupby("value", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if 'total_cost_drug_' in col]].mean()).set_index(
|
||||
"value", drop=True)
|
||||
cost_average = cost_average.round(2)
|
||||
cost_average['combined'] = cost_average.values.tolist()
|
||||
cost_average["ids"] = cost_average.index
|
||||
cost_average["ids"] = cost_average["ids"].str.split(',').str[2]
|
||||
cost_average.set_index("ids", inplace=True)
|
||||
|
||||
|
||||
# Calculate average number of doses
|
||||
freq_average = pd.DataFrame(upid_drugs_df.groupby("ids", as_index=False)[
|
||||
[col for col in upid_drugs_df.columns if 'freq_' in col]].mean()).set_index("ids",
|
||||
drop=True)
|
||||
# freq_average = freq_average.round()
|
||||
freq_average['combined'] = freq_average.values.tolist()
|
||||
|
||||
# Remove negative totals from "Cost" column
|
||||
num = cost_dfs._get_numeric_data()
|
||||
num[num < 0] = 0
|
||||
|
||||
value_dfs["Cost"] = value_dfs["value"].map(cost_dfs["Cost"])
|
||||
|
||||
ice_df[['parents', 'labels', 'ids']] = value_dfs["value"].str.split(',', expand=True)
|
||||
# ice_df["index"] = ice_df.ids
|
||||
# ice_df.set_index("index", inplace=True)
|
||||
|
||||
ice_df["average_administered"] = ice_df["ids"].map(freq_average["combined"])
|
||||
ice_df["cost"] = value_dfs["Cost"]
|
||||
ice_df["value"] = value_dfs["size"]
|
||||
|
||||
ice_df["average_cost"] = ice_df["ids"].map(cost_average["combined"])
|
||||
ice_df["average_cost"] = ice_df["average_cost"].apply(remove_nan_string)
|
||||
|
||||
ice_df["average_spacing"] = ice_df["ids"].map(spacing_average["combined"])
|
||||
ice_df["average_spacing"] = ice_df["average_spacing"].apply(remove_nan_string)
|
||||
ice_df["average_spacing"] = ice_df.apply(lambda x: list_to_string(x), axis=1)
|
||||
ice_df["average_spacing"] = ice_df["average_spacing"].str.replace("nan", "N/A")
|
||||
|
||||
|
||||
logger.info("Building graph dataframe structure.")
|
||||
# Add very top level of Trust
|
||||
new_row = pd.DataFrame({'parents': '', 'ids': "N&WICS", 'labels': 'N&WICS', 'value': 0, "cost": 0}, index=[0])
|
||||
ice_df = pd.concat(objs=[ice_df, new_row], ignore_index=True, axis=0)
|
||||
|
||||
# need to add parents as blocks...
|
||||
l3 = [x for x in ice_df.parents.unique() if x not in ice_df.ids]
|
||||
while len(l3) > 1:
|
||||
for l in l3:
|
||||
z = l.rfind("-")
|
||||
if z > 0:
|
||||
l_dict = {"parents": l[:z - 1], "ids": l, "value": 0, "labels": l[z + 2:], "cost": 0}
|
||||
l_df = pd.concat([l_df, pd.DataFrame(l_dict, index=[0])], ignore_index=True)
|
||||
ice_df2 = pd.concat([ice_df, l_df], ignore_index=True)
|
||||
l3 = [x for x in ice_df2.parents.unique() if x not in ice_df2.ids.unique()]
|
||||
ice_df = ice_df2.drop_duplicates("ids")
|
||||
|
||||
ice_df["level"] = ice_df["ids"].str.count('-')
|
||||
ice_df = ice_df[~ice_df['labels'].isin(["COST", "CHARGE", "N/A"])]
|
||||
ice_df.sort_values(by=["level"], ascending=False, inplace=True, ignore_index=True)
|
||||
|
||||
for index, row in ice_df.iterrows():
|
||||
lookup_index = ice_df.index[ice_df['ids'] == row['parents']]
|
||||
ice_df.loc[lookup_index, 'value'] = ice_df.loc[lookup_index, "value"] + ice_df.loc[index, "value"]
|
||||
ice_df.loc[lookup_index, 'cost'] = ice_df.loc[lookup_index, "cost"] + ice_df.loc[index, 'cost']
|
||||
|
||||
# Sum of parent values to create denominator for percentage - FOR PATIENT NUMBER COLOUR GRADING
|
||||
colour_df = pd.DataFrame(ice_df.groupby(["parents"])["value"].sum())
|
||||
ice_df['colour'] = ice_df["parents"].map(colour_df["value"])
|
||||
ice_df['colour'] = ice_df['value']/ice_df['colour']
|
||||
|
||||
# Sum of parent values to create denominator for percentage - FOR COST COLOUR GRADING
|
||||
#colour_df = pd.DataFrame(ice_df.groupby(["parents"])["cost"].sum())
|
||||
#ice_df['colour'] = ice_df["parents"].map(colour_df["cost"])
|
||||
#ice_df['colour'] = ice_df['cost'] / ice_df['colour']
|
||||
|
||||
|
||||
ice_df['costpp'] = ice_df['cost'] / ice_df['value']
|
||||
# Treatment length info
|
||||
ice_df['avg_days'] = ice_df["ids"].map(avg_treatment_dfs["Days treated"])
|
||||
ice_df['First seen'] = ice_df["ids"].map(first_seen_treatment_dfs["First seen"])
|
||||
ice_df['Last seen'] = ice_df["ids"].map(last_seen_treatment_dfs["Last seen"])
|
||||
|
||||
ice_df["dates"] = ice_df.apply(lambda x: min_max_treatment_dates(ice_df, x), axis=1)
|
||||
ice_df[['First seen (Parent)', 'Last seen (Parent)']] = ice_df["dates"].str.split(',', expand=True)
|
||||
|
||||
# Sort labels to be alphabetical
|
||||
# ice_df.sort_values(by=["labels"], ascending=True, inplace=True, ignore_index=True)
|
||||
ice_df['First seen'] = pd.to_datetime(ice_df['First seen'])
|
||||
ice_df['Last seen'] = pd.to_datetime(ice_df['Last seen'])
|
||||
ice_df["cost_pp_pa"] = ice_df.apply(lambda x: cost_pp_pa(x), axis=1)
|
||||
|
||||
# Filter out rows where value is less than minimum number of patients
|
||||
ice_df = ice_df[ice_df['value'] >= minimum_num_patients]
|
||||
|
||||
logger.info("Generating graph.")
|
||||
|
||||
figure(ice_df, title, save_dir)
|
||||
return
|
||||
|
||||
|
||||
def figure(ice_df4, dir_string, save_dir):
|
||||
"""
|
||||
Create and display icicle figure (legacy interface).
|
||||
|
||||
This function delegates to visualization.plotly_generator.figure_legacy()
|
||||
for backward compatibility.
|
||||
|
||||
Args:
|
||||
ice_df4: DataFrame with chart data
|
||||
dir_string: Title string (used for filename and chart title)
|
||||
save_dir: Directory to save the HTML file
|
||||
"""
|
||||
_figure_legacy(ice_df4, dir_string, save_dir)
|
||||
return
|
||||
|
||||
|
||||
# fig = go.Figure(go.Icicle(
|
||||
# labels=ice_df4.labels,
|
||||
# ids=ice_df4.ids,
|
||||
# # count="branches",
|
||||
# parents=ice_df4.parents,
|
||||
# customdata=np.stack((ice_df4.value, ice_df4.colour, ice_df4.cost, ice_df4.costpp, first_seen, last_seen,
|
||||
# first_seen_parent, last_seen_parent, average_spacing, ice_df4.cost_pp_pa), axis=1),
|
||||
# values=ice_df4.value,
|
||||
# branchvalues="total",
|
||||
# marker=dict(
|
||||
# colors=ice_df4.colour,
|
||||
# colorscale='Viridis'),
|
||||
# maxdepth=3,
|
||||
# texttemplate='<b>%{label}</b> '
|
||||
# '<br><b>Total patients:</b> %{customdata[0]} - %{customdata[1]:.3p} of patients in level'
|
||||
# '<br><b>Total cost:</b> £%{customdata[2]:.3~s}'
|
||||
# '<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}'
|
||||
# '<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}',
|
||||
# hovertemplate='<b>%{label}</b>'
|
||||
# '<br><b>Total patients:</b> %{customdata[0]} - %{customdata[1]:.3p} of patients in level'
|
||||
# '<br><b>Total cost:</b> £%{customdata[2]:.3~s}'
|
||||
# '<br><b>Average cost per patient:</b> £%{customdata[3]:.3~s}'
|
||||
# '<br><b>Average cost per patient per annum:</b> £%{customdata[9]:.3~s}'
|
||||
# '<br><b>First seen:</b> %{customdata[4]}'
|
||||
# '<br><b>Last seen (including further treatments):</b> %{customdata[7]}'
|
||||
# '<br><b>Average treatment duration:</b>'
|
||||
# '%{customdata[8]}'
|
||||
# '<extra></extra>',
|
||||
# ))
|
||||
#
|
||||
#import os
|
||||
#def main():
|
||||
# input = "ice_df.csv"
|
||||
# save_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
# dir = "debugging"
|
||||
# ice_df4 = pd.read_csv(input)
|
||||
#
|
||||
# ice_df4['First seen'] = pd.to_datetime(ice_df4['First seen'])
|
||||
# ice_df4['avg_days'] = pd.to_timedelta(ice_df4['avg_days'])
|
||||
# ice_df4['Last seen'] = pd.to_datetime(ice_df4['Last seen'])
|
||||
# figure(ice_df4, dir, save_dir)
|
||||
#
|
||||
#if __name__ == "__main__":
|
||||
# main()
|
||||
|
||||
|
||||
def generate_graph_v2(
|
||||
df: pd.DataFrame,
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
last_seen_date: str,
|
||||
save_dir: str,
|
||||
trust_filter: list[str],
|
||||
drug_filter: list[str],
|
||||
directory_filter: list[str],
|
||||
minimum_num_patients: int = 0,
|
||||
title: str = "",
|
||||
paths: Optional[PathConfig] = None,
|
||||
) -> Optional[go.Figure]:
|
||||
"""
|
||||
Generate patient pathway icicle chart using refactored pipeline.
|
||||
|
||||
This is the modern API that uses the refactored analysis functions.
|
||||
It provides cleaner parameter names and returns the figure instead of
|
||||
automatically opening it in a browser.
|
||||
|
||||
Args:
|
||||
df: DataFrame with processed patient intervention data
|
||||
start_date: Start date for patient initiation filter (YYYY-MM-DD)
|
||||
end_date: End date for patient initiation filter (YYYY-MM-DD)
|
||||
last_seen_date: Filter for patients last seen after this date
|
||||
save_dir: Directory to save the HTML file
|
||||
trust_filter: List of trust names to include
|
||||
drug_filter: List of drug names to include
|
||||
directory_filter: List of directories to include
|
||||
minimum_num_patients: Minimum number of patients to include a pathway
|
||||
title: Chart title (auto-generated from dates if empty)
|
||||
paths: PathConfig for file paths (uses default if None)
|
||||
|
||||
Returns:
|
||||
Plotly Figure object, or None if no data
|
||||
"""
|
||||
if paths is None:
|
||||
paths = default_paths
|
||||
|
||||
ice_df, final_title = _generate_icicle_chart(
|
||||
df=df,
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
last_seen_date=last_seen_date,
|
||||
trust_filter=trust_filter,
|
||||
drug_filter=drug_filter,
|
||||
directory_filter=directory_filter,
|
||||
minimum_num_patients=minimum_num_patients,
|
||||
title=title,
|
||||
paths=paths,
|
||||
)
|
||||
|
||||
if ice_df is None or len(ice_df) == 0:
|
||||
return None
|
||||
|
||||
fig = create_icicle_figure(ice_df, final_title)
|
||||
|
||||
if save_dir:
|
||||
fig.write_html(f"{save_dir}/{final_title}.html")
|
||||
logger.info(f"Success! File saved to {save_dir}/{final_title}.html")
|
||||
|
||||
return fig
|
||||
|
||||
|
||||
def create_icicle_figure(ice_df: pd.DataFrame, title: str) -> go.Figure:
|
||||
"""
|
||||
Create Plotly icicle figure from prepared DataFrame.
|
||||
|
||||
This function delegates to visualization.plotly_generator.create_icicle_figure()
|
||||
for the actual figure generation.
|
||||
|
||||
Args:
|
||||
ice_df: DataFrame with parents, ids, labels, value, colour etc.
|
||||
title: Chart title
|
||||
|
||||
Returns:
|
||||
Plotly Figure object
|
||||
"""
|
||||
return _create_icicle_figure(ice_df, title)
|
||||
@@ -1,77 +0,0 @@
|
||||
|
||||
|
||||
I'll start by reading the required files in order.
|
||||
Reading progress.txt, IMPLEMENTATION_PLAN.md, and guardrails.md. The iteration log in progress.txt is empty ÔÇö this is the **first iteration**. All tasks in IMPLEMENTATION_PLAN.md are `[ ]` (not started).
|
||||
|
||||
Let me check recent commits and read the key source files I'll be modifying.
|
||||
Good. I've read all the key files. Let me assess the situation:
|
||||
|
||||
**This is Iteration 1.** All tasks are `[ ]`. No previous work has been done.
|
||||
|
||||
**Task selection reasoning:**
|
||||
|
||||
Looking at the IMPLEMENTATION_PLAN.md phases:
|
||||
- **Phase 1** (1.1 and 1.2) ÔÇö foundational: update the Snowflake query + build drug mapping
|
||||
- **Phase 2** (2.1 and 2.2) ÔÇö depends on Phase 1 outputs
|
||||
- **Phase 3** (3.1 and 3.2) ÔÇö depends on Phase 2
|
||||
- **Phase 4** (4.1-4.3) ÔÇö depends on Phase 3
|
||||
|
||||
The natural starting point is **Task 1.1** (update `get_patient_indication_groups()`) AND **Task 1.2** (build drug-to-Search_Term mapping). These are independent of each other ÔÇö 1.1 modifies the Snowflake query, 1.2 loads from CSV. However, per the "one task per iteration" rule, I'll start with **Task 1.2** because it's self-contained, requires no Snowflake access to test, and can be validated locally. Task 1.1 modifies the Snowflake query which can't be tested without a live connection.
|
||||
|
||||
Let me mark Task 1.2 as in-progress and implement it.
|
||||
Now let me mark Task 1.2 as in-progress in IMPLEMENTATION_PLAN.md.
|
||||
Now I'll implement the two functions. Looking at the DimSearchTerm.csv structure:
|
||||
- `Search_Term` ÔÇö clinical condition
|
||||
- `CleanedDrugName` ÔÇö pipe-separated drug fragments
|
||||
- `PrimaryDirectorate` ÔÇö the directorate
|
||||
|
||||
Note: `diabetes` appears twice (DIABETIC MEDICINE and OPHTHALMOLOGY directorates). For indication charts, we use Search_Term not directorate, so the drug fragments from both rows should be combined.
|
||||
|
||||
Let me add the functions to `diagnosis_lookup.py`. I'll add them right before the `CLUSTER_MAPPING_SQL` constant since they're logically in the "drug mapping" category.
|
||||
Now I need to add the new functions to the `__all__` export list.
|
||||
Now let me validate the code. First a syntax check, then import check, then a functionality test.
|
||||
Syntax check passed. Now import check.
|
||||
Now let me run a functionality test to verify the mapping loads correctly and drug matching works.
|
||||
All tests pass:
|
||||
|
||||
- **164 Search_Terms** loaded (close to expected ~165 ÔÇö the slightly lower count is because `diabetes` appears twice with different directorates but is correctly combined into one entry)
|
||||
- **346 drug fragments** loaded
|
||||
- **ADALIMUMAB** correctly matches 7 indications (rheumatoid arthritis, plaque psoriasis, etc.)
|
||||
- **OMALIZUMAB** matches 4 asthma/urticaria-related indications
|
||||
- **Substring matching works**: "PEGYLATED LIPOSOMAL DOXORUBICIN" matches via the "PEGYLATED" fragment
|
||||
- **Drug name with dosage works**: "ADALIMUMAB 40MG" matches the same 7 indications as plain "ADALIMUMAB"
|
||||
- **Duplicate Search_Term handling works**: `diabetes` drug fragments combined from both CSV rows
|
||||
- **Reverse lookup works**: `ADALIMUMAB` fragment correctly maps back to all 7 Search_Terms
|
||||
|
||||
Note the PEGYLATED match includes hepatitis c and acute myeloid leukaemia ÔÇö these broad fragments will be constrained by the GP diagnosis requirement in Phase 2 (a patient must have GP dx for the condition AND the drug must be listed for that condition). This is exactly as designed.
|
||||
|
||||
Let me also verify the IMPLEMENTATION_PLAN.md subtasks for 1.2:
|
||||
|
||||
1. `load_drug_indication_mapping()` ÔÇö Done, builds both dicts
|
||||
2. `drug_matches_search_term()` ÔÇö I implemented this as `get_search_terms_for_drug()` instead, which is more useful (returns all matching terms at once rather than checking one at a time). The plan specified a boolean check function, but the batch approach is more practical for Phase 2 where we need all matching terms per drug.
|
||||
3. Verify: ADALIMUMAB matches "axial spondyloarthritis" ÔÇö Confirmed. OMALIZUMAB matches "asthma" ÔÇö Confirmed.
|
||||
|
||||
Now let me mark the subtasks complete and update IMPLEMENTATION_PLAN.md.
|
||||
Now commit, then update progress.txt.
|
||||
Good, committed as `0779df7`. Now update progress.txt.
|
||||
**Iteration 1 complete.**
|
||||
|
||||
**Summary:**
|
||||
- **Task 1.2** ÔÇö Build drug-to-Search_Term lookup from DimSearchTerm.csv ÔÇö DONE
|
||||
- Added `load_drug_indication_mapping()` and `get_search_terms_for_drug()` to `diagnosis_lookup.py`
|
||||
- 164 Search_Terms, 346 drug fragments loaded
|
||||
- Substring matching verified: ADALIMUMAB  7 indications, OMALIZUMAB  4, PEGYLATED LIPOSOMAL DOXORUBICIN  4
|
||||
- Committed as `0779df7`
|
||||
|
||||
**Next iteration should:** Work on Task 1.1 (update Snowflake query to return ALL matches with code_frequency) or Task 2.1 (assign_drug_indications function) if Snowflake isn't available for testing.
|
||||
**Iteration 1 complete.**
|
||||
|
||||
**Summary:**
|
||||
- **Task 1.2** ÔÇö Build drug-to-Search_Term lookup from DimSearchTerm.csv ÔÇö DONE
|
||||
- Added `load_drug_indication_mapping()` and `get_search_terms_for_drug()` to `diagnosis_lookup.py`
|
||||
- 164 Search_Terms, 346 drug fragments loaded
|
||||
- Substring matching verified: ADALIMUMAB  7 indications, OMALIZUMAB  4, PEGYLATED LIPOSOMAL DOXORUBICIN  4
|
||||
- Committed as `0779df7`
|
||||
|
||||
**Next iteration should:** Work on Task 1.1 (update Snowflake query to return ALL matches with code_frequency) or Task 2.1 (assign_drug_indications function) if Snowflake isn't available for testing.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,99 +0,0 @@
|
||||
|
||||
|
||||
I'll start by reading the required files in order to understand the current state of the project.
|
||||
Now let me check recent git history and read the current state of the key file I'll need to modify.
|
||||
Good. I've read all the required files. Let me assess the current state:
|
||||
|
||||
**Completed tasks:**
|
||||
- Task 1.3: `load_drug_indication_mapping()` and `get_search_terms_for_drug()` ÔÇö done in Iteration 1
|
||||
|
||||
**Remaining tasks (Phase 1):**
|
||||
- Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
|
||||
- Task 1.2: Merge asthma Search_Terms in CLUSTER_MAPPING_SQL and `load_drug_indication_mapping()`
|
||||
|
||||
**Task selection reasoning:**
|
||||
Progress.txt recommends either Task 1.2 (asthma merge ÔÇö self-contained, testable locally) or Task 1.1 (Snowflake query update). I'll choose **Task 1.2** because:
|
||||
1. It's fully self-contained and testable without Snowflake connectivity
|
||||
2. It's a prerequisite for getting clean data from both the Snowflake query AND the drug mapping
|
||||
3. It's simpler and lower risk ÔÇö a good foundation before modifying the Snowflake query
|
||||
|
||||
Let me start by understanding what needs to change. I need to:
|
||||
1. Update `CLUSTER_MAPPING_SQL` to merge 3 asthma Search_Terms into one `"asthma"` entry
|
||||
2. Update `load_drug_indication_mapping()` to merge asthma variants when loading DimSearchTerm.csv
|
||||
3. Verify the changes work correctly
|
||||
|
||||
First, let me check DimSearchTerm.csv to see the asthma-related rows.
|
||||
Let me check the asthma-related entries in DimSearchTerm.csv.
|
||||
Now let me check the current CLUSTER_MAPPING_SQL entries for asthma. I can see from the file I already read:
|
||||
- `('allergic asthma', 'AST_COD')` ÔÇö line 1201
|
||||
- `('asthma', 'eFI2_Asthma')` ÔÇö line 1210
|
||||
- `('severe persistent allergic asthma', 'SEVAST_COD')` ÔÇö line 1316
|
||||
- `('urticaria', 'XSAL_COD')` ÔÇö line 1337
|
||||
|
||||
The plan for CLUSTER_MAPPING_SQL is to map all 3 asthma Cluster_IDs to `Search_Term = 'asthma'`:
|
||||
- `('asthma', 'AST_COD')` ÔÇö was allergic asthma
|
||||
- `('asthma', 'eFI2_Asthma')` ÔÇö stays the same
|
||||
- `('asthma', 'SEVAST_COD')` ÔÇö was severe persistent allergic asthma
|
||||
|
||||
`urticaria` stays as-is.
|
||||
Now updating CLUSTER_MAPPING_SQL. I need to change `'allergic asthma'`  `'asthma'` and `'severe persistent allergic asthma'`  `'asthma'`. This means the CTE will have multiple rows mapping to `Search_Term = 'asthma'` with different Cluster_IDs (AST_COD, eFI2_Asthma, SEVAST_COD). That's fine  the UNION in AllIndicationCodes will just give us more SNOMED codes mapping to `'asthma'`, which is what we want.
|
||||
Good. Now the CLUSTER_MAPPING_SQL has 3 rows for `'asthma'` with Cluster_IDs: AST_COD, eFI2_Asthma, SEVAST_COD. The GP lookup will return `Search_Term = 'asthma'` for any of these clusters. `urticaria` with `XSAL_COD` stays separate.
|
||||
|
||||
Now I need to update `load_drug_indication_mapping()` to merge the asthma variants. When loading DimSearchTerm.csv, rows with Search_Term = "allergic asthma" or "severe persistent allergic asthma" should have their drug fragments merged under a single "asthma" key.
|
||||
I need to add a Search_Term normalization step in `load_drug_indication_mapping()`. The approach: define a mapping of asthma variant names  "asthma", and apply it when reading each row. This keeps the merge logic contained in one place.
|
||||
Now I need to update the `__all__` exports to include the new constant, and update the log message to note the merge.
|
||||
|
||||
Let me also update the `__all__` list to include `SEARCH_TERM_MERGE_MAP` for potential use by other modules.
|
||||
Now let me validate the changes.
|
||||
Syntax check passed. Now let me test the import and verify the asthma merge works correctly.
|
||||
All validations pass:
|
||||
|
||||
1. **Asthma merge works**: `asthma` now has the combined drug list: OMALIZUMAB, BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, RESLIZUMAB (from all 3 original rows)
|
||||
2. **Variants removed**: `allergic asthma` and `severe persistent allergic asthma` no longer exist as separate entries
|
||||
3. **Urticaria separate**: `urticaria`  `['OMALIZUMAB']`  correctly kept separate
|
||||
4. **OMALIZUMAB maps correctly**: to both `asthma` and `urticaria` (not just one)
|
||||
5. **Drug matching still works**: ADALIMUMAB maps to 7 indications as expected
|
||||
6. **Count changes**: 162 Search_Terms (was 164 ÔÇö 2 fewer because 3 asthma entries merged into 1)
|
||||
7. **Fragment count unchanged**: 346 (no fragments lost, just redistributed)
|
||||
|
||||
Now let me also verify the CLUSTER_MAPPING_SQL changes by checking the specific lines.
|
||||
CLUSTER_MAPPING_SQL now has 3 entries for `'asthma'`:
|
||||
- `('asthma', 'AST_COD')` ÔÇö was `allergic asthma`
|
||||
- `('asthma', 'eFI2_Asthma')` ÔÇö unchanged
|
||||
- `('asthma', 'SEVAST_COD')` ÔÇö was `severe persistent allergic asthma`
|
||||
|
||||
And `('urticaria', 'XSAL_COD')` is untouched.
|
||||
|
||||
Now let me mark the task in the IMPLEMENTATION_PLAN.md.
|
||||
Now let me commit the changes and update progress.txt.
|
||||
I see many modified files from prior work. Let me check what my specific changes are in the files I modified.
|
||||
The diff is clean and precise. Now let me update progress.txt and then commit both files together.
|
||||
Now let me commit the changes.
|
||||
Now let me update the commit hash in progress.txt.
|
||||
**Iteration 2 complete.** Here's the summary:
|
||||
|
||||
**Task 1.2: Merge related asthma Search_Terms** ÔÇö DONE
|
||||
|
||||
Changes made:
|
||||
1. **CLUSTER_MAPPING_SQL**: `'allergic asthma'`  `'asthma'` (AST_COD) and `'severe persistent allergic asthma'`  `'asthma'` (SEVAST_COD). Now 3 Cluster_IDs all map to `'asthma'`.
|
||||
2. **SEARCH_TERM_MERGE_MAP**: New constant mapping asthma variants  canonical `"asthma"`. Reusable by other modules.
|
||||
3. **load_drug_indication_mapping()**: Applies merge map at CSV load time. Combined asthma drug list: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB.
|
||||
4. **urticaria** stays separate (XSAL_COD, OMALIZUMAB only).
|
||||
|
||||
All validation passed. Commits: `b0a8a9d`, `4fed0e5`.
|
||||
|
||||
**Next iteration**: Task 1.1 (update Snowflake query to return ALL matches with code_frequency) or Task 2.1 (assign_drug_indications function).
|
||||
**Iteration 2 complete.** Here's the summary:
|
||||
|
||||
**Task 1.2: Merge related asthma Search_Terms** ÔÇö DONE
|
||||
|
||||
Changes made:
|
||||
1. **CLUSTER_MAPPING_SQL**: `'allergic asthma'`  `'asthma'` (AST_COD) and `'severe persistent allergic asthma'`  `'asthma'` (SEVAST_COD). Now 3 Cluster_IDs all map to `'asthma'`.
|
||||
2. **SEARCH_TERM_MERGE_MAP**: New constant mapping asthma variants  canonical `"asthma"`. Reusable by other modules.
|
||||
3. **load_drug_indication_mapping()**: Applies merge map at CSV load time. Combined asthma drug list: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB.
|
||||
4. **urticaria** stays separate (XSAL_COD, OMALIZUMAB only).
|
||||
|
||||
All validation passed. Commits: `b0a8a9d`, `4fed0e5`.
|
||||
|
||||
**Next iteration**: Task 1.1 (update Snowflake query to return ALL matches with code_frequency) or Task 2.1 (assign_drug_indications function).
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,100 +0,0 @@
|
||||
|
||||
|
||||
I'll start by reading the required files in order, as specified by the Ralph loop instructions.
|
||||
Good, I've read the three critical files. Now let me check recent commits and understand the current state.
|
||||
|
||||
Reading progress.txt: Last iteration (2) completed Task 1.2 (merge asthma Search_Terms). The recommended next task is **Task 1.1** (update `get_patient_indication_groups()` to return ALL matches with code_frequency) or **Task 2.1** (create `assign_drug_indications()` function).
|
||||
|
||||
Reading IMPLEMENTATION_PLAN.md: Tasks 1.2 and 1.3 are `[x]`. Task 1.1 is `[ ]` (ready). Tasks 2.1, 2.2, 3.1, 3.2, 4.1-4.3 are all `[ ]`.
|
||||
|
||||
Let me check recent commits and read the current diagnosis_lookup.py to understand the code I need to modify.
|
||||
Now I have full context. Let me make my task selection decision.
|
||||
|
||||
**Task Selection Reasoning:**
|
||||
|
||||
Looking at the remaining tasks:
|
||||
- **Task 1.1** (Update `get_patient_indication_groups()` to return ALL matches with code_frequency) ÔÇö This is the next logical dependency. Tasks 2.1 and 2.2 need the GP matches data to be in the right format (multiple rows per patient with code_frequency). The previous iteration explicitly recommended this as the next task.
|
||||
- **Task 2.1** (Create `assign_drug_indications()`) ÔÇö Could be built with mock data, but it's cleaner to do 1.1 first so 2.1 can use the real return type.
|
||||
|
||||
**Decision: I'll work on Task 1.1** ÔÇö Update `get_patient_indication_groups()` to return ALL matches with `code_frequency` instead of just the most recent match per patient.
|
||||
|
||||
The changes needed:
|
||||
1. Add `earliest_hcd_date` parameter
|
||||
2. Replace `QUALIFY ROW_NUMBER()` with `GROUP BY + COUNT(*)`
|
||||
3. Change return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`
|
||||
4. Update empty DataFrame columns to match
|
||||
5. Update docstring
|
||||
|
||||
Let me start implementing.
|
||||
First, let me mark Task 1.1 as in-progress in IMPLEMENTATION_PLAN.md.
|
||||
Now implementing the changes to `get_patient_indication_groups()`. The key changes are:
|
||||
|
||||
1. **Add `earliest_hcd_date` parameter** ÔÇö restricts GP codes to the HCD data window
|
||||
2. **Replace `QUALIFY ROW_NUMBER()` with `GROUP BY + COUNT(*)`** ÔÇö returns ALL matches with frequency
|
||||
3. **Update return columns** ÔÇö `code_frequency` instead of `EventDateTime`
|
||||
4. **Update empty DataFrame columns** to match new return type
|
||||
5. **Update docstring**
|
||||
Good, the core function is updated. Let me verify the changes compile correctly.
|
||||
Running syntax and import checks to validate the code.
|
||||
Syntax and imports both pass. Let me also verify the function signature is correct by checking the inspect output.
|
||||
The function signature is correct. Now let me check if any callers of this function need to be updated. The function has a new `earliest_hcd_date` parameter with `None` default, so existing callers remain backward-compatible. But I should verify the callers handle the new return format (multiple rows per patient, `code_frequency` instead of `EventDateTime`).
|
||||
The main caller is in `cli/refresh_pathways.py`. Let me check how it uses the result to understand if the caller needs updating (though per the plan, caller changes are Task 3.1, not 1.1).
|
||||
I can see the caller at line 424-428 does `dict(zip(gp_matches_df['PatientPseudonym'], gp_matches_df['Search_Term']))`. This creates a 1:1 mapping ÔÇö with the old query this was fine (one row per patient), but with the new query returning multiple rows per patient, `dict(zip(...))` would only keep the last match.
|
||||
|
||||
However, this is the **current** caller code and updating it is **Task 3.1** (Pipeline Integration), not Task 1.1. Task 1.1 only covers updating the function itself. The caller changes will be done in a future iteration when we integrate `assign_drug_indications()`.
|
||||
|
||||
The key thing is that the function change is backward-compatible ÔÇö the new parameter `earliest_hcd_date` defaults to `None`, so the existing caller still works. The return format change (multiple rows per patient, `code_frequency` instead of `EventDateTime`) means the caller needs updating, but that's explicitly Task 3.1.
|
||||
|
||||
Now let me also verify the empty DataFrame columns match the new return type throughout.
|
||||
All validation passes. Let me update the IMPLEMENTATION_PLAN.md to mark Task 1.1 subtasks complete.
|
||||
The last subtask "Verify: Query returns more rows than before" requires a live Snowflake connection ÔÇö that will be verified during Phase 3/4 integration testing. The code change itself is complete and validated locally. Let me note this in the implementation plan.
|
||||
Now let me update progress.txt with the iteration results.
|
||||
There are two instances of "### Blocked items: - None". Let me use more context to uniquely identify the last one.
|
||||
Now let me commit the changes.
|
||||
I need to commit only the files I changed in this iteration: `data_processing/diagnosis_lookup.py`, `IMPLEMENTATION_PLAN.md`, and `progress.txt`.
|
||||
Commit successful. Let me update progress.txt with the commit hash.
|
||||
## Iteration 3 Summary
|
||||
|
||||
**Task completed: 1.1 ÔÇö Update `get_patient_indication_groups()` to return ALL matches with code_frequency**
|
||||
|
||||
**Changes made to `data_processing/diagnosis_lookup.py`:**
|
||||
1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1` ÔÇö was returning only the single most recent match per patient
|
||||
2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"` ÔÇö now returns ALL matching Search_Terms per patient with a frequency count
|
||||
3. **Added** `earliest_hcd_date: Optional[str] = None` parameter ÔÇö restricts GP codes to the HCD data window when provided
|
||||
4. **Updated** return columns: `code_frequency` replaces `EventDateTime`
|
||||
5. **Updated** all empty DataFrame returns and docstring
|
||||
|
||||
**Validation:**
|
||||
- py_compile: PASSED
|
||||
- Import check: PASSED
|
||||
- Function signature: Verified (4 params, new `earliest_hcd_date` defaults to None)
|
||||
- Empty DataFrame columns: `['PatientPseudonym', 'Search_Term', 'code_frequency']` verified
|
||||
|
||||
**Commits:**
|
||||
- `c93417f` ÔÇö feat: return ALL GP matches with code_frequency
|
||||
- `947b87a` ÔÇö docs: update progress.txt
|
||||
|
||||
**Next iteration should work on Task 2.1:** Create `assign_drug_indications()` function ÔÇö the core logic that cross-references GP matches with drug mappings to produce modified UPIDs.
|
||||
## Iteration 3 Summary
|
||||
|
||||
**Task completed: 1.1 ÔÇö Update `get_patient_indication_groups()` to return ALL matches with code_frequency**
|
||||
|
||||
**Changes made to `data_processing/diagnosis_lookup.py`:**
|
||||
1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1` ÔÇö was returning only the single most recent match per patient
|
||||
2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"` ÔÇö now returns ALL matching Search_Terms per patient with a frequency count
|
||||
3. **Added** `earliest_hcd_date: Optional[str] = None` parameter ÔÇö restricts GP codes to the HCD data window when provided
|
||||
4. **Updated** return columns: `code_frequency` replaces `EventDateTime`
|
||||
5. **Updated** all empty DataFrame returns and docstring
|
||||
|
||||
**Validation:**
|
||||
- py_compile: PASSED
|
||||
- Import check: PASSED
|
||||
- Function signature: Verified (4 params, new `earliest_hcd_date` defaults to None)
|
||||
- Empty DataFrame columns: `['PatientPseudonym', 'Search_Term', 'code_frequency']` verified
|
||||
|
||||
**Commits:**
|
||||
- `c93417f` ÔÇö feat: return ALL GP matches with code_frequency
|
||||
- `947b87a` ÔÇö docs: update progress.txt
|
||||
|
||||
**Next iteration should work on Task 2.1:** Create `assign_drug_indications()` function ÔÇö the core logic that cross-references GP matches with drug mappings to produce modified UPIDs.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,80 +0,0 @@
|
||||
|
||||
|
||||
I'll start by reading the required files in order to understand the current state of the project.
|
||||
Now checking recent commits and reading the current state of the key file I'll be modifying.
|
||||
Good. I've read all the required files. Let me understand the current state:
|
||||
|
||||
**Phase 1 is complete** (Tasks 1.1, 1.2, 1.3 all done). The next task is **Task 2.1: Create `assign_drug_indications()` function** ÔÇö this is the core drug-aware matching logic.
|
||||
|
||||
**Why Task 2.1**:
|
||||
- All Phase 1 dependencies are complete (Snowflake query returns ALL matches with code_frequency, drug mapping loaded from CSV, asthma merged)
|
||||
- Task 2.1 is the foundation for Phase 2 ÔÇö Task 2.2 (tiebreaker logic) is naturally part of 2.1
|
||||
- This can be built and tested with mock data (no Snowflake needed)
|
||||
- Phase 3 (pipeline integration) depends on this function existing
|
||||
|
||||
Let me read the current refresh_pathways.py to understand the integration point and how gp_matches_df is currently used.
|
||||
Now I have the full picture. I can see the critical issue in `refresh_pathways.py` lines 424-428 ÔÇö `dict(zip(...))` which only keeps the LAST match per patient (since the new query returns multiple rows per patient). This will be fixed in Task 3.1.
|
||||
|
||||
Now I'm ready to build Task 2.1: `assign_drug_indications()`. Let me mark it in-progress and start implementing.
|
||||
Now implementing the `assign_drug_indications()` function. The key logic:
|
||||
|
||||
1. For each UPID + Drug Name pair in the HCD data:
|
||||
- Look up patient's GP-matched Search_Terms (from gp_matches_df via PseudoNHSNoLinked)
|
||||
- Look up which Search_Terms list this drug (from drug mapping)
|
||||
- Intersect  valid indications
|
||||
- Pick highest code_frequency as tiebreaker (or alphabetical if tied)
|
||||
- Modify UPID: `{UPID}|{search_term}` or fallback `{UPID}|{Directory} (no GP dx)`
|
||||
|
||||
2. Build indication_df mapping modified UPID  Search_Term
|
||||
|
||||
I'll add this function to `diagnosis_lookup.py` since that's where all the related functions live.
|
||||
Now I'll add the `assign_drug_indications()` function. I'll insert it after `get_search_terms_for_drug()` (line 1204) and before the `CLUSTER_MAPPING_SQL` section (line 1207).
|
||||
Now I need to update the `__all__` exports to include `assign_drug_indications`.
|
||||
Now let me validate the code. Running py_compile and import checks.
|
||||
Syntax check passed. Now import check.
|
||||
Import check passed. Now let me test with mock data to verify the function works correctly ÔÇö this covers Task 2.2 (tiebreaker logic) as well.
|
||||
All tests pass. Let me also test the tiebreaker with equal frequencies to verify alphabetical fallback.
|
||||
The alphabetical tiebreaker is comparing with `(-freq, term)` ÔÇö so for equal freq, "crohn's disease" < "psoriatic arthritis" alphabetically, which means crohn's should come first. But the apostrophe in "crohn's disease" might be causing a sorting issue. Let me check.
|
||||
Hmm, "crohn's disease" IS alphabetically first. Let me debug what's happening in the function.
|
||||
I see the issue. "crohn's disease" from DimSearchTerm.csv is not in the ADALIMUMAB Search_Terms list ÔÇö looking at the output, ADALIMUMAB matches: ankylosing spondylitis, axial spondyloarthritis, plaque psoriasis, psoriatic arthritis, rheumatoid arthritis, ulcerative colitis, uveitis. "crohn's disease" is NOT in that list (ADALIMUMAB doesn't match crohn's in DimSearchTerm.csv). So only psoriatic arthritis is in the intersection, giving a single match ÔÇö not a tiebreaker.
|
||||
|
||||
My test was wrong about the data. Let me fix the test to use two Search_Terms that actually overlap with ADALIMUMAB.
|
||||
All tests pass:
|
||||
- Equal frequency  alphabetical tiebreaker works ("psoriatic arthritis" < "rheumatoid arthritis")
|
||||
- Higher frequency wins (47 RA codes vs 3 psoriatic codes  RA)
|
||||
- Empty GP matches  fallback to directory
|
||||
|
||||
Now let me update the IMPLEMENTATION_PLAN.md to mark Task 2.1 and 2.2 complete, then commit and update progress.txt.
|
||||
Now commit the changes.
|
||||
Now update progress.txt with the iteration log.
|
||||
**Iteration 4 complete.** Here's the summary:
|
||||
|
||||
**Task 2.1 + 2.2: `assign_drug_indications()` ÔÇö COMPLETE**
|
||||
|
||||
Added the core drug-aware indication matching function to `diagnosis_lookup.py`. For each UPID + Drug Name pair:
|
||||
1. Gets patient's GP-matched Search_Terms (via PseudoNHSNoLinked)
|
||||
2. Gets which Search_Terms list this drug (from DimSearchTerm.csv mapping)
|
||||
3. Intersects to find valid indications
|
||||
4. Tiebreaker: highest code_frequency wins, alphabetical if tied
|
||||
5. Fallback: `{Directory} (no GP dx)` for unmatched drugs
|
||||
6. Modifies UPID: `{UPID}|{search_term}`
|
||||
|
||||
All mock data tests pass. Commits: `408976e` (function) and `d9891c8` (progress).
|
||||
|
||||
**Next iteration**: Task 3.1 ÔÇö integrate `assign_drug_indications()` into `refresh_pathways.py`, replacing the old `dict(zip())` approach that only kept one match per patient.
|
||||
**Iteration 4 complete.** Here's the summary:
|
||||
|
||||
**Task 2.1 + 2.2: `assign_drug_indications()` ÔÇö COMPLETE**
|
||||
|
||||
Added the core drug-aware indication matching function to `diagnosis_lookup.py`. For each UPID + Drug Name pair:
|
||||
1. Gets patient's GP-matched Search_Terms (via PseudoNHSNoLinked)
|
||||
2. Gets which Search_Terms list this drug (from DimSearchTerm.csv mapping)
|
||||
3. Intersects to find valid indications
|
||||
4. Tiebreaker: highest code_frequency wins, alphabetical if tied
|
||||
5. Fallback: `{Directory} (no GP dx)` for unmatched drugs
|
||||
6. Modifies UPID: `{UPID}|{search_term}`
|
||||
|
||||
All mock data tests pass. Commits: `408976e` (function) and `d9891c8` (progress).
|
||||
|
||||
**Next iteration**: Task 3.1 ÔÇö integrate `assign_drug_indications()` into `refresh_pathways.py`, replacing the old `dict(zip())` approach that only kept one match per patient.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,90 +0,0 @@
|
||||
|
||||
|
||||
I'll start by reading the required files in order to understand the current state of the project.
|
||||
Now let me check the recent git log and read the current state of `refresh_pathways.py` since Task 3.1 is next.
|
||||
Reading progress.txt, IMPLEMENTATION_PLAN.md, and guardrails.md. I can see:
|
||||
|
||||
**Status**: Phase 1 (Tasks 1.1, 1.2, 1.3) and Phase 2 (Tasks 2.1, 2.2) are all complete. The next task is **Task 3.1: Update `refresh_pathways.py` indication processing to use `assign_drug_indications()`**.
|
||||
|
||||
**Why Task 3.1**: This is the first task in Phase 3 and the natural next step. All Phase 1 & 2 dependencies are complete (the query returns all matches, drug mapping is loaded, and `assign_drug_indications()` exists). Now I need to wire them together in the pipeline.
|
||||
|
||||
The current indication processing block (lines 361-518) uses the old approach:
|
||||
- Line 424-428: `dict(zip(...))` which only keeps the LAST match per patient
|
||||
- Lines 430-452: Manual indication_df building per UPID (no drug awareness)
|
||||
|
||||
I need to replace this with the new drug-aware approach:
|
||||
1. Load drug mapping via `load_drug_indication_mapping()`
|
||||
2. Pass `earliest_hcd_date` to `get_patient_indication_groups()`
|
||||
3. Call `assign_drug_indications()` to get (modified_df, indication_df)
|
||||
4. Use modified_df for pathway processing
|
||||
|
||||
Let me first mark Task 3.1 as in-progress, then look at the `assign_drug_indications` function signature to ensure I wire it correctly.
|
||||
Reading the relevant function signatures from diagnosis_lookup.py.
|
||||
Good. I now have a clear picture of the function signatures:
|
||||
|
||||
- `load_drug_indication_mapping(csv_path=None)`  `(fragment_to_search_terms, search_term_to_fragments)`
|
||||
- `assign_drug_indications(df, gp_matches_df, search_term_to_fragments)`  `(modified_df, indication_df)`
|
||||
- `indication_df` is already indexed by modified UPID with a `'Directory'` column ÔÇö exactly what `process_indication_pathway_for_date_filter` expects
|
||||
|
||||
Now I'll update `refresh_pathways.py`. The key changes to the `elif current_chart_type == "indication":` block (lines 361-518):
|
||||
|
||||
1. Import `assign_drug_indications` and `load_drug_indication_mapping` at the top
|
||||
2. Load drug mapping
|
||||
3. Pass `earliest_hcd_date` to `get_patient_indication_groups()`
|
||||
4. Call `assign_drug_indications()` instead of the manual dict(zip) approach
|
||||
5. Use modified_df for pathway processing (instead of original df)
|
||||
6. indication_df is already in the right format ÔÇö no need for the manual building code
|
||||
|
||||
Let me first update the imports, then replace the indication processing block.
|
||||
Now replace the indication processing block. The old code spans from the `elif current_chart_type == "indication":` block at line 361 to the end of the try/except at line 518. I need to replace the inner logic (lines 362-512) with the new drug-aware approach.
|
||||
|
||||
Let me replace the entire indication block. The key section to replace is from `elif current_chart_type == "indication":` through the end of the inner try block.
|
||||
Now let me verify the file looks correct by reading the updated indication block.
|
||||
The code looks correct. Let me also verify the imports at the top are clean.
|
||||
Good, imports look clean. Now let me run the validation checks.
|
||||
All validation checks pass:
|
||||
- `py_compile` ÔÇö no syntax errors
|
||||
- Individual imports ÔÇö `assign_drug_indications`, `get_patient_indication_groups`, `load_drug_indication_mapping` all import successfully
|
||||
- Full module import ÔÇö `cli.refresh_pathways` imports without error
|
||||
|
||||
Now let me update IMPLEMENTATION_PLAN.md to mark Task 3.1 subtasks as complete, then update progress.txt and commit.
|
||||
Now update progress.txt with the iteration log.
|
||||
Now let me commit the changes.
|
||||
I'll only stage the files I actually changed in this iteration: `cli/refresh_pathways.py`, `IMPLEMENTATION_PLAN.md`, and `progress.txt`.
|
||||
Good ÔÇö 3 files, net reduction of 7 lines in refresh_pathways.py (cleaner code), plus progress notes.
|
||||
Now update the commit hash in progress.txt.
|
||||
**Task 3.1 complete.** Here's what was done:
|
||||
|
||||
**Changes to `cli/refresh_pathways.py`:**
|
||||
- Added imports for `assign_drug_indications` and `load_drug_indication_mapping`
|
||||
- Replaced the old indication processing block (~90 lines  ~60 lines) with drug-aware matching:
|
||||
1. Loads drug mapping from DimSearchTerm.csv
|
||||
2. Computes `earliest_hcd_date` from HCD data to restrict GP code window
|
||||
3. Gets ALL GP matches per patient (with code_frequency) via updated query
|
||||
4. Calls `assign_drug_indications()` to cross-reference drugs against GP diagnoses
|
||||
5. Passes `modified_df` (with indication-aware UPIDs like `RMV12345|rheumatoid arthritis`) to pathway processing
|
||||
6. Uses `indication_df` directly ÔÇö already in the correct format (indexed by modified UPID, `Directory` column)
|
||||
|
||||
**Validation:** `py_compile` passed, all imports verified, full module import OK.
|
||||
|
||||
**Commits:** `920570b` (code) + `73088b0` (progress docs)
|
||||
|
||||
**Next task:** 3.2 ÔÇö Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v` to test the integrated pipeline with live Snowflake data.
|
||||
**Task 3.1 complete.** Here's what was done:
|
||||
|
||||
**Changes to `cli/refresh_pathways.py`:**
|
||||
- Added imports for `assign_drug_indications` and `load_drug_indication_mapping`
|
||||
- Replaced the old indication processing block (~90 lines  ~60 lines) with drug-aware matching:
|
||||
1. Loads drug mapping from DimSearchTerm.csv
|
||||
2. Computes `earliest_hcd_date` from HCD data to restrict GP code window
|
||||
3. Gets ALL GP matches per patient (with code_frequency) via updated query
|
||||
4. Calls `assign_drug_indications()` to cross-reference drugs against GP diagnoses
|
||||
5. Passes `modified_df` (with indication-aware UPIDs like `RMV12345|rheumatoid arthritis`) to pathway processing
|
||||
6. Uses `indication_df` directly ÔÇö already in the correct format (indexed by modified UPID, `Directory` column)
|
||||
|
||||
**Validation:** `py_compile` passed, all imports verified, full module import OK.
|
||||
|
||||
**Commits:** `920570b` (code) + `73088b0` (progress docs)
|
||||
|
||||
**Next task:** 3.2 ÔÇö Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v` to test the integrated pipeline with live Snowflake data.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,330 +0,0 @@
|
||||
|
||||
|
||||
I'll start by reading the required files in order to understand the current state of the project.
|
||||
Reading progress.txt, IMPLEMENTATION_PLAN.md, and guardrails.md. Now checking recent git history.
|
||||
Good. I can see the state of the project:
|
||||
|
||||
**Completed**: Tasks 1.1, 1.2, 1.3 (Phase 1), Tasks 2.1, 2.2 (Phase 2), Task 3.1 (Phase 3 integration)
|
||||
**Next up**: Task 3.2 ÔÇö Test with dry run (`python -m cli.refresh_pathways --chart-type indication --dry-run -v`)
|
||||
|
||||
This is the first task that requires a live Snowflake connection. Let me examine the current state of the refresh_pathways.py code and understand the dry run flag behavior before running it.
|
||||
|
||||
**Why Task 3.2**: It's the next logical step ÔÇö 3.1 (pipeline integration) is complete, and 3.2 validates that the integrated pipeline actually works end-to-end. All later tasks (4.1, 4.2, 4.3) depend on 3.2 passing.
|
||||
Good, the pipeline code looks clean. Now let me mark Task 3.2 as in-progress and run the dry run test. This requires a live Snowflake connection. Let me first check if Snowflake is accessible.
|
||||
Running the dry run test. This will fetch data from Snowflake and process indication charts without inserting to the database.
|
||||
The command is running in the background. Let me check the output.
|
||||
Every batch is timing out after ~30 seconds. The query is too slow with the `GROUP BY + COUNT(*)` change. The previous version (with `QUALIFY ROW_NUMBER()`) worked because it returned 1 row per patient. Now it's doing `GROUP BY + COUNT(*)` which is much more expensive on the large GP records table.
|
||||
|
||||
Let me stop the background task since it's going to keep failing, and investigate the query.
|
||||
The issue is clear: the updated query with `GROUP BY + COUNT(*)` is timing out at 30 seconds per batch. The original `QUALIFY ROW_NUMBER()` version worked within the timeout. I need to understand the query structure and the timeout setting.
|
||||
|
||||
Let me look at the actual query and the timeout configuration.
|
||||
Let me check the default query timeout and the `execute_dict` method to understand the 30s timeout.
|
||||
The config is not defined in this file. Let me find where the timeout defaults are.
|
||||
The default query_timeout is 300 seconds. But the logs show queries being cancelled after ~30 seconds: `000604 (57014): SQL execution was cancelled by the client due to a timeout`. Wait ÔÇö the error says "SQL execution cancelled by the client due to a timeout" but the Snowflake session timeout is set to 300s. Let me look more carefully. The `execute_dict` method is what's being called.
|
||||
The log says `Executing query (timeout=300s)` but the queries are dying at ~30s. Wait ÔÇö let me re-read the error timestamps more carefully:
|
||||
|
||||
- Batch 1: 23:18:58  23:19:29 = 31s
|
||||
- Batch 2: 23:19:29  23:20:00 = 31s
|
||||
|
||||
Each batch is timing out at ~30s. The error code is `000604 (57014)` which is Snowflake's SQL cancellation. But the timeout is set to 300s. This suggests something else is cancelling ÔÇö maybe the warehouse has a statement timeout configured at the Snowflake level (30s), which would override the session setting.
|
||||
|
||||
The real issue is the query performance. The `GROUP BY + COUNT(*)` with the large CTE and join on `PrimaryCareClinicalCoding` is expensive. The original `QUALIFY ROW_NUMBER()` query worked because it could short-circuit once it found one match per patient.
|
||||
|
||||
Let me understand the problem better. The key difference is:
|
||||
- **Old query**: `QUALIFY ROW_NUMBER() = 1`  returns 1 row per patient (efficient, limits result set)
|
||||
- **New query**: `GROUP BY + COUNT(*)`  scans ALL matching rows, groups them, counts  much heavier
|
||||
|
||||
The solution is to optimize the query. Let me think about approaches:
|
||||
|
||||
1. **Increase timeout** ÔÇö Won't help if the query is inherently slow for 500-patient batches
|
||||
2. **Reduce batch size** ÔÇö smaller batches might complete faster
|
||||
3. **Optimize query structure** ÔÇö maybe restructure to be more efficient
|
||||
|
||||
Let me test with a direct Snowflake query to understand the actual query plan. First, let me check if we can run a smaller test batch.
|
||||
Let me test the query directly against Snowflake to understand the performance characteristics. I'll run a small batch (10 patients) to see if the query works at all, and also test the current timeout behavior.
|
||||
Good progress ÔÇö I can see two things:
|
||||
1. The session timeout IS set to 300s, but queries still fail at ~30s. This suggests the warehouse itself has a timeout (e.g., XSMALL warehouse with its own statement timeout).
|
||||
2. The query test failed because I used the wrong database name. Let me fix that and test properly.
|
||||
|
||||
Let me check the warehouse statement timeout and also run the actual test with the correct database.
|
||||
The warehouse timeout is 7200s (2 hours) and the session timeout is 300s. Neither would cause a 30s timeout. Let me re-read the error messages more carefully ÔÇö maybe it's a network/connection timeout rather than a query execution timeout.
|
||||
|
||||
Actually, looking at the log more carefully: the session sets `STATEMENT_TIMEOUT_IN_SECONDS = 300` but the queries fail at 30s. The error is `000604 (57014): SQL execution was cancelled by the client due to a timeout`. The key phrase is "**by the client**" ÔÇö this is the Python client (snowflake-connector-python) timing out, not Snowflake server.
|
||||
|
||||
Let me check if there's a Python-side network timeout or client-side configuration.
|
||||
I see `network_timeout` being set. Let me check the timeout defaults.
|
||||
`connection_timeout: int = 30` ÔÇö that's the network_timeout which is being passed to `snowflake.connector.connect()` as `network_timeout=30`. This is the **network-level timeout** for Snowflake operations. According to the Snowflake Python connector docs, `network_timeout` is the timeout for socket-level operations, which includes query execution time. When a query takes longer than 30 seconds, the client kills it.
|
||||
|
||||
This is the root cause: `network_timeout=30` limits ALL query execution to 30 seconds, regardless of the `STATEMENT_TIMEOUT_IN_SECONDS` setting.
|
||||
|
||||
Now I have two options to fix this:
|
||||
1. **Increase `network_timeout`** ÔÇö but this would affect all queries
|
||||
2. **Pass a larger timeout to `execute_dict` calls in `get_patient_indication_groups`**
|
||||
|
||||
Actually wait ÔÇö the `network_timeout` parameter in the Snowflake connector is for network-level operations. Let me double-check. The old query (with `QUALIFY ROW_NUMBER() = 1`) worked within 30s for 500-patient batches, but the new `GROUP BY + COUNT(*)` doesn't.
|
||||
|
||||
The cleanest fix is to make the `get_patient_indication_groups()` function pass a longer timeout when calling `connector.execute_dict()`. But `execute_dict` doesn't have a way to change the network timeout ÔÇö it only changes the Snowflake-side `STATEMENT_TIMEOUT_IN_SECONDS`.
|
||||
|
||||
The real issue is the client-side `network_timeout=30`. This was fine for the old query but not for the new one. Let me think about the best approach:
|
||||
|
||||
1. **Optimize the query** to run within 30s ÔÇö this is the best long-term fix
|
||||
2. **Increase connection_timeout** ÔÇö easy but affects everything
|
||||
3. **Create a new connection with higher timeout** for GP lookups
|
||||
|
||||
Let me first try to understand if the query can be optimized. The main issue is the `GROUP BY + COUNT(*)` across potentially millions of GP records. Let me check the query structure and think about optimization.
|
||||
|
||||
Actually, looking at this more carefully, I think the most pragmatic approach is a combination:
|
||||
|
||||
1. **Optimize the query** ÔÇö we can use `COUNT(DISTINCT ...)` or add better filtering
|
||||
2. **Reduce batch size** from 500 to something smaller (say 100)
|
||||
3. **Increase the timeout** for this specific query path
|
||||
|
||||
But first, let me try running the query with just a few patients to confirm it works at all.
|
||||
Wrong table name. Let me check the actual Snowflake schema being used for the HCD data.
|
||||
OK. Let me now directly test the GP lookup query with a small number of patients. I'll pull from the same table used in the actual pipeline.
|
||||
Even with just 5 patients, the query times out at 30 seconds. This confirms the problem is the query execution time, not batch size. The `GROUP BY + COUNT(*)` version is fundamentally slower than the `QUALIFY ROW_NUMBER()` version.
|
||||
|
||||
The `PrimaryCareClinicalCoding` table is massive (national GP records). The join with the CTE + GROUP BY is too expensive. The original QUALIFY ROW_NUMBER() was fast because Snowflake could optimize it ÔÇö find the first match and stop.
|
||||
|
||||
Let me think about the optimization approach. The core problem is:
|
||||
1. Large CTE (148 rows  joined with SNOMED codes  potentially thousands of codes)
|
||||
2. Join with massive `PrimaryCareClinicalCoding` table
|
||||
3. GROUP BY forces a full scan
|
||||
|
||||
**Strategy**: Keep the efficient QUALIFY approach but return multiple rows. Instead of `ROW_NUMBER()`, use `DENSE_RANK()` or just return distinct Search_Terms with a `COUNT(*)` computed differently.
|
||||
|
||||
Actually, the best approach: split the query into two steps:
|
||||
1. First, find which patients have ANY match (fast with the existing join pattern)
|
||||
2. Then, for matched patients, compute the code_frequency
|
||||
|
||||
Or better: use a different approach to get code_frequency without GROUP BY on the massive table:
|
||||
|
||||
```sql
|
||||
-- Use QUALIFY to get distinct Search_Terms per patient (no aggregation)
|
||||
SELECT DISTINCT pc."PatientPseudonym", aic.Search_Term
|
||||
FROM PrimaryCareClinicalCoding pc
|
||||
JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
|
||||
WHERE pc."PatientPseudonym" IN (...)
|
||||
```
|
||||
|
||||
This returns distinct patient-SearchTerm pairs without the expensive GROUP BY + COUNT(*). For the tiebreaker, we could:
|
||||
- Use a separate query to count codes (but that's also slow)
|
||||
- Use a window function instead of GROUP BY
|
||||
|
||||
Let me try the DISTINCT approach first ÔÇö it should be much faster.
|
||||
Even SELECT DISTINCT times out. The fundamental issue is the join between `PrimaryCareClinicalCoding` and the CTE-expanded SNOMED codes, filtered by patient and date. This is consistently timing out at ~30s even for 5 patients.
|
||||
|
||||
But wait ÔÇö the **old query** (with QUALIFY ROW_NUMBER()) worked! Let me verify that the old query still works, and check how long it takes.
|
||||
The old query takes 19.7s for 5 patients. That's already very close to the 30s timeout! With 500 patients, it would definitely take much longer. The fact that it worked before was likely because of caching or fewer patients at the time.
|
||||
|
||||
Now I understand the full picture:
|
||||
- Old query: ~20s for 5 patients (borderline, 500 patients would be ~30s+ with caching help)
|
||||
- New query (GROUP BY): even slower, consistently >30s
|
||||
- The `network_timeout=30` in the connector config kills both
|
||||
|
||||
The fix needs two parts:
|
||||
1. **Increase the network_timeout** to allow these queries to complete
|
||||
2. **Possibly optimize the query** for better performance
|
||||
|
||||
But the network_timeout is the immediate blocker. Let me fix that first. The cleanest approach is to pass a longer timeout when creating the connection for GP lookups. However, modifying the connector's network_timeout affects all queries.
|
||||
|
||||
Actually, looking at the Snowflake Python connector documentation, `network_timeout` controls the maximum time the client waits for the server response. To allow longer-running queries, we should increase this. The question is whether to increase it globally or per-connection.
|
||||
|
||||
The simplest approach: increase `connection_timeout` (network_timeout) in the config from 30 to 300 (or more). This is safe because the query-level `STATEMENT_TIMEOUT_IN_SECONDS` still controls how long Snowflake will run a query before killing it server-side.
|
||||
|
||||
Let me also check if we can pass timeout directly to `get_patient_indication_groups`  `execute_dict`.
|
||||
The most surgical fix: change `connection_timeout` from 30 to a higher value. But modifying a shared config file affects all Snowflake operations.
|
||||
|
||||
Actually, looking at `get_patient_indication_groups`, it calls `connector.execute_dict(query, tuple(params))` ÔÇö without passing a timeout parameter. But `execute_dict` DOES accept a `timeout` parameter which sets `STATEMENT_TIMEOUT_IN_SECONDS`. However, that only controls the server-side timeout, NOT the client-side `network_timeout`.
|
||||
|
||||
The real fix is to increase `network_timeout` when connecting. Let me look at whether `get_patient_indication_groups` can create its own connection with a higher timeout, or if we should just increase the default.
|
||||
|
||||
The pragmatic approach: increase `connection_timeout` from 30 to 600 (10 minutes). This is the **network-level** timeout (how long the client waits for any response from Snowflake). The query-level `query_timeout=300` still controls how long Snowflake will run a query. This is safe because:
|
||||
- Activity data fetch already takes ~10s and uses chunked queries (separate timeout mechanism)
|
||||
- GP lookup queries can take 20-30+ seconds per batch
|
||||
- The query_timeout (300s) is the actual safety net against runaway queries
|
||||
Now, there are two approaches to fix this:
|
||||
|
||||
**Option A**: Increase the global `connection_timeout` from 30 to 600 in the config. Simple but affects all queries.
|
||||
|
||||
**Option B**: Have `get_patient_indication_groups()` create its own connector with a higher network_timeout. More surgical but adds complexity.
|
||||
|
||||
I think Option A is the right call. The `connection_timeout=30` is just a network wait time, not a query execution limit. All other queries (like the activity data fetch) already work within 30s, so increasing it won't change their behavior. It just gives the GP lookup queries room to complete.
|
||||
|
||||
But I also need to consider query optimization. Even with a higher timeout, 20s per 5 patients ÔåÆ ~20-30 minutes for 36K patients in 500-patient batches (74 batches ├ù 20s = ~25 minutes). The old approach took ~9 minutes (according to CLAUDE.md). So the new GROUP BY query will be significantly slower.
|
||||
|
||||
Let me think about a better query strategy. Instead of `GROUP BY + COUNT(*)`, what about keeping the existing query pattern but removing the `QUALIFY ROW_NUMBER() = 1` to return all matches, and computing code_frequency client-side?
|
||||
|
||||
```sql
|
||||
-- Return ALL matches per patient (like the old query but without ROW_NUMBER() = 1 filter)
|
||||
SELECT DISTINCT
|
||||
pc."PatientPseudonym",
|
||||
aic.Search_Term,
|
||||
pc."SNOMEDCode"
|
||||
FROM PrimaryCareClinicalCoding pc
|
||||
JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
|
||||
WHERE pc."PatientPseudonym" IN (...)
|
||||
AND pc."EventDateTime" >= ...
|
||||
```
|
||||
|
||||
Then compute `code_frequency` in Python: `df.groupby(['PatientPseudonym', 'Search_Term']).size()`.
|
||||
|
||||
Wait, that still returns all individual SNOMED codes which could be huge. And `SELECT DISTINCT` already timed out.
|
||||
|
||||
Let me try a different approach ÔÇö use `QUALIFY` with a higher rank instead of removing it entirely:
|
||||
|
||||
```sql
|
||||
-- Return multiple rows per patient but use window function to get count
|
||||
SELECT
|
||||
pc."PatientPseudonym",
|
||||
aic.Search_Term,
|
||||
COUNT(*) OVER (PARTITION BY pc."PatientPseudonym", aic.Search_Term) AS code_frequency
|
||||
FROM PrimaryCareClinicalCoding pc
|
||||
JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
|
||||
WHERE ...
|
||||
QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym", aic.Search_Term ORDER BY pc."EventDateTime" DESC) = 1
|
||||
```
|
||||
|
||||
This uses `ROW_NUMBER() = 1` but partitioned by `PatientPseudonym + Search_Term` (not just PatientPseudonym), so it returns one row per patient-SearchTerm pair. And the `COUNT(*) OVER(...)` window function gives us the code_frequency without a GROUP BY.
|
||||
|
||||
This should be similar in performance to the old query because it still uses the `QUALIFY ROW_NUMBER() = 1` optimization, just with a different partition.
|
||||
|
||||
Let me test this.
|
||||
Still timing out at 30s. The window function approach is just as slow. The fundamental problem is the `network_timeout=30` is too low for ANY variant of this query. The old query barely squeaked through at 19.7s for 5 patients.
|
||||
|
||||
The key insight: even the OLD working query takes ~20s per batch. When we ran the full pipeline in earlier iterations (according to CLAUDE.md, "~9 minutes for 37K patients"), that's ~74 batches × ~7s per batch with Snowflake query caching helping subsequent batches. But the first batch is always the slowest.
|
||||
|
||||
The fix is clear: **increase the connection_timeout** and then also increase it for `get_patient_indication_groups`. Let me also check if there's a way to make the query faster.
|
||||
|
||||
Actually, I realize the old query that took 19.7s was for the FIRST 5 random patients. In production, Snowflake caches the query plan and data after the first batch, so subsequent batches are much faster. This matches the "9 minutes for 37K patients" ÔÇö that's ~7-8 seconds per batch on average, with the first batch being much slower.
|
||||
|
||||
The fix is straightforward: increase `connection_timeout` to give the first batch enough time to complete. Let me do that.
|
||||
Now let me test again with the new timeout. Let me try the window function + QUALIFY approach first.
|
||||
Still timing out. The config change didn't take effect. The connector might be caching its configuration or reusing a connection. Let me check if the connector singleton is caching the old config.
|
||||
Let me check how the config is loaded and passed to the connector.
|
||||
The config is cached. Let me check what `load_snowflake_config` does ÔÇö it might read from snowflake.toml and override the defaults.
|
||||
I see ÔÇö line 185 has a hardcoded fallback: `connection_timeout=timeout_data.get("connection_timeout", 30)`. The TOML file overrides the dataclass default. Let me check the TOML file.
|
||||
Found it. Line 40: `connection_timeout = 30` in `config/snowflake.toml`. This is what gets loaded. The dataclass default I changed doesn't matter because the TOML overrides it. Let me update the TOML file.
|
||||
Also revert the dataclass default change since the TOML is the authoritative source.
|
||||
Wait ÔÇö the `load_snowflake_config` also has a hardcoded fallback on line 185: `timeout_data.get("connection_timeout", 30)`. I should update this fallback too for consistency, but actually let me leave it. The TOML value (600) will be used since the key exists. The fallback 30 only applies if the key is missing from TOML.
|
||||
|
||||
Actually, I should also update the fallback to match the new intent. Let me do that for consistency:
|
||||
Now let me test the query again with the updated timeout. The test will create a fresh connector since each `python -c` call starts fresh.
|
||||
The query now works with the 600s timeout. It took 39.6s for 5 patients ÔÇö significantly slower than the old QUALIFY query (19.7s). Let me now test with the 500-patient batch size to estimate total time.
|
||||
|
||||
Actually, let me first consider: at 39.6s per batch with batch_size=5, that's ~8s per patient. For 36K patients, that would be absurd. But the first batch is always the slowest (cold cache). Let me test with 500 patients to see if Snowflake's caching kicks in.
|
||||
Excellent. The 500-patient batch takes 38.4s ÔÇö very similar to the 5-patient batch (39.6s). This confirms the bottleneck is query compilation/planning, not the data volume. Snowflake's query plan for the CTE+JOIN is the expensive part, and it's roughly constant regardless of batch size.
|
||||
|
||||
Key data:
|
||||
- 500 patients  38.4s, 5517 rows, 488 matched (97.6% match rate)
|
||||
- Avg 11.3 indications per patient (this includes non-drug-related diagnoses like "sepsis", "influenza")
|
||||
- 115 unique Search_Terms found
|
||||
|
||||
For 36K patients at 500/batch = 72 batches × ~38s = ~46 minutes. That's significantly longer than the old approach (9 minutes). But subsequent batches might be faster due to Snowflake query caching.
|
||||
|
||||
Let me test a second batch to see if caching helps.
|
||||
No caching benefit ÔÇö still ~40s per batch. This means 72 batches ├ù 40s = ~48 minutes just for the GP lookup. That's too slow. Let me think about optimization.
|
||||
|
||||
The core problem: each batch re-executes the massive CTE (148 cluster mappings  SNOMED codes expansion) and joins with the huge `PrimaryCareClinicalCoding` table. The CTE is identical across all batches  only the patient list changes.
|
||||
|
||||
**Optimization ideas:**
|
||||
1. **Larger batch size** ÔÇö If 500 patients takes 40s and 5 patients also takes 40s, then the batch size doesn't matter much. Try 2000 or 5000 patients per batch to reduce the number of CTE compilations.
|
||||
2. **Skip code_frequency** ÔÇö Use DISTINCT instead of GROUP BY + COUNT(*), compute frequency client-side later if needed at all
|
||||
3. **Pre-create temp table** for the CTE results, then query against that
|
||||
|
||||
Let me try option 1 first ÔÇö increase batch size to 5000.
|
||||
Huge improvement. 5000 patients takes 41.3s ÔÇö almost the same as 500 patients. The overhead is in the CTE compilation, not the data volume. With 5000-patient batches:
|
||||
- 36K patients / 5000 per batch = ~8 batches
|
||||
- 8 × 41s = ~5.5 minutes for GP lookups
|
||||
|
||||
That's reasonable and much better than the 48 minutes with 500-patient batches. Let me try an even larger batch.
|
||||
10K patients takes ~43s. The time is nearly constant regardless of batch size (40-43s), confirming the overhead is CTE compilation. With 10K-patient batches:
|
||||
- ~42K patients / 10000 = ~5 batches × 43s = ~3.5 minutes
|
||||
|
||||
That's excellent. Let me now update the batch size in `get_patient_indication_groups()` from 500 to 5000. I'll use 5000 rather than 10000 to stay conservative (10K placeholders in an IN clause is pushing it).
|
||||
|
||||
Actually wait ÔÇö the caller in `refresh_pathways.py` passes `batch_size=500`. Let me check.
|
||||
I need to change the batch_size in the caller. Let me update both the default in the function and the call site.
|
||||
There are two functions with `batch_size: int = 500`. Let me update the specific one.
|
||||
Now update the caller in refresh_pathways.py:
|
||||
Let me verify both files compile correctly.
|
||||
Now let me run the full dry run test with the fixes applied.
|
||||
The dry run is running in the background. It will take a while ÔÇö Snowflake fetch (~10s), data transformations (~6 min), GP lookup (~5 min with new batch sizes), pathway processing (~1 min). Let me check on it periodically.
|
||||
Data transformations are running. Let me wait for them to complete and check the GP lookup phase.
|
||||
The first batch completed successfully: **Batch 1: 54,300 rows in 48.72s**. The GP lookup is now working with the 5000-patient batches and 600s network timeout. Batch 2 is running. 8 batches total.
|
||||
|
||||
Let me wait for the full pipeline to complete.
|
||||
The dry run completed successfully! Let me read the full output to get all the key metrics.
|
||||
Let me also see the tiebreaker and top indications stats.
|
||||
Excellent. Here's the full picture of the dry run results:
|
||||
|
||||
**GP Lookup Results:**
|
||||
- 36,628 patients queried  33,642 with GP match (91.8%)
|
||||
- 8 batches of 5000 patients, ~40-48s each
|
||||
- 537,794 total patient-indication matches (avg 16 per patient ÔÇö includes non-drug-related like sepsis, influenza)
|
||||
- Top GP indications: sepsis (32K), drug misuse (31K), influenza (28K)
|
||||
|
||||
**Drug-Aware Matching Results:**
|
||||
- 50,797 UPID-Drug pairs processed
|
||||
- **25,059 matched (49.3%)** ÔÇö drug has a DimSearchTerm mapping AND patient has GP dx for that condition
|
||||
- **25,738 fallback (50.7%)** ÔÇö no match, falls back to directory
|
||||
- **15,238 tiebreaker used** ÔÇö drug matched multiple indications, picked by highest code_frequency
|
||||
- **42,072 unique modified UPIDs** (up from original ~37K unique patients)
|
||||
- Top assigned indications: "Undefined (no GP dx)" (8,110), macular degeneration (5,256), diabetes (3,841), "OPHTHALMOLOGY (no GP dx)" (3,040), rheumatoid arthritis (2,216)
|
||||
|
||||
**Pathway Node Counts:**
|
||||
| Date Filter | Nodes |
|
||||
|---|---|
|
||||
| all_6mo | 438 |
|
||||
| all_12mo | 484 |
|
||||
| 1yr_6mo | 181 |
|
||||
| 1yr_12mo | 199 |
|
||||
| 2yr_6mo | 257 |
|
||||
| 2yr_12mo | 287 |
|
||||
| **Total** | **1,846** |
|
||||
|
||||
This is comparable to the old approach (~300 per date filter for the wider filters). The 49.3% match rate seems reasonable ÔÇö drugs without a DimSearchTerm mapping (or where the patient has no GP coding for the drug's indication) fall back to directory labels.
|
||||
|
||||
The dry run is a success. Let me now mark Task 3.2 as complete, commit the changes, and update progress.
|
||||
Let me update the IMPLEMENTATION_PLAN.md to mark Task 3.2 complete.
|
||||
Now update progress.txt with the iteration log.
|
||||
There are 4 occurrences of "### Blocked items:\n- None" in the file. I need to be more specific. Let me append to the end of the file instead.
|
||||
Now let me add a guardrail for the timeout issue since it's a failure pattern future iterations should know about.
|
||||
Now let me commit all the changes.
|
||||
There are many changed files from prior uncommitted work (DESIGN_SYSTEM.md, RALPH_PROMPT.md, etc.). I should only commit the files I changed for Task 3.2. Let me stage just the relevant files.
|
||||
Now update progress.txt with the commit hash.
|
||||
Task 3.2 is complete. Let me verify the final state.
|
||||
## Summary ÔÇö Iteration 6 Complete
|
||||
|
||||
**Task 3.2: Test with dry run** ÔÇö COMPLETE
|
||||
|
||||
### Key findings and fixes:
|
||||
1. **Root cause**: GP lookup queries timing out at 30s due to `connection_timeout=30` in snowflake.toml (controls Python client `network_timeout`, not server-side timeout)
|
||||
2. **Fix 1**: Increased `connection_timeout` from 30  600 seconds
|
||||
3. **Fix 2**: Increased GP lookup batch_size from 500  5000 (query time is ~40s regardless of batch size  bottleneck is CTE compilation, not data volume)
|
||||
4. **Dry run successful**:
|
||||
- 91.8% GP match rate
|
||||
- 49.3% drug-indication match rate, 50.7% fallback to directory
|
||||
- 15,238 tiebreaker decisions made
|
||||
- 42,072 unique modified UPIDs
|
||||
- 1,846 total pathway nodes across 6 date filters
|
||||
|
||||
### Next: Task 4.1 (Full refresh with both chart types)
|
||||
## Summary ÔÇö Iteration 6 Complete
|
||||
|
||||
**Task 3.2: Test with dry run** ÔÇö COMPLETE
|
||||
|
||||
### Key findings and fixes:
|
||||
1. **Root cause**: GP lookup queries timing out at 30s due to `connection_timeout=30` in snowflake.toml (controls Python client `network_timeout`, not server-side timeout)
|
||||
2. **Fix 1**: Increased `connection_timeout` from 30  600 seconds
|
||||
3. **Fix 2**: Increased GP lookup batch_size from 500  5000 (query time is ~40s regardless of batch size  bottleneck is CTE compilation, not data volume)
|
||||
4. **Dry run successful**:
|
||||
- 91.8% GP match rate
|
||||
- 49.3% drug-indication match rate, 50.7% fallback to directory
|
||||
- 15,238 tiebreaker decisions made
|
||||
- 42,072 unique modified UPIDs
|
||||
- 1,846 total pathway nodes across 6 date filters
|
||||
|
||||
### Next: Task 4.1 (Full refresh with both chart types)
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,132 +0,0 @@
|
||||
|
||||
|
||||
I'll start by reading the required files in order to understand the current state of the project.
|
||||
Good. I've read progress.txt, IMPLEMENTATION_PLAN.md, and guardrails.md. Now let me check recent git history.
|
||||
Now I have the full picture. Let me assess the state:
|
||||
|
||||
**Completed**: Phase 1 (1.1, 1.2, 1.3), Phase 2 (2.1, 2.2), Phase 3 (3.1, 3.2)
|
||||
**Remaining**: Phase 4 ÔÇö Tasks 4.1, 4.2, 4.3
|
||||
|
||||
The previous iteration (6) recommended working on **Task 4.1: Full refresh with both chart types**. This requires running `python -m cli.refresh_pathways --chart-type all` (a live Snowflake refresh that takes ~15 minutes).
|
||||
|
||||
**Why Task 4.1 next**: It's the next logical step ÔÇö dry run passed, now we do the actual insert. Tasks 4.2 and 4.3 depend on 4.1 completing (they validate the results).
|
||||
|
||||
Let me mark Task 4.1 as in-progress in the implementation plan, then run the full refresh.
|
||||
Running the full refresh now. This will fetch from Snowflake, transform data, run GP lookups, and insert both directory and indication pathway nodes into SQLite. Expected time: ~15 minutes.
|
||||
The refresh is running in the background. Let me check the output so far.
|
||||
The refresh is progressing well. Directory charts completed (1,101 nodes across 6 filters), and now indication charts are being processed. GP lookup completed successfully (91.8% match rate, 42,072 modified UPIDs). Indication chart processing is underway ÔÇö I can see all_6mo (438 nodes) and all_12mo starting. Let me wait for it to finish.
|
||||
Full refresh completed successfully. Let me summarize the results:
|
||||
|
||||
**Full Refresh Results (Task 4.1):**
|
||||
- **Total time**: 738.4 seconds (~12 minutes)
|
||||
- **Total nodes inserted**: 2,947 (cleared 3,633 old nodes first)
|
||||
- **Directory charts**: 1,101 nodes across 6 date filters
|
||||
- **Indication charts**: 1,846 nodes across 6 date filters
|
||||
- **GP lookup**: 36,628 patients, 33,642 matched (91.8%), 8 batches in ~30s total
|
||||
- **Drug-indication matching**: 50,797 UPID-Drug pairs  25,059 matched (49.3%), 15,238 tiebreakers, 25,738 fallback
|
||||
- **Modified UPIDs**: 42,072 (up from 36,628 original patients)
|
||||
|
||||
Both chart types generated data successfully. Let me update the todo list and proceed to Task 4.2 (validate indication chart correctness).
|
||||
Now validating the indication chart data in SQLite. I'll run several queries to verify correctness ÔÇö checking drug groupings, separate pathways for different indications, and fallback behavior.
|
||||
I need you to validate the data in a SQLite database at `C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data\pathways.db`.
|
||||
|
||||
Run these validation queries using Bash with `python -c "..."` or `sqlite3` to check the pathway_nodes table. The working directory is `C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign`.
|
||||
|
||||
**Context**: This is a pre-computed pathway hierarchy for NHS patient treatment data. There are two chart types: "directory" and "indication". We just ran a full refresh and need to validate the indication charts specifically.
|
||||
|
||||
The `ids` column contains a hierarchy like: `Trust - Indication - Drug - Pathway`, separated by " - ".
|
||||
For indication charts, level 2 = indication/search_term (e.g., "rheumatoid arthritis", "asthma").
|
||||
Modified UPIDs embed the indication: `{UPID}|{search_term}`.
|
||||
|
||||
**Run these validation queries** (each as a separate `sqlite3` command or python script):
|
||||
|
||||
1. **Chart type counts**: Count nodes per chart_type to confirm both exist:
|
||||
```sql
|
||||
SELECT chart_type, COUNT(*) FROM pathway_nodes GROUP BY chart_type;
|
||||
```
|
||||
|
||||
2. **Indication node counts by date filter**: For indication charts, count nodes per date_filter_id:
|
||||
```sql
|
||||
SELECT date_filter_id, COUNT(*) FROM pathway_nodes WHERE chart_type='indication' GROUP BY date_filter_id;
|
||||
```
|
||||
|
||||
3. **Level distribution for indication charts**: Check hierarchy levels are correct (0=Root, 1=Trust, 2=Indication, 3=Drug, 4+=Pathway):
|
||||
```sql
|
||||
SELECT level, COUNT(*) FROM pathway_nodes WHERE chart_type='indication' GROUP BY level ORDER BY level;
|
||||
```
|
||||
|
||||
4. **Top indications by patient count** (level 2 = indication): Show the labels and patient counts for the biggest indication groups (default date filter 'all_6mo'):
|
||||
```sql
|
||||
SELECT labels, value FROM pathway_nodes WHERE chart_type='indication' AND date_filter_id='all_6mo' AND level=2 ORDER BY CAST(value AS INTEGER) DESC LIMIT 15;
|
||||
```
|
||||
|
||||
5. **Drugs under "rheumatoid arthritis"**: Check that drugs under RA are RA drugs. For date_filter 'all_6mo', find level 3 nodes whose parent contains "rheumatoid arthritis":
|
||||
```sql
|
||||
SELECT labels, value FROM pathway_nodes WHERE chart_type='indication' AND date_filter_id='all_6mo' AND level=3 AND ids LIKE '%rheumatoid arthritis%' ORDER BY CAST(value AS INTEGER) DESC LIMIT 10;
|
||||
```
|
||||
|
||||
6. **Drugs under "asthma"**: Similarly check asthma drugs:
|
||||
```sql
|
||||
SELECT labels, value FROM pathway_nodes WHERE chart_type='indication' AND date_filter_id='all_6mo' AND level=3 AND ids LIKE '%asthma%' ORDER BY CAST(value AS INTEGER) DESC LIMIT 10;
|
||||
```
|
||||
|
||||
7. **Fallback nodes** (no GP dx): Count fallback indication labels:
|
||||
```sql
|
||||
SELECT labels, value FROM pathway_nodes WHERE chart_type='indication' AND date_filter_id='all_6mo' AND level=2 AND labels LIKE '%(no GP dx)%' ORDER BY CAST(value AS INTEGER) DESC LIMIT 10;
|
||||
```
|
||||
|
||||
8. **Directory charts unchanged**: Verify directory chart node count is reasonable and unaffected:
|
||||
```sql
|
||||
SELECT date_filter_id, COUNT(*) FROM pathway_nodes WHERE chart_type='directory' GROUP BY date_filter_id;
|
||||
```
|
||||
|
||||
Run all 8 queries and report the results. I need to confirm:
|
||||
- Both chart types have data
|
||||
- Indication chart has correct hierarchy levels
|
||||
- RA drugs under RA indication look correct (should include ADALIMUMAB, RITUXIMAB, etc.)
|
||||
- Asthma drugs under asthma look correct (should include OMALIZUMAB, MEPOLIZUMAB, etc.)
|
||||
- Fallback nodes exist for unmatched drugs
|
||||
- Directory charts are unchanged
|
||||
All 8 validation queries passed. The indication chart data looks correct:
|
||||
|
||||
- RA drugs (ADALIMUMAB, RITUXIMAB, BARICITINIB, etc.) correctly grouped under "rheumatoid arthritis"
|
||||
- Asthma drugs (DUPILUMAB, OMALIZUMAB) correctly grouped under "asthma"
|
||||
- Fallback nodes properly labeled (e.g., "RHEUMATOLOGY (no GP dx)")
|
||||
- Directory charts unchanged
|
||||
|
||||
Now let me validate the Reflex UI compilation (Task 4.3).
|
||||
Reflex compiles successfully. Now let me mark all the Phase 4 tasks as complete in the implementation plan and update progress.txt.
|
||||
Also mark the last Task 1.1 subtask as verified:
|
||||
Now update progress.txt with this iteration's results.
|
||||
Need more context to uniquely identify the last occurrence. Let me target it specifically.
|
||||
Now append the new iteration entry at the end of the file.
|
||||
Now let me commit the changes.
|
||||
I see there are many changes beyond just my iteration's work (DESIGN_SYSTEM.md, RALPH_PROMPT.md, ralph.ps1, uv.lock, etc. from previous uncommitted changes). I should only commit the files relevant to this iteration's task: IMPLEMENTATION_PLAN.md and progress.txt. The other changes are pre-existing and not part of this task.
|
||||
Now let me verify all tasks are marked complete in IMPLEMENTATION_PLAN.md.
|
||||
No incomplete tasks remain. All tasks are `[x]`. Let me do the final completion check ÔÇö verify all completion criteria are satisfied.
|
||||
Wait, that output is confusing ÔÇö let me re-check.
|
||||
All 47 items marked complete, zero incomplete. All completion criteria are satisfied:
|
||||
|
||||
- App compiles without errors (reflex compile succeeded in 16.6s)
|
||||
- Both chart types generate pathway data (directory: 1,101, indication: 1,846)
|
||||
- Indication charts show drug-specific indication matching (49.3% match rate)
|
||||
- Drugs under same indication grouped together (validated via SQLite queries)
|
||||
- Drugs under different indications create separate pathways (42,072 modified UPIDs > 36,628 patients)
|
||||
- Fallback works for unmatched drugs (labeled with directorate "(no GP dx)")
|
||||
- Full refresh completes successfully (2,947 records in 738.4s)
|
||||
- Existing directory charts unaffected (1,101 nodes, same distribution)
|
||||
|
||||
<promise>COMPLETE</promise>
|
||||
All 47 items marked complete, zero incomplete. All completion criteria are satisfied:
|
||||
|
||||
- App compiles without errors (reflex compile succeeded in 16.6s)
|
||||
- Both chart types generate pathway data (directory: 1,101, indication: 1,846)
|
||||
- Indication charts show drug-specific indication matching (49.3% match rate)
|
||||
- Drugs under same indication grouped together (validated via SQLite queries)
|
||||
- Drugs under different indications create separate pathways (42,072 modified UPIDs > 36,628 patients)
|
||||
- Fallback works for unmatched drugs (labeled with directorate "(no GP dx)")
|
||||
- Full refresh completes successfully (2,947 records in 738.4s)
|
||||
- Existing directory charts unaffected (1,101 nodes, same distribution)
|
||||
|
||||
<promise>COMPLETE</promise>
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
@@ -169,6 +169,93 @@ If you discover a new failure pattern during your work, add it to this file.
|
||||
- **Rule**: Run `python run_dash.py` (or `python -c "from dash_app.app import app"` for import checks). The app must start without errors after EVERY task.
|
||||
- **Why**: Broken imports or circular dependencies compound across tasks. Catch them immediately.
|
||||
|
||||
---
|
||||
|
||||
## Icicle Chart Hierarchy Integrity
|
||||
|
||||
### Drug/trust/directorate filters must preserve ancestor nodes
|
||||
- **When**: Building WHERE clauses in `pathway_queries.py:load_pathway_nodes()` that filter by drug, trust, or directorate
|
||||
- **Rule**: Ancestor nodes (levels 0, 1, 2) must ALWAYS be included in query results regardless of drug/trust/directorate filters. Only apply those filters to level 3+ nodes. The icicle chart requires an unbroken parent→child chain from a single root node. If you filter out the root, trust, or directory-level nodes, Plotly will report "Multiple implied roots" and the chart will be blank.
|
||||
- **Why**: The drug filter `drug_sequence LIKE %DRUG%` was applied to all levels, dropping level 0-2 nodes that have NULL drug_sequence. This broke the icicle hierarchy.
|
||||
|
||||
### Pattern-matching IDs must be globally unique
|
||||
- **When**: Creating components with pattern-matching IDs like `{"type": "drug-fragment", "index": "..."}`
|
||||
- **Rule**: The `index` value must be globally unique across the ENTIRE layout, not just within a section. If the same entity (e.g., a drug fragment name) can appear in multiple contexts (e.g., multiple indications within a directorate), the ID must include ALL distinguishing context. Use `f"{directorate}|{search_term}|{fragment}"` not just `f"{directorate}|{fragment}"`.
|
||||
- **Why**: RIVAROXABAN appeared under multiple indications within CARDIOLOGY, causing `DuplicateIdError` on first page load.
|
||||
|
||||
---
|
||||
|
||||
## UI Structure
|
||||
|
||||
### Sidebar is for views, filter bar is for filters
|
||||
- **When**: Deciding where to place UI controls
|
||||
- **Rule**: The sidebar should contain chart VIEW selectors (Icicle, Sankey, Timeline) and navigation. Filter controls (drug selection, trust selection, date range, chart type toggle) belong in the top filter bar. Do NOT put filter triggers in the sidebar.
|
||||
- **Why**: Having 3-4 sidebar buttons that all open the same drawer was confusing. Filters should be grouped together in the persistent filter bar.
|
||||
|
||||
### Use dmc.Modal for filter selection, not dmc.Drawer
|
||||
- **When**: Building filter selection UI (drugs, trusts, directorates)
|
||||
- **Rule**: Use `dmc.Modal` (centered popup) for filter selection, not `dmc.Drawer` (slide-in panel). Each filter type gets its own modal. Trigger buttons go in the filter bar with selection count badges (e.g., "Drugs (3)").
|
||||
- **Why**: A single drawer with all filters required excessive scrolling and was confusing. Modals provide focused, per-filter selection with clear open/close behavior.
|
||||
|
||||
### Use frontend-developer agent for UX decisions
|
||||
- **When**: Designing modal layouts, filter bar organization, or other UX-sensitive components
|
||||
- **Rule**: Spawn the `frontend-developer` agent to review the data shapes and recommend optimal layout, sizing, and interaction patterns before building.
|
||||
- **Why**: Good UX requires considering the data (42 drugs, 7 trusts, 19 directorates × 163 indications) and choosing appropriate patterns for each.
|
||||
|
||||
## Tab Architecture (Phase 9)
|
||||
|
||||
### Only render the active tab's chart
|
||||
- **When**: Building the tab switching callback in `chart.py`
|
||||
- **Rule**: Check `app-state.active_tab` and ONLY compute the figure for the active tab. Return `no_update` or a cached placeholder for inactive tabs. Do NOT render all 8 charts on every filter change.
|
||||
- **Why**: Computing 8 Plotly figures on every filter change would be extremely slow. Lazy rendering is essential — only the visible chart needs computation.
|
||||
|
||||
### Chart figure functions go in src/visualization/, not dash_app/
|
||||
- **When**: Creating new chart figures (market share, lollipop, waterfall, Sankey, etc.)
|
||||
- **Rule**: Create figure builder functions in `src/visualization/` (e.g., `plotly_generator.py` or new files). Dash callbacks call these shared functions. Do NOT put Plotly figure construction logic directly in `dash_app/callbacks/`.
|
||||
- **Why**: Shared figure functions can be tested independently and reused outside Dash. Same pattern as `create_icicle_from_nodes()`.
|
||||
|
||||
### New query functions use same pattern as existing ones
|
||||
- **When**: Adding query functions to `src/data_processing/pathway_queries.py`
|
||||
- **Rule**: Follow the same pattern as `load_pathway_nodes()`: accept `db_path` parameter, use `sqlite3.connect()` with `row_factory = sqlite3.Row`, parameterized queries, return JSON-serializable dicts/lists. Add thin wrappers in `dash_app/data/queries.py`.
|
||||
- **Why**: Consistency with existing code. The thin wrapper pattern ensures DB path resolution is centralized.
|
||||
|
||||
### Parsing utilities must handle missing/null data gracefully
|
||||
- **When**: Parsing `average_spacing` HTML strings or `ids` column values
|
||||
- **Rule**: Always handle `None`, empty string `""`, and malformed data. Return sensible defaults (empty dict, empty list) rather than raising exceptions. The pathway_nodes data has NULLs at ancestor levels and some nodes may have empty statistics.
|
||||
- **Why**: Not all nodes have `average_spacing` or `average_administered` populated. Level 0-2 nodes have no drug-level statistics.
|
||||
|
||||
### Tab bar replaces sidebar chart views
|
||||
- **When**: Implementing tab infrastructure in Task 9.1
|
||||
- **Rule**: Add the tab bar INSIDE `chart_card.py`. Remove the "Chart Views" section from `sidebar.py` (Icicle/Sankey/Timeline items). The sidebar should only have the "Overview" section after this change.
|
||||
- **Why**: Tabs are the standard UI pattern for switching between chart views in the same content area. The sidebar was a temporary placeholder.
|
||||
|
||||
## Two-View Architecture (Phase 10)
|
||||
|
||||
### Patient Pathways and Trust Comparison are separate views with separate state
|
||||
- **When**: Building the two-view navigation (sidebar switching between Patient Pathways and Trust Comparison)
|
||||
- **Rule**: `active_view` in app-state controls which view is shown. Patient Pathways has `active_tab` (icicle/sankey) + drug/trust/directorate filters. Trust Comparison has `selected_comparison_directorate` (null = landing page, string = dashboard). Drug/trust/directorate filters do NOT affect Trust Comparison. Date filter + chart type toggle are GLOBAL and affect both views.
|
||||
- **Why**: The whole point of the restructure is that these views have different analytical perspectives. Leaking Patient Pathways filters into Trust Comparison would break the directorate-focused comparison.
|
||||
|
||||
### Trust Comparison queries must break down BY trust, not aggregate across trusts
|
||||
- **When**: Writing query functions for Trust Comparison charts (Task 10.6)
|
||||
- **Rule**: Existing Phase 9 query functions (e.g., `get_drug_market_share`) aggregate across trusts for same (directory, drug) pair. Trust Comparison needs the OPPOSITE — show per-trust breakdown within a single directorate. Create NEW query functions (`get_trust_market_share`, `get_trust_cost_waterfall`, etc.) rather than modifying existing ones. Existing functions are still used by Patient Pathways.
|
||||
- **Why**: Modifying existing queries would break Patient Pathways charts. The trust-comparison perspective is fundamentally different from the existing aggregate perspective.
|
||||
|
||||
### Cost Effectiveness in Trust Comparison is directorate-scoped but NOT trust-split
|
||||
- **When**: Building the Cost Effectiveness chart in the Trust Comparison 6-chart dashboard
|
||||
- **Rule**: Cost Effectiveness shows pathway costs for the selected directorate only. It does NOT split by trust — it's a single chart showing pathways within the directorate. Use existing `get_pathway_costs(directory=selected)` query, not a new trust-split variant.
|
||||
- **Why**: Splitting pathway costs by trust muddies the water — pathway analysis is more meaningful at the directorate level.
|
||||
|
||||
### Use /frontend-design skill for UX design, not frontend-developer agent
|
||||
- **When**: Designing layouts for header, sub-header, landing page, dashboard, filter placement (Task 10.1)
|
||||
- **Rule**: Use the `/frontend-design` skill (invoked via the Skill tool) for design consultation. Do NOT spawn a `frontend-developer` Task agent.
|
||||
- **Why**: User explicitly requested the frontend-design skill for design work.
|
||||
|
||||
### Trust Comparison landing page respects chart type toggle
|
||||
- **When**: Building the directorate/indication selector buttons (Task 10.7)
|
||||
- **Rule**: When chart type is "By Directory", show ~14 directorate buttons. When "By Indication", show ~32 indication buttons. The button list comes from reference-data store (available_directorates or available_indications). Changing the chart type toggle while on the landing page should refresh the button list. If a directorate was selected and user switches to indication mode, clear `selected_comparison_directorate` and return to landing page.
|
||||
- **Why**: The chart type toggle is global and the Trust Comparison view must respect it.
|
||||
|
||||
<!--
|
||||
ADD NEW GUARDRAILS BELOW as failures are observed during the loop.
|
||||
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Ralph Wiggum Loop - Reflex UI Redesign variant.
|
||||
Ralph Wiggum Loop - Dash Migration variant.
|
||||
|
||||
.DESCRIPTION
|
||||
Outer loop for iterative Reflex frontend development.
|
||||
Outer loop for iterative Dash frontend development (Reflex -> Dash migration).
|
||||
Each iteration spawns a fresh `claude --print` invocation.
|
||||
Memory persists via filesystem only: git commits, progress.txt, IMPLEMENTATION_PLAN.md, guardrails.md.
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
- Same error repeated N consecutive iterations (stuck)
|
||||
|
||||
.PARAMETER Model
|
||||
Claude model to use. Default: "sonnet".
|
||||
Claude model to use. Default: "opus".
|
||||
|
||||
.PARAMETER BranchName
|
||||
Optional git branch name. If provided, creates/checks out the branch before starting.
|
||||
@@ -27,10 +27,10 @@
|
||||
Number of consecutive iterations with the same error before circuit breaker trips. Default: 3.
|
||||
|
||||
.EXAMPLE
|
||||
.\ralph.ps1 -Model "sonnet" -BranchName "feature/ui-redesign"
|
||||
.\ralph.ps1 -Model "opus" -BranchName "feature/dash-migration"
|
||||
|
||||
.EXAMPLE
|
||||
.\ralph.ps1 -Model "opus" -MaxNoProgress 2
|
||||
.\ralph.ps1 -Model "sonnet" -MaxNoProgress 2
|
||||
#>
|
||||
|
||||
param(
|
||||
@@ -45,7 +45,6 @@ $ErrorActionPreference = "Stop"
|
||||
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
$promptFile = Join-Path $scriptDir "RALPH_PROMPT.md"
|
||||
$planFile = Join-Path $scriptDir "IMPLEMENTATION_PLAN.md"
|
||||
$designFile = Join-Path $scriptDir "DESIGN_SYSTEM.md"
|
||||
$guardrailsFile = Join-Path $scriptDir "guardrails.md"
|
||||
$progressFile = Join-Path $scriptDir "progress.txt"
|
||||
$logDir = Join-Path $scriptDir "logs"
|
||||
@@ -62,11 +61,6 @@ if (-not (Test-Path $planFile)) {
|
||||
exit 1
|
||||
}
|
||||
|
||||
if (-not (Test-Path $designFile)) {
|
||||
Write-Error "DESIGN_SYSTEM.md not found at $designFile"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if (-not (Test-Path $guardrailsFile)) {
|
||||
Write-Warning "guardrails.md not found at $guardrailsFile - loop may miss known failure patterns"
|
||||
}
|
||||
@@ -147,7 +141,7 @@ if (Test-Path $progressFile) {
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "===== Ralph Wiggum Loop (Reflex UI) =====" -ForegroundColor Cyan
|
||||
Write-Host "===== Ralph Wiggum Loop (Dash Migration) =====" -ForegroundColor Cyan
|
||||
Write-Host "Model: $Model | Runs until COMPLETE" -ForegroundColor Cyan
|
||||
Write-Host "Circuit breakers: no-progress=$MaxNoProgress, same-error=$MaxSameError" -ForegroundColor Cyan
|
||||
if ($BranchName) { Write-Host "Branch: $BranchName" -ForegroundColor Cyan }
|
||||
@@ -334,7 +328,7 @@ while ($true) {
|
||||
if ($outputString -match "<promise>COMPLETE</promise>") {
|
||||
Write-Host ""
|
||||
Write-Host "===== COMPLETE =====" -ForegroundColor Green
|
||||
Write-Host "UI redesign finished after $i iteration(s) this run ($totalIteration total)." -ForegroundColor Green
|
||||
Write-Host "Dash migration finished after $i iteration(s) this run ($totalIteration total)." -ForegroundColor Green
|
||||
exit 0
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
@echo off
|
||||
setlocal EnableDelayedExpansion
|
||||
|
||||
title HCD Patient Pathway Analysis
|
||||
echo.
|
||||
echo ==========================================
|
||||
echo HCD Patient Pathway Analysis
|
||||
echo NHS High-Cost Drug Treatment Pathways
|
||||
echo ==========================================
|
||||
echo.
|
||||
|
||||
:: -------------------------------------------------------
|
||||
:: First run vs subsequent run
|
||||
:: -------------------------------------------------------
|
||||
if exist ".venv\Scripts\activate.bat" (
|
||||
echo Ready to launch.
|
||||
goto :run_app
|
||||
)
|
||||
|
||||
echo First-time setup detected. This will:
|
||||
echo 1. Install uv (Python package manager)
|
||||
echo 2. Install Python 3.12 and dependencies
|
||||
echo 3. Build and start the application
|
||||
echo.
|
||||
echo Requires internet access. May take 3-5 minutes.
|
||||
echo.
|
||||
pause
|
||||
|
||||
:: -------------------------------------------------------
|
||||
:: Install uv if not available
|
||||
:: -------------------------------------------------------
|
||||
where uv >nul 2>&1
|
||||
if %ERRORLEVEL% neq 0 (
|
||||
echo.
|
||||
echo [1/3] Installing uv...
|
||||
powershell -ExecutionPolicy Bypass -Command "irm https://astral.sh/uv/install.ps1 | iex"
|
||||
|
||||
set "PATH=%USERPROFILE%\.local\bin;%PATH%"
|
||||
set "PATH=%USERPROFILE%\.cargo\bin;%PATH%"
|
||||
|
||||
where uv >nul 2>&1
|
||||
if !ERRORLEVEL! neq 0 (
|
||||
echo.
|
||||
echo ERROR: uv installation failed.
|
||||
echo Try installing manually: https://docs.astral.sh/uv/getting-started/installation/
|
||||
echo Then re-run this script.
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
echo uv installed successfully.
|
||||
) else (
|
||||
echo [1/3] uv already installed.
|
||||
)
|
||||
|
||||
:: -------------------------------------------------------
|
||||
:: Sync dependencies
|
||||
:: -------------------------------------------------------
|
||||
echo.
|
||||
echo [2/3] Installing Python and dependencies...
|
||||
echo (First run only — please wait)
|
||||
echo.
|
||||
|
||||
uv sync
|
||||
if %ERRORLEVEL% neq 0 (
|
||||
echo.
|
||||
echo ERROR: Dependency installation failed.
|
||||
echo Check your internet connection and try again.
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
echo.
|
||||
echo Setup complete.
|
||||
|
||||
:: -------------------------------------------------------
|
||||
:: Run application
|
||||
:: -------------------------------------------------------
|
||||
:run_app
|
||||
echo.
|
||||
echo [3/3] Starting application...
|
||||
echo.
|
||||
echo App will open at: http://localhost:3000
|
||||
echo First launch builds the frontend (~60 seconds).
|
||||
echo Subsequent launches are fast.
|
||||
echo.
|
||||
echo To stop: close this window or press Ctrl+C
|
||||
echo ==========================================
|
||||
echo.
|
||||
|
||||
start "" cmd /c "timeout /t 8 /nobreak >nul && start http://localhost:3000"
|
||||
|
||||
uv run reflex run
|
||||
if %ERRORLEVEL% neq 0 (
|
||||
echo.
|
||||
echo Application exited with an error.
|
||||
echo Try deleting .web\ and running again.
|
||||
echo.
|
||||
pause
|
||||
)
|
||||
Reference in New Issue
Block a user