HighCostDrugsDemo/progress.txt

# Progress Log — Reflex → Dash Migration

## Project Context

Migrating the HCD Analysis frontend from Reflex to Dash (Plotly) + Dash Mantine Components. Pipeline/analysis logic in `src/` is untouched, but shared utilities (data queries, figure construction) should be added TO `src/` so Dash callbacks call into them rather than duplicating code.

**Previous state**: Fully working Reflex app with pre-computed pathway architecture (SQLite), dual chart types (directory + indication), drug-aware indication matching. All pipeline work is done.

**New goal**: Replace Reflex with Dash for better control over layout, CSS, and component behavior. Add a dmc.Drawer-based "card browser" for drug/indication selection organized by clinical directorate.

## Key Data Patterns

### SQLite pathway_nodes table
- ~3,600 rows across 12 datasets (6 date filters × 2 chart types)
- Key columns: `parents, ids, labels, level, value, cost, costpp, cost_pp_pa, colour, first_seen, last_seen, first_seen_parent, last_seen_parent, average_spacing, trust_name, directory, drug_sequence, chart_type, date_filter_id`
- Level 0 = Root, Level 1 = Trust, Level 2 = Directory/Indication, Level 3 = Drug, Level 4+ = Pathway
- `chart_type`: "directory" or "indication"
- `date_filter_id`: "all_6mo" (default), "all_12mo", "1yr_6mo", "1yr_12mo", "2yr_6mo", "2yr_12mo"
- UNIQUE constraint: (date_filter_id, chart_type, ids)

### DimSearchTerm.csv (for card browser)
- Located at `data/DimSearchTerm.csv`
- Columns: Search_Term, CleanedDrugName (pipe-separated drug fragments), PrimaryDirectorate
- ~165 rows; some Search_Terms appear twice (e.g., "diabetes" under DIABETIC MEDICINE and OPHTHALMOLOGY)
- Drug fragments are UPPERCASE substrings matched against standardized drug names
- SEARCH_TERM_MERGE_MAP in `src/data_processing/diagnosis_lookup.py` merges asthma variants: {"allergic asthma": "asthma", "severe persistent allergic asthma": "asthma"}

### Data loading logic to extract
- `pathways_app/pathways_app.py` lines 407-488: `load_data()` — loads available drugs, directorates, indications, total records, last updated from SQLite
- `pathways_app/pathways_app.py` lines 490-642: `load_pathway_data()` — queries pathway_nodes with date_filter_id + chart_type + optional drug/directory filters
- `pathways_app/pathways_app.py` lines 769-920: `icicle_figure` — builds go.Icicle with 10-field customdata, NHS colorscale, texttemplate, hovertemplate

### CSS from 01_nhs_classic.html
- Lines 8-314 contain the full CSS (copy to dash_app/assets/nhs.css)
- Google Fonts: `Source Sans 3` weights 300,400,600,700,900
- CSS variables: `--nhs-blue: #005EB8`, `--nhs-dark-blue: #003087`, `--nhs-light-blue: #41B6E6`, etc.
- Key classes: `.top-header`, `.sidebar`, `.main`, `.kpi-row`, `.kpi-card`, `.filter-bar`, `.toggle-pill`, `.chart-card`, `.chart-tab`, `.page-footer`
- Remove `.icicle`, `.icicle__row`, `.icicle__cell`, `.lvl-*` classes — those are mock chart CSS, Plotly handles the real chart

### Dash-specific patterns
- State via `dcc.Store`: 3 stores (app-state, chart-data, reference-data)
- Callbacks: unidirectional flow (filter change → app-state → chart-data → UI components)
- DMC components: `dmc.MantineProvider` wraps everything, `dmc.Drawer` for card browser
- Pattern-matching callbacks: `{"type": "drug-chip", "index": drug_name}` for dynamic drug chip selection
- Assets auto-served from `dash_app/assets/` directory

### Database path from dash_app/
- From `dash_app/data/queries.py`: `Path(__file__).resolve().parents[2] / "data" / "pathways.db"`
- From `dash_app/data/card_browser.py`: same pattern for `data/DimSearchTerm.csv`

### Existing src/ code to build on (not duplicate)
- `src/visualization/plotly_generator.py` already has `create_icicle_figure(ice_df, title)` that takes a DataFrame with columns like `"First seen"`, `"Last seen (Parent)"` (with spaces). The Reflex AppState `icicle_figure` (pathways_app.py:769) takes list-of-dicts with keys like `first_seen`, `last_seen_parent` (underscores). For Dash, add a NEW function `create_icicle_from_nodes(nodes, title)` that accepts list-of-dicts. Don't modify the existing DataFrame-based function.
- `src/data_processing/database.py` has `DatabaseManager` class. Add standalone query functions here (or a new `pathway_queries.py`) so Dash and Reflex share the same SQL.
- The existing `create_icicle_figure` uses Viridis colorscale; the Reflex version uses NHS blue gradient. The Dash version should use the NHS blue gradient from pathways_app.py.

### Architecture decision: shared code in src/
- Pipeline/analysis logic is OFF LIMITS: pathway_pipeline.py, transforms.py, diagnosis_lookup.py (matching), pathway_analyzer.py, statistics.py, refresh_pathways.py
- Shared utilities are ENCOURAGED to add to src/: plotly_generator.py (new dict-based function), database.py (query functions)
- dash_app/data/queries.py should be a thin wrapper that calls into src/

## Iteration Log

## Iteration 1 — 2026-02-06
### Task: Phase 0 — Tasks 0.1 + 0.2 (Project Scaffolding + CSS Extraction)
### Why this task:
- This is the first iteration. Phase 0 scaffolding is the foundation everything else depends on.
- Tasks 0.1 and 0.2 are tightly coupled (CSS needs the assets dir from 0.1), so both done together.
### Status: COMPLETE
### What was done:
- Created `dash_app/` directory with subdirectories: `assets/`, `data/`, `components/`, `callbacks/`, `utils/`
- Created `__init__.py` in all packages
- Created `run_dash.py` entry point at project root
- Updated `pyproject.toml` with `dash>=2.14.0` and `dash-mantine-components>=0.14.0`
- Ran `uv sync` — installed Dash 4.0.0 and DMC 2.5.1 (newer than plan expected)
- Created `dash_app/app.py` with `MantineProvider` wrapper, 3 `dcc.Store` components (app-state, chart-data, reference-data), and placeholder layout
- Extracted CSS from `01_nhs_classic.html` into `dash_app/assets/nhs.css` (7.5KB)
  - Added Google Fonts `@import` for Source Sans 3
  - Removed mock icicle chart CSS (`.icicle`, `.icicle__row`, `.icicle__cell`, `.lvl-*`)
  - Kept all real component CSS: header, sidebar, KPI, filter bar, chart card, footer, responsive
### Validation results:
- Tier 1 (Code): `python -c "from dash_app.app import app"` — OK, layout type is MantineProvider
- Tier 1 (App starts): `python run_dash.py` — serves at http://127.0.0.1:8050/
- CSS: nhs.css (7,491 bytes) found in Dash assets folder, auto-served
### Files changed:
- `pyproject.toml` — added dash + dmc dependencies
- `run_dash.py` — new entry point
- `dash_app/__init__.py` — new package
- `dash_app/app.py` — new app with MantineProvider + stores + placeholder
- `dash_app/assets/nhs.css` — extracted from 01_nhs_classic.html
- `dash_app/data/__init__.py` — empty package
- `dash_app/components/__init__.py` — empty package
- `dash_app/callbacks/__init__.py` — empty package
- `dash_app/utils/__init__.py` — empty package
### Committed: 1c3ece6 "feat: create dash_app skeleton with nhs.css and MantineProvider (Phase 0)"
### Patterns discovered:
- Dash 4.0.0 and DMC 2.5.1 installed (plan said 2.x and 0.14.x). The API is compatible — `MantineProvider`, `Drawer`, `Accordion`, `Chip` all available. DMC 2.x is based on Mantine v7.
- `dmc.MantineProvider(children=[...])` works for wrapping the layout in both DMC versions.
- `suppress_callback_exceptions=True` needed in Dash app since callbacks reference IDs not yet in layout during phased development.
### Next iteration should:
- Start Phase 1: Task 1.1 — Create shared data access functions
- Read `pathways_app/pathways_app.py` lines 407-642 to extract `load_data()` and `load_pathway_data()` logic
- Create shared functions in `src/data_processing/pathway_queries.py` (database.py is already used for DatabaseManager)
- Create thin wrapper `dash_app/data/queries.py` that calls the shared functions
- Verify with `python -c "from dash_app.data.queries import load_initial_data; print(load_initial_data())"`
### Blocked items:
- None

## Iteration 2 — 2026-02-06
### Task: Phase 1 — Task 1.1 (Create shared data access functions)
### Why this task:
- Phase 0 complete in iteration 1; Phase 1 is next in dependency order
- progress.txt from iteration 1 explicitly recommended this task
- All UI components and callbacks (Phases 2-5) depend on having data access working
### Status: COMPLETE
### What was done:
- Created `src/data_processing/pathway_queries.py` with two shared functions:
  - `load_initial_data(db_path)` — returns available drugs (42), directorates (14), indications (32), total_records, last_updated
  - `load_pathway_nodes(db_path, filter_id, chart_type, selected_drugs, selected_directorates)` — returns nodes list, unique_patients, total_drugs, total_cost, last_updated
- Both functions extracted directly from AppState methods in pathways_app.py (lines 407-642), with Reflex `self.*` references replaced by function parameters
- All return values are plain dicts/lists — JSON-serializable for dcc.Store
- Created thin wrapper `dash_app/data/queries.py` that resolves DB_PATH and delegates to shared functions
- Used separate file (pathway_queries.py) rather than adding to database.py because database.py is connection management (240 lines), queries are a distinct concern
### Validation results:
- Tier 1 (Code): `python -c "from dash_app.data.queries import load_initial_data"` — OK (requires uv run for .pth file)
- Tier 1 (App starts): `from dash_app.app import app` — OK, layout type is MantineProvider
- Tier 3 (Functional):
  - `load_initial_data()`: 42 drugs, 14 directorates, 32 indications, last_updated=2026-02-06T00:08:55
  - `load_pathway_data("all_6mo", "directory")`: 293 nodes, 11,118 patients, 39 drugs, £130.5M cost
  - `load_pathway_data("all_6mo", "indication")`: 438 nodes, 11,252 patients
  - `load_pathway_data("all_6mo", "directory", selected_drugs=["ADALIMUMAB"])`: 70 nodes (drug filter works)
### Files changed:
- `src/data_processing/pathway_queries.py` — NEW: shared query functions
- `dash_app/data/queries.py` — NEW: thin Dash wrapper with DB_PATH resolution
- `IMPLEMENTATION_PLAN.md` — Task 1.1 marked [x]
### Committed: b71748f "feat: add shared pathway query functions for Dash data access (Task 1.1)"
### Patterns discovered:
- `src/` is on sys.path only when using `uv run` (via .pth file created by setup_dev.py). Running `python` directly won't find `data_processing` module. Always use `uv run python` for testing.
- `total_records` from `pathway_refresh_log` returns 0 — the refresh log's `source_row_count` field appears empty despite `completed_at` having a value. This is cosmetic — the KPI can use `unique_patients` from chart-data instead.
- Drug filtering correctly includes nodes with NULL drug_sequence (root, trust, directory levels) alongside matching drug nodes. Root node patient count becomes 0 when drug filter is active — this matches Reflex behavior.
### Next iteration should:
- Start Task 1.2 — Build directorate card tree from DimSearchTerm.csv
- Create `dash_app/data/card_browser.py` with `build_directorate_tree()` and `get_all_drugs()`
- Read `data/DimSearchTerm.csv` to understand the data format
- Import SEARCH_TERM_MERGE_MAP from `data_processing.diagnosis_lookup` for asthma normalization
- Remember: drug fragments in CleanedDrugName are UPPERCASE substrings, not exact matches
### Blocked items:
- None

## Iteration 3 — 2026-02-06
### Task: Phase 1 — Task 1.2 (Build directorate card tree from DimSearchTerm.csv)
### Why this task:
- Phase 1 data access layer needs both query functions (done in 1.1) AND the card browser data (this task)
- Progress.txt from iteration 2 explicitly recommended this task
- The drawer component (Phase 4) depends on this data structure
### Status: COMPLETE
### What was done:
- Created `dash_app/data/card_browser.py` with two functions:
  - `build_directorate_tree()` — reads DimSearchTerm.csv, applies SEARCH_TERM_MERGE_MAP, returns nested dict: {PrimaryDirectorate: {Search_Term: [drug_fragment, ...]}}
  - `get_all_drugs()` — delegates to `load_initial_data()` from queries.py (no SQL duplication)
- SEARCH_TERM_MERGE_MAP correctly imported from `data_processing.diagnosis_lookup`
- Asthma variants ("allergic asthma", "severe persistent allergic asthma") merge into single "asthma" entry with combined/deduplicated drug fragments
- Tree uses sets during construction to deduplicate drug fragments from merged terms, then converts to sorted lists
### Validation results:
- Tier 1 (Code): `from dash_app.data.card_browser import build_directorate_tree` — OK
- Tier 1 (App starts): `from dash_app.app import app` — OK, layout type is MantineProvider
- Tier 3 (Functional):
  - `build_directorate_tree()`: 19 directorates, correctly nested dict
  - CARDIOLOGY has 10 indications, CLINICAL HAEMATOLOGY has many more
  - Asthma merge verified: only "asthma" present (not "allergic asthma" or "severe persistent allergic asthma"), OMALIZUMAB included in merged drug list
  - `get_all_drugs()`: 42 drugs from pathway_nodes level 3
### Files changed:
- `dash_app/data/card_browser.py` — NEW: directorate tree builder + drug list accessor
- `IMPLEMENTATION_PLAN.md` — Task 1.2 marked [x]
### Committed: 7654942 "feat: add directorate card tree builder for drug browser drawer (Task 1.2)"
### Patterns discovered:
- `get_all_drugs()` uses a lazy import (`from dash_app.data.queries import load_initial_data`) to avoid circular imports since both modules are in `dash_app/data/`
- Drug fragments in DimSearchTerm.csv are already UPPERCASE in the CSV, but `.upper()` is applied defensively
- "diabetes" appears twice in DimSearchTerm.csv (DIABETIC MEDICINE and OPHTHALMOLOGY) — this is handled naturally by the directorate grouping (same search_term under two different directorates)
### Next iteration should:
- Start Phase 2: Task 2.1 — Header + sidebar components
- Read `01_nhs_classic.html` to get the exact HTML structure, CSS classes, and SVG icons
- Create `dash_app/components/header.py` with `make_header()` function
- Create `dash_app/components/sidebar.py` with `make_sidebar()` function
- Use CSS classes from nhs.css (`.top-header`, `.sidebar`, etc.) — NOT inline styles
- Header needs placeholder `html.Span` with IDs for callback updates: `id="header-record-count"`, `id="header-last-updated"`
- Sidebar needs click targets for drawer open (Drug Selection, Indications items)
### Blocked items:
- None

## Iteration 4 — 2026-02-06
### Task: Phase 2 — Task 2.1 (Header + sidebar components)
### Why this task:
- Phases 0 and 1 complete. Phase 2 is the static layout phase, and 2.1 is the first task.
- Previous iteration (3) explicitly recommended this as the next task.
- Header and sidebar are foundational layout components — everything else renders inside this shell.
### Status: COMPLETE
### What was done:
- Created `dash_app/components/header.py` with `make_header()` function:
  - Top bar with `.top-header` class, NHS logo badge, "HCD Analysis" title, breadcrumb
  - Data freshness indicators: green status dot + `html.Span(id="header-record-count")` + `html.Span(id="header-last-updated")` for callback updates
  - Structure matches `01_nhs_classic.html` lines 319-333 exactly
- Created `dash_app/components/sidebar.py` with `make_sidebar()` function:
  - 7 navigation items across 2 sections (Analysis: 5 items, Reports: 2 items)
  - SVG icons from the HTML concept, embedded via data URI `html.Img` elements (Dash lacks inline SVG support)
  - Added `.sidebar__icon` CSS class matching `.sidebar__item svg` sizing (18x18px)
  - "Drug Selection" (`id="sidebar-drug-selection"`) and "Indications" (`id="sidebar-indications"`) have IDs + `n_clicks=0` for drawer callbacks in Phase 4
  - Footer: "NHS Norfolk & Waveney ICB / High Cost Drugs Programme"
- Updated `dash_app/app.py`:
  - Imports and uses `make_header()` and `make_sidebar()`
  - Layout: MantineProvider → [3 stores, Header, Nav(sidebar), Main(placeholder)]
  - Main content area uses `html.Main(className="main")` — ready for KPIs/filter bar/chart card in Task 2.2
### Validation results:
- Tier 1 (Code): All imports OK — `from dash_app.components.header import make_header`, sidebar, app all pass
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 2 (Layout): Programmatic check confirms structure: MantineProvider with 6 children (3 stores + Header(top-header) + Nav(sidebar) + Main(main))
### Files changed:
- `dash_app/components/header.py` — NEW: header component
- `dash_app/components/sidebar.py` — NEW: sidebar component with SVG icons
- `dash_app/app.py` — Updated to import and assemble header + sidebar
- `dash_app/assets/nhs.css` — Added `.sidebar__icon` rule
- `IMPLEMENTATION_PLAN.md` — Task 2.1 marked [x]
### Committed: bdc1690 "feat: add header and sidebar components for Dash layout (Task 2.1)"
### Patterns discovered:
- Dash doesn't support inline SVG elements natively (no `html.Svg` or similar). Workaround: embed SVG as data URI in `html.Img` elements using `urllib.parse.quote()`. Limitation: `stroke="currentColor"` doesn't inherit from parent CSS, so icons appear in their default SVG color (black) rather than inheriting the sidebar item color. This is cosmetic and acceptable for now.
- Dash serves a minimal HTML shell; all React components render client-side. Don't test for CSS classes in the HTTP response — use programmatic layout inspection instead.
- `html.A` elements in Dash accept `n_clicks=0` property, making them callback-compatible for click events (needed for sidebar drawer triggers).
### Next iteration should:
- Start Task 2.2 — Main content area: KPI row + filter bar + chart card
- Create three component files:
  - `dash_app/components/kpi_row.py` with `make_kpi_row()` — 4 KPI cards with IDs for callback updates
  - `dash_app/components/filter_bar.py` with `make_filter_bar()` — chart type toggle pills + date dropdowns
  - `dash_app/components/chart_card.py` with `make_chart_card()` — chart area with tabs + `dcc.Graph(id="pathway-chart")`
- Read `01_nhs_classic.html` lines 377-516 for the exact structure
- Use CSS classes: `.kpi-row`, `.kpi-card`, `.filter-bar`, `.toggle-pill`, `.chart-card`, `.chart-tab`
- Filter bar needs: chart type toggle pills (By Directory / By Indication), Initiated dropdown, Last seen dropdown
- Chart card needs: title, dynamic subtitle (hierarchy label), tab row (Icicle active, Sankey/Timeline disabled), `dcc.Graph`
- KPI card IDs: `id="kpi-patients"`, `id="kpi-drugs"`, `id="kpi-cost"`, `id="kpi-match"`
- Toggle pill IDs: `id="chart-type-directory"`, `id="chart-type-indication"`
- Filter IDs: `id="filter-initiated"`, `id="filter-last-seen"`
### Blocked items:
- None

## Iteration 5 — 2026-02-06
### Task: Phase 2 — Task 2.2 (Main content area: KPI row + filter bar + chart card)
### Why this task:
- Phase 2.1 (header + sidebar) complete. Task 2.2 fills the main content area with actual components.
- Previous iteration (4) explicitly recommended this as the next task.
- The main area was a placeholder — this task adds the three core content sections.
### Status: COMPLETE
### What was done:
- Created `dash_app/components/kpi_row.py` with `make_kpi_row()`:
  - 4 KPI cards in a grid: Unique Patients, Drug Types, Total Cost, Indication Match
  - Fourth card has `.kpi-card--green` modifier for green top border
  - Each value has an ID for callback updates: `kpi-patients`, `kpi-drugs`, `kpi-cost`, `kpi-match`
  - Default value is "—" (em dash) — populated by callbacks in Phase 3
  - Helper `_kpi_card()` builds individual cards with CSS classes from nhs.css
- Created `dash_app/components/filter_bar.py` with `make_filter_bar()`:
  - Chart type toggle pills: "By Directory" (active) / "By Indication" with `.toggle-pill` CSS
  - Button IDs: `chart-type-directory`, `chart-type-indication` with `n_clicks=0` for callbacks
  - Initiated dropdown via `dcc.Dropdown`: All years (default), Last 2 years, Last 1 year
  - Last seen dropdown via `dcc.Dropdown`: Last 6 months (default), Last 12 months
  - Used `dcc.Dropdown` (not `html.Select`) for native Dash callback compatibility
  - Drug/directorate dropdowns omitted — those go in the drawer (Phase 4)
  - Added `.filter-dropdown` CSS to nhs.css for dcc.Dropdown sizing within filter bar
- Created `dash_app/components/chart_card.py` with `make_chart_card()`:
  - Card header with title "Patient Pathway Visualization" and dynamic subtitle (`id="chart-subtitle"`)
  - Default subtitle: "Trust → Directorate → Drug → Patient Pathway"
  - Tab row: Icicle (active), Sankey (disabled), Timeline (disabled)
  - `dcc.Graph(id="pathway-chart")` with min-height 500px and display config
  - Sankey/Timeline tabs are `disabled=True` placeholders for future expansion
- Updated `dash_app/app.py`:
  - Added imports for `make_kpi_row`, `make_filter_bar`, `make_chart_card`
  - Main content area now: `[make_kpi_row(), make_filter_bar(), make_chart_card()]`
  - Removed placeholder text
### Validation results:
- Tier 1 (Code): All imports pass — kpi_row, filter_bar, chart_card, app all import OK
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 2 (Layout): Verified programmatically:
  - MantineProvider has 6 children (3 stores + header + sidebar + main)
  - Main has 3 children: Section(kpi-row), Section(filter-bar), Section(chart-card)
  - All 13 expected IDs found: app-state, chart-data, reference-data, kpi-patients, kpi-drugs, kpi-cost, kpi-match, chart-type-directory, chart-type-indication, filter-initiated, filter-last-seen, pathway-chart, chart-subtitle
### Files changed:
- `dash_app/components/kpi_row.py` — NEW: 4 KPI cards with callback-updatable values
- `dash_app/components/filter_bar.py` — NEW: toggle pills + date filter dropdowns
- `dash_app/components/chart_card.py` — NEW: chart header + tabs + dcc.Graph
- `dash_app/app.py` — Updated: imports and assembles new components in main area
- `dash_app/assets/nhs.css` — Added: `.filter-dropdown` CSS for dcc.Dropdown styling
- `IMPLEMENTATION_PLAN.md` — Task 2.2 items marked [x]
### Committed: 307563b "feat: add KPI row, filter bar, and chart card components (Task 2.2)"
### Patterns discovered:
- `dcc.Dropdown` must be used instead of `html.Select` for Dash callback compatibility. `html.Select` doesn't have a `value` property that triggers callbacks. `dcc.Dropdown` renders its own React-Select widget that needs CSS customization to match the `.filter-select` styling from the HTML concept.
- `html.Button` with `n_clicks=0` works for toggle pills — callbacks can use `callback_context.triggered_id` to determine which button was clicked.
- `dcc.Graph` accepts a `config` dict for mode bar customization (removing lasso/select, hiding logo).
- Unicode arrow `\u2192` (→) used in chart subtitle for cross-platform compatibility.
### Next iteration should:
- Start Task 2.3 — Footer + full page assembly
- Create `dash_app/components/footer.py` with `make_footer()` using `.page-footer` CSS class
- Add footer to the main content area in `app.py` (below chart card)
- Verify full page layout matches `01_nhs_classic.html` visually
- The 3 dcc.Store components are already in place (added in Phase 0), so 2.3's store bullet is already done
- After 2.3, Phase 2 will be complete — Phase 3 (Core Callbacks) is next
- Phase 3.1 will need: reference data loading callback (on page load), filter state management callback (toggle pills + dropdowns → app-state), and `register_callbacks(app)` wiring
### Blocked items:
- None

## Iteration 6 — 2026-02-06
### Task: Phase 2 — Task 2.3 (Footer + full page assembly)
### Why this task:
- Tasks 2.1 and 2.2 complete. Task 2.3 is the final piece of Phase 2.
- Previous iteration (5) explicitly recommended this as the next task.
- Completing this marks Phase 2 done, enabling Phase 3 (Core Callbacks).
### Status: COMPLETE
### What was done:
- Created `dash_app/components/footer.py` with `make_footer()`:
  - Returns `html.Footer(className="page-footer")` matching 01_nhs_classic.html line 520
  - Text: "NHS Norfolk and Waveney ICB — High Cost Drugs Analysis"
- Updated `dash_app/app.py`:
  - Added `make_footer` import
  - Added `make_footer()` as 4th child in main content area (after chart card)
- Full page layout now complete: MantineProvider → [3 stores, Header, Sidebar, Main(KPIs, Filter bar, Chart card, Footer)]
- Note: dcc.Store components and page assembly were already done in earlier iterations. Task 2.3 effectively just needed the footer component.
### Validation results:
- Tier 1 (Code): `from dash_app.components.footer import make_footer` — OK, returns Footer with className="page-footer"
- Tier 1 (App starts): `from dash_app.app import app` — OK, no errors
- Tier 2 (Layout): Verified programmatically:
  - MantineProvider: 6 children (3 stores + Header(top-header) + Nav(sidebar) + Main(main))
  - Main: 4 children (Section(kpi-row), Section(filter-bar), Section(chart-card), Footer(page-footer))
  - All component IDs present and correct
### Files changed:
- `dash_app/components/footer.py` — NEW: footer component
- `dash_app/app.py` — Updated: import + added footer to main area
- `IMPLEMENTATION_PLAN.md` — Task 2.3 marked [x]
### Committed: 3568e03 "feat: add footer component and complete Phase 2 static layout (Task 2.3)"
### Patterns discovered:
- Phase 2 was incrementally built across iterations 4-6. The layout assembly (app.py) evolved naturally — each iteration added its components. No "big bang" assembly step was needed.
### Next iteration should:
- Start Phase 3: Task 3.1 — Reference data loading + filter state management
- Create `dash_app/callbacks/filters.py` with two callbacks:
  1. `load_reference_data`: fires on page load, calls `queries.load_initial_data()`, populates `reference-data` store + header record count + header last updated
  2. `update_app_state`: fires when chart-type toggle buttons or date dropdowns change, computes `date_filter_id` from initiated+last_seen combo, updates `app-state` store, manages toggle pill active classes
- Create `dash_app/callbacks/__init__.py` with `register_callbacks(app)` function
- Wire `register_callbacks(app)` call in `app.py` after layout definition
- Key IDs to use as inputs: `chart-type-directory`, `chart-type-indication` (n_clicks), `filter-initiated` (value), `filter-last-seen` (value)
- Key IDs to use as outputs: `app-state` (data), `reference-data` (data), `header-record-count` (children), `header-last-updated` (children)
- Date filter ID mapping: initiated "all"/"2yr"/"1yr" + last_seen "6mo"/"12mo" → e.g. "all_6mo", "1yr_12mo"
- For toggle pills, use Output for className to toggle `toggle-pill--active` class
- Read pathways_app/pathways_app.py lines 407-488 to understand the reference data structure
### Blocked items:
- None

## Iteration 7 — 2026-02-06
### Task: Phase 3 — Task 3.1 (Reference data loading + filter state management)
### Why this task:
- Phases 0-2 complete. Phase 3 is next in dependency order.
- Previous iteration (6) explicitly recommended this as the next task.
- All subsequent callbacks (3.2 chart data, 3.3 KPIs, 3.4 icicle chart) depend on filter state management from 3.1.
### Status: COMPLETE
### What was done:
- Created `dash_app/callbacks/filters.py` with two callbacks:
  1. `load_reference_data`: fires on page load via `dcc.Location` pathname Input, calls `queries.load_initial_data()`, populates `reference-data` store + header record count ("Data loaded" or "N records") + header last updated (date string)
  2. `update_app_state`: fires on chart-type toggle clicks (n_clicks) + date dropdown changes (value), uses `ctx.triggered_id` to determine chart type, computes `date_filter_id` from `{initiated}_{last_seen}`, updates `app-state` store, toggles `toggle-pill--active` CSS class on pills
- Created `dash_app/callbacks/__init__.py` with `register_callbacks(app)` function that imports and registers all callback modules
- Wired `register_callbacks(app)` in `app.py` after layout definition
- Added `dcc.Location(id="url", refresh=False)` to layout for page-load trigger
- Initially used `app-state` as Input for `load_reference_data` but changed to `dcc.Location` to avoid re-firing on every filter change (would cause circular updates)
### Validation results:
- Tier 1 (Code): All imports pass — `filters.py`, `callbacks/__init__.py`, `app.py` all OK
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional):
  - Both callbacks registered in `app.callback_map`: reference-data.data, app-state.data
  - Reference data loads correctly: 42 drugs, 14 directorates, 32 indications, last updated 2026-02-06
  - Date filter ID computation: all 6 combinations verified correct
  - Toggle pill class toggling: active = "toggle-pill toggle-pill--active", inactive = "toggle-pill"
### Files changed:
- `dash_app/callbacks/filters.py` — NEW: load_reference_data + update_app_state callbacks
- `dash_app/callbacks/__init__.py` — Updated: register_callbacks(app) with filter callback registration
- `dash_app/app.py` — Updated: added dcc.Location, import + call register_callbacks(app)
- `IMPLEMENTATION_PLAN.md` — Task 3.1 items marked [x]
### Committed: eda35c7 "feat: add reference data loading and filter state callbacks (Task 3.1)"
### Patterns discovered:
- Dash 4.0.0 uses `ctx.triggered_id` (direct string) rather than the older `callback_context.triggered[0]["prop_id"]` parsing pattern. Simpler and more reliable.
- Using `app-state` as Input for reference data loading creates circular updates: `update_app_state` writes app-state → `load_reference_data` fires → queries DB unnecessarily. Use `dcc.Location(id="url")` pathname as the page-load trigger instead.
- `dcc.Location(refresh=False)` fires once on page load with `pathname="/"` — perfect for initialization callbacks.
- `total_records` from `pathway_refresh_log.source_row_count` is still 0 (known since iteration 2). The header shows "Data loaded" as fallback text instead of "0 records".
### Next iteration should:
- Start Task 3.2 — Pathway data loading callback
- Create the callback in `dash_app/callbacks/chart.py`:
  - Input: `app-state` store (fires when filters change)
  - Output: `chart-data` store
  - Calls `queries.load_pathway_data(filter_id, chart_type, selected_drugs, selected_directorates)`
  - Must handle initial page load (when app-state has default values) AND subsequent filter changes
- Register in `dash_app/callbacks/__init__.py` alongside filter callbacks
- Verify: changing date filter dropdown → app-state updates → chart-data store gets new pathway nodes
- This is the bridge between filter state and chart rendering — 3.3 (KPIs) and 3.4 (icicle chart) both depend on chart-data being populated
### Blocked items:
- None

## Iteration 8 — 2026-02-06
### Task: Phase 3 — Task 3.2 (Pathway data loading callback)
### Why this task:
- Phase 3.1 (filter state management) is complete. Task 3.2 is the next dependency in the callback chain.
- Previous iteration (7) explicitly recommended this task.
- Tasks 3.3 (KPIs) and 3.4 (icicle chart) both depend on chart-data being populated by this callback.
### Status: COMPLETE
### What was done:
- Created `dash_app/callbacks/chart.py` with `register_chart_callbacks(app)`:
  - `load_pathway_data` callback: Input=`app-state` store, Output=`chart-data` store
  - Extracts `date_filter_id`, `chart_type`, `selected_drugs`, `selected_directorates` from app-state
  - Converts empty lists to None (so query function skips WHERE clause for unfiltered state)
  - Lazy imports `queries.load_pathway_data` to avoid circular import at module level
  - Returns `no_update` when app-state is empty (prevents query on initial empty state)
- Registered in `dash_app/callbacks/__init__.py` alongside filter callbacks
- Callback chain now complete: filter change → app-state → chart-data
### Validation results:
- Tier 1 (Code): `from dash_app.callbacks.chart import register_chart_callbacks` — OK
- Tier 1 (App starts): `from dash_app.app import app` — OK, all 3 callbacks registered
- Tier 3 (Functional): Verified callback chain with real data:
  - Default (all_6mo, directory): 293 nodes, 11,118 patients, 39 drugs, £130.6M
  - Indication chart: 438 nodes, 11,252 patients
  - Drug filter (ADALIMUMAB): 70 nodes
  - Date filter (2yr_12mo): 147 nodes, 5,104 patients
  - JSON serialization verified — dcc.Store compatible
### Files changed:
- `dash_app/callbacks/chart.py` — NEW: pathway data loading callback
- `dash_app/callbacks/__init__.py` — Updated: registers chart callback
- `IMPLEMENTATION_PLAN.md` — Task 3.2 marked [x]
### Committed: ad9fa1c "feat: add pathway data loading callback bridging filters to chart-data (Task 3.2)"
### Patterns discovered:
- Empty list `[]` from app-state needs conversion to `None` before passing to `load_pathway_nodes()`. The query function treats `None` as "no filter" (skips WHERE clause) but `[]` would create an empty IN clause returning zero results.
- `no_update` is the correct return for when app-state is not yet populated (rather than returning an empty dict), since it prevents downstream callbacks from firing unnecessarily.
### Next iteration should:
- Start Task 3.3 — KPI update callback
- Create `dash_app/callbacks/kpi.py`:
  - Input: `chart-data` store
  - Output: 4 KPI card values (`kpi-patients`, `kpi-drugs`, `kpi-cost`, `kpi-match`)
  - Extract `unique_patients`, `total_drugs`, `total_cost` from chart-data
  - Format: patients with commas (e.g., "11,118"), cost as "£130.6M", drugs as plain number
  - Indication match rate: could be "~93%" (from CLAUDE.md) or computed from data — check if chart-data has match info
- Register in callbacks/__init__.py
- After 3.3, move to Task 3.4 (icicle chart rendering) — the final core callback
- Task 3.4 requires reading `pathways_app/pathways_app.py` lines 769-920 for the icicle figure construction, then adding `create_icicle_from_nodes()` to `src/visualization/plotly_generator.py`
### Blocked items:
- None

## Iteration 9 — 2026-02-06
### Task: Phase 3 — Task 3.3 (KPI update callback)
### Why this task:
- Phase 3.2 (pathway data loading) is complete. Task 3.3 is next in the callback chain.
- Previous iteration (8) explicitly recommended this task.
- KPI cards currently show "—" placeholders — this callback populates them with real data.
### Status: COMPLETE
### What was done:
- Created `dash_app/callbacks/kpi.py` with `register_kpi_callbacks(app)`:
  - `update_kpis` callback: Input=`chart-data` + `app-state`, Output=4 KPI card values
  - Formats: patients with commas (11,118), drugs as plain number (39), cost as £130.6M
  - `_format_cost()` helper handles M/K/plain thresholds
  - 4th KPI (Indication Match): shows "~93%" when chart_type is "indication", "—" otherwise
  - Returns `no_update` when chart-data is empty (prevents clearing KPIs during load)
- Registered in `dash_app/callbacks/__init__.py` — now 4 callbacks total
- Decision: Indication match rate is static "~93%" because:
  - Reflex app's `indication_match_rate` was never dynamically computed (always 0.0 → "—")
  - Computing from nodes would require detecting "(no GP dx)" labels, adding complexity for marginal value
  - ~93% is the documented rate from the SNOMED cluster matching pipeline
### Validation results:
- Tier 1 (Code): `from dash_app.callbacks.kpi import register_kpi_callbacks` — OK
- Tier 1 (App starts): `from dash_app.app import app` — OK, 4 callbacks registered
- Tier 3 (Functional):
  - Cost formatting: 130.5M → £130.5M, 50K → £50.0K, 999 → £999
  - Directory chart: 11,118 patients, 39 drugs, £130.6M, Match = —
  - Indication chart: Match = ~93%
  - All 4 KPI output IDs registered: kpi-patients, kpi-drugs, kpi-cost, kpi-match
### Files changed:
- `dash_app/callbacks/kpi.py` — NEW: KPI update callback with formatting
- `dash_app/callbacks/__init__.py` — Updated: registers kpi callback
- `IMPLEMENTATION_PLAN.md` — Task 3.3 marked [x]
### Committed: 9c971c0 "feat: add KPI update callback with formatted patient/drug/cost display (Task 3.3)"
### Patterns discovered:
- `\u00a3` (£) is the safe way to embed the pound sign in Python strings — avoids encoding issues with source files
- `no_update` tuple must match the number of outputs: `return no_update, no_update, no_update, no_update` for 4 outputs
- The `app-state` is used as a secondary Input (not just `chart-data`) to get `chart_type` for the indication match KPI — this is fine since `app-state` changes trigger `chart-data` changes which also trigger this callback
### Next iteration should:
- Start Task 3.4 — Icicle chart rendering callback (the final core callback)
- Two sub-tasks:
  1. Add `create_icicle_from_nodes(nodes, title)` to `src/visualization/plotly_generator.py`
     - Read `pathways_app/pathways_app.py` lines 769-920 for the exact Plotly go.Icicle construction
     - Extract the 10-field customdata, NHS blue colorscale, texttemplate, hovertemplate
     - Keep existing `create_icicle_figure(ice_df)` untouched — add new function alongside
  2. Add `update_chart` callback to `dash_app/callbacks/chart.py`
     - Input: `chart-data` store, Output: `pathway-chart` figure
     - Calls `create_icicle_from_nodes(chart_data["nodes"], title)`
     - Dynamic title based on chart type
- IMPORTANT: Read the guardrails about the icicle figure — must use NHS blue gradient, not Viridis
- After 3.4, Phase 3 is complete — Phase 4 (Drawer) is next
### Blocked items:
- None

## Iteration 10 — 2026-02-06
### Task: Phase 3 — Task 3.4 (Icicle chart rendering callback)
### Why this task:
- Tasks 3.1-3.3 complete. Task 3.4 is the final task in Phase 3.
- Previous iteration (9) explicitly recommended this task.
- This is the most critical callback — it renders the actual Plotly icicle chart from pathway data.
### Status: COMPLETE
### What was done:
- Added `create_icicle_from_nodes(nodes, title)` to `src/visualization/plotly_generator.py`:
  - Accepts list-of-dicts (the JSON format from dcc.Store / load_pathway_nodes)
  - Same 10-field customdata structure as Reflex icicle_figure (value, colour, cost, costpp, first_seen, last_seen, first_seen_parent, last_seen_parent, average_spacing, cost_pp_pa)
  - NHS blue gradient colorscale: Heritage Blue #003087 → Pale Blue #E3F2FD (5 stops)
  - Same texttemplate and hovertemplate strings as Reflex version
  - maxdepth=3, branchvalues="total", sort=False
  - Layout: transparent background, reduced margins (t:40, l:8, r:8, b:24), autosize
  - Font: Source Sans 3 (matching nhs.css) instead of Inter (Reflex used Inter)
  - Existing `create_icicle_figure(ice_df)` left completely untouched
- Added `update_chart` callback to `dash_app/callbacks/chart.py`:
  - Input: chart-data store + app-state store
  - Output: pathway-chart figure + chart-subtitle children
  - Calls `create_icicle_from_nodes(chart_data["nodes"], title)`
  - Dynamic subtitle: "Trust → Directorate → Drug → Pathway" or "Trust → Indication → Drug → Pathway"
- Added `_generate_chart_title()` helper mirroring Reflex `_generate_pathway_chart_title()`:
  - Formats: "By Directory | All years / Last 6 months" (default)
  - Handles drug/directorate selections in title
### Validation results:
- Tier 1 (Code): `from visualization.plotly_generator import create_icicle_from_nodes` — OK
- Tier 1 (Code): `from dash_app.callbacks.chart import register_chart_callbacks` — OK
- Tier 1 (App starts): `from dash_app.app import app` — OK, 5 callbacks registered
- Tier 3 (Functional):
  - Directory chart: 293 nodes → Figure with 1 icicle trace, 293 labels, 10-field customdata
  - Indication chart: 438 nodes → Figure renders correctly
  - Colorscale starts at (0.0, '#003087') — NHS Heritage Blue confirmed
  - Title: "Patient Pathways — By Directory | All years / Last 6 months"
  - All 14 expected layout IDs present
### Files changed:
- `src/visualization/plotly_generator.py` — Added: create_icicle_from_nodes() function (115 lines)
- `dash_app/callbacks/chart.py` — Rewritten: added update_chart callback + _generate_chart_title helper
- `IMPLEMENTATION_PLAN.md` — Task 3.4 items marked [x]
### Committed: 40ce7fc "feat: add icicle chart rendering with NHS colorscale and dynamic titles (Task 3.4)"
### Patterns discovered:
- The Reflex version uses "Inter" font; switched to "Source Sans 3" to match the nhs.css design reference. This is the only intentional deviation from the Reflex code.
- `\u00a3` (£) works in Plotly template strings for the pound sign — no encoding issues.
- The `update_chart` callback uses both `chart-data` (for nodes) and `app-state` (for chart_type to determine subtitle) as Inputs. Both fire when filters change, but Dash deduplicates — the callback runs once with both updated values.
- `_generate_chart_title()` is a module-level function (not inside `register_chart_callbacks`) so it can be tested independently.
### Next iteration should:
- Start Phase 4: Task 4.1 — dmc.Drawer layout
- Create `dash_app/components/drawer.py` with `make_drawer()` function:
  - `dmc.Drawer(id="drug-drawer", position="right", size="480px")`
  - Top section: "All Drugs" card with flat alphabetical drug list from pathway_nodes level 3
  - Below: one card per PrimaryDirectorate from DimSearchTerm.csv with dmc.Accordion per Search_Term
  - Drug chips with pattern-matching IDs: `{"type": "drug-chip", "index": drug_name}`
  - Bottom: "Clear Filters" button
- Read `01_nhs_classic.html` for drawer styling if applicable (though drawer is mostly DMC-styled)
- Use `build_directorate_tree()` from `dash_app/data/card_browser.py` (already built in Task 1.2)
- Use `get_all_drugs()` from `dash_app/data/card_browser.py` for the flat drug list
- Add drawer to the app layout in `app.py`
- IMPORTANT: DMC version is 2.5.1 (Mantine v7 based). Check DMC 2.x API for Drawer, Accordion, Chip — prop names may differ from 0.14.x. E.g., `dmc.Drawer` may use `opened` instead of `open`, `onClose` instead of `on_close`.
### Blocked items:
- None

## Iteration 11 — 2026-02-06
### Task: Phase 4 — Task 4.1 (dmc.Drawer layout)
### Why this task:
- Phases 0-3 are complete. Phase 4 (Directorate Card Browser) is next in dependency order.
- Iteration 10 explicitly recommended this as the next task.
- The drawer layout is needed before drawer callbacks (Task 4.2) can be built.
### Status: COMPLETE
### What was done:
- Created `dash_app/components/drawer.py` with `make_drawer()` function:
  - `dmc.Drawer(id="drug-drawer", position="right", size="480px", opened=False)`
  - Title: "Drug & Indication Browser" via `dmc.Text`
  - ScrollArea wrapping all content for vertical overflow
  - **All Drugs section**: `dmc.ChipGroup(id="all-drugs-chips", multiple=True)` with 42 drug chips from pathway_nodes level 3
  - **Directorate section**: Nested `dmc.Accordion` with 19 directorates, each containing an inner accordion of indications (163 total). Drug fragments shown as `dmc.Badge` with pattern-match IDs `{"type": "drug-fragment", "index": "DIRECTORATE|FRAGMENT"}`
  - **Clear button**: `dmc.Button("Clear All Filters", id="clear-drug-filters", variant="outline", color="red")`
- Added drawer CSS to `dash_app/assets/nhs.css`:
  - `.drawer-section`, `.drawer-section-title`, `.drawer-chips-wrap`, `.drawer-drug-badge`, `.drawer-directorate-accordion`, `.drawer-clear-btn`
  - Chip font-family override to Source Sans 3
  - Badge hover effect for drug fragments
- Updated `dash_app/app.py`:
  - Added `make_drawer()` import and placed drawer in MantineProvider children (after sidebar, before main)
### Validation results:
- Tier 1 (Code): `from dash_app.components.drawer import make_drawer` — OK
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 2 (Layout): Verified programmatically:
  - MantineProvider: 8 children (4 stores/location + Header + Nav + Drawer + Main)
  - Drawer: position=right, size=480px, opened=False
  - All Drugs ChipGroup: 42 chips, multiple=True
  - Directorate Accordion: 19 directorates, 163 indications
  - 433 IDs in layout (mostly drug-fragment pattern-match IDs)
  - Key IDs present: drug-drawer, all-drugs-chips, clear-drug-filters
  - App has 5 callbacks registered (same as before — no new callbacks in this layout-only task)
### Files changed:
- `dash_app/components/drawer.py` — NEW: drawer component with drug chips and directorate cards
- `dash_app/app.py` — Updated: imports and adds drawer to layout
- `dash_app/assets/nhs.css` — Added: drawer-specific CSS classes
- `IMPLEMENTATION_PLAN.md` — Task 4.1 marked [x]
### Committed: 5dc552f "feat: add dmc.Drawer drug browser with directorate cards and drug chips (Task 4.1)"
### Patterns discovered:
- DMC 2.5.1 `dmc.Chip` doesn't expose `n_clicks`. It's a controlled component that works through `dmc.ChipGroup` with `value` prop (list of selected values when `multiple=True`). This is simpler than pattern-matching callbacks for drug selection — a single callback on `all-drugs-chips.value` replaces N individual chip click handlers.
- DMC 2.5.1 `dmc.Drawer` uses `opened` (not `open`), consistent with Mantine v7 API.
- `dmc.AccordionControl` + `dmc.AccordionPanel` inside `dmc.AccordionItem` is the DMC 2.x pattern. Nesting accordions works — directorate accordion → indication accordion inside.
- Drug fragments use `dmc.Badge` (not `dmc.Chip`) because they're informational/action triggers, not multi-select toggles. Badge IDs use pattern-matching format `{"type": "drug-fragment", "index": "DIR|FRAG"}` for callback handling in 4.2.
- The drawer data is loaded at layout time (module-level `make_drawer()` call during app import), not lazily. This is fine since the data is small (~165 CSV rows + 42 drugs from SQLite) and only runs once at app startup.
### Next iteration should:
- Start Task 4.2 — Drawer callbacks
- Create `dash_app/callbacks/drawer.py` with `register_drawer_callbacks(app)`:
  1. **Open/close drawer**: sidebar "Drug Selection" (`id="sidebar-drug-selection"`, n_clicks) or "Indications" (`id="sidebar-indications"`, n_clicks) click → set `drug-drawer.opened=True`. Drawer's built-in close (X button, overlay click) handles closing via `opened` prop.
  2. **Drug chip selection**: `all-drugs-chips.value` (ChipGroup value change) → update `app-state.selected_drugs` → triggers chain: app-state → chart-data → KPIs + chart
  3. **Drug fragment click**: `{"type": "drug-fragment", "index": ALL}.n_clicks` → match fragment against full drug names → select matching drugs in `all-drugs-chips.value` + `app-state.selected_drugs`
  4. **Clear filters**: `clear-drug-filters.n_clicks` → reset `app-state.selected_drugs=[]` + `all-drugs-chips.value=[]`
- Register in `dash_app/callbacks/__init__.py`
- Key consideration: The `all-drugs-chips` ChipGroup value must stay in sync with `app-state.selected_drugs`. Consider whether to use `all-drugs-chips.value` as the source of truth (simpler) or `app-state.selected_drugs` (needed for other callbacks). May need a two-way sync or just derive one from the other.
- IMPORTANT: Clicking a drug fragment badge should filter to drugs whose FULL names contain that fragment (substring match). The fragment "PEGYLATED" would match "PEGYLATED INTERFERON ALFA-2A" for example. The matching logic should use `drug_name.upper().find(fragment) >= 0`.
### Blocked items:
- None

## Iteration 12 — 2026-02-06
### Task: Phase 4 — Task 4.2 (Drawer callbacks)
### Why this task:
- Phase 4.1 (drawer layout) complete. Task 4.2 adds the interactivity to the drawer.
- Iteration 11 explicitly recommended this as the next task.
- Without callbacks, the drawer can't open or affect the chart — it's just a static component.
### Status: COMPLETE
### What was done:
- Created `dash_app/callbacks/drawer.py` with `register_drawer_callbacks(app)`:
  1. `open_drawer`: Input = sidebar-drug-selection + sidebar-indications n_clicks → Output = drug-drawer.opened = True. Drawer's built-in close (X / overlay) handles closing.
  2. `handle_fragment_or_clear`: Input = drug-fragment badges (pattern-matching ALL) n_clicks + clear-drug-filters n_clicks → Output = all-drugs-chips.value
     - Fragment click: extracts fragment from `{"type": "drug-fragment", "index": "DIR|FRAG"}`, finds all drugs containing the fragment (substring match), toggles them in chip selection
     - Clear click: sets chip value to empty list `[]`
     - Toggle behavior: if all matching drugs already selected, deselects them; otherwise adds them
- Modified `dash_app/callbacks/filters.py` `update_app_state`:
  - Added `all-drugs-chips.value` as 5th Input (after date filter dropdowns)
  - When chip selection changes, `selected_drugs` in app-state is updated
  - This completes the callback chain: chip change → app-state → chart-data → chart + KPIs
- Registered drawer callbacks in `dash_app/callbacks/__init__.py` — now 7 callbacks total
- Design decision: `all-drugs-chips.value` (ChipGroup) is the visual source of truth for drug selection. The drawer callbacks write to it, and `update_app_state` reads from it to sync into `app-state.selected_drugs`. No `allow_duplicate` needed — clean unidirectional flow.
### Validation results:
- Tier 1 (Code): All imports pass — drawer.py, filters.py, callbacks/__init__.py, app.py
- Tier 1 (App starts): MantineProvider layout, 7 callbacks registered, app ready to serve
- Tier 3 (Functional):
  - Callback chain verified: sidebar clicks → drawer opens (drug-drawer.opened Output)
  - Fragment matching: "ADALIMUMAB" → matches ["ADALIMUMAB"], "RITUXI" → matches ["RITUXIMAB"]
  - Some fragments (PEGYLATED, INHALED) match 0 drugs in pathway_nodes — returns no_update (correct behavior: those fragments may reference drugs not in current data)
  - Clear button → empty chip list → app-state.selected_drugs = [] → full chart reload
  - update_app_state Input list confirmed: 5 Inputs (2 toggle pills + 2 dropdowns + 1 ChipGroup)
### Files changed:
- `dash_app/callbacks/drawer.py` — NEW: open_drawer + handle_fragment_or_clear callbacks
- `dash_app/callbacks/__init__.py` — Updated: registers drawer callbacks
- `dash_app/callbacks/filters.py` — Updated: added all-drugs-chips.value as Input to update_app_state
- `IMPLEMENTATION_PLAN.md` — Task 4.2 marked [x]
### Committed: fe76e5a "feat: add drawer callbacks for drug selection, fragment matching, and clear (Task 4.2)"
### Patterns discovered:
- `dmc.ChipGroup.value` is the cleanest way to manage multi-select drug state. The ChipGroup handles visual selection state internally; callbacks only need to update the `value` prop (list of selected values). No per-chip n_clicks tracking needed.
- Fragment badge IDs use format `{"type": "drug-fragment", "index": "DIRECTORATE|FRAGMENT"}`. The directorate prefix is stripped with `split("|", 1)[-1]` to get the pure fragment for substring matching.
- Pattern-matching `Input({"type": "drug-fragment", "index": ALL}, "n_clicks")` with `prevent_initial_call=True` is essential — without it, the callback fires on page load with all n_clicks as None/0, which would trigger `no_update`.
- The toggle behavior (click fragment → select matching drugs; click again → deselect) is implemented by checking if all matching drugs are already in the current selection.
- `allow_duplicate=True` was avoided by routing all drug selection through `all-drugs-chips.value` → `update_app_state`. Only one callback writes to each Output.
### Next iteration should:
- Start Phase 5: Task 5.1 — Trust selection
- Options for trust selection:
  - Add a "Trusts" section to the drawer (preferred — keeps all filters together)
  - OR add trust checkboxes in the sidebar
- Implementation: add `dmc.ChipGroup(id="trust-chips", multiple=True)` to the drawer with available trusts from reference-data
- Wire trust selection similarly to drug selection: trust-chips.value → update_app_state → app-state.selected_trusts → chart reload
- Check `src/data_processing/pathway_queries.py` `load_pathway_nodes()` — does it support a `selected_trusts` parameter? If not, add trust filtering to the query.
- Read `pathways_app/pathways_app.py` for how trust filtering was handled in Reflex
- ALTERNATIVE: Skip 5.1 for now and do 5.2 (loading/error/empty states) which is more impactful for UX polish. Trust selection is less critical since there are few trusts.
### Blocked items:
- None

## Iteration 13 — 2026-02-06
### Task: Phase 5 — Task 5.1 (Trust selection)
### Why this task:
- Phases 0-4 complete. Phase 5 is the final polish phase.
- Task 5.1 is the first task in Phase 5 and adds the last major filter dimension (trust).
- Previous iteration (12) recommended this as the next task.
### Status: COMPLETE
### What was done:
- Added `available_trusts` to `load_initial_data()` in `src/data_processing/pathway_queries.py`:
  - Queries `SELECT DISTINCT trust_name FROM pathway_nodes WHERE level = 1` — returns 7 trusts
  - Updated both error return paths to include `available_trusts: []`
- Added `selected_trusts` parameter to `load_pathway_nodes()`:
  - Trust WHERE clause: `trust_name IN (...) OR trust_name IS NULL OR trust_name = ''`
  - The NULL/empty check keeps root nodes (level 0) which have no trust_name
- Updated `dash_app/data/queries.py` wrapper to pass `selected_trusts` through
- Added `get_all_trusts()` to `dash_app/data/card_browser.py` (same pattern as `get_all_drugs()`)
- Added trust ChipGroup to drawer (`dash_app/components/drawer.py`):
  - `dmc.ChipGroup(id="trust-chips", multiple=True)` with 7 trust chips
  - New "Trusts" section between "All Drugs" and "By Directorate" with dividers
- Updated sidebar (`dash_app/components/sidebar.py`):
  - "Trust Selection" item now has `item_id="sidebar-trust-selection"` for drawer open callback
- Updated callbacks:
  - `drawer.py`: Added `sidebar-trust-selection` as Input to `open_drawer`; added `trust-chips.value` as second Output to `handle_fragment_or_clear` (clear button resets both drug and trust chips)
  - `filters.py`: Added `trust-chips.value` as 6th Input to `update_app_state`; added `selected_trusts` to app-state dict
  - `chart.py`: Passes `selected_trusts` to query function; includes trusts in chart title
- Updated `app.py`: Added `selected_trusts: []` to default app-state
### Validation results:
- Tier 1 (Code): All imports pass, 7 callbacks registered
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional):
  - `load_initial_data()`: 7 trusts returned
  - Unfiltered: 293 nodes, 11,118 patients
  - One trust (NNUH): 107 nodes, 33 drugs — correct reduction
  - Two trusts: 181 nodes — correct intermediate reduction
  - Indication + trust filter: 158 nodes — works across chart types
  - Chart figure renders correctly from filtered nodes (107 labels)
  - Trust chips in drawer: 7 chips, multiple=True
### Files changed:
- `src/data_processing/pathway_queries.py` — Added: available_trusts query + selected_trusts filter
- `dash_app/data/queries.py` — Updated: pass selected_trusts parameter
- `dash_app/data/card_browser.py` — Added: get_all_trusts()
- `dash_app/components/drawer.py` — Added: trust ChipGroup section
- `dash_app/components/sidebar.py` — Updated: Trust Selection gets item_id for drawer open
- `dash_app/callbacks/drawer.py` — Updated: trust sidebar opens drawer; clear resets trust chips
- `dash_app/callbacks/filters.py` — Updated: trust-chips.value as Input + selected_trusts in state
- `dash_app/callbacks/chart.py` — Updated: pass selected_trusts to query; trusts in chart title
- `dash_app/app.py` — Updated: selected_trusts in default app-state
- `IMPLEMENTATION_PLAN.md` — Task 5.1 marked [x]
### Committed: f0505ee "feat: add trust selection to drawer with filter wiring (Task 5.1)"
### Patterns discovered:
- Trust filtering needs `trust_name IS NULL OR trust_name = ''` in the WHERE clause to keep root nodes (level 0) and the "Total" parent that have empty trust_name. Without this, the icicle chart would lose its root.
- Adding a second Output (`trust-chips.value`) to `handle_fragment_or_clear` requires returning `no_update` for the trust output on fragment clicks (only the clear button affects trusts). All return paths must return a tuple of 2 values.
- The `get_all_trusts()` / `get_all_drugs()` pattern delegates to `load_initial_data()` to avoid duplicating SQL queries. This means only one SQLite connection for all reference data.
### Next iteration should:
- Start Task 5.2 — Loading/error/empty states + dynamic hierarchy label
- Add `dcc.Loading` wrapper around the chart card's `dcc.Graph` component
- Show a "No data" message when chart-data has no nodes (e.g., after filtering to a trust with no matching drugs)
- Show error feedback when database query fails
- Note: Dynamic chart subtitle is ALREADY implemented (Task 3.4 added `chart-subtitle` Output). Check if 5.2's subtitle bullet can be marked done.
- Consider: `dcc.Loading(type="circle", color="#005EB8")` wrapping the `dcc.Graph` for a loading spinner during SQLite queries
### Blocked items:
- None

## Iteration 14 — 2026-02-06
### Task: Phase 5 — Task 5.2 (Loading/error/empty states + dynamic hierarchy label)
### Why this task:
- Task 5.1 complete. Task 5.2 is the next incomplete task in Phase 5.
- Iteration 13 explicitly recommended this task.
- Adds UX polish: loading spinner during data fetch, empty state when filters match nothing, error handling for database failures.
### Status: COMPLETE
### What was done:
- Added `dcc.Loading(type="circle", color="#005EB8")` wrapper around `dcc.Graph` in `chart_card.py`:
  - Wraps a `html.Div(id="chart-container")` containing the `dcc.Graph(id="pathway-chart")`
  - NHS blue spinner appears while chart data is loading
- Added `_empty_figure(message)` helper in `dash_app/callbacks/chart.py`:
  - Returns a blank Plotly figure with a centered annotation message
  - Uses `#768692` (NHS mid-grey) for text color, Source Sans 3 font
  - Used for both empty state and error state
- Modified `update_chart` callback to handle three states:
  1. **Error**: `chart_data.get("error")` → shows error message in empty figure
  2. **Empty**: `not chart_data.get("nodes")` → shows "No matching pathways found. Try adjusting your filters."
  3. **Normal**: renders icicle chart as before
  - Subtitle is always computed (even for empty/error states) to keep the hierarchy label consistent
- Added error handling to `load_pathway_data` callback:
  - `try/except Exception` wraps the query call
  - On failure, logs the exception and returns `{"nodes": [], "error": "Database query failed. Check logs for details."}`
  - Downstream callbacks (KPIs, chart) gracefully handle empty data
- Dynamic chart subtitle was already implemented in Task 3.4 — marked as done in IMPLEMENTATION_PLAN.md
### Validation results:
- Tier 1 (Code): All imports pass — chart.py, chart_card.py, app.py
- Tier 1 (App starts): `from dash_app.app import app` — OK, 7 callbacks registered
- Tier 2 (Layout): dcc.Loading found in chart card: type=circle, color=#005EB8, wrapping chart-container → pathway-chart
- Tier 3 (Functional):
  - Empty figure renders annotation: "No matching pathways found.\nTry adjusting your filters."
  - Error figure renders annotation: "Database query failed. Check logs for details."
  - KPIs show em-dash for all values when data is empty (0 patients, 0 drugs, £0)
### Files changed:
- `dash_app/components/chart_card.py` — Updated: added dcc.Loading wrapper around dcc.Graph
- `dash_app/callbacks/chart.py` — Updated: added _empty_figure helper, error handling in load_pathway_data, empty/error states in update_chart
- `IMPLEMENTATION_PLAN.md` — Task 5.2 marked [x]
### Committed: 5593d08 "feat: add loading spinner, empty state, and error handling to chart area (Task 5.2)"
### Patterns discovered:
- `_empty_figure()` uses Plotly annotations on a blank figure (no traces) for empty/error states. This is cleaner than returning an empty `go.Figure()` (which shows grid lines) because `xaxis.visible=False, yaxis.visible=False` hides all axes.
- `dcc.Loading` wraps children — when any Output inside changes, the spinner appears. Wrapping `dcc.Graph` directly in `dcc.Loading` is sufficient since `pathway-chart.figure` is the Output that triggers loading state.
- The `update_chart` callback now computes subtitle BEFORE checking for empty/error data. This ensures the hierarchy label stays consistent even when showing empty state (previously `no_update` was returned for both figure and subtitle, which could leave a stale subtitle).
### Next iteration should:
- Start Task 5.3 — Data freshness indicator
- The header already has `id="header-record-count"` and `id="header-last-updated"` spans (created in Task 2.1)
- The `load_reference_data` callback in `dash_app/callbacks/filters.py` already populates these on page load
- Check what the current values show — may just need formatting improvements (relative time like "2h ago")
- `load_initial_data()` returns `last_updated` as an ISO datetime string and `total_records` (which is 0 from pathway_refresh_log)
- Consider using `datetime` to compute relative time from `last_updated`, or just show the date
- After 5.3, only 5.4 (Remove Reflex + final validation) remains
### Blocked items (iter 14):
- None

## Iteration 15 — 2026-02-06
### Task: Phase 5 — Task 5.3 (Data freshness indicator)
### Why this task:
- Task 5.2 complete. Task 5.3 is the next incomplete task in Phase 5.
- Iteration 14 explicitly recommended this task.
- The header already had placeholder spans for record count and last updated — this task makes them useful.
### Status: COMPLETE
### What was done:
- Added `_format_relative_time(iso_timestamp)` helper to `dash_app/callbacks/filters.py`:
  - Parses ISO timestamp and returns relative time: "just now", "15m ago", "3h ago", "yesterday", "4d ago", "1w ago", or "23 Dec 2025" for older dates
  - Handles empty strings, invalid dates gracefully
- Added `total_patients` field to `load_initial_data()` in `src/data_processing/pathway_queries.py`:
  - Queries root node (level 0) value from default filter (all_6mo, directory) as patient count
  - Only runs when `total_records` is 0 (source_row_count empty in refresh log — known issue)
  - Returns 11,118 patients for current data
- Updated `load_reference_data` callback to use both improvements:
  - Record text: "11,118 patients" (comma-formatted, falls back to "Data loaded" if no data)
  - Updated text: "14h ago" (relative time from last_updated ISO timestamp)
  - Matches design reference in 01_nhs_classic.html: green dot + "{N} records" + "Last updated: 2h ago"
### Validation results:
- Tier 1 (Code): All imports pass, app starts, 7 callbacks registered
- Tier 1 (App starts): `from dash_app.app import app` — OK
- Tier 3 (Functional):
  - Relative time formatting: tested 10 cases (just now, minutes, hours, yesterday, days, weeks, months, empty, invalid, actual value) — all correct
  - Record text: "11,118 patients" — correct
  - Updated text: "14h ago" — correct (refresh was at 2026-02-06T00:08:55)
### Files changed:
- `src/data_processing/pathway_queries.py` — Added: total_patients query from root node
- `dash_app/callbacks/filters.py` — Added: _format_relative_time helper, updated load_reference_data
- `IMPLEMENTATION_PLAN.md` — Task 5.3 marked [x]
### Committed: e877268 "feat: add data freshness indicator with relative time and patient count (Task 5.3)"
### Patterns discovered:
- `total_records` from `pathway_refresh_log.source_row_count` is always 0 for this project. The `total_patients` field from the root node's `value` is a better alternative for the header display.
- `datetime.fromisoformat()` handles the SQLite timestamp format (`2026-02-06T00:08:55.812828`) natively in Python 3.10+.
- The relative time helper is a standalone function (not inside `register_filter_callbacks`) for testability.
### Next iteration should:
- Start Task 5.4 — Remove Reflex + final validation
- This is the FINAL task. Sub-items:
  1. Remove `reflex` from `pyproject.toml` dependencies
  2. Move `pathways_app/` to `archive/` (don't delete — keep for reference)
  3. Delete `pathways_app/styles.py` and any Reflex-specific files
  4. Update `CLAUDE.md` to document the Dash app structure, new run command (`python run_dash.py`), callback architecture
  5. Run `python run_dash.py` — verify app starts cleanly
  6. Verify no Reflex imports in `dash_app/`
  7. Run `uv sync` after removing reflex dependency
- IMPORTANT: Read the completion criteria at bottom of IMPLEMENTATION_PLAN.md — all must be satisfied
- After 5.4, check if ALL tasks are [x] and output the COMPLETE signal
### Blocked items:
- None

## Iteration 16 — 2026-02-06
### Task: Phase 5 — Task 5.4 (Remove Reflex + final validation)
### Why this task:
- Tasks 5.1-5.3 complete. Task 5.4 is the FINAL task in the entire implementation plan.
- Iteration 15 explicitly recommended this task.
- This removes the Reflex dependency, archives the old app, updates documentation, and validates everything.
### Status: COMPLETE
### What was done:
- Removed `reflex>=0.6.0` from `pyproject.toml` dependencies
- Ran `uv sync` — removed ~30 Reflex-related packages (reflex, starlette, uvicorn, sqlalchemy, etc.)
- Moved `pathways_app/` to `archive/pathways_app/` via `git mv` (preserves git history)
- Moved `rxconfig.py` to `archive/rxconfig.py` via `git mv`
- Verified zero Reflex imports in `dash_app/` (grep returned no matches)
- Updated `CLAUDE.md` comprehensively:
  - "Reflex framework" → "Dash (Plotly) + Dash Mantine Components"
  - Run command: `reflex run` → `python run_dash.py`, port 3000 → 8050
  - Package structure: `pathways_app/` section replaced with full `dash_app/` tree
  - New sections: Shared Data Queries (pathway_queries.py), Dash Application state/callbacks/components
  - Data flow diagram: Reflex AppState → Dash callback chain
  - Updated all "Reflex" references to "Dash" throughout
  - Visualization module: documented `create_icicle_from_nodes()` alongside existing function
- Ran comprehensive validation — all 10 completion criteria PASS:
  1. App starts cleanly at localhost:8050
  2. Layout matches design (MantineProvider → stores + header + sidebar + drawer + main)
  3. Icicle chart renders 293 nodes with NHS blue gradient and 10-field customdata
  4. Callback chain fires correctly through all 7 callbacks
  5. Drawer contains 19 directorates, 163 indications, drug fragment badges
  6. Drug selection filters chart (ADALIMUMAB: 293 → 70 nodes)
  7. ChipGroup with 42 drugs, multiple=True
  8. Clear button resets drug and trust chips → full chart reload
  9. KPIs update: 11,118 patients, 39 drugs, £130.6M
  10. Zero Reflex imports in dash_app/
### Validation results:
- Tier 1 (Code): All imports pass, app starts, 7 callbacks registered
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional): All 10 completion criteria pass (validated by subagent)
### Files changed:
- `pyproject.toml` — Removed: reflex dependency
- `uv.lock` — Updated: removed Reflex packages
- `pathways_app/` → `archive/pathways_app/` — Archived via git mv
- `rxconfig.py` → `archive/rxconfig.py` — Archived via git mv
- `CLAUDE.md` — Rewritten: Reflex → Dash documentation
- `IMPLEMENTATION_PLAN.md` — Task 5.4 marked [x], all completion criteria marked [x]
### Committed: fe8642d "feat: remove Reflex, archive old app, update docs for Dash migration (Task 5.4)"
### Patterns discovered:
- `uv sync` after removing a large dependency (reflex) cleanly removes all transitive dependencies (~30 packages). No manual cleanup needed.
- `git mv` preserves rename history — `git log --follow archive/pathways_app/pathways_app.py` shows the full commit history.
- `.web/` (Reflex's generated frontend) is gitignored so doesn't need archiving. Can be manually deleted.
### Next iteration should:
- ALL TASKS COMPLETE. No next iteration needed.
### Blocked items (iter 16):
- None

## Iteration 17 — 2026-02-06
### Task: Phase 6 — Update all documentation
### Why this task:
- Phases 0-5 complete. Phase 6 (documentation cleanup) was the only remaining work.
- All non-archive .md files still referenced Reflex (commands, architecture, deployment).
### Status: COMPLETE
### What was done:
- **README.md**: Complete rewrite — replaced `reflex run` with `python run_dash.py`, updated features list (dual charts, pre-computed pathways, drug browser), updated project structure (dash_app/ instead of pathways_app/), updated quick start, usage, troubleshooting
- **docs/USER_GUIDE.md**: Complete rewrite — new single-page Dash UI guide covering header, sidebar, KPI row, filter bar, chart card, drawer, drug browser, indication matching
- **docs/DEPLOYMENT.md**: Complete rewrite — Dash/Gunicorn deployment replacing Reflex/FastAPI/Next.js, Docker examples, nginx config, systemd service, single port 8050
- **docs/DESIGN_SYSTEM.md**: Updated "Reflex Implementation" section → "Dash Implementation" with correct CSS/component references
- **RALPH_PROMPT.md**: Updated title ("Dash Application Maintenance" not "Reflex → Dash Migration"), updated data reference section to point to src/ shared functions instead of pathways_app/
- **guardrails.md**: Updated 3 rules to reflect current Dash architecture (shared utilities exist, icicle function exists, data queries exist) instead of migration instructions
- **IMPLEMENTATION_PLAN.md**: Phase 6 tasks marked [x]
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 7 callbacks registered
- Grep for Reflex in non-archive .md files: only CLAUDE.md line 140 (archive description — accurate) and IMPLEMENTATION_PLAN.md (historical migration log — accurate)
### Files changed:
- README.md — Rewritten for Dash
- docs/USER_GUIDE.md — Rewritten for Dash
- docs/DEPLOYMENT.md — Rewritten for Dash
- docs/DESIGN_SYSTEM.md — Updated implementation section
- RALPH_PROMPT.md — Updated title and references
- guardrails.md — Updated 3 rules
- IMPLEMENTATION_PLAN.md — Phase 6 marked [x]
- progress.txt — This entry
### Patterns discovered:
- archive/ files (IMPROVEMENT_RECOMMENDATIONS.md) retain Reflex references intentionally — they're historical
- IMPLEMENTATION_PLAN.md retains Reflex references in completed task descriptions — these are accurate migration history
### Next iteration should:
- Phase 6 is complete. All tasks across all phases are now [x].
### Blocked items:
- None

## Phases 0-6 COMPLETE (17 iterations)
- Phase 0: Scaffolding (2 tasks) — iteration 1
- Phase 1: Data Access (2 tasks) — iterations 2-3
- Phase 2: Static Layout (3 tasks) — iterations 4-6
- Phase 3: Core Callbacks (4 tasks) — iterations 7-10
- Phase 4: Drawer (2 tasks) — iterations 11-12
- Phase 5: Polish & Cleanup (4 tasks) — iterations 13-16
- Phase 6: Documentation (4 tasks) — iteration 17

---

## Phase 7: Bug Fixes & UI Restructure (NEW)

User testing revealed 2 bugs and several UX issues. Phase 7 addresses all of them.

### Known Bugs

**Bug 1 — Duplicate ID on first load:**
`DuplicateIdError: Duplicate component id found in the initial layout: {"index":"CARDIOLOGY|RIVAROXABAN","type":"drug-fragment"}`
- Only happens on first `python run_dash.py` load. Subsequent refreshes are fine.
- Root cause: In `drawer.py:66`, badge IDs are `f"{directorate}|{frag}"` but the same fragment can appear under multiple indications within the same directorate. E.g., RIVAROXABAN appears under both "acute coronary syndrome" and "atrial fibrillation" within CARDIOLOGY.
- Fix: Include search_term in the ID: `f"{directorate}|{search_term}|{frag}"`. Update callback parsing in `drawer.py` to handle the 3-part key.

**Bug 2 — Drug filter breaks icicle chart ("multiple implied roots"):**
Console error: `WARN: Multiple implied roots, cannot build icicle hierarchy of trace 0`
- Happens when selecting ANY drug from the All Drugs chip group — chart goes blank.
- Root cause: `pathway_queries.py:load_pathway_nodes()` applies the `drug_sequence LIKE` filter to ALL levels. This drops ancestor nodes (root, trust, directory levels 0-2) that don't have a drug_sequence matching the filter. Without ancestors, the icicle chart sees disconnected subtrees.
- Fix: Restructure the WHERE clause so drug/directorate/trust filters only apply to the appropriate levels. Ancestor nodes (level 0-2) must ALWAYS be included. The WHERE logic should be: `WHERE date_filter_id = ? AND chart_type = ? AND (level < 3 OR drug_sequence LIKE ?)` — or similar approach that preserves the parent chain.
- IMPORTANT: Check if the same issue affects trust and directorate filters too.

### UX Issues

**UX 1 — Sidebar has wrong items:**
- "Cost Analysis" and "Export Data" have no functionality — remove them.
- Drug/Trust/Directory/Indication items open the same drawer — confusing. These filter triggers should move to the filter bar.
- Sidebar should contain chart VIEW selectors (Icicle, Sankey, Timeline) instead.

**UX 2 — Single drawer for all filters doesn't work:**
- Too much scrolling, confusing that 3-4 buttons open the same panel.
- Replace `dmc.Drawer` with `dmc.Modal` dialogs — one per filter type (Drugs, Trusts, Directorates).
- Each modal opened by a button in the filter bar (with selection count badge).

### Task Dependency Order for Phase 7

1. **Task 7.1** (duplicate ID fix) — standalone, no dependencies. Do this first.
2. **Task 7.2** (filter query fix) — standalone, no dependencies. Can be done right after 7.1.
3. **Task 7.3** (sidebar restructure) — remove items, add chart view buttons. No modal dependency.
4. **Task 7.4** (replace drawer with modals) — depends on 7.3 (sidebar items removed). This is the biggest task. **Use the frontend-developer agent** to design the modal UX.
5. **Task 7.5** (filter triggers in filter bar) — depends on 7.4 (modals exist to open).

### Next iteration should:
- Start with Task 7.1 (duplicate ID fix) — it's the simplest and most isolated.
- Read `dash_app/components/drawer.py` line 66 — the badge ID construction.
- Read `dash_app/callbacks/drawer.py` line 48 — the fragment key parsing.
- Read `dash_app/data/card_browser.py` — `build_directorate_tree()` to understand the data shape and where duplicates come from.
- Fix the ID to include search_term, update the callback parsing, verify no DuplicateIdError on first load.

## Iteration 18 — 2026-02-06
### Task: Phase 7 — Task 7.1 (Fix duplicate component ID error on first load)
### Why this task:
- First task in Phase 7, simplest and most isolated bug fix
- Progress.txt from Phase 7 setup explicitly recommended starting here
- No dependencies on other Phase 7 tasks
### Status: COMPLETE
### What was done:
- Changed badge ID format in `drawer.py:66` from `f"{directorate}|{frag}"` to `f"{directorate}|{search_term}|{frag}"`
- Updated callback fragment parsing in `callbacks/drawer.py:48` from `split("|", 1)[-1]` to `rsplit("|", 1)[-1]` (split from right to extract just the fragment from the 3-part key)
- Verified 544 IDs in drawer component — all unique, zero duplicates
- Confirmed 90+ duplicate fragments existed across directorates (CARDIOLOGY, CLINICAL HAEMATOLOGY, MEDICAL ONCOLOGY, RHEUMATOLOGY, etc.)
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 7 callbacks registered
- Tier 1 (App starts): Layout access succeeds, no DuplicateIdError
- Tier 3 (Functional): Fragment parsing verified for 3-part key ("CARDIOLOGY|acute coronary syndrome|RIVAROXABAN" → "RIVAROXABAN"), 2-part key fallback, and standalone key
### Files changed:
- `dash_app/components/drawer.py` — Updated: badge ID includes search_term (line 66)
- `dash_app/callbacks/drawer.py` — Updated: fragment parsing uses rsplit (line 48)
- `IMPLEMENTATION_PLAN.md` — Task 7.1 marked [x]
### Committed: 7be136a "fix: resolve DuplicateIdError by including search_term in drug-fragment badge IDs (Task 7.1)"
### Patterns discovered:
- `rsplit("|", 1)[-1]` is the correct way to extract the last segment from a multi-part key. `split("|", 1)[-1]` would return everything after the first pipe, which fails for 3+ part keys.
- The DuplicateIdError only occurred on first load because Dash validates layout IDs during initial render. `suppress_callback_exceptions=True` only suppresses callback-related ID checks, not layout duplication.
- Over 90 drug fragments are shared across multiple indications within the same directorate — this is expected because drugs are prescribed for multiple conditions.
### Next iteration should:
- Start Task 7.2 — Fix drug filter breaking the icicle chart ("multiple implied roots")
- Read `src/data_processing/pathway_queries.py` — the `load_pathway_nodes()` function
- The WHERE clause applies `drug_sequence LIKE %DRUG%` to ALL levels, dropping ancestor nodes (root, trust, directory levels 0-2) that have NULL drug_sequence
- Fix: restructure WHERE so levels 0-2 are always included, drug filter only applies to level >= 3
- Also check if trust/directorate filters have the same issue (ancestor nodes dropping)
- Verify: select a drug → chart renders correctly with full hierarchy intact
### Blocked items:
- None

## Iteration 19 — 2026-02-06
### Task: Phase 7 — Task 7.2 (Fix drug filter breaking icicle chart)
### Why this task:
- Task 7.1 complete in iteration 18. Task 7.2 is next in Phase 7 dependency order.
- Progress.txt from iteration 18 explicitly recommended this task.
- Critical bug — selecting ANY drug made the chart go blank with "multiple implied roots" error.
### Status: COMPLETE
### What was done:
- Diagnosed root cause in `src/data_processing/pathway_queries.py:load_pathway_nodes()`:
  - Levels 0-2 have `drug_sequence = ''` (empty string), NOT `drug_sequence IS NULL`
  - The drug filter's `OR drug_sequence IS NULL` check didn't match empty strings
  - Same bug existed for directorate filter: `directory = ''` at levels 0-1, but check was `OR directory IS NULL`
  - Trust filter was already correct (had `OR trust_name = ''`)
- Fixed drug filter: `(level < 3 OR drug_sequence LIKE ...)` — levels 0-2 always included
- Fixed directorate filter: `(level < 2 OR directory IN (...) OR directory IS NULL OR directory = '')` — levels 0-1 always included
- Verified all filter combinations preserve single-root hierarchy:
  - Drug only (ADALIMUMAB): 120 nodes, root present
  - Directorate only (RHEUMATOLOGY): 94 nodes, root present
  - Trust only (NNUH): 107 nodes, root present
  - Drug + Trust combined: 45 nodes, root present
  - Indication chart + drug: 195 nodes, root present
- Verified icicle charts render correctly from all filtered datasets
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 7 callbacks registered
- Tier 1 (App starts): `python run_dash.py` — serves at http://127.0.0.1:8050/ — no errors
- Tier 3 (Functional): All 5 filter combinations produce valid hierarchy with single root node; icicle charts render correctly
### Files changed:
- `src/data_processing/pathway_queries.py` — Fixed: drug and directorate WHERE clauses use level-based gating
- `IMPLEMENTATION_PLAN.md` — Task 7.2 marked [x], completion criterion for drug filter marked [x]
### Committed: 00627a7 "fix: preserve ancestor nodes in drug/directorate filters to prevent broken icicle hierarchy (Task 7.2)"
### Patterns discovered:
- `drug_sequence` and `directory` columns use empty string `''` (not NULL) for ancestor nodes (levels 0-2). The `IS NULL` check is insufficient — must use level-based gating (`level < N`) to unconditionally include ancestor nodes.
- Level-based gating is cleaner than checking for NULL/empty: `(level < 3 OR drug_filter_condition)` clearly expresses the intent that ancestors are always kept.
- Trust filter already worked because it had `OR trust_name = ''` in addition to `OR trust_name IS NULL`.
### Next iteration should:
- Start Task 7.3 — Restructure sidebar: remove placeholder items, add chart view buttons
- Read `dash_app/components/sidebar.py` to see current items
- Remove: "Cost Analysis", "Export Data" (no functionality)
- Remove: "Drug Selection", "Trust Selection", "Directory Selection", "Indications" (these become filter bar buttons in Task 7.5)
- Add: chart view buttons — "Icicle Chart" (active), "Sankey Diagram" (disabled), "Timeline" (disabled)
- Keep: "Pathway Overview" as the top active item
- Remove the tab row from `dash_app/components/chart_card.py` (chart view selection moves to sidebar)
- Update sidebar IDs and any callbacks that reference removed IDs (e.g., `sidebar-drug-selection`, `sidebar-trust-selection`, `sidebar-indications` are used in drawer.py `open_drawer` callback)
### Blocked items:
- None

## Iteration 20 — 2026-02-06
### Task: Phase 7 — Task 7.3 (Restructure sidebar: chart views, remove placeholder items)
### Why this task:
- Tasks 7.1 and 7.2 (bug fixes) complete. Task 7.3 is next in Phase 7 dependency order.
- Progress.txt from iteration 19 explicitly recommended this task.
- Tasks 7.4 (modals) and 7.5 (filter bar triggers) depend on the sidebar being cleaned up first.
### Status: COMPLETE
### What was done:
- **Sidebar restructured** (`dash_app/components/sidebar.py`):
  - Removed 6 items: Drug Selection, Trust Selection, Directory Selection, Indications, Cost Analysis, Export Data
  - Renamed "Analysis" section to "Overview" (now contains only "Pathway Overview")
  - Added "Chart Views" section with: "Icicle Chart" (active), "Sankey Diagram" (disabled), "Timeline" (disabled)
  - New SVG icons for chart views: icicle (stacked bars), sankey (flow lines), timeline (dots on line)
  - Sidebar footer preserved
- **Chart card tab row removed** (`dash_app/components/chart_card.py`):
  - Removed the 3-tab row (Icicle/Sankey/Timeline) — chart view selection is now in the sidebar
  - Chart card now contains only header (title + subtitle) and the loading-wrapped dcc.Graph
- **Drawer open callback removed** (`dash_app/callbacks/drawer.py`):
  - Removed `open_drawer` callback that referenced removed sidebar IDs (`sidebar-drug-selection`, `sidebar-indications`, `sidebar-trust-selection`)
  - Fragment matching and clear callbacks preserved (still functional via drawer component in layout)
  - Callback count dropped from 7 to 6
- **CSS updated** (`dash_app/assets/nhs.css`):
  - Added `.sidebar__item--disabled` class (opacity 0.4, cursor not-allowed, pointer-events none)
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 6 callbacks registered
- Tier 2 (Layout):
  - Sidebar: 2 sections (Overview: 1 item, Chart Views: 3 items), footer present
  - Icicle Chart: active, Sankey/Timeline: disabled
  - Chart card: no tab row, header + graph only
  - Removed IDs confirmed absent: sidebar-drug-selection, sidebar-indications, sidebar-trust-selection
  - Drawer still in layout (will be replaced in Task 7.4): drug-drawer, all-drugs-chips, trust-chips all present
### Files changed:
- `dash_app/components/sidebar.py` — Rewritten: removed filter items, added chart views
- `dash_app/components/chart_card.py` — Updated: removed tab row
- `dash_app/callbacks/drawer.py` — Updated: removed open_drawer callback
- `dash_app/assets/nhs.css` — Added: .sidebar__item--disabled CSS
- `IMPLEMENTATION_PLAN.md` — Task 7.3 marked [x], completion criteria updated
### Committed: 7aa49b0 "refactor: restructure sidebar with chart views, remove placeholder items (Task 7.3)"
### Patterns discovered:
- When removing sidebar items that are referenced by callbacks (e.g., `sidebar-drug-selection` as Input to `open_drawer`), the callback must also be removed or updated. Dash will error on startup if a callback references an ID not in the layout and `suppress_callback_exceptions=False`, but with `suppress_callback_exceptions=True` it would silently fail. Removing the callback is cleaner.
- The drawer component (and its drug/trust chips) can remain in the layout even after removing the way to open it. The `all-drugs-chips` and `trust-chips` IDs are still referenced by `update_app_state`, so they must remain until the modals replace them in Task 7.4.
### Next iteration should:
- Start Task 7.4 — Replace dmc.Drawer with dmc.Modal for filter selection
- This is the biggest remaining task. Sub-steps:
  1. **Use the frontend-developer agent** to design the modal UX: 42 drugs, 7 trusts, 19 directorates × 163 indications. Agent should recommend optimal modal layout, sizing, and interaction patterns.
  2. Create `dash_app/components/modals.py` (or separate files per modal) with:
     - Drug Selection modal (`dmc.Modal`, contains current `all-drugs-chips` ChipGroup)
     - Trust Selection modal (`dmc.Modal`, contains current `trust-chips` ChipGroup)
     - Directorate Browser modal (`dmc.Modal`, contains nested directorate accordion)
  3. Remove `dash_app/components/drawer.py` and the `make_drawer()` call from `app.py`
  4. Update callback wiring — `update_app_state` Inputs change from chip IDs in drawer to chip IDs in modals (or keep same IDs if modal components use the same IDs)
  5. Modal IDs: `drug-modal`, `trust-modal`, `directorate-modal` (each needs `opened` state)
- Read `dash_app/components/drawer.py` to understand the current chip/accordion structure
- Read `dash_app/callbacks/drawer.py` to understand fragment matching logic
- The simplest approach: move the same ChipGroup and accordion components from the drawer into modals with the SAME IDs, avoiding any callback rewiring
- After 7.4, Task 7.5 adds trigger buttons in the filter bar to open each modal
### Blocked items:
- None

## Iteration 21 — 2026-02-06
### Task: Phase 7 — Tasks 7.4 + 7.5 (Replace drawer with modals + filter bar triggers)
### Why this task:
- Tasks 7.1-7.3 complete. Task 7.4 is next in Phase 7 dependency order.
- Progress.txt from iteration 20 explicitly recommended this task.
- Tasks 7.4 and 7.5 are tightly coupled — modals (7.4) need trigger buttons (7.5) to open them, so both done together.
### Status: COMPLETE
### What was done:
- **Used frontend-developer agent** to design modal UX for 3 filter types:
  - Drug Selection: `size="lg"`, centered, with selection count badge in title
  - Trust Selection: `size="sm"`, centered, minimal layout for 7 items
  - Directorate Browser: `size="xl"`, centered, with nested accordion
  - Live filtering (no "Apply" button) — matches existing behavior
  - Per-modal and global "Clear" buttons
- **Created `dash_app/components/modals.py`**:
  - `make_drug_modal()`: dmc.Modal with ChipGroup (id=all-drugs-chips, same as drawer), count badge, clear button
  - `make_trust_modal()`: dmc.Modal with ChipGroup (id=trust-chips, same as drawer), count badge, clear button
  - `make_directorate_modal()`: dmc.Modal with nested accordion (same structure as drawer), drug fragment badges with pattern-matching IDs, clear button
  - `make_modals()`: returns all three modals in a wrapper div
  - Component IDs preserved from drawer — no callback rewiring needed
- **Updated `dash_app/components/filter_bar.py`**:
  - Added 3 filter trigger buttons: "Drugs", "Trusts", "Directorates" with count badge spans
  - Added "Clear All" button for global filter reset
  - Buttons use new `.filter-btn` CSS class
- **Created `dash_app/callbacks/modals.py`**:
  - 3 modal open callbacks (one per modal, triggered by filter bar buttons)
  - `handle_selection_actions`: unified callback for fragment matching + 4 clear triggers (per-drug, per-trust, directorate modal, global)
  - `update_count_badges`: updates both filter bar badges and modal header badges
- **Updated `dash_app/app.py`**: replaced `make_drawer()` with `make_modals()`
- **Updated `dash_app/callbacks/__init__.py`**: registers modal callbacks instead of drawer
- **Deleted `dash_app/components/drawer.py`** and **`dash_app/callbacks/drawer.py`** (dead code after modal replacement)
- **Updated `dash_app/assets/nhs.css`**: replaced drawer CSS with filter-btn and modal CSS classes
- **Removed search inputs** from drug and directorate modals — dynamic chip filtering conflicts with Dash's callback architecture (ChipGroup children can't be both layout-defined and callback-output). 42 drugs is scannable without search.
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 10 callbacks registered
- Tier 1 (App starts): `python run_dash.py` — "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 2 (Layout):
  - 3 modals in layout: drug-modal, trust-modal, directorate-modal (all centered, correctly sized)
  - 32 string IDs, 540 pattern-match IDs — zero duplicates
  - All key IDs present: modal IDs, trigger buttons, count badges, chip groups
  - Filter bar has: toggle pills + date dropdowns + divider + Drugs/Trusts/Directorates buttons + Clear All
- Tier 3 (Functional):
  - Callback chain verified: open-drug-modal → drug-modal.opened, etc.
  - Fragment matching preserved: drug-fragment badges → all-drugs-chips.value
  - Clear buttons: clear-drug-selection → drugs only, clear-trust-selection → trusts only, clear-all-filters → both
  - Count badges: all-drugs-chips.value + trust-chips.value → filter-btn badges + modal header badges
  - App-state chain: chip values → update_app_state → chart-data → chart + KPIs (unchanged)
### Files changed:
- `dash_app/components/modals.py` — NEW: 3 filter modals (drug, trust, directorate)
- `dash_app/callbacks/modals.py` — NEW: modal open/close, selection actions, count badges
- `dash_app/components/filter_bar.py` — Updated: added trigger buttons + clear all
- `dash_app/app.py` — Updated: replaced drawer with modals
- `dash_app/callbacks/__init__.py` — Updated: registers modal callbacks
- `dash_app/assets/nhs.css` — Updated: replaced drawer CSS with filter-btn + modal CSS
- `dash_app/components/drawer.py` — DELETED (replaced by modals.py)
- `dash_app/callbacks/drawer.py` — DELETED (replaced by modals.py)
- `IMPLEMENTATION_PLAN.md` — Tasks 7.4, 7.5 marked [x], all completion criteria marked [x]
### Committed: f2c5b26 "refactor: replace dmc.Drawer with dmc.Modal for filter selection (Task 7.4 + 7.5)"
### Patterns discovered:
- Reusing the same component IDs (all-drugs-chips, trust-chips, drug-fragment pattern) when migrating from drawer to modals means zero callback rewiring. The callbacks don't care whether the component is inside a Drawer or Modal — they reference by ID.
- dmc.Modal title prop can contain components with IDs (e.g., dmc.Badge with id="drug-modal-count"). Dash finds these during callback resolution even though they're in the title prop, not children.
- dmc.Modal with `centered=True` and `overlayProps={"backgroundOpacity": 0.55, "blur": 3}` gives a clean modern feel.
- Dynamic chip filtering in Dash is architecturally difficult because ChipGroup children are set at layout time and can't easily be toggled without making children an Output (which conflicts with value already being an Output). Search input deferred to future enhancement.
- Filter bar button count badges use CSS class toggling (hidden/visible) rather than conditional rendering — simpler with Dash's callback model.
### All tasks COMPLETE:
- ALL tasks in IMPLEMENTATION_PLAN.md are now marked [x]
- ALL completion criteria are satisfied
### Next iteration should:
- ALL TASKS COMPLETE. No next iteration needed.
- Run final validation: `python run_dash.py`, verify all 13 completion criteria at bottom of IMPLEMENTATION_PLAN.md.
### Blocked items:
- None

## Iteration 22 — 2026-02-06
### Task: Section 8 — Both tasks (prune empty ancestors + KPIs reflect filters)
### Why this task:
- All Phase 7 tasks complete. Section 8 had 2 remaining unchecked items.
- Task 8.1 (hide empty directorates when filtering drugs) is a visible UX issue: filtering by Immunoglobulin showed 42 empty directorate boxes.
- Task 8.2 (KPIs reflect filtered data) was partially working but KPIs showed unfiltered root node totals (11,118 patients / £130.5M) even when a single drug was selected.
- Both tasks are in the same file (`pathway_queries.py`) so done together.
### Status: COMPLETE
### What was done:
- **Empty ancestor pruning** (`_prune_empty_ancestors()`):
  - Added after row fetch in `load_pathway_nodes()`, activated when drug or directorate filters are present
  - Two-pass algorithm: first keeps root (L0), leaves (L3+), and nodes referenced as parents; second pass rechecks L1 nodes still have L2 children
  - IMMUNOGLOBULIN filter: 54 nodes → 10 nodes (1 root, 1 trust, 4 directorates, 4 drugs)
  - All filter combinations tested: single root, zero level 1-2 orphans, icicle charts render correctly
  - Pre-existing level 3+ orphans (pathway sequences like INFLIXIMAB→ADALIMUMAB where parent drug doesn't match filter) are unrelated to this change and don't affect chart rendering
- **Filtered KPI values**:
  - When any entity filter (drug, directorate, trust) is active, `unique_patients` and `total_cost` are summed from level-3 drug nodes instead of using root node's pre-computed totals
  - IMMUNOGLOBULIN: 78 patients, £3.8M (was 11,118 / £130.5M)
  - ADALIMUMAB: 3,236 patients, £29.2M
  - NNUH trust: 5,887 patients, £74.5M
  - No filter: unchanged (11,118 / £130.6M — uses root node as before)
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 10 callbacks registered
- Tier 1 (App starts): `python run_dash.py` — "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional):
  - 8 filter combinations tested: no filter, IMMUNOGLOBULIN, ADALIMUMAB, multi-drug, directorate, trust, drug+trust, indication+drug
  - All produce valid icicle hierarchy (single root, zero level 1-2 orphans)
  - All icicle charts render via create_icicle_from_nodes()
  - KPI values reflect filtered data in all cases
### Files changed:
- `src/data_processing/pathway_queries.py` — Added: _prune_empty_ancestors(), filtered KPI summation
- `IMPLEMENTATION_PLAN.md` — Section 8 tasks marked [x]
### Committed: de08d4b "fix: prune empty ancestor nodes and update KPIs for filtered views (Section 8)"
### Patterns discovered:
- `_prune_empty_ancestors` uses a two-pass approach: first pass identifies nodes referenced as parents, second pass verifies L1 nodes still have L2 children after the first prune. This handles cascading (trust with no kept directorates → trust removed).
- Level 3+ orphans in drug-filtered results are expected: pathway sequences like INFLIXIMAB→ADALIMUMAB have a parent drug node (INFLIXIMAB) that doesn't match the ADALIMUMAB filter. Plotly silently drops these orphaned leaf nodes, which is correct behavior for displaying filtered pathways.
- KPI summation at level 3 (drug nodes) is the right level because drug nodes represent the total patients/cost for that drug under a directorate. Summing level 4+ (pathway nodes) would double-count.
### All tasks COMPLETE:
- ALL tasks in IMPLEMENTATION_PLAN.md are now marked [x], including Section 8
- ALL completion criteria are satisfied
### Next iteration should:
- ALL TASKS COMPLETE. No next iteration needed.
### Blocked items:
- None

---

## Manual Intervention — 2026-02-06
### Reason: Add 7 new analytics chart tabs (Phase 9)
### Changes made:
- IMPLEMENTATION_PLAN.md: Added Phase 9 with tasks 9.1–9.10 (10 tasks covering infrastructure, queries, 7 charts, and polish)
- progress.txt: Added this intervention entry with context for next iteration
- guardrails.md: Added tab architecture guardrails (lazy rendering, single dcc.Graph, active_tab in app-state)
- RALPH_PROMPT.md: Updated focus description to reference Phase 9 analytics charts
### Tasks reset: none (all previous work valid — this is purely additive)
### Tasks added:
- 9.1: Parsing utilities + tab infrastructure (tabs in chart_card.py, remove sidebar chart views)
- 9.2: Query functions for all 7 chart types (in src/data_processing/pathway_queries.py)
- 9.3: First-Line Market Share — horizontal grouped bar chart
- 9.4: Pathway Cost Effectiveness — lollipop/dot plot with retention annotations
- 9.5: Cost Waterfall — Plotly waterfall chart by directorate
- 9.6: Drug Switching Sankey — Sankey diagram of treatment line transitions
- 9.7: Dosing Interval Comparison — grouped bars from parsed average_spacing
- 9.8: Directorate × Drug Heatmap — matrix chart with count/cost toggle
- 9.9: Treatment Duration — horizontal bars of avg_days by drug
- 9.10: Final integration, polish, documentation
### Context for next iteration:
- Previous 22 iterations completed the full Dash migration (Phases 0–8). The app runs at http://localhost:8050 with icicle chart, filters, KPIs.
- Phase 9 adds 7 new chart tabs alongside the existing icicle chart.
- **Architecture**: Tab bar goes INSIDE `chart_card.py` (replacing the sidebar chart view items). Use `dmc.Tabs` or `dcc.Tabs`. Store `active_tab` in `app-state`.
- **Sidebar changes**: Remove "Chart Views" section from sidebar.py. Keep "Overview" section.
- **Lazy rendering**: Only compute the active tab's chart. Check `active_tab` in the chart callback and dispatch to the correct figure builder.
- **Shared code**: New query functions in `src/data_processing/pathway_queries.py`. New figure functions in `src/visualization/`. Thin wrappers in `dash_app/data/queries.py`.
- **Parsing**: `ids` column at level 4+ has format `"TRUST - DIRECTORY - DRUG_A - DRUG_A|DRUG_B"`. Split by ` - `, take level 3+ segments, split by `|` for drug sequence.
- **average_spacing**: HTML string like `"given X times with Y weekly interval"` — needs regex parsing.
- **Existing chart callback**: `dash_app/callbacks/chart.py:update_chart()` currently renders icicle only. Needs conditional dispatch based on `active_tab`.
- **Start with Task 9.1** (parsing utilities + tab infrastructure) — it's the foundation for all other tasks.
### New guardrails added:
- Tab architecture: lazy rendering, single dcc.Graph, active_tab in app-state
- Chart figure functions: shared in src/visualization/, not duplicated in dash_app/
- Parsing utilities: shared in src/data_processing/, tested independently

## Iteration 23 — 2026-02-06
### Task: Phase 9 — Task 9.1 (Parsing utilities + tab infrastructure)
### Why this task:
- First task in Phase 9, foundation for all other analytics chart tasks
- Progress.txt explicitly recommends starting here
- No dependencies — pure infrastructure
### Status: COMPLETE
### What was done:
- **Created `src/data_processing/parsing.py`** with 3 shared parsing functions:
  1. `parse_average_spacing(html)` — regex extracts drug_name, dose_count, weekly_interval, total_weeks from HTML strings. Returns list of dicts (supports multi-drug entries). Tested with real data.
  2. `parse_pathway_drugs(ids, level)` — splits `ids` column by ` - ` and returns drug names from index 3 onwards. Returns empty list for level < 3.
  3. `calculate_retention_rate(nodes)` — for each level 4+ pathway, calculates what % of patients do NOT escalate to the next treatment line. Tested with real RHEUMATOLOGY data (e.g., ADALIMUMAB→ETANERCEPT: 82.6% retained).
- **Updated `dash_app/components/chart_card.py`**:
  - Added `TAB_DEFINITIONS` list of 8 (id, label) tuples exported for use by callbacks
  - Tab bar uses plain `html.Button` elements with existing `.chart-tab` / `.chart-tab--active` CSS classes
  - Single `dcc.Graph` shared across all tabs (lazy rendering)
- **Updated `dash_app/components/sidebar.py`**:
  - Removed "Chart Views" section (Icicle/Sankey/Timeline items) — chart selection is now in the tab bar
  - Only "Overview" section with "Pathway Overview" remains
  - Removed unused icon definitions (icicle, sankey, timeline SVGs)
- **Updated `dash_app/callbacks/chart.py`**:
  - Added `switch_tab` callback: 8 tab button Inputs → `active-tab` store + 8 CSS class Outputs
  - Modified `update_chart` to read `active-tab` store and dispatch: icicle renders via `create_icicle_from_nodes()`; other tabs show "coming soon" placeholder
  - `_empty_figure()` and `_generate_chart_title()` preserved unchanged
- **Updated `dash_app/app.py`**:
  - Added `dcc.Store(id="active-tab", storage_type="memory", data="icicle")` — separate store for tab state, avoids `allow_duplicate` conflicts with `app-state`
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 2 (Layout):
  - 8 tab IDs present: tab-icicle, tab-market-share, tab-cost-effectiveness, tab-cost-waterfall, tab-sankey, tab-dosing, tab-heatmap, tab-duration
  - active-tab store present with default "icicle"
  - Sidebar: "Chart Views" section removed, only "Overview" remains
  - 11 callbacks registered (was 10, +1 for switch_tab)
- Tier 3 (Functional):
  - Icicle chart still renders: 293 nodes, 11,118 patients, correct figure
  - Parsing utilities: all 3 functions tested with real SQLite data
  - Tab label lookup works for all 8 tabs
  - Placeholder figures render with correct "coming soon" message
### Files changed:
- `src/data_processing/parsing.py` — NEW: 3 shared parsing functions
- `dash_app/components/chart_card.py` — Updated: 8-tab bar with TAB_DEFINITIONS
- `dash_app/components/sidebar.py` — Updated: removed Chart Views section
- `dash_app/callbacks/chart.py` — Updated: switch_tab callback + tab-based dispatch
- `dash_app/app.py` — Updated: added active-tab dcc.Store
- `IMPLEMENTATION_PLAN.md` — Task 9.1 marked [x]
### Committed: fe2d048 "feat: add parsing utilities and 8-tab chart infrastructure (Task 9.1)"
### Patterns discovered:
- `active_tab` stored in a SEPARATE `dcc.Store(id="active-tab")` rather than inside `app-state`. This avoids `allow_duplicate=True` (since `app-state` is already Output of `update_app_state` in filters.py). Tab switching doesn't trigger pathway data reload — only visual switching.
- Tab buttons use plain `html.Button` with existing `.chart-tab` CSS classes from nhs.css. This matches the design reference exactly, unlike DMC Tabs or dcc.Tabs which would need CSS overrides.
- `TAB_DEFINITIONS` exported from `chart_card.py` as a module-level constant, imported by `chart.py` for callback Input/Output construction. Keeps tab names defined in one place.
- `switch_tab` callback uses `*tab_outputs` and `*tab_inputs` unpacking to dynamically wire all 8 tabs. Adding a new tab only requires adding to `TAB_DEFINITIONS`.
- `parse_average_spacing` returns a LIST of dicts (not a single dict) because level 4+ nodes can have multi-drug entries in the average_spacing HTML.
### Next iteration should:
- Start Task 9.2 — Query functions for all 7 chart types
- Add 7 new query functions to `src/data_processing/pathway_queries.py`:
  - `get_drug_market_share()` — Level 3 nodes grouped by directory
  - `get_pathway_costs()` — Level 4+ nodes with cost_pp_pa
  - `get_cost_waterfall()` — Level 2 nodes with cost_pp_pa
  - `get_drug_transitions()` — Level 3+ for Sankey transitions
  - `get_dosing_intervals()` — Level 3 nodes with parsed average_spacing
  - `get_drug_directory_matrix()` — Level 3 pivoted as directory × drug
  - `get_treatment_durations()` — Level 3 with avg_days
- Add thin wrappers in `dash_app/data/queries.py`
- Follow same pattern as existing `load_pathway_nodes()`: accept db_path, use sqlite3.Row, parameterized queries, return JSON-serializable data
- Read `src/data_processing/pathway_queries.py` first to understand existing patterns
- Read `AdditionalAnalytics.md` for each chart's data requirements
- Test each query with `python -c "..."` against real data
### Blocked items:
- None

## Iteration 24 — 2026-02-06
### Task: Phase 9 — Task 9.2 (Query functions for all 7 chart types)
### Why this task:
- Task 9.1 (parsing + tab infrastructure) complete in iteration 23
- Progress.txt explicitly recommended this task next
- All 7 chart implementations (9.3–9.9) depend on these query functions
### Status: COMPLETE
### What was done:
- **Added 7 query functions to `src/data_processing/pathway_queries.py`**:
  1. `get_drug_market_share()` — Level 3 nodes aggregated across trusts by directory+drug, with computed proportions. 63 rows for directory chart, 111 for indication.
  2. `get_pathway_costs()` — Level 4+ nodes with pathway labels (drug_sequence joined with →). 38 pathways for RHEUMATOLOGY.
  3. `get_cost_waterfall()` — Aggregates level 3 cost/patients by directory (level 2 cost_pp_pa is always "N/A"). 12 directorates sorted by cost_pp desc.
  4. `get_drug_transitions()` — Parses drug_sequence into source→target transitions with ordinal line labels (e.g., "ADALIMUMAB (1st)" → "ETANERCEPT (2nd)"). Returns {nodes, links} for Sankey. 49 nodes, 65 links.
  5. `get_dosing_intervals()` — Uses `parse_average_spacing()` to extract weekly_interval/dose_count/total_weeks. 124 rows for all drugs.
  6. `get_drug_directory_matrix()` — Pivots level 3 into directory × drug matrix with patients/cost/cost_pp_pa. 12×39 matrix, 63 non-empty cells.
  7. `get_treatment_durations()` — Weighted avg of avg_days by patients across trusts. 59 entries unfiltered.
- **Added helpers**: `_safe_float()` for None/N/A handling, `_ordinal()` for Sankey node labels
- **Added 7 thin wrappers** to `dash_app/data/queries.py` (resolve DB_PATH, delegate to src/)
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 11 callbacks registered
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional): All 7 queries tested with real data, both chart types
### Files changed:
- `src/data_processing/pathway_queries.py` — Added: 7 query functions + 2 helpers
- `dash_app/data/queries.py` — Added: 7 thin wrapper functions
- `IMPLEMENTATION_PLAN.md` — Task 9.2 marked [x]
### Committed: d98cd4f "feat: add 7 analytics chart query functions (Task 9.2)"
### Patterns discovered:
- `cost_pp_pa` at level 2 is always "N/A". Waterfall must compute from level 3 aggregation.
- `cost_pp_pa` at level 3 is a string (including "N/A"). Use `_safe_float()`.
- Sankey ordinal suffix ("1st", "2nd") prevents self-loops for same-drug transitions.
- Treatment duration uses weighted average for cross-trust aggregation.
- All queries work seamlessly with both "directory" and "indication" chart types.
### Next iteration should:
- Start Task 9.3 — First-Line Market Share chart (Tab 2)
- Sub-steps:
  1. Create figure function in `src/visualization/` — `create_market_share_figure(data)` for horizontal grouped bar chart
  2. Wire into `update_chart` in `dash_app/callbacks/chart.py` — dispatch on active_tab="market-share"
  3. The query `get_drug_market_share()` returns [{directory, drug, patients, proportion, cost, cost_pp_pa}] sorted by total desc
  4. Use NHS blue palette, one cluster per directorate, drugs as bars within
- Read `dash_app/callbacks/chart.py` to understand the tab dispatch pattern
- Read `src/visualization/plotly_generator.py` to see existing figure function pattern
### Blocked items (iter 24):
- None

## Iteration 25 — 2026-02-06
### Task: Phase 9 — Task 9.3 (First-Line Market Share chart — Tab 2)
### Why this task:
- Tasks 9.1 (tab infra) and 9.2 (query functions) complete. Task 9.3 is the first actual chart.
- Progress.txt from iteration 24 explicitly recommended this task.
- Simplest chart — validates the tab-switching + figure-building pattern for all subsequent charts.
### Status: COMPLETE
### What was done:
- **Created `create_market_share_figure(data, title)` in `src/visualization/plotly_generator.py`**:
  - Horizontal stacked bar chart: one row per directorate (sorted by total patients desc), drugs stacked within
  - NHS colour palette (15 colours cycling for different drugs)
  - Hover shows drug name, directorate, patient count, share %, cost, cost p.p.p.a.
  - Dynamic height based on number of directorates (60px per row + margins)
  - Legend positioned below chart in horizontal orientation
  - Transparent backgrounds, Source Sans 3 font, matching NHS design aesthetic
- **Added `_render_market_share(app_state, title)` helper in `dash_app/callbacks/chart.py`**:
  - Extracts filter params from app-state (date_filter_id, chart_type, directory, trust)
  - Calls `get_drug_market_share()` wrapper then `create_market_share_figure()`
  - Handles empty data and exceptions gracefully
  - For single-directory filter: passes to query as `directory=` param. For multi: shows all.
  - For single-trust filter: passes to query as `trust=` param. For multi: shows all.
- **Updated `update_chart` dispatch**: `active_tab == "market-share"` → `_render_market_share()`
- **Architecture note**: Separate `dash_app/callbacks/market_share.py` file not needed — the dispatch pattern uses a single `update_chart` callback with helper functions. This avoids `allow_duplicate=True` on `Output("pathway-chart", "figure")`.
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 11 callbacks registered
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional):
  - Directory chart: 63 rows, 12 directories, 39 drugs, 39 traces — renders correctly
  - Indication chart: 111 rows, 39 traces — renders correctly
  - Trust filter (NNUH): 45 rows — filters correctly
  - Date filter (2yr_12mo): 47 rows — filters correctly
  - Empty data: returns empty figure — handled correctly
  - Icicle still works: 293 nodes, 11,118 patients — no regression
### Files changed:
- `src/visualization/plotly_generator.py` — Added: `create_market_share_figure()` (120 lines)
- `dash_app/callbacks/chart.py` — Added: `_render_market_share()` helper + dispatch branch
- `IMPLEMENTATION_PLAN.md` — Task 9.3 marked [x]
### Committed: f8960a3 "feat: add First-Line Market Share chart (Task 9.3)"
### Patterns discovered:
- Market share chart uses `barmode="stack"` for cleaner visualization — stacked bars show proportion within each directorate clearly
- The `_render_X()` helper pattern (query + figure builder in one function) is clean and reusable for all subsequent chart types
- Market share query returns data sorted by directory total patients desc — reversed for horizontal bars so highest total is at top
- 39 unique drugs creates many traces, but most drugs only appear in 1-2 directorates so visual complexity is manageable
- `get_drug_market_share()` accepts single directory/trust — for multi-select, pass None to show all
### Next iteration should:
- Start Task 9.4 — Pathway Cost Effectiveness chart (Tab 3)
- Sub-steps:
  1. Create figure function in `src/visualization/plotly_generator.py` — `create_cost_effectiveness_figure(data, retention)`
  2. Build horizontal lollipop/dot chart: Y-axis = pathway label, X-axis = £ per patient per annum
  3. Dot size = patient count, colour gradient: green (cheap) → amber → red (expensive)
  4. Add retention rate annotations using `calculate_retention_rate()` from `src/data_processing/parsing.py`
  5. Wire into `update_chart` via `_render_cost_effectiveness()` helper
  6. The query `get_pathway_costs()` returns [{ids, pathway_label, cost_pp_pa, patients, directory, drug_sequence}]
  7. Use `parse_pathway_drugs()` if needed for pathway labels
- Read `src/data_processing/parsing.py` for `calculate_retention_rate()` signature
- Read `get_pathway_costs()` data shape
### Blocked items:
- None

## Iteration 26 — 2026-02-06
### Task: Phase 9 — Task 9.4 (Pathway Cost Effectiveness chart — Tab 3)
### Why this task:
- Tasks 9.1 (tab infra), 9.2 (queries), 9.3 (market share) complete. Task 9.4 is next in sequence.
- Progress.txt from iteration 25 explicitly recommended this task.
- This is the user's key insight chart — shows cost effectiveness of different treatment pathways with retention annotations.
### Status: COMPLETE
### What was done:
- **Fixed `calculate_retention_rate()` in `src/data_processing/parsing.py`**:
  - Added `_get_patients()` helper that accepts either `"value"` or `"patients"` key
  - Bug: function returned 0 entries because `get_pathway_costs()` returns data with `"patients"` key, not `"value"`. The original function only checked `node.get("value", 0)`.
  - Now works with both chart data nodes (`value`) and pathway cost data (`patients`).
- **Created `create_cost_effectiveness_figure(data, retention, title)` in `src/visualization/plotly_generator.py`**:
  - Horizontal lollipop chart: stick lines from 0 to cost, dot at cost value
  - Y-axis = pathway label (e.g., "Adalimumab → Secukinumab → Rituximab"), X-axis = £ per patient per annum
  - Dot size scaled by patient count (min 8px, max 30px)
  - Colour gradient: green (#009639) for cheap, amber (#ED8B00) for mid, red (#DA291C) for expensive
  - Hover shows: pathway name, cost p.p.p.a., patients, total cost, avg duration, directorate, treatment lines, retention rate
  - Retention annotations for pathways with <90% retention and ≥10 patients (up to 8 annotations)
  - Caps at top 40 pathways by cost for readability
  - Dynamic height based on pathway count
  - NHS design aesthetic: Source Sans 3, transparent backgrounds, clean gridlines
- **Added `_render_cost_effectiveness(app_state, chart_data, title)` helper in `dash_app/callbacks/chart.py`**:
  - Extracts filter params (date_filter_id, chart_type, single directory/trust)
  - Calls `get_pathway_costs()` → `calculate_retention_rate()` → `create_cost_effectiveness_figure()`
  - Handles empty data and exceptions gracefully
- **Wired into `update_chart` dispatch**: `active_tab == "cost-effectiveness"` → `_render_cost_effectiveness()`
- **Architecture note**: Like market share, uses the `_render_X()` helper pattern within the single `update_chart` callback. No separate callback file needed.
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 11 callbacks registered
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional):
  - Directory chart: 111 pathways, 111 retention entries, 41 traces — renders correctly
  - Indication chart: 92 pathways, 92 retention entries — renders correctly
  - RHEUMATOLOGY filter: 38 pathways, 39 traces — filters correctly
  - NNUH trust filter: 50 pathways — filters correctly
  - Empty filter (nonexistent directory): 0 pathways — returns empty figure
  - Date filter (2yr_12mo): 17 pathways — filters correctly
  - Icicle still works: 293 nodes, 11,118 patients — no regression
  - Market share still works — no regression
### Files changed:
- `src/data_processing/parsing.py` — Fixed: calculate_retention_rate() accepts both 'value' and 'patients' keys
- `src/visualization/plotly_generator.py` — Added: create_cost_effectiveness_figure() (~170 lines)
- `dash_app/callbacks/chart.py` — Added: _render_cost_effectiveness() helper + dispatch branch
- `IMPLEMENTATION_PLAN.md` — Task 9.4 marked [x]
### Committed: 4ef7239 "feat: add Pathway Cost Effectiveness lollipop chart (Task 9.4)"
### Patterns discovered:
- `get_pathway_costs()` returns `"patients"` as the key while chart data nodes use `"value"`. Shared parsing functions that operate on either data format need to handle both keys.
- Lollipop chart pattern: separate `go.Scatter` with `mode="lines"` for sticks + one `go.Scatter` with `mode="markers"` for dots. Each stick is its own trace (40 pathways = 41 total traces).
- Retention annotation threshold: `rate < 90% AND patients >= 10` avoids annotating trivial pathways. Cap of 8 annotations prevents visual clutter.
- The `_render_X(app_state, chart_data, title)` pattern differs slightly from market share (`_render_market_share(app_state, title)`) by accepting `chart_data` — this is because cost effectiveness queries directly from SQLite rather than using chart-data nodes. Future charts may also use this pattern.
### Next iteration should:
- Start Task 9.5 — Cost Waterfall chart (Tab 4)
- Sub-steps:
  1. Create figure function in `src/visualization/plotly_generator.py` — `create_cost_waterfall_figure(data, title)`
  2. Build Plotly waterfall chart from `get_cost_waterfall()` data
  3. Each bar = one directorate's average cost_pp_pa, sorted highest to lowest
  4. NHS colours, responds to chart_type toggle, date filter, trust filter
  5. Wire into `update_chart` via `_render_cost_waterfall()` helper
- The query `get_cost_waterfall()` returns [{directory, cost_pp, patients, total_cost}] sorted by cost_pp desc
- Read `get_cost_waterfall()` in pathway_queries.py for exact data shape
### Blocked items:
- None

## Iteration 28 — 2026-02-06
### Task: Phase 9 — Task 9.6 (Drug Switching Sankey chart — Tab 5)
### Why this task:
- Tasks 9.1–9.5 complete. Task 9.6 is next in sequence.
- Progress.txt from iteration 27 explicitly recommended this task.
- Sankey is a well-defined chart type with pre-existing query function (`get_drug_transitions()`).
### Status: COMPLETE
### What was done:
- **Created `create_sankey_figure(data, title)` in `src/visualization/plotly_generator.py`** (~130 lines):
  - Plotly `go.Sankey` diagram: nodes = drugs with ordinal suffixes (e.g., "ADALIMUMAB (1st)"), links = patient transitions
  - Nodes coloured by base drug name — same drug gets same colour across treatment lines (1st, 2nd, 3rd)
  - Link colours use source node colour at 40% opacity for visual flow
  - NHS colour palette (15 colours cycling), Source Sans 3 font
  - Custom hover: node shows drug name + total patients, link shows source → target + patient count
  - Dynamic height: scales with number of unique drug bases
  - `arrangement="snap"` for clean node positioning
- **Added `_render_sankey(app_state, title)` helper in `dash_app/callbacks/chart.py`**:
  - Extracts filter params (date_filter_id, chart_type, single directory/trust)
  - Calls `get_drug_transitions()` wrapper then `create_sankey_figure()`
  - Handles empty data and exceptions gracefully
- **Wired into `update_chart` dispatch**: `active_tab == "sankey"` → `_render_sankey()`
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional):
  - Directory chart: 49 nodes, 65 links — renders correctly
  - Indication chart: 34 nodes, 35 links — renders correctly
  - RHEUMATOLOGY filter: 24 nodes, 27 links — filters correctly
  - NNUH trust filter: 38 nodes, 47 links — filters correctly
  - Date filter (2yr_12mo): 13 nodes, 12 links — filters correctly
  - Empty data: returns empty figure — handled correctly
  - Icicle, market share, cost effectiveness, cost waterfall — no regressions
### Files changed:
- `src/visualization/plotly_generator.py` — Added: `create_sankey_figure()` (~130 lines)
- `dash_app/callbacks/chart.py` — Added: `_render_sankey()` helper + dispatch branch
- `IMPLEMENTATION_PLAN.md` — Task 9.6 marked [x]
### Committed: 4ffcdf4 "feat: add Drug Switching Sankey diagram (Task 9.6)"
### Patterns discovered:
- Sankey node colouring by base drug name (stripping ordinal suffix) provides visual continuity — users can track the same drug across treatment lines by colour.
- `hex_to_rgba()` helper for link transparency is a reusable pattern. Using 35% opacity keeps links readable without obscuring overlapping flows.
- `arrangement="snap"` works better than "perpendicular" for this data shape — it allows Plotly to optimally position nodes vertically.
- The `_render_sankey(app_state, title)` pattern matches `_render_market_share()` and `_render_cost_waterfall()` — all take app_state and title, query independently from SQLite.
### Next iteration should:
- Start Task 9.7 — Dosing Interval Comparison chart (Tab 6)
- Sub-steps:
  1. Create figure function in `src/visualization/plotly_generator.py` — `create_dosing_figure(data, title)`
  2. Build horizontal grouped bar chart from `get_dosing_intervals()` data
  3. Uses `parse_average_spacing()` to extract weekly interval numbers from HTML strings
  4. Y-axis = trust or directorate, X-axis = weekly interval, grouped by drug
  5. Wire into `update_chart` via `_render_dosing()` helper
  6. Responds to drug filter, date filter, chart type toggle
- Read `get_dosing_intervals()` in pathway_queries.py for exact data shape
- Read `parse_average_spacing()` in `src/data_processing/parsing.py` for HTML parsing logic
### Blocked items:
- None

## Iteration 27 — 2026-02-06
### Task: Phase 9 — Task 9.5 (Cost Waterfall chart — Tab 4)
### Why this task:
- Tasks 9.1 (tab infra), 9.2 (queries), 9.3 (market share), 9.4 (cost effectiveness) complete. Task 9.5 is next in sequence.
- Progress.txt from iteration 26 explicitly recommended this task.
- Cost Waterfall is a straightforward bar chart — validates the pattern for remaining charts.
### Status: COMPLETE
### What was done:
- **Created `create_cost_waterfall_figure(data, title)` in `src/visualization/plotly_generator.py`**:
  - Vertical bar chart: one bar per directorate, sorted by cost_pp (cost per patient) descending
  - NHS colour palette cycling through 12 colours for visual distinction
  - Text labels above bars showing £ values, patient count annotations below
  - Weighted average reference line (dashed red) across all directorates
  - Hover shows: directorate name, cost per patient, patient count, total cost
  - Auto tick angle rotation for >6 bars, dynamic sizing
  - NHS design aesthetic: Source Sans 3, transparent backgrounds, clean gridlines
- **Added `_render_cost_waterfall(app_state, title)` helper in `dash_app/callbacks/chart.py`**:
  - Extracts filter params (date_filter_id, chart_type, single trust)
  - Calls `get_cost_waterfall()` wrapper then `create_cost_waterfall_figure()`
  - Handles empty data and exceptions gracefully
- **Wired into `update_chart` dispatch**: `active_tab == "cost-waterfall"` → `_render_cost_waterfall()`
- **Architecture note**: Uses `go.Bar` (not `go.Waterfall`) for cleaner control — each bar shows absolute cost_pp for a directorate. The weighted average reference line gives the "running total" context that a true waterfall would provide.
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` — OK, 11 callbacks registered
- Tier 1 (App starts): `python run_dash.py` → "Dash is running on http://127.0.0.1:8050/" — no errors
- Tier 3 (Functional):
  - Directory chart: 12 bars (directorates), sorted by cost_pp desc — renders correctly
  - Indication chart: 39 bars (indications) — renders correctly
  - Trust filter (NNUH): 10 bars — filters correctly
  - Date filter (2yr_12mo): 12 bars — filters correctly
  - Empty data: returns empty figure — handled correctly
  - Icicle still works: 293 nodes, 11,118 patients — no regression
  - Market share still works: 63 rows, 39 traces — no regression
  - Cost effectiveness still works — no regression
### Files changed:
- `src/visualization/plotly_generator.py` — Added: `create_cost_waterfall_figure()` (~130 lines)
- `dash_app/callbacks/chart.py` — Added: `_render_cost_waterfall()` helper + dispatch branch
- `IMPLEMENTATION_PLAN.md` — Task 9.5 marked [x]
### Committed: 73a8d1a "feat: add Cost Waterfall bar chart (Task 9.5)"
### Patterns discovered:
- `go.Bar` is better than `go.Waterfall` for this use case — we want absolute values per directorate, not cumulative increments. Waterfall would show how each directorate adds to a running total, but the spec asks for comparing cost_pp across directorates independently.
- The weighted average reference line (`fig.add_hline()`) provides the "big picture" context — users can quickly see which directorates are above/below average cost per patient.
- Patient count annotations below bars (`yshift=-18`) give context on sample size without cluttering the main bars.
- The `_render_cost_waterfall(app_state, title)` pattern matches `_render_market_share()` — both take app_state and title, query independently from SQLite (not from chart-data nodes).
### Next iteration should:
- Start Task 9.6 — Drug Switching Sankey chart (Tab 5)
- Sub-steps:
  1. Create figure function in `src/visualization/plotly_generator.py` — `create_sankey_figure(data, title)`
  2. Build Plotly Sankey diagram from `get_drug_transitions()` data
  3. The query returns `{nodes: [{label, colour}], links: [{source, target, value, source_label, target_label}]}`
  4. Left nodes = 1st-line drugs, middle = 2nd-line, right = 3rd-line. Link width = patient count.
  5. Wire into `update_chart` via `_render_sankey()` helper
  6. Responds to all existing filters (directory, trust, date, chart type)
- Read `get_drug_transitions()` in pathway_queries.py for exact data shape
- Note: Sankey has ordinal suffixes (e.g., "ADALIMUMAB (1st)") to prevent self-loops
### Blocked items:
- None