# Progress Log — Dashboard Visualization Improvements

## Project Context

Working Dash application with 2 views (Patient Pathways + Trust Comparison), 13 chart functions in `plotly_generator.py`, and a complete callback chain. Now improving chart quality: bug fixes, visual polish, and new analytics.

**Current state**: Fully functional Dash app at http://localhost:8050 with icicle, Sankey, market share, cost effectiveness, cost waterfall, dosing, heatmap, and duration charts. Trust Comparison has 6 dedicated charts. All filters work.

**New goal**: Fix chart bugs (heatmap colorscale, legend overflow, trust color differentiation), add visual polish (consistent styling, smooth gradients), add new analytics (retention funnel, pathway depth, scatter, network), and new backend analytics (trends, dose distribution, timeline, NICE compliance).

## Key Architecture Patterns

### plotly_generator.py (PRIMARY target file)
- 13 chart functions, all accept list-of-dicts, return `go.Figure`
- Located at `src/visualization/plotly_generator.py` (~1782 lines)
- Key functions and approximate line numbers:
  - `create_icicle_from_nodes(nodes, title)` — L113
  - `create_market_share_figure(data, title)` — L247
  - `create_cost_effectiveness_figure(data, retention, title)` — L384
  - `create_cost_waterfall_figure(data, title)` — L562
  - `create_sankey_figure(data, title)` — L706
  - `create_dosing_figure(data, title, group_by)` — L837
  - `_dosing_by_drug(data, colours)` — L926
  - `_dosing_by_trust(data, colours)` — L1007
  - `create_heatmap_figure(data, title, metric)` — L1189
  - `create_duration_figure(data, title, show_directory)` — L1329
  - `create_trust_market_share_figure(data, title)` — L1481
  - `create_trust_heatmap_figure(data, title, metric)` — L1582
  - `create_trust_duration_figure(data, title)` — L1689
- NOTE: Line numbers will shift as you edit. Re-read the file each iteration.

### Callback chain
- `dash_app/callbacks/chart.py` — Patient Pathways tab dispatch (`_render_*` helpers → `update_chart`)
- `dash_app/callbacks/trust_comparison.py` — 6 Trust Comparison chart callbacks
- Tab switching: `active-tab` dcc.Store, tab IDs = `"tab-{short_id}"`
- TAB_DEFINITIONS in `chart_card.py` — currently: icicle, sankey

### Adding a new Patient Pathways tab
1. Query function in `src/data_processing/pathway_queries.py` (accept `db_path` param)
2. Thin wrapper in `dash_app/data/queries.py` (resolve DB_PATH)
3. Figure function in `src/visualization/plotly_generator.py`
4. Add to `TAB_DEFINITIONS` in `dash_app/components/chart_card.py`
5. Add `_render_*()` helper in `dash_app/callbacks/chart.py`
6. Add elif case in `update_chart()` dispatch

### State management
- 4 `dcc.Store` components: app-state, chart-data, reference-data, active-tab
- Unidirectional: filter inputs → app-state → chart-data → UI
- 20 registered callbacks total

### DMC version
- Dash 4.0.0 + DMC 2.5.1 (Mantine v7 based)
- `dmc.MantineProvider` wraps layout
- `dmc.SegmentedControl` available for metric toggles

### Flex chain for chart filling viewport
- Full flex chain: `.main` → `#view-container` → `#patient-pathways-view` → `.chart-card` → loading wrapper → `#chart-container` → `#pathway-chart`
- `responsive=True` on dcc.Graph + `autosize=True` in figure layout
- `dcc.Loading` wraps children in `.dash-loading-callback > div` — CSS must propagate flex through both

### Known heatmap bugs (to fix)
- Non-linear colorscale compresses 99% of range into identical blues
- No cell text — must hover every cell
- Light end (#F0F4F8) invisible against transparent background
- Fixed width can overflow container
- Fixed l=200 left margin wastes space

### Known legend bugs (to fix)
- Horizontal legends at y=-0.15 with fixed bottom margins overflow with 42 drugs
- Affects: market_share, trust_market_share, dosing, trust_duration

### Known color bugs (to fix)
- First 6 of 10 trust palette colors are blue variants — nearly indistinguishable
- _dosing_by_drug interpolates from one blue to another blue

## Iteration Log

## Iteration 1 — 2026-02-07
### Task: A.1 — Extract shared styling constants + `_base_layout()` helper
### Why this task:
- A.1 is the foundation for all subsequent Phase A tasks (A.2-A.4 all reference `_base_layout()` and the palette constants). Must be done first.
### Status: COMPLETE
### What was done:
- Added 7 module-level constants after `logger` line: `CHART_FONT_FAMILY`, `CHART_TITLE_SIZE`, `CHART_TITLE_COLOR`, `GRID_COLOR`, `ANNOTATION_COLOR`, `TRUST_PALETTE` (7 colors), `DRUG_PALETTE` (15 colors)
- Created `_base_layout(title, **overrides)` helper returning dict with: title (font family/size/color, centered), hoverlabel (white bg, #CBD5E1 border, font), paper/plot bgcolor transparent, autosize=True, base font family
- Applied `_base_layout()` to `create_icicle_from_nodes()` — replaced 20-line explicit layout block with `_base_layout()` call + 3 overrides (margin, hoverlabel size=14, clickmode)
- Also replaced hardcoded `"Source Sans 3, system-ui, sans-serif"` in icicle textfont with `CHART_FONT_FAMILY` constant
### Validation results:
- Tier 1 (Code): all imports pass, `python run_dash.py` starts cleanly
- Tier 2 (Visual): icicle figure title correct ("Patient Pathways — By Directory"), font family/size/color all match expected values
### Files changed:
- `src/visualization/plotly_generator.py` — added constants + `_base_layout()` + refactored icicle layout
- `IMPLEMENTATION_PLAN.md` — marked A.1 subtasks [x]
### Committed: 63c1801 "feat: add shared styling constants and _base_layout() helper (Task A.1)"
### Patterns discovered:
- The `_base_layout()` returns a plain dict that gets unpacked via `fig.update_layout(**layout)`. Callers pass chart-specific overrides as kwargs.
- Icicle hoverlabel uses `size=14` (slightly larger than base `13`) — preserved as override.
- Constants are at module level, so all functions in the file can reference them directly.
- Line numbers in IMPLEMENTATION_PLAN.md are now stale (shifted ~70 lines due to constants/helper insertion). Future iterations should search by function name.
### Next iteration should:
- Start with Task A.2: Fix heatmap colorscale + cell annotations. Read `create_heatmap_figure()` and `create_trust_heatmap_figure()` functions by searching for their names (line numbers have shifted).
- The `_base_layout()` and palette constants are now available — A.2 should use `_base_layout()` for both heatmap functions.
- Key heatmap fixes: linear 5-stop colorscale, cell text annotations, zmin=0, autosize, automargin, subtitle when >25 drugs.
### Blocked items:
- None

## Iteration 2 — 2026-02-07
### Task: A.2 — Fix heatmap colorscale + cell annotations
### Why this task:
- A.2 is the next Phase A task after A.1. It depends on `_base_layout()` (done in A.1). It addresses the most-reported heatmap bugs: non-linear colorscale, missing cell text, fixed width overflow, fixed left margin.
### Status: COMPLETE
### What was done:
- **create_heatmap_figure()**: Replaced non-linear 7-stop colorscale with linear 5-stop `[0.0 #E3F2FD, 0.25 #90CAF9, 0.5 #42A5F5, 0.75 #1E88E5, 1.0 #003087]`. Added `text=text_values, texttemplate="%{text}"` with metric-specific formatting (patients: `N`, cost: `£Nk`, cost_pp_pa: `£N`). Set `zmin=0`. Removed fixed `width`, using `autosize=True` from `_base_layout()`. Replaced `l=200` with `l=8` + `yaxis automargin=True`. Added subtitle annotation when 25-drug cap is hit. Reduced `xgap/ygap` to 1 when >15 columns.
- **create_trust_heatmap_figure()**: Applied identical fixes. Also fixed Unicode escapes (`\u00a3`, `\u00d7`, `\u2014`) back to literal characters.
- Both functions now use `_base_layout(chart_title)` for consistent styling — title color is now `#1E293B` (CHART_TITLE_COLOR) instead of old hardcoded `#003087`, title size is 18 (CHART_TITLE_SIZE) for both (trust heatmap was 16 before).
### Validation results:
- Tier 1 (Code): imports pass, `uv run python -c "from dash_app.app import app"` OK, `uv run python run_dash.py` starts cleanly
- Tier 2 (Visual): Functional test confirms: zmin=0, text present, texttemplate="%{text}", 5-stop colorscale, autosize=True, yaxis automargin=True, title color #1E293B
### Files changed:
- `src/visualization/plotly_generator.py` — both heatmap functions rewritten
- `IMPLEMENTATION_PLAN.md` — marked A.2 subtasks [x]
### Committed: 56ca11e "fix: heatmap linear colorscale, cell annotations, autosize (Task A.2)"
### Patterns discovered:
- The `_base_layout()` helper makes chart function updates significantly easier — just `layout = _base_layout(title)` + `layout.update(...)` + `fig.update_layout(**layout)`. No need to repeat font/bgcolor/autosize.
- `textfont=dict(size=10)` on the heatmap trace controls cell annotation size — needs to be small enough not to overflow cells.
- Cost formatting: `£Nk` for cost ≥ 1000, `£N` for small values. This avoids `£125,000` flooding cells.
### Next iteration should:
- Start with Task A.3: Fix legend overflow in 4 charts. This requires creating `_smart_legend(n_items)` helper and applying it + `_base_layout()` to: `create_market_share_figure()`, `create_trust_market_share_figure()`, `create_dosing_figure()`, `create_trust_duration_figure()`.
- Search for these function names by `def create_market_share_figure` etc. — line numbers have shifted again.
- The `_smart_legend()` helper should: >15 items → vertical right legend; ≤15 → horizontal with dynamic bottom margin.
### Blocked items:
- None

## Iteration 3 — 2026-02-07
### Task: A.3 — Fix legend overflow in 4 charts
### Why this task:
- A.3 is the next Phase A task after A.1 and A.2. It has no blockers and was explicitly recommended by Iteration 2. It addresses the legend overflow bug reported for charts with 42 drugs.
### Status: COMPLETE
### What was done:
- **Created `_smart_legend(n_items, legend_title)` helper**: Returns legend dict with adaptive positioning. >15 items → vertical right legend (orientation="v", x=1.02, xanchor="left"). ≤15 items → horizontal below chart (orientation="h", y=-0.12, xanchor="center").
- **Created `_smart_legend_margin(n_items)` helper**: Returns margin dict. >15 items → r=140, b=40. ≤15 items → dynamic b based on estimated row count (~6 items per row), r=24.
- **`create_market_share_figure()`**: Replaced local `nhs_colours` with `DRUG_PALETTE`. Replaced manual layout block with `_base_layout()` + `_smart_legend()`. Replaced hardcoded `GRID_COLOR` reference.
- **`create_trust_market_share_figure()`**: Same treatment. Also replaced Unicode escapes (`\u00a3`, `\u2014`) with literal characters (`£`, `—`).
- **`create_dosing_figure()`**: Replaced local `nhs_colours` with `DRUG_PALETTE`. Legend adapts to trace count using `sum(1 for t in fig.data if t.showlegend is not False)`. Uses `_base_layout()`.
- **`create_trust_duration_figure()`**: Replaced local `nhs_colours` with `TRUST_PALETTE` (7 maximally-distinct colors — also satisfies A.4 subtask). Fixed `l=200` → `l=8` + automargin. Uses `_base_layout()`.
- **Note**: This also completed two A.4 subtasks: `create_trust_duration_figure()` now uses `TRUST_PALETTE`, and `create_trust_market_share_figure()` now uses `DRUG_PALETTE`.
### Validation results:
- Tier 1 (Code): All imports pass. `from dash_app.app import app` OK. `python run_dash.py` starts cleanly, HTTP 200.
- Tier 2 (Visual): Functional tests confirm: 20-drug chart → vertical right legend; 5-drug → horizontal; 7-trust → horizontal with TRUST_PALETTE colors [#005EB8, #DA291C, #009639, #ED8B00, #7C2855, #00A499, #330072].
- Tier 3 (Functional): All 4 chart functions produce correct figures with sample data. Legend orientation switches at 15-item threshold.
### Files changed:
- `src/visualization/plotly_generator.py` — added `_smart_legend()` + `_smart_legend_margin()`, refactored 4 chart functions
- `IMPLEMENTATION_PLAN.md` — marked A.3 subtasks [x], updated A.4 for completed subtasks
### Committed: 90de24c "fix: adaptive legends + _base_layout for 4 chart functions (Task A.3)"
### Patterns discovered:
- Plotly trace.showlegend defaults to None (not True), so checking `t.showlegend is not False` correctly identifies visible legend items.
- The `_smart_legend_margin()` helper returns only `b` and `r` keys, allowing callers to merge with their own `t` and `l` via `dict(t=60, l=8, **legend_margins)`.
- Net code reduction of 48 lines across 4 functions by replacing repeated layout blocks with `_base_layout()`.
- Several A.4 subtasks were completed opportunistically during A.3 (replacing nhs_colours with palette constants).
### Next iteration should:
- Start with Task A.4: Fix remaining trust comparison color differentiation. Two subtasks remain:
  1. Add `is_trust_comparison=False` param to `create_cost_waterfall_figure()` — use `TRUST_PALETTE` when True. Update `tc_cost_waterfall` callback to pass `is_trust_comparison=True`.
  2. Fix `_dosing_by_drug()` blue→blue interpolation: replace with `plotly.colors.sample_colorscale("Viridis", ...)` for meaningful gradient.
- The `create_trust_duration_figure()` TRUST_PALETTE fix and `create_trust_market_share_figure()` DRUG_PALETTE fix are already done (marked [x] in plan).
- Search for `create_cost_waterfall_figure` and `_dosing_by_drug` by function name (line numbers have shifted).
### Blocked items:
- None

## Iteration 4 — 2026-02-07
### Task: A.4 — Fix trust comparison color differentiation (remaining 3 subtasks)
### Why this task:
- A.4 is the last Phase A task. A.1-A.3 are complete. Two A.4 subtasks were already done opportunistically in A.3. The remaining 3 subtasks complete Phase A.
### Status: COMPLETE
### What was done:
- **`create_cost_waterfall_figure()`**: Added `is_trust_comparison=False` parameter. When True, uses `TRUST_PALETTE` (7 distinct colors) instead of `DRUG_PALETTE`. Replaced local `nhs_colours` list. Applied `_base_layout()` — removed 20+ lines of hardcoded layout. Updated annotation fonts to use `ANNOTATION_COLOR` and `CHART_FONT_FAMILY` constants. Used `GRID_COLOR` constant for gridcolor.
- **`tc_cost_waterfall` callback**: Updated call to pass `is_trust_comparison=True`, so Trust Comparison cost waterfall now uses 7 maximally-distinct trust colors.
- **`_dosing_by_drug()`**: Replaced manual RGB interpolation (blue `#005EB8` → blue `#41B6E6`) with `plotly.colors.sample_colorscale("Viridis", ratios)`. Result: bars now range from yellow (high interval) through teal to purple (low interval) — clearly distinguishable.
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` OK. `python run_dash.py` starts cleanly, HTTP 200.
- Tier 2 (Visual): Cost waterfall normal mode uses DRUG_PALETTE colors (#005EB8, #DA291C, #009639...). Trust mode uses TRUST_PALETTE. Dosing by drug uses Viridis: rgb(253,231,37), rgb(34,144,140), rgb(59,81,138) — visually distinct.
- Tier 3 (Functional): is_trust_comparison=False (default) preserves existing behavior. is_trust_comparison=True switches to TRUST_PALETTE. Viridis sampling produces correct gradients.
### Files changed:
- `src/visualization/plotly_generator.py` — `create_cost_waterfall_figure()` + `_dosing_by_drug()`
- `dash_app/callbacks/trust_comparison.py` — `tc_cost_waterfall` callback
- `IMPLEMENTATION_PLAN.md` — marked A.4 subtasks [x]
### Committed: 950d93b "fix: trust palette for cost waterfall + Viridis dosing gradient (Task A.4)"
### Patterns discovered:
- `plotly.colors.sample_colorscale("Viridis", ratios)` returns a list of `rgb(r,g,b)` strings — can be passed directly to `marker.color` as a list. Very clean replacement for manual interpolation.
- The `_base_layout()` + `layout.update()` pattern removed ~30 lines from `create_cost_waterfall_figure()` (net -33 lines in diff). Each function converted gets simpler.
- Phase A is now COMPLETE. All 4 tasks (A.1-A.4) are done.
### Next iteration should:
- Start with Task B.1: Fix title inconsistencies across all charts. This requires:
  1. Apply `_base_layout()` to remaining unconverted chart functions: `create_sankey_figure()`, `create_cost_effectiveness_figure()`, `create_duration_figure()`
  2. These functions still have hardcoded `#003087` title colors and `"Source Sans 3"` font strings
  3. Some Trust Comparison functions may still use `size=16` instead of `CHART_TITLE_SIZE` (18)
  4. Search for `def create_sankey_figure`, `def create_cost_effectiveness_figure`, `def create_duration_figure` to find current line numbers
- After B.1, the next tasks are B.2 (cost effectiveness gradient) and B.3 (Sankey narrow-screen fix) — both are small and independent.
### Blocked items:
- None

## Iteration 5 — 2026-02-07
### Task: B.1 — Fix title inconsistencies across all charts
### Why this task:
- B.1 is the first Phase B task. Phase A is complete. Progress.txt from Iteration 4 explicitly recommended B.1 next. It ensures all chart functions use consistent styling via `_base_layout()`.
### Status: COMPLETE
### What was done:
- **`create_sankey_figure()`**: Replaced local `nhs_colours` (15 blue-heavy colors) with `DRUG_PALETTE`. Replaced 20-line hardcoded layout block (title color `#003087`, manual font/bgcolor) with `_base_layout()` call + 2 overrides (font size 12, margin/height).
- **`create_cost_effectiveness_figure()`**: Replaced 38-line manual layout block (title, xaxis, yaxis, margin, bgcolor, hoverlabel, font) with `_base_layout()` + 5-key update. Replaced hardcoded annotation font strings with `ANNOTATION_COLOR` and `CHART_FONT_FAMILY` constants. Replaced `gridcolor="#E2E8F0"` with `GRID_COLOR`.
- **`create_duration_figure()`**: Replaced 30-line manual layout (title color `#003087`, l=200 fixed margin, manual bgcolor/font) with `_base_layout()` + 6-key update. Fixed `margin.l` from 200 → 8 + `yaxis automargin=True`. Replaced hardcoded annotation font with constants. Used `ANNOTATION_COLOR` in subtitle HTML span.
- Net result: -52 lines (24 added, 76 removed). All 11 chart functions now use `_base_layout()`.
### Validation results:
- Tier 1 (Code): `from dash_app.app import app` OK. `python run_dash.py` starts cleanly, no errors.
- Tier 2 (Visual): All three functions produce figures with title color `#1E293B`, title size 18, font family `Source Sans 3, system-ui, sans-serif`, transparent bgcolor. Duration uses automargin instead of fixed l=200.
### Files changed:
- `src/visualization/plotly_generator.py` — converted 3 remaining chart functions to `_base_layout()`
- `IMPLEMENTATION_PLAN.md` — marked B.1 subtasks [x]
### Committed: 8d05adc "fix: consistent titles via _base_layout() for Sankey, Cost Effectiveness, Duration (Task B.1)"
### Patterns discovered:
- All 11 chart functions in plotly_generator.py now use `_base_layout()`. No more hardcoded `"Source Sans 3"` strings or `#003087` title colors exist outside of constants/colorscale definitions.
- Sankey's local `nhs_colours` was different from `DRUG_PALETTE` (had more blue variants like `#003087`, `#41B6E6`, `#0066CC`). Replacing with `DRUG_PALETTE` gives more distinguishable drug colors.
- The `_base_layout()` conversion is a reliable pattern: replace the entire `fig.update_layout(...)` call with `layout = _base_layout(title)` + `layout.update({chart-specific})` + `fig.update_layout(**layout)`. Removes ~15-30 lines per function.
### Next iteration should:
- Choose Task B.2 (cost effectiveness smooth gradient) or B.3 (Sankey narrow-screen fix) — both are small, independent tasks with no blockers.
- B.2: In `create_cost_effectiveness_figure()`, replace the 3-bin hard threshold (green/amber/red at 0.33/0.66) with smooth RGB interpolation. Green (#009639) → Amber (#ED8B00) at ratio 0–0.5, Amber → Red (#DA291C) at 0.5–1.0.
- B.3: In `create_sankey_figure()`, change `arrangement="snap"` → `arrangement="freeform"` and increase `pad` from 20 → 25.
- Both are quick changes — could potentially do B.2+B.3 together if scoped carefully, but the loop rules say one task per iteration.
### Blocked items:
- None
