# Progress Log — Dashboard Visualization Improvements

## Project Context

Working Dash application with 2 views (Patient Pathways + Trust Comparison), 13 chart functions in `plotly_generator.py`, and a complete callback chain. Now improving chart quality: bug fixes, visual polish, and new analytics.

**Current state**: Fully functional Dash app at http://localhost:8050 with icicle, Sankey, market share, cost effectiveness, cost waterfall, dosing, heatmap, and duration charts. Trust Comparison has 6 dedicated charts. All filters work.

**New goal**: Fix chart bugs (heatmap colorscale, legend overflow, trust color differentiation), add visual polish (consistent styling, smooth gradients), add new analytics (retention funnel, pathway depth, scatter, network), and new backend analytics (trends, dose distribution, timeline, NICE compliance).

## Key Architecture Patterns

### plotly_generator.py (PRIMARY target file)
- 13 chart functions, all accept list-of-dicts, return `go.Figure`
- Located at `src/visualization/plotly_generator.py` (~1782 lines)
- Key functions and approximate line numbers:
  - `create_icicle_from_nodes(nodes, title)` — L113
  - `create_market_share_figure(data, title)` — L247
  - `create_cost_effectiveness_figure(data, retention, title)` — L384
  - `create_cost_waterfall_figure(data, title)` — L562
  - `create_sankey_figure(data, title)` — L706
  - `create_dosing_figure(data, title, group_by)` — L837
  - `_dosing_by_drug(data, colours)` — L926
  - `_dosing_by_trust(data, colours)` — L1007
  - `create_heatmap_figure(data, title, metric)` — L1189
  - `create_duration_figure(data, title, show_directory)` — L1329
  - `create_trust_market_share_figure(data, title)` — L1481
  - `create_trust_heatmap_figure(data, title, metric)` — L1582
  - `create_trust_duration_figure(data, title)` — L1689
- NOTE: Line numbers will shift as you edit. Re-read the file each iteration.

### Callback chain
- `dash_app/callbacks/chart.py` — Patient Pathways tab dispatch (`_render_*` helpers → `update_chart`)
- `dash_app/callbacks/trust_comparison.py` — 6 Trust Comparison chart callbacks
- Tab switching: `active-tab` dcc.Store, tab IDs = `"tab-{short_id}"`
- TAB_DEFINITIONS in `chart_card.py` — currently: icicle, sankey

### Adding a new Patient Pathways tab
1. Query function in `src/data_processing/pathway_queries.py` (accept `db_path` param)
2. Thin wrapper in `dash_app/data/queries.py` (resolve DB_PATH)
3. Figure function in `src/visualization/plotly_generator.py`
4. Add to `TAB_DEFINITIONS` in `dash_app/components/chart_card.py`
5. Add `_render_*()` helper in `dash_app/callbacks/chart.py`
6. Add elif case in `update_chart()` dispatch

### State management
- 4 `dcc.Store` components: app-state, chart-data, reference-data, active-tab
- Unidirectional: filter inputs → app-state → chart-data → UI
- 20 registered callbacks total

### DMC version
- Dash 4.0.0 + DMC 2.5.1 (Mantine v7 based)
- `dmc.MantineProvider` wraps layout
- `dmc.SegmentedControl` available for metric toggles

### Flex chain for chart filling viewport
- Full flex chain: `.main` → `#view-container` → `#patient-pathways-view` → `.chart-card` → loading wrapper → `#chart-container` → `#pathway-chart`
- `responsive=True` on dcc.Graph + `autosize=True` in figure layout
- `dcc.Loading` wraps children in `.dash-loading-callback > div` — CSS must propagate flex through both

### Known heatmap bugs (to fix)
- Non-linear colorscale compresses 99% of range into identical blues
- No cell text — must hover every cell
- Light end (#F0F4F8) invisible against transparent background
- Fixed width can overflow container
- Fixed l=200 left margin wastes space

### Known legend bugs (to fix)
- Horizontal legends at y=-0.15 with fixed bottom margins overflow with 42 drugs
- Affects: market_share, trust_market_share, dosing, trust_duration

### Known color bugs (to fix)
- First 6 of 10 trust palette colors are blue variants — nearly indistinguishable
- _dosing_by_drug interpolates from one blue to another blue

## Iteration Log

## Iteration 1 — 2026-02-07
### Task: A.1 — Extract shared styling constants + `_base_layout()` helper
### Why this task:
- A.1 is the foundation for all subsequent Phase A tasks (A.2-A.4 all reference `_base_layout()` and the palette constants). Must be done first.
### Status: COMPLETE
### What was done:
- Added 7 module-level constants after `logger` line: `CHART_FONT_FAMILY`, `CHART_TITLE_SIZE`, `CHART_TITLE_COLOR`, `GRID_COLOR`, `ANNOTATION_COLOR`, `TRUST_PALETTE` (7 colors), `DRUG_PALETTE` (15 colors)
- Created `_base_layout(title, **overrides)` helper returning dict with: title (font family/size/color, centered), hoverlabel (white bg, #CBD5E1 border, font), paper/plot bgcolor transparent, autosize=True, base font family
- Applied `_base_layout()` to `create_icicle_from_nodes()` — replaced 20-line explicit layout block with `_base_layout()` call + 3 overrides (margin, hoverlabel size=14, clickmode)
- Also replaced hardcoded `"Source Sans 3, system-ui, sans-serif"` in icicle textfont with `CHART_FONT_FAMILY` constant
### Validation results:
- Tier 1 (Code): all imports pass, `python run_dash.py` starts cleanly
- Tier 2 (Visual): icicle figure title correct ("Patient Pathways — By Directory"), font family/size/color all match expected values
### Files changed:
- `src/visualization/plotly_generator.py` — added constants + `_base_layout()` + refactored icicle layout
- `IMPLEMENTATION_PLAN.md` — marked A.1 subtasks [x]
### Committed: 63c1801 "feat: add shared styling constants and _base_layout() helper (Task A.1)"
### Patterns discovered:
- The `_base_layout()` returns a plain dict that gets unpacked via `fig.update_layout(**layout)`. Callers pass chart-specific overrides as kwargs.
- Icicle hoverlabel uses `size=14` (slightly larger than base `13`) — preserved as override.
- Constants are at module level, so all functions in the file can reference them directly.
- Line numbers in IMPLEMENTATION_PLAN.md are now stale (shifted ~70 lines due to constants/helper insertion). Future iterations should search by function name.
### Next iteration should:
- Start with Task A.2: Fix heatmap colorscale + cell annotations. Read `create_heatmap_figure()` and `create_trust_heatmap_figure()` functions by searching for their names (line numbers have shifted).
- The `_base_layout()` and palette constants are now available — A.2 should use `_base_layout()` for both heatmap functions.
- Key heatmap fixes: linear 5-stop colorscale, cell text annotations, zmin=0, autosize, automargin, subtitle when >25 drugs.
### Blocked items:
- None

## Iteration 2 — 2026-02-07
### Task: A.2 — Fix heatmap colorscale + cell annotations
### Why this task:
- A.2 is the next Phase A task after A.1. It depends on `_base_layout()` (done in A.1). It addresses the most-reported heatmap bugs: non-linear colorscale, missing cell text, fixed width overflow, fixed left margin.
### Status: COMPLETE
### What was done:
- **create_heatmap_figure()**: Replaced non-linear 7-stop colorscale with linear 5-stop `[0.0 #E3F2FD, 0.25 #90CAF9, 0.5 #42A5F5, 0.75 #1E88E5, 1.0 #003087]`. Added `text=text_values, texttemplate="%{text}"` with metric-specific formatting (patients: `N`, cost: `£Nk`, cost_pp_pa: `£N`). Set `zmin=0`. Removed fixed `width`, using `autosize=True` from `_base_layout()`. Replaced `l=200` with `l=8` + `yaxis automargin=True`. Added subtitle annotation when 25-drug cap is hit. Reduced `xgap/ygap` to 1 when >15 columns.
- **create_trust_heatmap_figure()**: Applied identical fixes. Also fixed Unicode escapes (`\u00a3`, `\u00d7`, `\u2014`) back to literal characters.
- Both functions now use `_base_layout(chart_title)` for consistent styling — title color is now `#1E293B` (CHART_TITLE_COLOR) instead of old hardcoded `#003087`, title size is 18 (CHART_TITLE_SIZE) for both (trust heatmap was 16 before).
### Validation results:
- Tier 1 (Code): imports pass, `uv run python -c "from dash_app.app import app"` OK, `uv run python run_dash.py` starts cleanly
- Tier 2 (Visual): Functional test confirms: zmin=0, text present, texttemplate="%{text}", 5-stop colorscale, autosize=True, yaxis automargin=True, title color #1E293B
### Files changed:
- `src/visualization/plotly_generator.py` — both heatmap functions rewritten
- `IMPLEMENTATION_PLAN.md` — marked A.2 subtasks [x]
### Committed: 56ca11e "fix: heatmap linear colorscale, cell annotations, autosize (Task A.2)"
### Patterns discovered:
- The `_base_layout()` helper makes chart function updates significantly easier — just `layout = _base_layout(title)` + `layout.update(...)` + `fig.update_layout(**layout)`. No need to repeat font/bgcolor/autosize.
- `textfont=dict(size=10)` on the heatmap trace controls cell annotation size — needs to be small enough not to overflow cells.
- Cost formatting: `£Nk` for cost ≥ 1000, `£N` for small values. This avoids `£125,000` flooding cells.
### Next iteration should:
- Start with Task A.3: Fix legend overflow in 4 charts. This requires creating `_smart_legend(n_items)` helper and applying it + `_base_layout()` to: `create_market_share_figure()`, `create_trust_market_share_figure()`, `create_dosing_figure()`, `create_trust_duration_figure()`.
- Search for these function names by `def create_market_share_figure` etc. — line numbers have shifted again.
- The `_smart_legend()` helper should: >15 items → vertical right legend; ≤15 → horizontal with dynamic bottom margin.
### Blocked items:
- None

## Iteration 3 — 2026-02-07
### Task: A.3 — Fix legend overflow in 4 charts
### Why this task:
- A.3 is the next Phase A task after A.1 and A.2. It has no blockers and was explicitly recommended by Iteration 2. It addresses the legend overflow bug reported for charts with 42 drugs.
### Status: COMPLETE
### What was done:
- **Created `_smart_legend(n_items, legend_title)` helper**: Returns legend dict with adaptive positioning. >15 items → vertical right legend (orientation="v", x=1.02, xanchor="left"). ≤15 items → horizontal below chart (orientation="h", y=-0.12, xanchor="center").
- **Created `_smart_legend_margin(n_items)` helper**: Returns margin dict. >15 items → r=140, b=40. ≤15 items → dynamic b based on estimated row count (~6 items per row), r=24.
- **`create_market_share_figure()`**: Replaced local `nhs_colours` with `DRUG_PALETTE`. Replaced manual layout block with `_base_layout()` + `_smart_legend()`. Replaced hardcoded `GRID_COLOR` reference.
- **`create_trust_market_share_figure()`**: Same treatment. Also replaced Unicode escapes (`\u00a3`, `\u2014`) with literal characters (`£`, `—`).
- **`create_dosing_figure()`**: Replaced local `nhs_colours` with `DRUG_PALETTE`. Legend adapts to trace count using `sum(1 for t in fig.data if t.showlegend is not False)`. Uses `_base_layout()`.
- **`create_trust_duration_figure()`**: Replaced local `nhs_colours` with `TRUST_PALETTE` (7 maximally-distinct colors — also satisfies A.4 subtask). Fixed `l=200` → `l=8` + automargin. Uses `_base_layout()`.
- **Note**: This also completed two A.4 subtasks: `create_trust_duration_figure()` now uses `TRUST_PALETTE`, and `create_trust_market_share_figure()` now uses `DRUG_PALETTE`.
### Validation results:
- Tier 1 (Code): All imports pass. `from dash_app.app import app` OK. `python run_dash.py` starts cleanly, HTTP 200.
- Tier 2 (Visual): Functional tests confirm: 20-drug chart → vertical right legend; 5-drug → horizontal; 7-trust → horizontal with TRUST_PALETTE colors [#005EB8, #DA291C, #009639, #ED8B00, #7C2855, #00A499, #330072].
- Tier 3 (Functional): All 4 chart functions produce correct figures with sample data. Legend orientation switches at 15-item threshold.
### Files changed:
- `src/visualization/plotly_generator.py` — added `_smart_legend()` + `_smart_legend_margin()`, refactored 4 chart functions
- `IMPLEMENTATION_PLAN.md` — marked A.3 subtasks [x], updated A.4 for completed subtasks
### Committed: 90de24c "fix: adaptive legends + _base_layout for 4 chart functions (Task A.3)"
### Patterns discovered:
- Plotly trace.showlegend defaults to None (not True), so checking `t.showlegend is not False` correctly identifies visible legend items.
- The `_smart_legend_margin()` helper returns only `b` and `r` keys, allowing callers to merge with their own `t` and `l` via `dict(t=60, l=8, **legend_margins)`.
- Net code reduction of 48 lines across 4 functions by replacing repeated layout blocks with `_base_layout()`.
- Several A.4 subtasks were completed opportunistically during A.3 (replacing nhs_colours with palette constants).
### Next iteration should:
- Start with Task A.4: Fix remaining trust comparison color differentiation. Two subtasks remain:
  1. Add `is_trust_comparison=False` param to `create_cost_waterfall_figure()` — use `TRUST_PALETTE` when True. Update `tc_cost_waterfall` callback to pass `is_trust_comparison=True`.
  2. Fix `_dosing_by_drug()` blue→blue interpolation: replace with `plotly.colors.sample_colorscale("Viridis", ...)` for meaningful gradient.
- The `create_trust_duration_figure()` TRUST_PALETTE fix and `create_trust_market_share_figure()` DRUG_PALETTE fix are already done (marked [x] in plan).
- Search for `create_cost_waterfall_figure` and `_dosing_by_drug` by function name (line numbers have shifted).
### Blocked items:
- None
