# Progress Log — Dashboard Visualization Improvements

## Project Context

Working Dash application with 2 views (Patient Pathways + Trust Comparison), 13 chart functions in `plotly_generator.py`, and a complete callback chain. Now improving chart quality: bug fixes, visual polish, and new analytics.

**Current state**: Fully functional Dash app at http://localhost:8050 with icicle, Sankey, market share, cost effectiveness, cost waterfall, dosing, heatmap, and duration charts. Trust Comparison has 6 dedicated charts. All filters work.

**New goal**: Fix chart bugs (heatmap colorscale, legend overflow, trust color differentiation), add visual polish (consistent styling, smooth gradients), add new analytics (retention funnel, pathway depth, scatter, network), and new backend analytics (trends, dose distribution, timeline, NICE compliance).

## Key Architecture Patterns

### plotly_generator.py (PRIMARY target file)
- 13 chart functions, all accept list-of-dicts, return `go.Figure`
- Located at `src/visualization/plotly_generator.py` (~1782 lines)
- Key functions and approximate line numbers:
  - `create_icicle_from_nodes(nodes, title)` — L113
  - `create_market_share_figure(data, title)` — L247
  - `create_cost_effectiveness_figure(data, retention, title)` — L384
  - `create_cost_waterfall_figure(data, title)` — L562
  - `create_sankey_figure(data, title)` — L706
  - `create_dosing_figure(data, title, group_by)` — L837
  - `_dosing_by_drug(data, colours)` — L926
  - `_dosing_by_trust(data, colours)` — L1007
  - `create_heatmap_figure(data, title, metric)` — L1189
  - `create_duration_figure(data, title, show_directory)` — L1329
  - `create_trust_market_share_figure(data, title)` — L1481
  - `create_trust_heatmap_figure(data, title, metric)` — L1582
  - `create_trust_duration_figure(data, title)` — L1689
- NOTE: Line numbers will shift as you edit. Re-read the file each iteration.

### Callback chain
- `dash_app/callbacks/chart.py` — Patient Pathways tab dispatch (`_render_*` helpers → `update_chart`)
- `dash_app/callbacks/trust_comparison.py` — 6 Trust Comparison chart callbacks
- Tab switching: `active-tab` dcc.Store, tab IDs = `"tab-{short_id}"`
- TAB_DEFINITIONS in `chart_card.py` — currently: icicle, sankey

### Adding a new Patient Pathways tab
1. Query function in `src/data_processing/pathway_queries.py` (accept `db_path` param)
2. Thin wrapper in `dash_app/data/queries.py` (resolve DB_PATH)
3. Figure function in `src/visualization/plotly_generator.py`
4. Add to `TAB_DEFINITIONS` in `dash_app/components/chart_card.py`
5. Add `_render_*()` helper in `dash_app/callbacks/chart.py`
6. Add elif case in `update_chart()` dispatch

### State management
- 4 `dcc.Store` components: app-state, chart-data, reference-data, active-tab
- Unidirectional: filter inputs → app-state → chart-data → UI
- 20 registered callbacks total

### DMC version
- Dash 4.0.0 + DMC 2.5.1 (Mantine v7 based)
- `dmc.MantineProvider` wraps layout
- `dmc.SegmentedControl` available for metric toggles

### Flex chain for chart filling viewport
- Full flex chain: `.main` → `#view-container` → `#patient-pathways-view` → `.chart-card` → loading wrapper → `#chart-container` → `#pathway-chart`
- `responsive=True` on dcc.Graph + `autosize=True` in figure layout
- `dcc.Loading` wraps children in `.dash-loading-callback > div` — CSS must propagate flex through both

### Known heatmap bugs (to fix)
- Non-linear colorscale compresses 99% of range into identical blues
- No cell text — must hover every cell
- Light end (#F0F4F8) invisible against transparent background
- Fixed width can overflow container
- Fixed l=200 left margin wastes space

### Known legend bugs (to fix)
- Horizontal legends at y=-0.15 with fixed bottom margins overflow with 42 drugs
- Affects: market_share, trust_market_share, dosing, trust_duration

### Known color bugs (to fix)
- First 6 of 10 trust palette colors are blue variants — nearly indistinguishable
- _dosing_by_drug interpolates from one blue to another blue

## Iteration Log

## Iteration 1 — 2026-02-07
### Task: A.1 — Extract shared styling constants + `_base_layout()` helper
### Why this task:
- A.1 is the foundation for all subsequent Phase A tasks (A.2-A.4 all reference `_base_layout()` and the palette constants). Must be done first.
### Status: COMPLETE
### What was done:
- Added 7 module-level constants after `logger` line: `CHART_FONT_FAMILY`, `CHART_TITLE_SIZE`, `CHART_TITLE_COLOR`, `GRID_COLOR`, `ANNOTATION_COLOR`, `TRUST_PALETTE` (7 colors), `DRUG_PALETTE` (15 colors)
- Created `_base_layout(title, **overrides)` helper returning dict with: title (font family/size/color, centered), hoverlabel (white bg, #CBD5E1 border, font), paper/plot bgcolor transparent, autosize=True, base font family
- Applied `_base_layout()` to `create_icicle_from_nodes()` — replaced 20-line explicit layout block with `_base_layout()` call + 3 overrides (margin, hoverlabel size=14, clickmode)
- Also replaced hardcoded `"Source Sans 3, system-ui, sans-serif"` in icicle textfont with `CHART_FONT_FAMILY` constant
### Validation results:
- Tier 1 (Code): all imports pass, `python run_dash.py` starts cleanly
- Tier 2 (Visual): icicle figure title correct ("Patient Pathways — By Directory"), font family/size/color all match expected values
### Files changed:
- `src/visualization/plotly_generator.py` — added constants + `_base_layout()` + refactored icicle layout
- `IMPLEMENTATION_PLAN.md` — marked A.1 subtasks [x]
### Committed: 63c1801 "feat: add shared styling constants and _base_layout() helper (Task A.1)"
### Patterns discovered:
- The `_base_layout()` returns a plain dict that gets unpacked via `fig.update_layout(**layout)`. Callers pass chart-specific overrides as kwargs.
- Icicle hoverlabel uses `size=14` (slightly larger than base `13`) — preserved as override.
- Constants are at module level, so all functions in the file can reference them directly.
- Line numbers in IMPLEMENTATION_PLAN.md are now stale (shifted ~70 lines due to constants/helper insertion). Future iterations should search by function name.
### Next iteration should:
- Start with Task A.2: Fix heatmap colorscale + cell annotations. Read `create_heatmap_figure()` and `create_trust_heatmap_figure()` functions by searching for their names (line numbers have shifted).
- The `_base_layout()` and palette constants are now available — A.2 should use `_base_layout()` for both heatmap functions.
- Key heatmap fixes: linear 5-stop colorscale, cell text annotations, zmin=0, autosize, automargin, subtitle when >25 drugs.
### Blocked items:
- None