HighCostDrugsDemo/RALPH_PROMPT.md

# Ralph Wiggum Loop — Dashboard Visualization Improvements

You are operating inside an automated loop improving Plotly charts in an NHS patient pathway analysis Dash application. Each iteration you receive fresh context — you have NO memory of previous iterations. Your only memory is the filesystem.

**Current Focus**: Fix chart bugs, improve visual polish, add new analytics charts. See IMPLEMENTATION_PLAN.md for the full task list organized into Phases A–D.

## First Actions Every Iteration

Read these files in this order before doing anything else:

1. `progress.txt` — What previous iterations accomplished, what's blocked, and what to do next.
2. `IMPLEMENTATION_PLAN.md` — Task list with status markers, architecture overview, and completion criteria.
3. `guardrails.md` — Known failure patterns to avoid. You MUST read and follow these.
4. `CLAUDE.md` — Project architecture and backend code patterns.

Then run `git log --oneline -5` to see recent commits.

## Key Files for This Phase

**When modifying chart functions**, always read first:
- `src/visualization/plotly_generator.py` — PRIMARY file. All chart generation functions live here (~1782 lines).
- `dash_app/callbacks/chart.py` — Patient Pathways tab dispatch and chart rendering helpers.
- `dash_app/callbacks/trust_comparison.py` — Trust Comparison 6-chart callbacks.

**When adding new analytics charts**, also read:
- `src/data_processing/pathway_queries.py` — All SQLite query functions. New queries go here.
- `dash_app/data/queries.py` — Thin wrappers. Add wrapper for each new query.
- `dash_app/components/chart_card.py` — TAB_DEFINITIONS for Patient Pathways tabs.

**When modifying UI components**, read:
- `dash_app/components/trust_comparison.py` — TC landing + dashboard layout.
- `dash_app/assets/nhs.css` — All CSS styles.

## Narration

Narrate your work as you go. Your output is the only visibility the operator has into what's happening. For every significant action, explain what you're doing and why:

- **Reading files**: "Reading plotly_generator.py to locate the heatmap colorscale..."
- **Creating code**: "Adding _base_layout() helper to DRY shared layout properties..."
- **Debugging**: "Chart title color is #003087 instead of CHART_TITLE_COLOR..."
- **Testing**: "Running python run_dash.py to verify the app starts..."
- **Committing**: "Committing heatmap fixes."

Do NOT just output a summary at the end. Narrate throughout.

## Task Selection

1. Read ALL tasks in IMPLEMENTATION_PLAN.md — understand the full picture
2. Skip any marked `[x]` (complete) or `[B]` (blocked)
3. Check progress.txt for guidance — the previous iteration may have recommendations
4. **Choose a task** based on:
   - Dependencies (A.1 shared constants before A.2-A.4 which use them)
   - Phase ordering (Phase A before B, B before C, C before D)
   - Previous iteration's recommendations
5. **Document your reasoning**: Before starting, explain WHY you chose this task
6. Mark your chosen task `[~]` (in progress) in IMPLEMENTATION_PLAN.md

If your chosen task is blocked:
- Mark it `[B]` with a reason
- Document the blocker in progress.txt
- Move to a different ready task

## Development

Work on ONE task per iteration. Build incrementally and verify as you go.

### Key Technologies

- **Dash 4.0.0**: `from dash import Dash, html, dcc, Input, Output, State, ctx, ALL`
- **Dash Mantine Components 2.5.1**: `import dash_mantine_components as dmc` — `MantineProvider` wraps layout
- **Plotly**: `import plotly.graph_objects as go` — all chart figures
- **SQLite**: `import sqlite3` — read-only access to `data/pathways.db`
- **CSS**: All in `dash_app/assets/nhs.css` — auto-served by Dash

### plotly_generator.py Patterns

All chart functions follow the same pattern:
```python
def create_CHART_figure(data: list[dict], title: str = "", ...) -> go.Figure:
    """Create CHART from prepared data."""
    if not data:
        return go.Figure()

    # Build traces from data
    fig = go.Figure(data=traces)

    # Apply layout
    layout = _base_layout(display_title)
    layout.update({...chart-specific overrides...})
    fig.update_layout(**layout)

    return fig
```

### Adding a New Chart Tab

1. Add query function to `src/data_processing/pathway_queries.py` (accept `db_path` param)
2. Add thin wrapper to `dash_app/data/queries.py` (resolve DB_PATH and delegate)
3. Add figure function to `src/visualization/plotly_generator.py`
4. Add tab to `TAB_DEFINITIONS` in `dash_app/components/chart_card.py`
5. Add `_render_*()` helper in `dash_app/callbacks/chart.py`
6. Add elif case in `update_chart()` callback

### Database Access Pattern

```python
# In src/data_processing/pathway_queries.py
def get_something(db_path: Path, filter_id: str, chart_type: str, ...) -> list[dict]:
    conn = sqlite3.connect(str(db_path))
    conn.row_factory = sqlite3.Row
    cursor = conn.cursor()
    cursor.execute("SELECT ... WHERE date_filter_id = ? AND chart_type = ?", [filter_id, chart_type])
    rows = [dict(row) for row in cursor.fetchall()]
    conn.close()
    return rows

# In dash_app/data/queries.py (thin wrapper)
from data_processing.pathway_queries import get_something as _get_something
DB_PATH = Path(__file__).resolve().parents[2] / "data" / "pathways.db"

def get_something(filter_id="all_6mo", chart_type="directory", ...):
    return _get_something(DB_PATH, filter_id, chart_type, ...)
```

### Verification Steps

After writing code, ALWAYS verify:

1. **Import check**: `python -c "from dash_app.app import app"` (or specific module)
2. **App starts**: `python run_dash.py` — must start without errors
3. **Visual check** (when modifying charts): describe what you expect to see at localhost:8050
4. **For callbacks**: verify the callback chain fires correctly

If any step fails, fix the issue before proceeding.

## Validation Protocol

Every task MUST pass validation before being marked complete:

### Tier 1: Code Validation (MANDATORY)
- Code compiles without Python syntax errors
- Imports work without errors
- `python run_dash.py` starts without exceptions

### Tier 2: Visual Validation (for chart modification tasks)
- Chart renders in the browser
- Colors, labels, legend layout match expectations
- No overflow or overlap issues

### Tier 3: Functional Validation (for callback/toggle tasks)
- Callbacks fire when inputs change
- Metric toggles switch correctly
- New tabs appear and render data

### Validation Failure

If any tier fails:
- DO NOT mark the task complete
- Document the failure in progress.txt
- Fix the issue within this iteration if possible
- If you cannot fix it, mark the task `[B]` with details

## Quality Gates

Before marking ANY task `[x]`, ALL of these must be true:

1. Code is saved to the appropriate file(s)
2. Tier 1 validation passed (imports + app starts)
3. Tier 2/3 validation passed (as applicable)
4. All changes committed to git with a descriptive message

These are non-negotiable.

## Update Progress

After completing your work, append to progress.txt using this format:

```
## Iteration [N] — [YYYY-MM-DD]
### Task: [which task you worked on]
### Why this task:
- [Brief explanation of why you chose this task over others]
### Status: COMPLETE | BLOCKED | IN PROGRESS
### What was done:
- [Specific actions taken]
### Validation results:
- Tier 1 (Code): [import check, app starts]
- Tier 2 (Visual): [chart renders, colors correct]
- Tier 3 (Functional): [callbacks fire, toggles work]
### Files changed:
- [list of files created/modified]
### Committed: [git hash] "[commit message]"
### Patterns discovered:
- [Any reusable learnings — Plotly quirks, layout gotchas, Dash patterns]
### Next iteration should:
- [Explicit guidance for what the next fresh instance should do first]
- [Note any context that would be lost without writing it here]
### Blocked items:
- [Any tasks that are blocked and why]
```

If you discover a failure pattern, add it to `guardrails.md`.

## Commit Changes

1. Stage changed files
2. Use a descriptive commit message referencing the task (e.g., "fix: heatmap colorscale + cell annotations (Task A.2)")
3. Commit after your task is validated and complete
4. If you updated progress.txt with a blocked status, commit that too

## Completion Check

If ALL tasks in IMPLEMENTATION_PLAN.md are marked `[x]`:

1. Run `python run_dash.py` to verify app starts cleanly
2. Verify all completion criteria at the bottom of IMPLEMENTATION_PLAN.md are satisfied
3. Only then output the completion signal on its own line:

```
<promise>COMPLETE</promise>
```

DO NOT output this string under any other circumstances.
DO NOT output it if any task is still `[ ]` or `[B]` or `[~]`.

## Rules

- Complete ONE task per iteration, then update progress and stop
- ALWAYS read progress.txt, guardrails.md before starting work
- **Read plotly_generator.py** when modifying ANY chart function (line numbers shift!)
- **DO NOT modify pipeline/analysis logic** in src/ (pathway_pipeline, transforms, diagnosis_lookup, pathway_analyzer, refresh_pathways)
- **DO add/modify** chart functions in `src/visualization/plotly_generator.py`
- **DO add** new query functions in `src/data_processing/pathway_queries.py`
- **New figure functions** go in `src/visualization/`, not in `dash_app/callbacks/`
- **New query functions** go in `src/data_processing/pathway_queries.py` with thin wrappers in `dash_app/data/queries.py`
- **dcc.Store for state** — no server-side globals
- **Lazy tab rendering** — only compute the active tab's chart
- Keep commits atomic and well-described
- If stuck for 2+ attempts, document in progress.txt and move on
- `python run_dash.py` must work after every task