HighCostDrugsDemo/guardrails.md

Guardrails

Known failure patterns. Read EVERY iteration. Follow ALL of these rules. If you discover a new failure pattern during your work, add it to this file.

---

Backend Isolation

Do NOT modify pipeline/analysis logic in src/

• When: Improving charts or adding analytics
• Rule: Do NOT change the logic in these files — they are the data pipeline and must stay as-is:
  • data_processing/pathway_pipeline.py, transforms.py, diagnosis_lookup.py (matching/query logic)
  • analysis/pathway_analyzer.py, statistics.py
  • cli/refresh_pathways.py
  • data_processing/schema.py, reference_data.py, cache.py, data_source.py
• Why: The pipeline is complete and tested. Changing it risks breaking the data refresh workflow.

DO use shared utilities in src/ rather than duplicating

• When: Adding chart functions or query functions
• Rule: Chart figure functions go in src/visualization/plotly_generator.py. Query functions go in src/data_processing/pathway_queries.py. Dash callbacks should CALL INTO src/, not duplicate the code.
• Why: Duplicating SQL queries and figure logic creates copies that drift apart.

Do NOT modify pathways.db schema or data from Dash callbacks

• When: Querying the database from Dash callbacks
• Rule: Read-only access from Dash. Use sqlite3.connect(db_path) with SELECT queries only. Never INSERT, UPDATE, DELETE, or ALTER from the Dash app.
• Exception: The standalone cli/compute_trends.py script may CREATE and INSERT into the pathway_trends table. This is a separate CLI command, not part of the Dash app or the main refresh pipeline.
• Why: pathways.db is populated by CLI commands. The Dash app is a read-only consumer.

Trend computation uses existing pipeline functions as-is

• When: Building cli/compute_trends.py
• Rule: Import and call fetch_and_transform_data() and process_pathway_for_date_filter() from pathway_pipeline.py. Do NOT modify these functions. Do NOT modify schema.py, reference_data.py, or refresh_pathways.py. The new script creates its own table via CREATE TABLE IF NOT EXISTS.
• Why: The historical snapshot approach works by calling existing functions with different max_date values. No pipeline changes needed.

---

Chart Generation (plotly_generator.py)

Use _base_layout() for all chart functions

• When: Modifying or creating any chart function after Task A.1
• Rule: Call _base_layout(title) to get shared layout properties, then update with chart-specific overrides. Do NOT hardcode font family, title font size, bgcolor, hoverlabel, or autosize in individual functions.
• Why: DRY principle. Inconsistent styling was a bug category (Tier 2 fix).

Use module-level palette constants

• When: Assigning colors to traces in any chart function
• Rule: Use TRUST_PALETTE (7 colors) for trust-comparison charts where bars/traces represent trusts. Use DRUG_PALETTE (15 colors) for charts where bars/traces represent drugs. Do NOT define local nhs_colours lists.
• Why: Local blue-heavy palettes made trusts indistinguishable (a reported bug).

Heatmaps must have cell text annotations

• When: Modifying create_heatmap_figure() or create_trust_heatmap_figure()
• Rule: Always include text=text_values, texttemplate="%{text}" on the heatmap trace. Format text per metric: patients → "N", cost → "£Nk", cost_pp_pa → "£N".
• Why: Without cell text, users must hover every cell to read values — a reported usability bug.

Heatmaps must use linear colorscale

• When: Setting colorscale on heatmap traces
• Rule: Use linear 5-stop colorscale: [0.0 #E3F2FD, 0.25 #90CAF9, 0.5 #42A5F5, 0.75 #1E88E5, 1.0 #003087]. Always set zmin=0. Do NOT use non-linear stops like [0.01, 0.1, 0.3, ...].
• Why: Non-linear stops compressed 99% of the value range into identical blues.

Charts must use autosize, not fixed width

• When: Setting chart dimensions
• Rule: Use autosize=True instead of explicit width=.... Dynamic height is fine (calculated from data). Use yaxis automargin=True instead of fixed left margins.
• Why: Fixed widths overflow their containers on different screen sizes.

Legends must adapt to item count

• When: Setting legend layout on charts with variable trace counts
• Rule: Use _smart_legend(n_items) helper (once created in Task A.3). >15 items = vertical right legend. ≤15 items = horizontal with dynamic bottom margin.
• Why: Horizontal legends with 42 drugs wrap 5+ rows and overlap chart content.

---

Callback Architecture

No circular callback dependencies

• When: Writing Dash callbacks
• Rule: Callbacks must flow unidirectionally: filter inputs → app-state store → chart-data store → UI components. Never have a component that is both Input and Output in the same callback chain without an intermediate store.
• Why: Dash raises DuplicateCallback errors for circular dependencies.

Use dcc.Store for all state, not server-side globals

• When: Managing application state
• Rule: ALL state lives in dcc.Store components. Never use module-level globals or class variables for state. The 4 stores: app-state (session), chart-data (memory), reference-data (session), active-tab (memory).
• Why: Dash is stateless per request. Server-side state breaks with multiple users.

Only render the active tab's chart

• When: Building tab switching or chart rendering callbacks
• Rule: Check active-tab store and ONLY compute the figure for the active tab. Return no_update or placeholder for inactive tabs.
• Why: Computing all charts on every filter change would be extremely slow.

Chart figure functions go in src/visualization/, not dash_app/

• When: Creating new chart figures
• Rule: Create figure builder functions in src/visualization/plotly_generator.py. Dash callbacks call these shared functions. Do NOT put Plotly figure construction logic directly in dash_app/callbacks/.
• Why: Shared figure functions can be tested independently and reused.

New query functions use same pattern as existing ones

• When: Adding query functions to src/data_processing/pathway_queries.py
• Rule: Follow the same pattern as load_pathway_nodes(): accept db_path parameter, use sqlite3.connect() with row_factory = sqlite3.Row, parameterized queries, return JSON-serializable dicts/lists. Add thin wrappers in dash_app/data/queries.py.
• Why: Consistency with existing code. The thin wrapper pattern ensures DB path resolution is centralized.

---

Data Patterns

Use parameterized queries for all filters

• When: Building WHERE clauses with user-selected values
• Rule: Use ? placeholders and pass params as a list. Never use f-strings or string interpolation for filter values.
• Why: Prevents SQL injection and handles special characters in drug/directory names (e.g., "CROHN'S DISEASE").

Parsing utilities must handle missing/null data gracefully

• When: Parsing average_spacing HTML strings, average_administered JSON, or ids column values
• Rule: Always handle None, empty string "", and malformed data. Return sensible defaults rather than raising exceptions.
• Why: Not all nodes have statistics populated. Level 0-2 nodes have no drug-level statistics.

---

Process Guardrails

One task per iteration

• When: Temptation to do additional tasks after completing the current one
• Rule: Complete ONE task, validate it, commit it, update progress, then stop
• Why: Multiple tasks increase error risk and make failures harder to diagnose

Never mark complete without validation

• When: Task feels "done" but hasn't been tested
• Rule: All validation tiers must pass before marking [x]
• Why: "Feels done" is not "is done"

Write explicit handoff notes

• When: Every iteration, before stopping
• Rule: The "Next iteration should" section must contain specific, actionable guidance
• Why: The next iteration has zero memory. If you don't write it down, it's lost.

Validate with python run_dash.py

• When: After completing any task
• Rule: Run python run_dash.py (or python -c "from dash_app.app import app" for import checks). The app must start without errors after EVERY task.
• Why: Broken imports or circular dependencies compound across tasks. Catch them immediately.

Re-read plotly_generator.py before editing

• When: Starting any task that modifies chart functions
• Rule: Always re-read src/visualization/plotly_generator.py at the start of the iteration. Line numbers in IMPLEMENTATION_PLAN.md are approximate and shift as edits accumulate. Search for function names, not line numbers.
• Why: Previous iterations may have changed the file, shifting all line numbers.

3-view navigation pattern

• When: Modifying switch_view() in navigation.py or update_app_state() in filters.py
• Rule: There are 3 views: patient-pathways, trust-comparison, trends. The switch_view() callback has 6 Outputs (3 view styles + 3 nav classNames). The update_app_state() callback has 3 nav Inputs. When updating either callback, ensure ALL return paths handle all 3 views correctly. Every return statement must include values for all 6 outputs / handle all 3 active_view values.
• Why: Adding a 3rd view to a previously binary toggle is error-prone — missing a return path causes Dash callback errors.

Trends view state in app-state

• When: Working on the Trends view (E.2–E.4)
• Rule: selected_trends_directorate must be initialized as None in the app-state dcc.Store initial data in app.py. The Trends view uses landing/detail toggle based on this value (same pattern as Trust Comparison's selected_comparison_directorate).
• Why: Missing initial state causes KeyError on first page load.

Removing callback Outputs/Inputs requires updating ALL return paths

• When: Removing Outputs or Inputs from an existing callback (e.g., E.1 removing trends toggle from update_chart)
• Rule: When removing an Output from a callback, you MUST update EVERY return statement in that callback to match the new Output count. Count the number of return statements before editing and verify the same count after. The update_chart() callback currently has 4+ return paths.
• Why: Mismatched return tuple length causes InvalidCallbackReturnValue at runtime.