     1ÔåÆ# Guardrails
     2ÔåÆ
     3ÔåÆKnown failure patterns. Read EVERY iteration. Follow ALL of these rules.
     4ÔåÆIf you discover a new failure pattern during your work, add it to this file.
     5ÔåÆ
     6ÔåÆ---
     7ÔåÆ
     8ÔåÆ## Drug-Indication Matching Guardrails
     9ÔåÆ
    10ÔåÆ### Match drugs to indications, not just patients to indications
    11ÔåÆ- **When**: Building the indication mapping for pathway charts
    12ÔåÆ- **Rule**: Each drug must be validated against BOTH the patient's GP diagnoses AND the drug-to-indication mapping from DimSearchTerm.csv. A patient being diagnosed with rheumatoid arthritis does NOT mean all their drugs are for rheumatoid arthritis.
    13ÔåÆ- **Why**: The previous approach assigned ONE indication per patient (most recent GP dx), ignoring which drugs actually treat which conditions. This produced misleading pathways.
    14ÔåÆ
    15ÔåÆ### Use DimSearchTerm.csv for drug-to-Search_Term mapping
    16ÔåÆ- **When**: Determining which Search_Term a drug belongs to
    17ÔåÆ- **Rule**: Load `data/DimSearchTerm.csv`. The `CleanedDrugName` column has pipe-separated drug name fragments. Match HCD drug names against these fragments using substring matching (case-insensitive).
    18ÔåÆ- **Why**: This CSV is the authoritative mapping of which drugs are used for which clinical indications.
    19ÔåÆ
    20ÔåÆ### Use substring matching for drug fragments
    21ÔåÆ- **When**: Matching HCD drug names against DimSearchTerm CleanedDrugName fragments
    22ÔåÆ- **Rule**: Check if any fragment from DimSearchTerm is a SUBSTRING of the HCD drug name (case-insensitive). E.g., "PEGYLATED" should match "PEGYLATED LIPOSOMAL DOXORUBICIN".
    23ÔåÆ- **Why**: DimSearchTerm contains both full drug names (ADALIMUMAB) and partial fragments (PEGYLATED, INHALED). Exact match would miss the partial ones.
    24ÔåÆ
    25ÔåÆ### Modified UPID uses pipe delimiter
    26ÔåÆ- **When**: Creating indication-aware UPIDs
    27ÔåÆ- **Rule**: Format is `{original_UPID}|{search_term}`. Use pipe `|` as delimiter. Do NOT use ` - ` (hyphen with spaces) as that's used for pathway hierarchy levels in the `ids` column.
    28ÔåÆ- **Why**: The `ids` column uses " - " to separate hierarchy levels (e.g., "N&WICS - NNUH - rheumatoid arthritis - ADALIMUMAB"). Using the same delimiter in UPIDs would break hierarchy parsing.
    29ÔåÆ
    30ÔåÆ### Return ALL GP matches per patient, not just most recent
    31ÔåÆ- **When**: Querying Snowflake for patient GP diagnoses
    32ÔåÆ- **Rule**: Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`. Return ALL matching Search_Terms per patient with `GROUP BY + COUNT(*)` for code_frequency.
    33ÔåÆ- **Why**: A patient may have GP diagnoses for both rheumatoid arthritis AND asthma. We need ALL matches to cross-reference with their drugs.
    34ÔåÆ
    35ÔåÆ### Restrict GP code lookup to HCD data window
    36ÔåÆ- **When**: Building the WHERE clause for the GP record query
    37ÔåÆ- **Rule**: Add `AND pc."EventDateTime" >= :earliest_hcd_date` where `earliest_hcd_date` is `MIN(Intervention Date)` from the HCD DataFrame. Pass this as a parameter to `get_patient_indication_groups()`.
    38ÔåÆ- **Why**: Old GP codes from years before treatment started add noise. A diagnosis coded 10 years ago may no longer be relevant. Restricting to the HCD window ensures code_frequency reflects recent clinical activity for the conditions being actively treated.
    39ÔåÆ
    40ÔåÆ### Tiebreaker: highest GP code frequency when a drug matches multiple indications
    41ÔåÆ- **When**: A single drug maps to multiple Search_Terms AND the patient has GP dx for multiple
    42ÔåÆ- **Rule**: Use `code_frequency` (COUNT of matching SNOMED codes per Search_Term per patient) from the GP query. The Search_Term with the most matching codes in the patient's GP record wins. If tied, use alphabetical Search_Term for determinism.
    43ÔåÆ- **Why**: E.g., ADALIMUMAB is listed under rheumatoid arthritis, crohn's disease, psoriatic arthritis, etc. A patient with 47 RA codes and 2 crohn's codes is almost certainly on ADALIMUMAB for RA. Frequency of GP coding is a much stronger signal of clinical intent than recency ÔÇö a recent one-off asthma check doesn't mean ADALIMUMAB is for asthma.
    44ÔåÆ
    45ÔåÆ### Same patient, different indications = separate modified UPIDs
    46ÔåÆ- **When**: A patient's drugs map to different Search_Terms
    47ÔåÆ- **Rule**: Create separate modified UPIDs for each indication. E.g., `RMV12345|rheumatoid arthritis` and `RMV12345|asthma`. These are treated as separate "patients" by the pathway analyzer.
    48ÔåÆ- **Why**: This is the core design ÔÇö drugs for different indications should create separate treatment pathways, even for the same physical patient.
    49ÔåÆ
    50ÔåÆ### Fallback to directory for unmatched drugs
    51ÔåÆ- **When**: A drug doesn't match any Search_Term OR the patient has no GP dx for any of the drug's Search_Terms
    52ÔåÆ- **Rule**: Use fallback format: `{UPID}|{Directory} (no GP dx)`. The indication_df maps this to `"{Directory} (no GP dx)"`.
    53ÔåÆ- **Why**: Maintains consistent behavior with the previous approach for patients/drugs without GP diagnosis matches.
    54ÔåÆ
    55ÔåÆ### Merge asthma Search_Terms but keep urticaria separate
    56ÔåÆ- **When**: Working with asthma-related Search_Terms from CLUSTER_MAPPING_SQL or DimSearchTerm.csv
    57ÔåÆ- **Rule**: Merge "allergic asthma", "asthma", and "severe persistent allergic asthma" into a single "asthma" Search_Term. Keep "urticaria" as a separate Search_Term ÔÇö do NOT merge it with asthma.
    58ÔåÆ- **Why**: These are clinically the same condition at different severity levels. Splitting them fragments the data. Urticaria is a distinct dermatological condition that happens to share OMALIZUMAB.
    59ÔåÆ
    60ÔåÆ### Don't modify directory chart processing
    61ÔåÆ- **When**: Making changes to the indication matching logic
    62ÔåÆ- **Rule**: Only modify the indication chart path (`elif current_chart_type == "indication":`). Directory charts use unmodified UPIDs and directory-based grouping.
    63ÔåÆ- **Why**: Directory charts work correctly and should not be affected by indication matching changes.
    64ÔåÆ
    65ÔåÆ---
    66ÔåÆ
    67ÔåÆ## Snowflake Query Guardrails
    68ÔåÆ
    69ÔåÆ### Use PseudoNHSNoLinked for GP record matching
    70ÔåÆ- **When**: Querying GP records (PrimaryCareClinicalCoding) for patient diagnoses
    71ÔåÆ- **Rule**: Use `PseudoNHSNoLinked` column from HCD data, NOT `PersonKey` (LocalPatientID)
    72ÔåÆ- **Why**: PersonKey is provider-specific local ID. Only PseudoNHSNoLinked matches PatientPseudonym in GP records.
    73ÔåÆ
    74ÔåÆ### Embed cluster query as CTE in Snowflake
    75ÔåÆ- **When**: Looking up patient indications during data refresh
    76ÔåÆ- **Rule**: Use the `CLUSTER_MAPPING_SQL` content as a WITH clause in the patient lookup query
    77ÔåÆ- **Why**: This ensures we always use the complete cluster mapping and don't need local storage
    78ÔåÆ
    79ÔåÆ### Quote mixed-case column aliases in Snowflake SQL
    80ÔåÆ- **When**: Writing SELECT queries that return results to Python code
    81ÔåÆ- **Rule**: Use `AS "ColumnName"` (quoted) for any column alias you'll access by name in Python
    82ÔåÆ- **Why**: Snowflake uppercases unquoted identifiers. `SELECT foo AS Search_Term` returns `SEARCH_TERM`, so `row.get('Search_Term')` returns None. Fix: `SELECT foo AS "Search_Term"`
    83ÔåÆ
    84ÔåÆ### Build indication_df from all unique UPIDs, not PseudoNHSNoLinked
    85ÔåÆ- **When**: Creating the indication mapping DataFrame for pathway processing
    86ÔåÆ- **Rule**: Use `df.drop_duplicates(subset=['UPID'])` not `drop_duplicates(subset=['PseudoNHSNoLinked'])`
    87ÔåÆ- **Why**: A patient visiting multiple providers has multiple UPIDs. Using unique PseudoNHSNoLinked only maps one UPID per patient, leaving others as NaN.
    88ÔåÆ
    89ÔåÆ---
    90ÔåÆ
    91ÔåÆ## Data Processing Guardrails
    92ÔåÆ
    93ÔåÆ### Copy DataFrames in functions that modify columns
    94ÔåÆ- **When**: Writing functions like `prepare_data()` that modify DataFrame columns
    95ÔåÆ- **Rule**: Always `df = df.copy()` at the start of any function that modifies column values on the input DataFrame
    96ÔåÆ- **Why**: `prepare_data()` mapped Provider Code ÔåÆ Name in-place. When called multiple times on the same DataFrame, only the first call worked. The fix: `df.copy()` prevents destructive mutation.
    97ÔåÆ
    98ÔåÆ### Include chart_type in UNIQUE constraints for pathway_nodes
    99ÔåÆ- **When**: Creating or modifying the pathway_nodes table schema
   100ÔåÆ- **Rule**: The UNIQUE constraint MUST include `chart_type`: `UNIQUE(date_filter_id, chart_type, ids)`
   101ÔåÆ- **Why**: Without `chart_type`, `INSERT OR REPLACE` silently overwrites directory chart nodes when indication chart nodes are inserted.
   102ÔåÆ
   103ÔåÆ### Handle NaN in Directory when building fallback labels
   104ÔåÆ- **When**: Creating fallback indication labels for patients without GP diagnosis match
   105ÔåÆ- **Rule**: Check `pd.notna(directory)` before concatenating to string. Use `"UNKNOWN (no GP dx)"` for NaN cases.
   106ÔåÆ- **Why**: NaN handling prevents TypeError and ensures meaningful fallback labels.
   107ÔåÆ
   108ÔåÆ### Use parameterized queries for SQLite
   109ÔåÆ- **When**: Building WHERE clauses with user-selected filters
   110ÔåÆ- **Rule**: Use `?` placeholders and pass params tuple ÔÇö never string interpolation
   111ÔåÆ- **Why**: Prevents SQL injection and handles special characters in drug/directory names
   112ÔåÆ
   113ÔåÆ### Use existing pathway_analyzer functions
   114ÔåÆ- **When**: Processing pathway data for the icicle chart
   115ÔåÆ- **Rule**: Reuse functions from `analysis/pathway_analyzer.py` ÔÇö don't reinvent
   116ÔåÆ- **Why**: The existing code handles edge cases (empty groups, statistics calculation, color mapping)
   117ÔåÆ
   118ÔåÆ---
   119ÔåÆ
   120ÔåÆ## Reflex Guardrails
   121ÔåÆ
   122ÔåÆ### Use .to() methods for Var operations in rx.foreach
   123ÔåÆ- **When**: Working with items inside `rx.foreach` render functions
   124ÔåÆ- **Rule**: Use `item.to(int)` for numeric comparisons, `item.to_string()` for text operations
   125ÔåÆ- **Why**: Items from rx.foreach are Var objects, not plain Python values.
   126ÔåÆ
   127ÔåÆ### Use rx.cond for conditional rendering, not Python if
   128ÔåÆ- **When**: Conditionally showing/hiding components or changing styles based on state
   129ÔåÆ- **Rule**: Use `rx.cond(condition, true_component, false_component)` ÔÇö not Python `if`
   130ÔåÆ- **Why**: Python `if` evaluates at definition time; `rx.cond` evaluates reactively at render time
   131ÔåÆ
   132ÔåÆ---
   133ÔåÆ
   134ÔåÆ## Process Guardrails
   135ÔåÆ
   136ÔåÆ### One task per iteration
   137ÔåÆ- **When**: Temptation to do additional tasks after completing the current one
   138ÔåÆ- **Rule**: Complete ONE task, validate it, commit it, update progress, then stop
   139ÔåÆ- **Why**: Multiple tasks increase error risk and make failures harder to diagnose
   140ÔåÆ
   141ÔåÆ### Never mark complete without validation
   142ÔåÆ- **When**: Task feels "done" but hasn't been tested
   143ÔåÆ- **Rule**: All validation tiers must pass before marking `[x]`
   144ÔåÆ- **Why**: "Feels done" is not "is done"
   145ÔåÆ
   146ÔåÆ### Write explicit handoff notes
   147ÔåÆ- **When**: Every iteration, before stopping
   148ÔåÆ- **Rule**: The "Next iteration should" section must contain specific, actionable guidance
   149ÔåÆ- **Why**: The next iteration has zero memory. If you don't write it down, it's lost.
   150ÔåÆ
   151ÔåÆ### Check existing code for patterns
   152ÔåÆ- **When**: Unsure how to implement something
   153ÔåÆ- **Rule**: Look at `pathways_app/pathways_app.py`, `analysis/pathway_analyzer.py`, `cli/refresh_pathways.py`
   154ÔåÆ- **Why**: The existing codebase has solved many quirks already
   155ÔåÆ
   156ÔåÆ<!--
   157ÔåÆADD NEW GUARDRAILS BELOW as failures are observed during the loop.
   158ÔåÆ
   159ÔåÆFormat:
   160ÔåÆ### [Short descriptive name]
   161ÔåÆ- **When**: What situation triggers this guardrail?
   162ÔåÆ- **Rule**: What must you do (or not do)?
   163ÔåÆ- **Why**: What failure prompted adding this guardrail?
   164ÔåÆ-->
   165ÔåÆ

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

# Guardrails

Known failure patterns. Read EVERY iteration. Follow ALL of these rules.
If you discover a new failure pattern during your work, add it to this file.

---

## Drug-Indication Matching Guardrails

### Match drugs to indications, not just patients to indications
- **When**: Building the indication mapping for pathway charts
- **Rule**: Each drug must be validated against BOTH the patient's GP diagnoses AND the drug-to-indication mapping from DimSearchTerm.csv. A patient being diagnosed with rheumatoid arthritis does NOT mean all their drugs are for rheumatoid arthritis.
- **Why**: The previous approach assigned ONE indication per patient (most recent GP dx), ignoring which drugs actually treat which conditions. This produced misleading pathways.

### Use DimSearchTerm.csv for drug-to-Search_Term mapping
- **When**: Determining which Search_Term a drug belongs to
- **Rule**: Load `data/DimSearchTerm.csv`. The `CleanedDrugName` column has pipe-separated drug name fragments. Match HCD drug names against these fragments using substring matching (case-insensitive).
- **Why**: This CSV is the authoritative mapping of which drugs are used for which clinical indications.

### Use substring matching for drug fragments
- **When**: Matching HCD drug names against DimSearchTerm CleanedDrugName fragments
- **Rule**: Check if any fragment from DimSearchTerm is a SUBSTRING of the HCD drug name (case-insensitive). E.g., "PEGYLATED" should match "PEGYLATED LIPOSOMAL DOXORUBICIN".
- **Why**: DimSearchTerm contains both full drug names (ADALIMUMAB) and partial fragments (PEGYLATED, INHALED). Exact match would miss the partial ones.

### Modified UPID uses pipe delimiter
- **When**: Creating indication-aware UPIDs
- **Rule**: Format is `{original_UPID}|{search_term}`. Use pipe `|` as delimiter. Do NOT use ` - ` (hyphen with spaces) as that's used for pathway hierarchy levels in the `ids` column.
- **Why**: The `ids` column uses " - " to separate hierarchy levels (e.g., "N&WICS - NNUH - rheumatoid arthritis - ADALIMUMAB"). Using the same delimiter in UPIDs would break hierarchy parsing.

### Return ALL GP matches per patient, not just most recent
- **When**: Querying Snowflake for patient GP diagnoses
- **Rule**: Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`. Return ALL matching Search_Terms per patient with `GROUP BY + COUNT(*)` for code_frequency.
- **Why**: A patient may have GP diagnoses for both rheumatoid arthritis AND asthma. We need ALL matches to cross-reference with their drugs.

### Restrict GP code lookup to HCD data window
- **When**: Building the WHERE clause for the GP record query
- **Rule**: Add `AND pc."EventDateTime" >= :earliest_hcd_date` where `earliest_hcd_date` is `MIN(Intervention Date)` from the HCD DataFrame. Pass this as a parameter to `get_patient_indication_groups()`.
- **Why**: Old GP codes from years before treatment started add noise. A diagnosis coded 10 years ago may no longer be relevant. Restricting to the HCD window ensures code_frequency reflects recent clinical activity for the conditions being actively treated.

### Tiebreaker: highest GP code frequency when a drug matches multiple indications
- **When**: A single drug maps to multiple Search_Terms AND the patient has GP dx for multiple
- **Rule**: Use `code_frequency` (COUNT of matching SNOMED codes per Search_Term per patient) from the GP query. The Search_Term with the most matching codes in the patient's GP record wins. If tied, use alphabetical Search_Term for determinism.
- **Why**: E.g., ADALIMUMAB is listed under rheumatoid arthritis, crohn's disease, psoriatic arthritis, etc. A patient with 47 RA codes and 2 crohn's codes is almost certainly on ADALIMUMAB for RA. Frequency of GP coding is a much stronger signal of clinical intent than recency ÔÇö a recent one-off asthma check doesn't mean ADALIMUMAB is for asthma.

### Same patient, different indications = separate modified UPIDs
- **When**: A patient's drugs map to different Search_Terms
- **Rule**: Create separate modified UPIDs for each indication. E.g., `RMV12345|rheumatoid arthritis` and `RMV12345|asthma`. These are treated as separate "patients" by the pathway analyzer.
- **Why**: This is the core design ÔÇö drugs for different indications should create separate treatment pathways, even for the same physical patient.

### Fallback to directory for unmatched drugs
- **When**: A drug doesn't match any Search_Term OR the patient has no GP dx for any of the drug's Search_Terms
- **Rule**: Use fallback format: `{UPID}|{Directory} (no GP dx)`. The indication_df maps this to `"{Directory} (no GP dx)"`.
- **Why**: Maintains consistent behavior with the previous approach for patients/drugs without GP diagnosis matches.

### Merge asthma Search_Terms but keep urticaria separate
- **When**: Working with asthma-related Search_Terms from CLUSTER_MAPPING_SQL or DimSearchTerm.csv
- **Rule**: Merge "allergic asthma", "asthma", and "severe persistent allergic asthma" into a single "asthma" Search_Term. Keep "urticaria" as a separate Search_Term ÔÇö do NOT merge it with asthma.
- **Why**: These are clinically the same condition at different severity levels. Splitting them fragments the data. Urticaria is a distinct dermatological condition that happens to share OMALIZUMAB.

### Don't modify directory chart processing
- **When**: Making changes to the indication matching logic
- **Rule**: Only modify the indication chart path (`elif current_chart_type == "indication":`). Directory charts use unmodified UPIDs and directory-based grouping.
- **Why**: Directory charts work correctly and should not be affected by indication matching changes.

---

## Snowflake Query Guardrails

### Use PseudoNHSNoLinked for GP record matching
- **When**: Querying GP records (PrimaryCareClinicalCoding) for patient diagnoses
- **Rule**: Use `PseudoNHSNoLinked` column from HCD data, NOT `PersonKey` (LocalPatientID)
- **Why**: PersonKey is provider-specific local ID. Only PseudoNHSNoLinked matches PatientPseudonym in GP records.

### Embed cluster query as CTE in Snowflake
- **When**: Looking up patient indications during data refresh
- **Rule**: Use the `CLUSTER_MAPPING_SQL` content as a WITH clause in the patient lookup query
- **Why**: This ensures we always use the complete cluster mapping and don't need local storage

### Quote mixed-case column aliases in Snowflake SQL
- **When**: Writing SELECT queries that return results to Python code
- **Rule**: Use `AS "ColumnName"` (quoted) for any column alias you'll access by name in Python
- **Why**: Snowflake uppercases unquoted identifiers. `SELECT foo AS Search_Term` returns `SEARCH_TERM`, so `row.get('Search_Term')` returns None. Fix: `SELECT foo AS "Search_Term"`

### Build indication_df from all unique UPIDs, not PseudoNHSNoLinked
- **When**: Creating the indication mapping DataFrame for pathway processing
- **Rule**: Use `df.drop_duplicates(subset=['UPID'])` not `drop_duplicates(subset=['PseudoNHSNoLinked'])`
- **Why**: A patient visiting multiple providers has multiple UPIDs. Using unique PseudoNHSNoLinked only maps one UPID per patient, leaving others as NaN.

---

## Data Processing Guardrails

### Copy DataFrames in functions that modify columns
- **When**: Writing functions like `prepare_data()` that modify DataFrame columns
- **Rule**: Always `df = df.copy()` at the start of any function that modifies column values on the input DataFrame
- **Why**: `prepare_data()` mapped Provider Code ÔåÆ Name in-place. When called multiple times on the same DataFrame, only the first call worked. The fix: `df.copy()` prevents destructive mutation.

### Include chart_type in UNIQUE constraints for pathway_nodes
- **When**: Creating or modifying the pathway_nodes table schema
- **Rule**: The UNIQUE constraint MUST include `chart_type`: `UNIQUE(date_filter_id, chart_type, ids)`
- **Why**: Without `chart_type`, `INSERT OR REPLACE` silently overwrites directory chart nodes when indication chart nodes are inserted.

### Handle NaN in Directory when building fallback labels
- **When**: Creating fallback indication labels for patients without GP diagnosis match
- **Rule**: Check `pd.notna(directory)` before concatenating to string. Use `"UNKNOWN (no GP dx)"` for NaN cases.
- **Why**: NaN handling prevents TypeError and ensures meaningful fallback labels.

### Use parameterized queries for SQLite
- **When**: Building WHERE clauses with user-selected filters
- **Rule**: Use `?` placeholders and pass params tuple ÔÇö never string interpolation
- **Why**: Prevents SQL injection and handles special characters in drug/directory names

### Use existing pathway_analyzer functions
- **When**: Processing pathway data for the icicle chart
- **Rule**: Reuse functions from `analysis/pathway_analyzer.py` ÔÇö don't reinvent
- **Why**: The existing code handles edge cases (empty groups, statistics calculation, color mapping)

---

## Reflex Guardrails

### Use .to() methods for Var operations in rx.foreach
- **When**: Working with items inside `rx.foreach` render functions
- **Rule**: Use `item.to(int)` for numeric comparisons, `item.to_string()` for text operations
- **Why**: Items from rx.foreach are Var objects, not plain Python values.

### Use rx.cond for conditional rendering, not Python if
- **When**: Conditionally showing/hiding components or changing styles based on state
- **Rule**: Use `rx.cond(condition, true_component, false_component)` ÔÇö not Python `if`
- **Why**: Python `if` evaluates at definition time; `rx.cond` evaluates reactively at render time

---

## Process Guardrails

### One task per iteration
- **When**: Temptation to do additional tasks after completing the current one
- **Rule**: Complete ONE task, validate it, commit it, update progress, then stop
- **Why**: Multiple tasks increase error risk and make failures harder to diagnose

### Never mark complete without validation
- **When**: Task feels "done" but hasn't been tested
- **Rule**: All validation tiers must pass before marking `[x]`
- **Why**: "Feels done" is not "is done"

### Write explicit handoff notes
- **When**: Every iteration, before stopping
- **Rule**: The "Next iteration should" section must contain specific, actionable guidance
- **Why**: The next iteration has zero memory. If you don't write it down, it's lost.

### Check existing code for patterns
- **When**: Unsure how to implement something
- **Rule**: Look at `pathways_app/pathways_app.py`, `analysis/pathway_analyzer.py`, `cli/refresh_pathways.py`
- **Why**: The existing codebase has solved many quirks already

<!--
ADD NEW GUARDRAILS BELOW as failures are observed during the loop.

Format:
### [Short descriptive name]
- **When**: What situation triggers this guardrail?
- **Rule**: What must you do (or not do)?
- **Why**: What failure prompted adding this guardrail?
-->


Reading progress.txt, IMPLEMENTATION_PLAN.md, and guardrails.md. Now checking recent git history.


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && git log --oneline -5

Show recent git commits


73088b0 docs: update progress.txt with Iteration 5 results (Task 3.1)
920570b feat: integrate drug-aware indication matching into refresh pipeline (Task 3.1)
d9891c8 docs: update progress.txt with Iteration 4 results (Task 2.1 + 2.2)
408976e feat: add assign_drug_indications() for drug-aware indication matching (Task 2.1 + 2.2)
947b87a docs: update progress.txt with Iteration 3 results (Task 1.1)


Good. I can see the state of the project:

**Completed**: Tasks 1.1, 1.2, 1.3 (Phase 1), Tasks 2.1, 2.2 (Phase 2), Task 3.1 (Phase 3 integration)
**Next up**: Task 3.2 ÔÇö Test with dry run (`python -m cli.refresh_pathways --chart-type indication --dry-run -v`)

This is the first task that requires a live Snowflake connection. Let me examine the current state of the refresh_pathways.py code and understand the dry run flag behavior before running it.

**Why Task 3.2**: It's the next logical step ÔÇö 3.1 (pipeline integration) is complete, and 3.2 validates that the integrated pipeline actually works end-to-end. All later tasks (4.1, 4.2, 4.3) depend on 3.2 passing.


     1ÔåÆ"""
     2ÔåÆCLI command for refreshing pathway data from Snowflake.
     3ÔåÆ
     4ÔåÆThis command fetches activity data from Snowflake, processes it through the
     5ÔåÆpathway pipeline for all 6 date filter combinations, and stores the results
     6ÔåÆin the SQLite pathway_nodes table. Supports two chart types:
     7ÔåÆ- "directory": Trust ÔåÆ Directory ÔåÆ Drug ÔåÆ Pathway (default)
     8ÔåÆ- "indication": Trust ÔåÆ Search_Term ÔåÆ Drug ÔåÆ Pathway (requires GP diagnosis lookup)
     9ÔåÆ
    10ÔåÆUsage:
    11ÔåÆ    python -m cli.refresh_pathways
    12ÔåÆ    python -m cli.refresh_pathways --minimum-patients 10
    13ÔåÆ    python -m cli.refresh_pathways --provider-codes RGT,RM1
    14ÔåÆ    python -m cli.refresh_pathways --chart-type all
    15ÔåÆ    python -m cli.refresh_pathways --chart-type directory
    16ÔåÆ    python -m cli.refresh_pathways --dry-run
    17ÔåÆ
    18ÔåÆRun `python -m cli.refresh_pathways --help` for full options.
    19ÔåÆ"""
    20ÔåÆ
    21ÔåÆimport argparse
    22ÔåÆimport json
    23ÔåÆimport sqlite3
    24ÔåÆimport sys
    25ÔåÆimport time
    26ÔåÆimport uuid
    27ÔåÆfrom datetime import datetime
    28ÔåÆfrom pathlib import Path
    29ÔåÆfrom typing import Optional
    30ÔåÆ
    31ÔåÆfrom core import PathConfig, default_paths
    32ÔåÆfrom core.logging_config import get_logger, setup_logging
    33ÔåÆfrom data_processing.database import DatabaseManager, DatabaseConfig
    34ÔåÆfrom data_processing.schema import (
    35ÔåÆ    clear_pathway_nodes,
    36ÔåÆ    get_pathway_table_counts,
    37ÔåÆ    verify_pathway_tables_exist,
    38ÔåÆ    create_pathway_tables,
    39ÔåÆ)
    40ÔåÆfrom data_processing.pathway_pipeline import (
    41ÔåÆ    ChartType,
    42ÔåÆ    DATE_FILTER_CONFIGS,
    43ÔåÆ    fetch_and_transform_data,
    44ÔåÆ    process_all_date_filters,
    45ÔåÆ    process_pathway_for_date_filter,
    46ÔåÆ    process_indication_pathway_for_date_filter,
    47ÔåÆ    extract_denormalized_fields,
    48ÔåÆ    extract_indication_fields,
    49ÔåÆ    convert_to_records,
    50ÔåÆ)
    51ÔåÆfrom data_processing.diagnosis_lookup import (
    52ÔåÆ    assign_drug_indications,
    53ÔåÆ    get_patient_indication_groups,
    54ÔåÆ    load_drug_indication_mapping,
    55ÔåÆ)
    56ÔåÆ
    57ÔåÆlogger = get_logger(__name__)
    58ÔåÆ
    59ÔåÆ
    60ÔåÆdef get_default_filters(paths: PathConfig) -> tuple[list[str], list[str], list[str]]:
    61ÔåÆ    """
    62ÔåÆ    Load default filter values from reference files.
    63ÔåÆ
    64ÔåÆ    Returns:
    65ÔåÆ        Tuple of (trust_filter, drug_filter, directory_filter)
    66ÔåÆ    """
    67ÔåÆ    import pandas as pd
    68ÔåÆ
    69ÔåÆ    # Load default trusts
    70ÔåÆ    trust_filter = []
    71ÔåÆ    if paths.default_trusts_csv.exists():
    72ÔåÆ        try:
    73ÔåÆ            trusts_df = pd.read_csv(paths.default_trusts_csv)
    74ÔåÆ            # Use the "Name" column which contains trust names
    75ÔåÆ            if 'Name' in trusts_df.columns:
    76ÔåÆ                trust_filter = trusts_df['Name'].dropna().tolist()
    77ÔåÆ            else:
    78ÔåÆ                # Fallback to first column if no Name column
    79ÔåÆ                trust_filter = trusts_df.iloc[:, 0].dropna().tolist()
    80ÔåÆ            logger.info(f"Loaded {len(trust_filter)} default trusts")
    81ÔåÆ        except Exception as e:
    82ÔåÆ            logger.warning(f"Could not load default trusts: {e}")
    83ÔåÆ
    84ÔåÆ    # Load default drugs (Include=1 in include.csv)
    85ÔåÆ    drug_filter = []
    86ÔåÆ    if paths.include_csv.exists():
    87ÔåÆ        try:
    88ÔåÆ            drugs_df = pd.read_csv(paths.include_csv)
    89ÔåÆ            if 'Include' in drugs_df.columns:
    90ÔåÆ                drug_filter = drugs_df[drugs_df['Include'] == 1].iloc[:, 0].dropna().tolist()
    91ÔåÆ            else:
    92ÔåÆ                # Assume first column contains drug names if no Include column
    93ÔåÆ                drug_filter = drugs_df.iloc[:, 0].dropna().tolist()
    94ÔåÆ            logger.info(f"Loaded {len(drug_filter)} default drugs")
    95ÔåÆ        except Exception as e:
    96ÔåÆ            logger.warning(f"Could not load default drugs: {e}")
    97ÔåÆ
    98ÔåÆ    # Load default directories
    99ÔåÆ    directory_filter = []
   100ÔåÆ    if paths.directory_list_csv.exists():
   101ÔåÆ        try:
   102ÔåÆ            dirs_df = pd.read_csv(paths.directory_list_csv)
   103ÔåÆ            # Assume first column contains directory names
   104ÔåÆ            directory_filter = dirs_df.iloc[:, 0].dropna().tolist()
   105ÔåÆ            logger.info(f"Loaded {len(directory_filter)} default directories")
   106ÔåÆ        except Exception as e:
   107ÔåÆ            logger.warning(f"Could not load default directories: {e}")
   108ÔåÆ
   109ÔåÆ    return trust_filter, drug_filter, directory_filter
   110ÔåÆ
   111ÔåÆ
   112ÔåÆdef insert_pathway_records(
   113ÔåÆ    conn: sqlite3.Connection,
   114ÔåÆ    records: list[dict],
   115ÔåÆ) -> int:
   116ÔåÆ    """
   117ÔåÆ    Insert pathway records into pathway_nodes table.
   118ÔåÆ
   119ÔåÆ    Uses INSERT OR REPLACE to handle updates to existing records.
   120ÔåÆ
   121ÔåÆ    Args:
   122ÔåÆ        conn: SQLite connection
   123ÔåÆ        records: List of record dicts from convert_to_records()
   124ÔåÆ
   125ÔåÆ    Returns:
   126ÔåÆ        Number of records inserted
   127ÔåÆ    """
   128ÔåÆ    if not records:
   129ÔåÆ        return 0
   130ÔåÆ
   131ÔåÆ    # Column order matching pathway_nodes schema (includes chart_type)
   132ÔåÆ    columns = [
   133ÔåÆ        'date_filter_id', 'chart_type', 'parents', 'ids', 'labels', 'level',
   134ÔåÆ        'value', 'cost', 'costpp', 'cost_pp_pa', 'colour',
   135ÔåÆ        'first_seen', 'last_seen', 'first_seen_parent', 'last_seen_parent',
   136ÔåÆ        'average_spacing', 'average_administered', 'avg_days',
   137ÔåÆ        'trust_name', 'directory', 'drug_sequence', 'data_refresh_id'
   138ÔåÆ    ]
   139ÔåÆ
   140ÔåÆ    placeholders = ', '.join(['?' for _ in columns])
   141ÔåÆ    column_names = ', '.join(columns)
   142ÔåÆ
   143ÔåÆ    insert_sql = f"""
   144ÔåÆ        INSERT OR REPLACE INTO pathway_nodes ({column_names})
   145ÔåÆ        VALUES ({placeholders})
   146ÔåÆ    """
   147ÔåÆ
   148ÔåÆ    # Convert records to tuples in column order
   149ÔåÆ    rows = []
   150ÔåÆ    for record in records:
   151ÔåÆ        row = tuple(record.get(col) for col in columns)
   152ÔåÆ        rows.append(row)
   153ÔåÆ
   154ÔåÆ    cursor = conn.executemany(insert_sql, rows)
   155ÔåÆ    return cursor.rowcount
   156ÔåÆ
   157ÔåÆ
   158ÔåÆdef log_refresh_start(
   159ÔåÆ    conn: sqlite3.Connection,
   160ÔåÆ    refresh_id: str,
   161ÔåÆ    date_from: Optional[str] = None,
   162ÔåÆ    date_to: Optional[str] = None,
   163ÔåÆ) -> None:
   164ÔåÆ    """Log the start of a refresh operation."""
   165ÔåÆ    conn.execute("""
   166ÔåÆ        INSERT INTO pathway_refresh_log
   167ÔåÆ        (refresh_id, started_at, status, snowflake_query_date_from, snowflake_query_date_to)
   168ÔåÆ        VALUES (?, ?, 'running', ?, ?)
   169ÔåÆ    """, (refresh_id, datetime.now().isoformat(), date_from, date_to))
   170ÔåÆ    conn.commit()
   171ÔåÆ
   172ÔåÆ
   173ÔåÆdef log_refresh_complete(
   174ÔåÆ    conn: sqlite3.Connection,
   175ÔåÆ    refresh_id: str,
   176ÔåÆ    record_count: int,
   177ÔåÆ    date_filter_counts: dict[str, int],
   178ÔåÆ    duration_seconds: float,
   179ÔåÆ) -> None:
   180ÔåÆ    """Log the successful completion of a refresh operation."""
   181ÔåÆ    conn.execute("""
   182ÔåÆ        UPDATE pathway_refresh_log
   183ÔåÆ        SET completed_at = ?,
   184ÔåÆ            status = 'completed',
   185ÔåÆ            record_count = ?,
   186ÔåÆ            date_filter_counts = ?,
   187ÔåÆ            processing_duration_seconds = ?
   188ÔåÆ        WHERE refresh_id = ?
   189ÔåÆ    """, (
   190ÔåÆ        datetime.now().isoformat(),
   191ÔåÆ        record_count,
   192ÔåÆ        json.dumps(date_filter_counts),
   193ÔåÆ        duration_seconds,
   194ÔåÆ        refresh_id,
   195ÔåÆ    ))
   196ÔåÆ    conn.commit()
   197ÔåÆ
   198ÔåÆ
   199ÔåÆdef log_refresh_failed(
   200ÔåÆ    conn: sqlite3.Connection,
   201ÔåÆ    refresh_id: str,
   202ÔåÆ    error_message: str,
   203ÔåÆ    duration_seconds: float,
   204ÔåÆ) -> None:
   205ÔåÆ    """Log a failed refresh operation."""
   206ÔåÆ    conn.execute("""
   207ÔåÆ        UPDATE pathway_refresh_log
   208ÔåÆ        SET completed_at = ?,
   209ÔåÆ            status = 'failed',
   210ÔåÆ            error_message = ?,
   211ÔåÆ            processing_duration_seconds = ?
   212ÔåÆ        WHERE refresh_id = ?
   213ÔåÆ    """, (
   214ÔåÆ        datetime.now().isoformat(),
   215ÔåÆ        error_message,
   216ÔåÆ        duration_seconds,
   217ÔåÆ        refresh_id,
   218ÔåÆ    ))
   219ÔåÆ    conn.commit()
   220ÔåÆ
   221ÔåÆ
   222ÔåÆdef refresh_pathways(
   223ÔåÆ    minimum_patients: int = 5,
   224ÔåÆ    provider_codes: Optional[list[str]] = None,
   225ÔåÆ    trust_filter: Optional[list[str]] = None,
   226ÔåÆ    drug_filter: Optional[list[str]] = None,
   227ÔåÆ    directory_filter: Optional[list[str]] = None,
   228ÔåÆ    db_path: Optional[Path] = None,
   229ÔåÆ    paths: Optional[PathConfig] = None,
   230ÔåÆ    dry_run: bool = False,
   231ÔåÆ    chart_type: str = "directory",
   232ÔåÆ) -> tuple[bool, str, dict]:
   233ÔåÆ    """
   234ÔåÆ    Main refresh function that orchestrates the full pipeline.
   235ÔåÆ
   236ÔåÆ    Args:
   237ÔåÆ        minimum_patients: Minimum patients to include a pathway
   238ÔåÆ        provider_codes: List of provider codes to filter Snowflake query
   239ÔåÆ        trust_filter: List of trust names to include in pathways
   240ÔåÆ        drug_filter: List of drug names to include in pathways
   241ÔåÆ        directory_filter: List of directories to include in pathways
   242ÔåÆ        db_path: Path to SQLite database (uses default if None)
   243ÔåÆ        paths: PathConfig for file paths
   244ÔåÆ        dry_run: If True, don't actually insert records
   245ÔåÆ        chart_type: Which chart type to process: "directory", "indication", or "all"
   246ÔåÆ
   247ÔåÆ    Returns:
   248ÔåÆ        Tuple of (success: bool, message: str, stats: dict)
   249ÔåÆ    """
   250ÔåÆ    if paths is None:
   251ÔåÆ        paths = default_paths
   252ÔåÆ
   253ÔåÆ    # Set up database connection
   254ÔåÆ    if db_path:
   255ÔåÆ        db_config = DatabaseConfig(db_path=db_path)
   256ÔåÆ    else:
   257ÔåÆ        db_config = DatabaseConfig(data_dir=paths.data_dir)
   258ÔåÆ
   259ÔåÆ    db_manager = DatabaseManager(db_config)
   260ÔåÆ
   261ÔåÆ    # Load default filters if not provided
   262ÔåÆ    default_trusts, default_drugs, default_dirs = get_default_filters(paths)
   263ÔåÆ
   264ÔåÆ    if trust_filter is None:
   265ÔåÆ        trust_filter = default_trusts
   266ÔåÆ    if drug_filter is None:
   267ÔåÆ        drug_filter = default_drugs
   268ÔåÆ    if directory_filter is None:
   269ÔåÆ        directory_filter = default_dirs
   270ÔåÆ
   271ÔåÆ    # Ensure we have some filters
   272ÔåÆ    if not drug_filter:
   273ÔåÆ        return False, "No drugs specified and could not load defaults", {}
   274ÔåÆ
   275ÔåÆ    # Determine which chart types to process
   276ÔåÆ    if chart_type == "all":
   277ÔåÆ        chart_types_to_process: list[ChartType] = ["directory", "indication"]
   278ÔåÆ    else:
   279ÔåÆ        chart_types_to_process = [chart_type]  # type: ignore
   280ÔåÆ
   281ÔåÆ    logger.info("=" * 60)
   282ÔåÆ    logger.info("Pathway Data Refresh Starting")
   283ÔåÆ    logger.info("=" * 60)
   284ÔåÆ    logger.info(f"Minimum patients: {minimum_patients}")
   285ÔåÆ    logger.info(f"Trust filter: {len(trust_filter)} trusts")
   286ÔåÆ    logger.info(f"Drug filter: {len(drug_filter)} drugs")
   287ÔåÆ    logger.info(f"Directory filter: {len(directory_filter)} directories")
   288ÔåÆ    logger.info(f"Provider codes: {provider_codes or 'All'}")
   289ÔåÆ    logger.info(f"Chart type(s): {', '.join(chart_types_to_process)}")
   290ÔåÆ    logger.info(f"Database: {db_manager.db_path}")
   291ÔåÆ    logger.info(f"Dry run: {dry_run}")
   292ÔåÆ    logger.info("=" * 60)
   293ÔåÆ
   294ÔåÆ    start_time = time.time()
   295ÔåÆ    refresh_id = str(uuid.uuid4())[:8]
   296ÔåÆ    stats = {
   297ÔåÆ        "refresh_id": refresh_id,
   298ÔåÆ        "date_filter_counts": {},
   299ÔåÆ        "total_records": 0,
   300ÔåÆ        "snowflake_rows": 0,
   301ÔåÆ    }
   302ÔåÆ
   303ÔåÆ    try:
   304ÔåÆ        # Verify database and tables
   305ÔåÆ        with db_manager.get_connection() as conn:
   306ÔåÆ            missing_tables = verify_pathway_tables_exist(conn)
   307ÔåÆ            if missing_tables:
   308ÔåÆ                logger.info(f"Creating missing tables: {missing_tables}")
   309ÔåÆ                create_pathway_tables(conn)
   310ÔåÆ
   311ÔåÆ            # Log refresh start
   312ÔåÆ            if not dry_run:
   313ÔåÆ                log_refresh_start(conn, refresh_id)
   314ÔåÆ
   315ÔåÆ        # Step 1: Fetch data from Snowflake
   316ÔåÆ        logger.info("")
   317ÔåÆ        logger.info("Step 1/4: Fetching data from Snowflake...")
   318ÔåÆ        df = fetch_and_transform_data(
   319ÔåÆ            provider_codes=provider_codes,
   320ÔåÆ            paths=paths,
   321ÔåÆ        )
   322ÔåÆ
   323ÔåÆ        if df.empty:
   324ÔåÆ            msg = "No data returned from Snowflake"
   325ÔåÆ            logger.error(msg)
   326ÔåÆ            with db_manager.get_connection() as conn:
   327ÔåÆ                log_refresh_failed(conn, refresh_id, msg, time.time() - start_time)
   328ÔåÆ            return False, msg, stats
   329ÔåÆ
   330ÔåÆ        stats["snowflake_rows"] = len(df)
   331ÔåÆ        logger.info(f"Fetched {len(df)} records from Snowflake")
   332ÔåÆ
   333ÔåÆ        # Step 2: Process all date filters for each chart type
   334ÔåÆ        num_date_filters = len(DATE_FILTER_CONFIGS)
   335ÔåÆ        num_chart_types = len(chart_types_to_process)
   336ÔåÆ        total_datasets = num_date_filters * num_chart_types
   337ÔåÆ
   338ÔåÆ        logger.info("")
   339ÔåÆ        logger.info(f"Step 2/4: Processing pathway data for {total_datasets} datasets "
   340ÔåÆ                   f"({num_date_filters} date filters x {num_chart_types} chart types)...")
   341ÔåÆ
   342ÔåÆ        # Store results keyed by "date_filter_id:chart_type"
   343ÔåÆ        results: dict[str, list[dict]] = {}
   344ÔåÆ
   345ÔåÆ        for current_chart_type in chart_types_to_process:
   346ÔåÆ            logger.info("")
   347ÔåÆ            logger.info(f"Processing chart type: {current_chart_type}")
   348ÔåÆ
   349ÔåÆ            if current_chart_type == "directory":
   350ÔåÆ                # Use existing process_all_date_filters for directory charts
   351ÔåÆ                dir_results = process_all_date_filters(
   352ÔåÆ                    df=df,
   353ÔåÆ                    trust_filter=trust_filter,
   354ÔåÆ                    drug_filter=drug_filter,
   355ÔåÆ                    directory_filter=directory_filter,
   356ÔåÆ                    minimum_patients=minimum_patients,
   357ÔåÆ                    refresh_id=refresh_id,
   358ÔåÆ                    paths=paths,
   359ÔåÆ                )
   360ÔåÆ                # Add results with chart_type suffix
   361ÔåÆ                for filter_id, records in dir_results.items():
   362ÔåÆ                    # Records already have chart_type set by convert_to_records
   363ÔåÆ                    results[f"{filter_id}:directory"] = records
   364ÔåÆ
   365ÔåÆ            elif current_chart_type == "indication":
   366ÔåÆ                # For indication charts, use drug-aware matching:
   367ÔåÆ                # 1. Get ALL GP diagnosis matches per patient (with code_frequency)
   368ÔåÆ                # 2. Cross-reference with drug-to-Search_Term mapping from DimSearchTerm.csv
   369ÔåÆ                # 3. Assign each drug to its matched indication via modified UPIDs
   370ÔåÆ                logger.info("Building drug-aware indication groups...")
   371ÔåÆ
   372ÔåÆ                # Check Snowflake availability
   373ÔåÆ                from data_processing.snowflake_connector import get_connector, is_snowflake_available
   374ÔåÆ
   375ÔåÆ                if not is_snowflake_available():
   376ÔåÆ                    logger.warning("Snowflake not available - cannot process indication charts")
   377ÔåÆ                    for config in DATE_FILTER_CONFIGS:
   378ÔåÆ                        results[f"{config.id}:indication"] = []
   379ÔåÆ                    continue
   380ÔåÆ
   381ÔåÆ                try:
   382ÔåÆ                    import pandas as pd
   383ÔåÆ                    connector = get_connector()
   384ÔåÆ
   385ÔåÆ                    if 'PseudoNHSNoLinked' not in df.columns:
   386ÔåÆ                        logger.error("DataFrame missing 'PseudoNHSNoLinked' column - cannot lookup GP records")
   387ÔåÆ                        for config in DATE_FILTER_CONFIGS:
   388ÔåÆ                            results[f"{config.id}:indication"] = []
   389ÔåÆ                        continue
   390ÔåÆ
   391ÔåÆ                    # Step 1: Load drug-to-Search_Term mapping from DimSearchTerm.csv
   392ÔåÆ                    _, search_term_to_fragments = load_drug_indication_mapping()
   393ÔåÆ                    logger.info(f"Loaded drug mapping: {len(search_term_to_fragments)} Search_Terms")
   394ÔåÆ
   395ÔåÆ                    # Step 2: Get ALL GP diagnosis matches per patient (with code_frequency)
   396ÔåÆ                    patient_pseudonyms = df['PseudoNHSNoLinked'].dropna().unique().tolist()
   397ÔåÆ                    logger.info(f"Looking up GP diagnoses for {len(patient_pseudonyms)} unique patients...")
   398ÔåÆ
   399ÔåÆ                    # Restrict GP codes to HCD data window (reduces noise from old diagnoses)
   400ÔåÆ                    earliest_hcd_date = df['Intervention Date'].min()
   401ÔåÆ                    if pd.notna(earliest_hcd_date):
   402ÔåÆ                        earliest_hcd_date_str = pd.Timestamp(earliest_hcd_date).strftime('%Y-%m-%d')
   403ÔåÆ                        logger.info(f"Restricting GP codes to HCD window: >= {earliest_hcd_date_str}")
   404ÔåÆ                    else:
   405ÔåÆ                        earliest_hcd_date_str = None
   406ÔåÆ
   407ÔåÆ                    gp_matches_df = get_patient_indication_groups(
   408ÔåÆ                        patient_pseudonyms=patient_pseudonyms,
   409ÔåÆ                        connector=connector,
   410ÔåÆ                        batch_size=500,
   411ÔåÆ                        earliest_hcd_date=earliest_hcd_date_str,
   412ÔåÆ                    )
   413ÔåÆ
   414ÔåÆ                    # Step 3: Assign drug-aware indications using cross-referencing
   415ÔåÆ                    # This replaces the old per-patient approach with per-drug matching
   416ÔåÆ                    modified_df, indication_df = assign_drug_indications(
   417ÔåÆ                        df=df,
   418ÔåÆ                        gp_matches_df=gp_matches_df,
   419ÔåÆ                        search_term_to_fragments=search_term_to_fragments,
   420ÔåÆ                    )
   421ÔåÆ
   422ÔåÆ                    logger.info(f"Drug-aware indication matching complete. "
   423ÔåÆ                               f"Modified UPIDs: {modified_df['UPID'].nunique()}, "
   424ÔåÆ                               f"Indication groups: {len(indication_df)}")
   425ÔåÆ
   426ÔåÆ                    if indication_df.empty:
   427ÔåÆ                        logger.warning("Empty indication_df - skipping indication charts")
   428ÔåÆ                        for config in DATE_FILTER_CONFIGS:
   429ÔåÆ                            results[f"{config.id}:indication"] = []
   430ÔåÆ                    else:
   431ÔåÆ                        # Process each date filter with drug-aware indication grouping
   432ÔåÆ                        # Use modified_df (with indication-aware UPIDs) instead of original df
   433ÔåÆ                        for config in DATE_FILTER_CONFIGS:
   434ÔåÆ                            logger.info(f"Processing indication pathway for {config.id}")
   435ÔåÆ
   436ÔåÆ                            ice_df = process_indication_pathway_for_date_filter(
   437ÔåÆ                                df=modified_df,
   438ÔåÆ                                indication_df=indication_df,
   439ÔåÆ                                config=config,
   440ÔåÆ                                trust_filter=trust_filter,
   441ÔåÆ                                drug_filter=drug_filter,
   442ÔåÆ                                directory_filter=directory_filter,
   443ÔåÆ                                minimum_patients=minimum_patients,
   444ÔåÆ                                paths=paths,
   445ÔåÆ                            )
   446ÔåÆ
   447ÔåÆ                            if ice_df is None:
   448ÔåÆ                                logger.warning(f"No indication pathway data for {config.id}")
   449ÔåÆ                                results[f"{config.id}:indication"] = []
   450ÔåÆ                                continue
   451ÔåÆ
   452ÔåÆ                            # Extract denormalized fields (using indication variant)
   453ÔåÆ                            ice_df = extract_indication_fields(ice_df)
   454ÔåÆ
   455ÔåÆ                            # Convert to records with chart_type="indication"
   456ÔåÆ                            records = convert_to_records(ice_df, config.id, refresh_id, chart_type="indication")
   457ÔåÆ                            results[f"{config.id}:indication"] = records
   458ÔåÆ
   459ÔåÆ                            logger.info(f"Completed {config.id}:indication: {len(records)} nodes")
   460ÔåÆ
   461ÔåÆ                except Exception as e:
   462ÔåÆ                    logger.error(f"Error processing indication charts: {e}")
   463ÔåÆ                    logger.exception(e)
   464ÔåÆ                    for config in DATE_FILTER_CONFIGS:
   465ÔåÆ                        results[f"{config.id}:indication"] = []
   466ÔåÆ
   467ÔåÆ        # Count records per filter and chart type
   468ÔåÆ        stats["chart_type_counts"] = {}
   469ÔåÆ        for key, records in results.items():
   470ÔåÆ            stats["date_filter_counts"][key] = len(records)
   471ÔåÆ            stats["total_records"] += len(records)
   472ÔåÆ            # Also track by chart type
   473ÔåÆ            _, ct = key.split(":")
   474ÔåÆ            stats["chart_type_counts"][ct] = stats["chart_type_counts"].get(ct, 0) + len(records)
   475ÔåÆ
   476ÔåÆ        logger.info("")
   477ÔåÆ        logger.info(f"Processed {stats['total_records']} total pathway nodes")
   478ÔåÆ        for chart_type_name, count in stats.get("chart_type_counts", {}).items():
   479ÔåÆ            logger.info(f"  {chart_type_name}: {count} nodes total")
   480ÔåÆ        for key, count in sorted(stats["date_filter_counts"].items()):
   481ÔåÆ            if count > 0:
   482ÔåÆ                logger.info(f"    {key}: {count} nodes")
   483ÔåÆ
   484ÔåÆ        if dry_run:
   485ÔåÆ            logger.info("")
   486ÔåÆ            logger.info("DRY RUN - Skipping database insertion")
   487ÔåÆ            elapsed = time.time() - start_time
   488ÔåÆ            return True, f"Dry run complete: {stats['total_records']} records would be inserted", stats
   489ÔåÆ
   490ÔåÆ        # Step 3: Clear existing data and insert new records
   491ÔåÆ        logger.info("")
   492ÔåÆ        logger.info("Step 3/4: Clearing existing pathway data and inserting new records...")
   493ÔåÆ
   494ÔåÆ        with db_manager.get_transaction() as conn:
   495ÔåÆ            # Clear all existing pathway nodes
   496ÔåÆ            deleted = clear_pathway_nodes(conn)
   497ÔåÆ            logger.info(f"Cleared {deleted} existing pathway nodes")
   498ÔåÆ
   499ÔåÆ            # Insert new records for each date filter + chart type combination
   500ÔåÆ            total_inserted = 0
   501ÔåÆ            for key, records in results.items():
   502ÔåÆ                if records:
   503ÔåÆ                    inserted = insert_pathway_records(conn, records)
   504ÔåÆ                    total_inserted += len(records)
   505ÔåÆ                    logger.info(f"  Inserted {len(records)} records for {key}")
   506ÔåÆ
   507ÔåÆ        # Step 4: Log completion
   508ÔåÆ        logger.info("")
   509ÔåÆ        logger.info("Step 4/4: Logging refresh completion...")
   510ÔåÆ
   511ÔåÆ        elapsed = time.time() - start_time
   512ÔåÆ
   513ÔåÆ        with db_manager.get_connection() as conn:
   514ÔåÆ            log_refresh_complete(
   515ÔåÆ                conn=conn,
   516ÔåÆ                refresh_id=refresh_id,
   517ÔåÆ                record_count=stats["total_records"],
   518ÔåÆ                date_filter_counts=stats["date_filter_counts"],
   519ÔåÆ                duration_seconds=elapsed,
   520ÔåÆ            )
   521ÔåÆ
   522ÔåÆ            # Verify final counts
   523ÔåÆ            counts = get_pathway_table_counts(conn)
   524ÔåÆ            logger.info(f"Final table counts: {counts}")
   525ÔåÆ
   526ÔåÆ        logger.info("")
   527ÔåÆ        logger.info("=" * 60)
   528ÔåÆ        logger.info(f"Refresh completed successfully in {elapsed:.1f} seconds")
   529ÔåÆ        logger.info(f"Total records: {stats['total_records']}")
   530ÔåÆ        logger.info(f"Refresh ID: {refresh_id}")
   531ÔåÆ        logger.info("=" * 60)
   532ÔåÆ
   533ÔåÆ        return True, f"Refresh complete: {stats['total_records']} records in {elapsed:.1f}s", stats
   534ÔåÆ
   535ÔåÆ    except Exception as e:
   536ÔåÆ        elapsed = time.time() - start_time
   537ÔåÆ        error_msg = f"Refresh failed: {e}"
   538ÔåÆ        logger.error(error_msg, exc_info=True)
   539ÔåÆ
   540ÔåÆ        try:
   541ÔåÆ            with db_manager.get_connection() as conn:
   542ÔåÆ                log_refresh_failed(conn, refresh_id, str(e), elapsed)
   543ÔåÆ        except Exception:
   544ÔåÆ            pass  # Don't fail the error handling
   545ÔåÆ
   546ÔåÆ        return False, error_msg, stats
   547ÔåÆ
   548ÔåÆ
   549ÔåÆdef main() -> int:
   550ÔåÆ    """CLI entry point."""
   551ÔåÆ    parser = argparse.ArgumentParser(
   552ÔåÆ        description="Refresh pathway data from Snowflake",
   553ÔåÆ        formatter_class=argparse.RawDescriptionHelpFormatter,
   554ÔåÆ        epilog="""
   555ÔåÆExamples:
   556ÔåÆ    # Basic refresh with defaults (directory chart only)
   557ÔåÆ    python -m cli.refresh_pathways
   558ÔåÆ
   559ÔåÆ    # Refresh both chart types (directory and indication)
   560ÔåÆ    python -m cli.refresh_pathways --chart-type all
   561ÔåÆ
   562ÔåÆ    # Refresh only indication-based charts
   563ÔåÆ    python -m cli.refresh_pathways --chart-type indication
   564ÔåÆ
   565ÔåÆ    # Refresh with custom minimum patients
   566ÔåÆ    python -m cli.refresh_pathways --minimum-patients 10
   567ÔåÆ
   568ÔåÆ    # Refresh specific providers only
   569ÔåÆ    python -m cli.refresh_pathways --provider-codes RGT,RM1
   570ÔåÆ
   571ÔåÆ    # Dry run to see what would be processed
   572ÔåÆ    python -m cli.refresh_pathways --dry-run
   573ÔåÆ
   574ÔåÆ    # Verbose output
   575ÔåÆ    python -m cli.refresh_pathways --verbose
   576ÔåÆ        """
   577ÔåÆ    )
   578ÔåÆ
   579ÔåÆ    parser.add_argument(
   580ÔåÆ        "--minimum-patients",
   581ÔåÆ        type=int,
   582ÔåÆ        default=5,
   583ÔåÆ        help="Minimum patients to include a pathway (default: 5)"
   584ÔåÆ    )
   585ÔåÆ
   586ÔåÆ    parser.add_argument(
   587ÔåÆ        "--provider-codes",
   588ÔåÆ        type=str,
   589ÔåÆ        default=None,
   590ÔåÆ        help="Comma-separated list of provider codes to filter (default: all)"
   591ÔåÆ    )
   592ÔåÆ
   593ÔåÆ    parser.add_argument(
   594ÔåÆ        "--db-path",
   595ÔåÆ        type=str,
   596ÔåÆ        default=None,
   597ÔåÆ        help="Path to SQLite database (default: data/pathways.db)"
   598ÔåÆ    )
   599ÔåÆ
   600ÔåÆ    parser.add_argument(
   601ÔåÆ        "--dry-run",
   602ÔåÆ        action="store_true",
   603ÔåÆ        help="Process data but don't insert into database"
   604ÔåÆ    )
   605ÔåÆ
   606ÔåÆ    parser.add_argument(
   607ÔåÆ        "--chart-type",
   608ÔåÆ        type=str,
   609ÔåÆ        choices=["directory", "indication", "all"],
   610ÔåÆ        default="directory",
   611ÔåÆ        help="Chart type to process: 'directory' (default), 'indication', or 'all'"
   612ÔåÆ    )
   613ÔåÆ
   614ÔåÆ    parser.add_argument(
   615ÔåÆ        "--verbose", "-v",
   616ÔåÆ        action="store_true",
   617ÔåÆ        help="Enable verbose logging"
   618ÔåÆ    )
   619ÔåÆ
   620ÔåÆ    args = parser.parse_args()
   621ÔåÆ
   622ÔåÆ    # Configure logging
   623ÔåÆ    import logging
   624ÔåÆ    log_level = logging.DEBUG if args.verbose else logging.INFO
   625ÔåÆ    setup_logging(level=log_level)
   626ÔåÆ
   627ÔåÆ    # Parse provider codes
   628ÔåÆ    provider_codes = None
   629ÔåÆ    if args.provider_codes:
   630ÔåÆ        provider_codes = [code.strip() for code in args.provider_codes.split(",")]
   631ÔåÆ
   632ÔåÆ    # Parse db path
   633ÔåÆ    db_path = Path(args.db_path) if args.db_path else None
   634ÔåÆ
   635ÔåÆ    # Run the refresh
   636ÔåÆ    success, message, stats = refresh_pathways(
   637ÔåÆ        minimum_patients=args.minimum_patients,
   638ÔåÆ        provider_codes=provider_codes,
   639ÔåÆ        db_path=db_path,
   640ÔåÆ        dry_run=args.dry_run,
   641ÔåÆ        chart_type=args.chart_type,
   642ÔåÆ    )
   643ÔåÆ
   644ÔåÆ    if success:
   645ÔåÆ        print(f"\n[OK] {message}")
   646ÔåÆ        return 0
   647ÔåÆ    else:
   648ÔåÆ        print(f"\n[FAILED] {message}", file=sys.stderr)
   649ÔåÆ        return 1
   650ÔåÆ
   651ÔåÆ
   652ÔåÆif __name__ == "__main__":
   653ÔåÆ    sys.exit(main())
   654ÔåÆ

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

"""
CLI command for refreshing pathway data from Snowflake.

This command fetches activity data from Snowflake, processes it through the
pathway pipeline for all 6 date filter combinations, and stores the results
in the SQLite pathway_nodes table. Supports two chart types:
- "directory": Trust ÔåÆ Directory ÔåÆ Drug ÔåÆ Pathway (default)
- "indication": Trust ÔåÆ Search_Term ÔåÆ Drug ÔåÆ Pathway (requires GP diagnosis lookup)

Usage:
    python -m cli.refresh_pathways
    python -m cli.refresh_pathways --minimum-patients 10
    python -m cli.refresh_pathways --provider-codes RGT,RM1
    python -m cli.refresh_pathways --chart-type all
    python -m cli.refresh_pathways --chart-type directory
    python -m cli.refresh_pathways --dry-run

Run `python -m cli.refresh_pathways --help` for full options.
"""

import argparse
import json
import sqlite3
import sys
import time
import uuid
from datetime import datetime
from pathlib import Path
from typing import Optional

from core import PathConfig, default_paths
from core.logging_config import get_logger, setup_logging
from data_processing.database import DatabaseManager, DatabaseConfig
from data_processing.schema import (
    clear_pathway_nodes,
    get_pathway_table_counts,
    verify_pathway_tables_exist,
    create_pathway_tables,
)
from data_processing.pathway_pipeline import (
    ChartType,
    DATE_FILTER_CONFIGS,
    fetch_and_transform_data,
    process_all_date_filters,
    process_pathway_for_date_filter,
    process_indication_pathway_for_date_filter,
    extract_denormalized_fields,
    extract_indication_fields,
    convert_to_records,
)
from data_processing.diagnosis_lookup import (
    assign_drug_indications,
    get_patient_indication_groups,
    load_drug_indication_mapping,
)

logger = get_logger(__name__)

def get_default_filters(paths: PathConfig) -> tuple[list[str], list[str], list[str]]:
    """
    Load default filter values from reference files.

    Returns:
        Tuple of (trust_filter, drug_filter, directory_filter)
    """
    import pandas as pd

    # Load default trusts
    trust_filter = []
    if paths.default_trusts_csv.exists():
        try:
            trusts_df = pd.read_csv(paths.default_trusts_csv)
            # Use the "Name" column which contains trust names
            if 'Name' in trusts_df.columns:
                trust_filter = trusts_df['Name'].dropna().tolist()
            else:
                # Fallback to first column if no Name column
                trust_filter = trusts_df.iloc[:, 0].dropna().tolist()
            logger.info(f"Loaded {len(trust_filter)} default trusts")
        except Exception as e:
            logger.warning(f"Could not load default trusts: {e}")

    # Load default drugs (Include=1 in include.csv)
    drug_filter = []
    if paths.include_csv.exists():
        try:
            drugs_df = pd.read_csv(paths.include_csv)
            if 'Include' in drugs_df.columns:
                drug_filter = drugs_df[drugs_df['Include'] == 1].iloc[:, 0].dropna().tolist()
            else:
                # Assume first column contains drug names if no Include column
                drug_filter = drugs_df.iloc[:, 0].dropna().tolist()
            logger.info(f"Loaded {len(drug_filter)} default drugs")
        except Exception as e:
            logger.warning(f"Could not load default drugs: {e}")

    # Load default directories
    directory_filter = []
    if paths.directory_list_csv.exists():
        try:
            dirs_df = pd.read_csv(paths.directory_list_csv)
            # Assume first column contains directory names
            directory_filter = dirs_df.iloc[:, 0].dropna().tolist()
            logger.info(f"Loaded {len(directory_filter)} default directories")
        except Exception as e:
            logger.warning(f"Could not load default directories: {e}")

    return trust_filter, drug_filter, directory_filter

def insert_pathway_records(
    conn: sqlite3.Connection,
    records: list[dict],
) -> int:
    """
    Insert pathway records into pathway_nodes table.

    Uses INSERT OR REPLACE to handle updates to existing records.

    Args:
        conn: SQLite connection
        records: List of record dicts from convert_to_records()

    Returns:
        Number of records inserted
    """
    if not records:
        return 0

    # Column order matching pathway_nodes schema (includes chart_type)
    columns = [
        'date_filter_id', 'chart_type', 'parents', 'ids', 'labels', 'level',
        'value', 'cost', 'costpp', 'cost_pp_pa', 'colour',
        'first_seen', 'last_seen', 'first_seen_parent', 'last_seen_parent',
        'average_spacing', 'average_administered', 'avg_days',
        'trust_name', 'directory', 'drug_sequence', 'data_refresh_id'
    ]

    placeholders = ', '.join(['?' for _ in columns])
    column_names = ', '.join(columns)

    insert_sql = f"""
        INSERT OR REPLACE INTO pathway_nodes ({column_names})
        VALUES ({placeholders})
    """

    # Convert records to tuples in column order
    rows = []
    for record in records:
        row = tuple(record.get(col) for col in columns)
        rows.append(row)

    cursor = conn.executemany(insert_sql, rows)
    return cursor.rowcount

def log_refresh_start(
    conn: sqlite3.Connection,
    refresh_id: str,
    date_from: Optional[str] = None,
    date_to: Optional[str] = None,
) -> None:
    """Log the start of a refresh operation."""
    conn.execute("""
        INSERT INTO pathway_refresh_log
        (refresh_id, started_at, status, snowflake_query_date_from, snowflake_query_date_to)
        VALUES (?, ?, 'running', ?, ?)
    """, (refresh_id, datetime.now().isoformat(), date_from, date_to))
    conn.commit()

def log_refresh_complete(
    conn: sqlite3.Connection,
    refresh_id: str,
    record_count: int,
    date_filter_counts: dict[str, int],
    duration_seconds: float,
) -> None:
    """Log the successful completion of a refresh operation."""
    conn.execute("""
        UPDATE pathway_refresh_log
        SET completed_at = ?,
            status = 'completed',
            record_count = ?,
            date_filter_counts = ?,
            processing_duration_seconds = ?
        WHERE refresh_id = ?
    """, (
        datetime.now().isoformat(),
        record_count,
        json.dumps(date_filter_counts),
        duration_seconds,
        refresh_id,
    ))
    conn.commit()

def log_refresh_failed(
    conn: sqlite3.Connection,
    refresh_id: str,
    error_message: str,
    duration_seconds: float,
) -> None:
    """Log a failed refresh operation."""
    conn.execute("""
        UPDATE pathway_refresh_log
        SET completed_at = ?,
            status = 'failed',
            error_message = ?,
            processing_duration_seconds = ?
        WHERE refresh_id = ?
    """, (
        datetime.now().isoformat(),
        error_message,
        duration_seconds,
        refresh_id,
    ))
    conn.commit()

def refresh_pathways(
    minimum_patients: int = 5,
    provider_codes: Optional[list[str]] = None,
    trust_filter: Optional[list[str]] = None,
    drug_filter: Optional[list[str]] = None,
    directory_filter: Optional[list[str]] = None,
    db_path: Optional[Path] = None,
    paths: Optional[PathConfig] = None,
    dry_run: bool = False,
    chart_type: str = "directory",
) -> tuple[bool, str, dict]:
    """
    Main refresh function that orchestrates the full pipeline.

    Args:
        minimum_patients: Minimum patients to include a pathway
        provider_codes: List of provider codes to filter Snowflake query
        trust_filter: List of trust names to include in pathways
        drug_filter: List of drug names to include in pathways
        directory_filter: List of directories to include in pathways
        db_path: Path to SQLite database (uses default if None)
        paths: PathConfig for file paths
        dry_run: If True, don't actually insert records
        chart_type: Which chart type to process: "directory", "indication", or "all"

    Returns:
        Tuple of (success: bool, message: str, stats: dict)
    """
    if paths is None:
        paths = default_paths

    # Set up database connection
    if db_path:
        db_config = DatabaseConfig(db_path=db_path)
    else:
        db_config = DatabaseConfig(data_dir=paths.data_dir)

    db_manager = DatabaseManager(db_config)

    # Load default filters if not provided
    default_trusts, default_drugs, default_dirs = get_default_filters(paths)

    if trust_filter is None:
        trust_filter = default_trusts
    if drug_filter is None:
        drug_filter = default_drugs
    if directory_filter is None:
        directory_filter = default_dirs

    # Ensure we have some filters
    if not drug_filter:
        return False, "No drugs specified and could not load defaults", {}

    # Determine which chart types to process
    if chart_type == "all":
        chart_types_to_process: list[ChartType] = ["directory", "indication"]
    else:
        chart_types_to_process = [chart_type]  # type: ignore

    logger.info("=" * 60)
    logger.info("Pathway Data Refresh Starting")
    logger.info("=" * 60)
    logger.info(f"Minimum patients: {minimum_patients}")
    logger.info(f"Trust filter: {len(trust_filter)} trusts")
    logger.info(f"Drug filter: {len(drug_filter)} drugs")
    logger.info(f"Directory filter: {len(directory_filter)} directories")
    logger.info(f"Provider codes: {provider_codes or 'All'}")
    logger.info(f"Chart type(s): {', '.join(chart_types_to_process)}")
    logger.info(f"Database: {db_manager.db_path}")
    logger.info(f"Dry run: {dry_run}")
    logger.info("=" * 60)

    start_time = time.time()
    refresh_id = str(uuid.uuid4())[:8]
    stats = {
        "refresh_id": refresh_id,
        "date_filter_counts": {},
        "total_records": 0,
        "snowflake_rows": 0,
    }

    try:
        # Verify database and tables
        with db_manager.get_connection() as conn:
            missing_tables = verify_pathway_tables_exist(conn)
            if missing_tables:
                logger.info(f"Creating missing tables: {missing_tables}")
                create_pathway_tables(conn)

            # Log refresh start
            if not dry_run:
                log_refresh_start(conn, refresh_id)

        # Step 1: Fetch data from Snowflake
        logger.info("")
        logger.info("Step 1/4: Fetching data from Snowflake...")
        df = fetch_and_transform_data(
            provider_codes=provider_codes,
            paths=paths,
        )

        if df.empty:
            msg = "No data returned from Snowflake"
            logger.error(msg)
            with db_manager.get_connection() as conn:
                log_refresh_failed(conn, refresh_id, msg, time.time() - start_time)
            return False, msg, stats

        stats["snowflake_rows"] = len(df)
        logger.info(f"Fetched {len(df)} records from Snowflake")

        # Step 2: Process all date filters for each chart type
        num_date_filters = len(DATE_FILTER_CONFIGS)
        num_chart_types = len(chart_types_to_process)
        total_datasets = num_date_filters * num_chart_types

        logger.info("")
        logger.info(f"Step 2/4: Processing pathway data for {total_datasets} datasets "
                   f"({num_date_filters} date filters x {num_chart_types} chart types)...")

        # Store results keyed by "date_filter_id:chart_type"
        results: dict[str, list[dict]] = {}

        for current_chart_type in chart_types_to_process:
            logger.info("")
            logger.info(f"Processing chart type: {current_chart_type}")

            if current_chart_type == "directory":
                # Use existing process_all_date_filters for directory charts
                dir_results = process_all_date_filters(
                    df=df,
                    trust_filter=trust_filter,
                    drug_filter=drug_filter,
                    directory_filter=directory_filter,
                    minimum_patients=minimum_patients,
                    refresh_id=refresh_id,
                    paths=paths,
                )
                # Add results with chart_type suffix
                for filter_id, records in dir_results.items():
                    # Records already have chart_type set by convert_to_records
                    results[f"{filter_id}:directory"] = records

            elif current_chart_type == "indication":
                # For indication charts, use drug-aware matching:
                # 1. Get ALL GP diagnosis matches per patient (with code_frequency)
                # 2. Cross-reference with drug-to-Search_Term mapping from DimSearchTerm.csv
                # 3. Assign each drug to its matched indication via modified UPIDs
                logger.info("Building drug-aware indication groups...")

                # Check Snowflake availability
                from data_processing.snowflake_connector import get_connector, is_snowflake_available

                if not is_snowflake_available():
                    logger.warning("Snowflake not available - cannot process indication charts")
                    for config in DATE_FILTER_CONFIGS:
                        results[f"{config.id}:indication"] = []
                    continue

                try:
                    import pandas as pd
                    connector = get_connector()

                    if 'PseudoNHSNoLinked' not in df.columns:
                        logger.error("DataFrame missing 'PseudoNHSNoLinked' column - cannot lookup GP records")
                        for config in DATE_FILTER_CONFIGS:
                            results[f"{config.id}:indication"] = []
                        continue

                    # Step 1: Load drug-to-Search_Term mapping from DimSearchTerm.csv
                    _, search_term_to_fragments = load_drug_indication_mapping()
                    logger.info(f"Loaded drug mapping: {len(search_term_to_fragments)} Search_Terms")

                    # Step 2: Get ALL GP diagnosis matches per patient (with code_frequency)
                    patient_pseudonyms = df['PseudoNHSNoLinked'].dropna().unique().tolist()
                    logger.info(f"Looking up GP diagnoses for {len(patient_pseudonyms)} unique patients...")

                    # Restrict GP codes to HCD data window (reduces noise from old diagnoses)
                    earliest_hcd_date = df['Intervention Date'].min()
                    if pd.notna(earliest_hcd_date):
                        earliest_hcd_date_str = pd.Timestamp(earliest_hcd_date).strftime('%Y-%m-%d')
                        logger.info(f"Restricting GP codes to HCD window: >= {earliest_hcd_date_str}")
                    else:
                        earliest_hcd_date_str = None

                    gp_matches_df = get_patient_indication_groups(
                        patient_pseudonyms=patient_pseudonyms,
                        connector=connector,
                        batch_size=500,
                        earliest_hcd_date=earliest_hcd_date_str,
                    )

                    # Step 3: Assign drug-aware indications using cross-referencing
                    # This replaces the old per-patient approach with per-drug matching
                    modified_df, indication_df = assign_drug_indications(
                        df=df,
                        gp_matches_df=gp_matches_df,
                        search_term_to_fragments=search_term_to_fragments,
                    )

                    logger.info(f"Drug-aware indication matching complete. "
                               f"Modified UPIDs: {modified_df['UPID'].nunique()}, "
                               f"Indication groups: {len(indication_df)}")

                    if indication_df.empty:
                        logger.warning("Empty indication_df - skipping indication charts")
                        for config in DATE_FILTER_CONFIGS:
                            results[f"{config.id}:indication"] = []
                    else:
                        # Process each date filter with drug-aware indication grouping
                        # Use modified_df (with indication-aware UPIDs) instead of original df
                        for config in DATE_FILTER_CONFIGS:
                            logger.info(f"Processing indication pathway for {config.id}")

                            ice_df = process_indication_pathway_for_date_filter(
                                df=modified_df,
                                indication_df=indication_df,
                                config=config,
                                trust_filter=trust_filter,
                                drug_filter=drug_filter,
                                directory_filter=directory_filter,
                                minimum_patients=minimum_patients,
                                paths=paths,
                            )

                            if ice_df is None:
                                logger.warning(f"No indication pathway data for {config.id}")
                                results[f"{config.id}:indication"] = []
                                continue

                            # Extract denormalized fields (using indication variant)
                            ice_df = extract_indication_fields(ice_df)

                            # Convert to records with chart_type="indication"
                            records = convert_to_records(ice_df, config.id, refresh_id, chart_type="indication")
                            results[f"{config.id}:indication"] = records

                            logger.info(f"Completed {config.id}:indication: {len(records)} nodes")

                except Exception as e:
                    logger.error(f"Error processing indication charts: {e}")
                    logger.exception(e)
                    for config in DATE_FILTER_CONFIGS:
                        results[f"{config.id}:indication"] = []

        # Count records per filter and chart type
        stats["chart_type_counts"] = {}
        for key, records in results.items():
            stats["date_filter_counts"][key] = len(records)
            stats["total_records"] += len(records)
            # Also track by chart type
            _, ct = key.split(":")
            stats["chart_type_counts"][ct] = stats["chart_type_counts"].get(ct, 0) + len(records)

        logger.info("")
        logger.info(f"Processed {stats['total_records']} total pathway nodes")
        for chart_type_name, count in stats.get("chart_type_counts", {}).items():
            logger.info(f"  {chart_type_name}: {count} nodes total")
        for key, count in sorted(stats["date_filter_counts"].items()):
            if count > 0:
                logger.info(f"    {key}: {count} nodes")

        if dry_run:
            logger.info("")
            logger.info("DRY RUN - Skipping database insertion")
            elapsed = time.time() - start_time
            return True, f"Dry run complete: {stats['total_records']} records would be inserted", stats

        # Step 3: Clear existing data and insert new records
        logger.info("")
        logger.info("Step 3/4: Clearing existing pathway data and inserting new records...")

        with db_manager.get_transaction() as conn:
            # Clear all existing pathway nodes
            deleted = clear_pathway_nodes(conn)
            logger.info(f"Cleared {deleted} existing pathway nodes")

            # Insert new records for each date filter + chart type combination
            total_inserted = 0
            for key, records in results.items():
                if records:
                    inserted = insert_pathway_records(conn, records)
                    total_inserted += len(records)
                    logger.info(f"  Inserted {len(records)} records for {key}")

        # Step 4: Log completion
        logger.info("")
        logger.info("Step 4/4: Logging refresh completion...")

        elapsed = time.time() - start_time

        with db_manager.get_connection() as conn:
            log_refresh_complete(
                conn=conn,
                refresh_id=refresh_id,
                record_count=stats["total_records"],
                date_filter_counts=stats["date_filter_counts"],
                duration_seconds=elapsed,
            )

            # Verify final counts
            counts = get_pathway_table_counts(conn)
            logger.info(f"Final table counts: {counts}")

        logger.info("")
        logger.info("=" * 60)
        logger.info(f"Refresh completed successfully in {elapsed:.1f} seconds")
        logger.info(f"Total records: {stats['total_records']}")
        logger.info(f"Refresh ID: {refresh_id}")
        logger.info("=" * 60)

        return True, f"Refresh complete: {stats['total_records']} records in {elapsed:.1f}s", stats

    except Exception as e:
        elapsed = time.time() - start_time
        error_msg = f"Refresh failed: {e}"
        logger.error(error_msg, exc_info=True)

        try:
            with db_manager.get_connection() as conn:
                log_refresh_failed(conn, refresh_id, str(e), elapsed)
        except Exception:
            pass  # Don't fail the error handling

        return False, error_msg, stats

def main() -> int:
    """CLI entry point."""
    parser = argparse.ArgumentParser(
        description="Refresh pathway data from Snowflake",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
    # Basic refresh with defaults (directory chart only)
    python -m cli.refresh_pathways

    # Refresh both chart types (directory and indication)
    python -m cli.refresh_pathways --chart-type all

    # Refresh only indication-based charts
    python -m cli.refresh_pathways --chart-type indication

    # Refresh with custom minimum patients
    python -m cli.refresh_pathways --minimum-patients 10

    # Refresh specific providers only
    python -m cli.refresh_pathways --provider-codes RGT,RM1

    # Dry run to see what would be processed
    python -m cli.refresh_pathways --dry-run

    # Verbose output
    python -m cli.refresh_pathways --verbose
        """
    )

    parser.add_argument(
        "--minimum-patients",
        type=int,
        default=5,
        help="Minimum patients to include a pathway (default: 5)"
    )

    parser.add_argument(
        "--provider-codes",
        type=str,
        default=None,
        help="Comma-separated list of provider codes to filter (default: all)"
    )

    parser.add_argument(
        "--db-path",
        type=str,
        default=None,
        help="Path to SQLite database (default: data/pathways.db)"
    )

    parser.add_argument(
        "--dry-run",
        action="store_true",
        help="Process data but don't insert into database"
    )

    parser.add_argument(
        "--chart-type",
        type=str,
        choices=["directory", "indication", "all"],
        default="directory",
        help="Chart type to process: 'directory' (default), 'indication', or 'all'"
    )

    parser.add_argument(
        "--verbose", "-v",
        action="store_true",
        help="Enable verbose logging"
    )

    args = parser.parse_args()

    # Configure logging
    import logging
    log_level = logging.DEBUG if args.verbose else logging.INFO
    setup_logging(level=log_level)

    # Parse provider codes
    provider_codes = None
    if args.provider_codes:
        provider_codes = [code.strip() for code in args.provider_codes.split(",")]

    # Parse db path
    db_path = Path(args.db_path) if args.db_path else None

    # Run the refresh
    success, message, stats = refresh_pathways(
        minimum_patients=args.minimum_patients,
        provider_codes=provider_codes,
        db_path=db_path,
        dry_run=args.dry_run,
        chart_type=args.chart_type,
    )

    if success:
        print(f"\n[OK] {message}")
        return 0
    else:
        print(f"\n[FAILED] {message}", file=sys.stderr)
        return 1

if __name__ == "__main__":
    sys.exit(main())


Good, the pipeline code looks clean. Now let me mark Task 3.2 as in-progress and run the dry run test. This requires a live Snowflake connection. Let me first check if Snowflake is accessible.


Edit

### 3.2 Test with dry run
- [ ] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`

### 3.2 Test with dry run
- [~] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`


The file C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\IMPLEMENTATION_PLAN.md has been updated successfully.

### 3.2 Test with dry run
- [~] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`

# Implementation Plan - Drug-Aware Indication Matching

## Project Overview

Update the indication-based pathway charts so that patient indications are matched **per drug**, not just per patient. Currently, each patient gets ONE indication (most recent GP diagnosis match). This ignores which drugs the patient is actually taking.

### The Problem

A patient on ADALIMUMAB + OMALIZUMAB currently gets assigned a single indication (e.g., "rheumatoid arthritis" ÔÇö the most recent GP match). But:
- ADALIMUMAB is used for rheumatoid arthritis, axial spondyloarthritis, crohn's disease, etc.
- OMALIZUMAB is used for asthma, allergic asthma, urticaria

These are different clinical pathways and should be treated as separate treatment journeys.

### The Solution

Match each drug to an indication by cross-referencing:
1. **GP diagnosis** ÔÇö which Search_Terms the patient has matching SNOMED codes for
2. **Drug mapping** ÔÇö which Search_Terms list each drug (from `DimSearchTerm.csv`)

Only assign a drug to an indication if BOTH conditions are met. If a patient's drugs map to different indications, they become separate pathways (via modified UPID).

### Key Design Decisions

| Aspect | Decision |
|--------|----------|
| Drug-indication source | `data/DimSearchTerm.csv` ÔÇö Search_Term ÔåÆ CleanedDrugName mapping |
| UPID modification | `{original_UPID}\|{search_term}` for drugs with matched indication |
| GP diagnosis matching | Return ALL matches per patient (not just most recent) |
| Drug matching | Substring match: HCD drug name contains DimSearchTerm fragment |
| Multiple indication matches per drug | Use highest GP code frequency as tiebreaker (COUNT of matching SNOMED codes per Search_Term) |
| GP code time range | Only codes from MIN(Intervention Date) onwards ÔÇö restricts to HCD data window |
| No indication match | Fallback to directory (same as current behavior) |
| Same patient, different indications | Separate pathways via different modified UPIDs |

### Examples

**Patient on ADALIMUMAB + GOLIMUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis drug list includes both ADALIMUMAB and GOLIMUMAB
- ÔåÆ Both drugs grouped under "axial spondyloarthritis", single pathway
- Modified UPID: `RMV12345|axial spondyloarthritis`

**Patient on ADALIMUMAB + OMALIZUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis lists ADALIMUMAB but not OMALIZUMAB
- asthma lists OMALIZUMAB but not ADALIMUMAB
- ÔåÆ Two separate pathways:
  - `RMV12345|axial spondyloarthritis` with ADALIMUMAB
  - `RMV12345|asthma` with OMALIZUMAB

**Patient on ADALIMUMAB, GP dx: rheumatoid arthritis (47 codes) + crohn's disease (2 codes)**
- Both Search_Terms list ADALIMUMAB AND patient has GP dx for both
- ÔåÆ Tiebreaker: highest code frequency ÔÇö rheumatoid arthritis has 47 matching SNOMED codes vs 2 for crohn's
- ÔåÆ Single pathway under rheumatoid arthritis (more clinical activity = more likely the treatment indication)

---

## Phase 1: Update Snowflake Query & Drug Mapping

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [x] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [x] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [x] Keep batch processing (500 patients per query)
- [x] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses) *(requires live Snowflake ÔÇö will be verified in Phase 3/4)*

### 1.2 Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
- [x] In `CLUSTER_MAPPING_SQL` (diagnosis_lookup.py), merge these 3 Search_Terms into one `"asthma"` entry:
  - `allergic asthma` (Cluster: OMALIZUMAB only)
  - `asthma` (Cluster: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB)
  - `severe persistent allergic asthma` (Cluster: OMALIZUMAB only)
- [x] Map all 3 Cluster_IDs to `Search_Term = 'asthma'` in the CTE VALUES
- [x] `urticaria` (OMALIZUMAB, DERMATOLOGY) stays SEPARATE ÔÇö do NOT merge with asthma
- [x] Also update `load_drug_indication_mapping()` to apply the same merge when loading DimSearchTerm.csv:
  - Combine drug lists from all 3 entries under a single `"asthma"` key
  - Deduplicate drug fragments (OMALIZUMAB appears in all 3)
- [x] Verify: GP code lookup returns `"asthma"` (not `"allergic asthma"` or `"severe persistent allergic asthma"`)
- [x] Verify: Drug mapping for `"asthma"` includes full combined drug list: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB

### 1.3 Build drug-to-Search_Term lookup from DimSearchTerm.csv
- [x] Add function `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
  - Loads `data/DimSearchTerm.csv`
  - Builds dict: `drug_fragment (uppercase) ÔåÆ list[Search_Term]`
  - Also builds reverse: `search_term ÔåÆ list[drug_fragments]`
  - CleanedDrugName is pipe-separated (e.g., "ADALIMUMAB|GOLIMUMAB|IXEKIZUMAB")
- [x] Add function `get_search_terms_for_drug(drug_name, search_term_to_fragments) -> list[str]`:
  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
  - More practical than per-term boolean check ÔÇö returns all matches at once for Phase 2 use
- [x] Verify: ADALIMUMAB matches "axial spondyloarthritis", OMALIZUMAB matches "asthma"

---

## Phase 2: Drug-Aware Indication Matching Logic

### 2.1 Create `assign_drug_indications()` function
- [x] Add to `diagnosis_lookup.py` or `pathway_pipeline.py`:
  ```
  def assign_drug_indications(
      df: pd.DataFrame,              # HCD data with UPID, Drug Name columns
      gp_matches_df: pd.DataFrame,   # PatientPseudonym ÔåÆ list of matched Search_Terms
      drug_mapping: dict,             # From load_drug_indication_mapping()
  ) -> tuple[pd.DataFrame, pd.DataFrame]:
      Returns: (modified_df, indication_df)
      - modified_df: HCD data with UPID replaced by {UPID}|{indication}
      - indication_df: mapping modified_UPID ÔåÆ Search_Term
  ```
- [x] Logic per UPID + Drug Name pair:
  1. Get patient's GP-matched Search_Terms with code_frequency (from gp_matches_df via PseudoNHSNoLinked)
  2. Get which Search_Terms include this drug (from drug_mapping)
  3. Intersection = valid indications for this drug-patient pair
  4. If 1 match: use it
  5. If multiple matches: use highest code_frequency as tiebreaker (most GP coding activity = most likely treatment indication)
  6. If 0 matches: use fallback directory
- [x] Modify UPID in df rows: `{original_UPID}|{matched_search_term}`
- [x] Build indication_df: `{modified_UPID}` ÔåÆ `Search_Term` (or fallback label)
- [x] Verify: Function compiles, handles edge cases (no GP match, no drug match)

### 2.2 Handle tiebreaker for multiple indication matches
- [x] When a drug matches multiple Search_Terms AND patient has GP dx for multiple:
  - Use `code_frequency` from the GP query (COUNT of matching SNOMED codes per Search_Term)
  - Higher code_frequency = more clinical activity for that condition = more likely treatment indication
  - E.g., patient with 47 RA codes and 2 crohn's codes ÔåÆ ADALIMUMAB assigned to RA
  - code_frequency is already returned by the updated query in Task 1.1
- [x] Verify: Tiebreaker logic correctly picks highest-frequency diagnosis
- [x] Verify: Tie on frequency (rare but possible) falls back to alphabetical Search_Term for determinism

---

## Phase 3: Pipeline Integration

### 3.1 Update `refresh_pathways.py` indication processing
- [x] In the `elif current_chart_type == "indication":` block:
  1. Call `get_patient_indication_groups()` as before (but now returns ALL matches)
  2. Load drug mapping: `drug_mapping = load_drug_indication_mapping()`
  3. Call `assign_drug_indications(df, gp_matches_df, drug_mapping)`
  4. Use modified_df (with indication-aware UPIDs) for pathway processing
  5. Use indication_df for the indication mapping
- [x] Pass modified_df (not original df) to `process_indication_pathway_for_date_filter()`
- [x] Verify: Pipeline compiles, `python -m py_compile cli/refresh_pathways.py`

### 3.2 Test with dry run
- [ ] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`
- [ ] Verify:
  - Modified UPIDs appear in pipeline log (e.g., `RMV12345|rheumatoid arthritis`)
  - Patient counts are reasonable (will be higher than before since same patient can appear under multiple indications)
  - Drug-indication matching is logged (match rate, fallback rate)
  - Pathway hierarchy shows drug-specific grouping under correct indications

---

## Phase 4: Full Refresh & Validation

### 4.1 Full refresh with both chart types
- [ ] Run `python -m cli.refresh_pathways --chart-type all`
- [ ] Verify:
  - Both chart types generate data
  - Directory charts unchanged (no modified UPIDs)
  - Indication charts reflect drug-aware matching

### 4.2 Validate indication chart correctness
- [ ] Check that drugs under an indication all appear in that Search_Term's drug list
- [ ] Verify that a patient on drugs for different indications creates separate pathway branches
- [ ] Verify that drugs sharing an indication are grouped in the same pathway
- [ ] Log: patient count comparison (old vs new approach)

### 4.3 Validate Reflex UI
- [ ] Run `python -m reflex compile` to verify app compiles
- [ ] Verify chart type toggle still works
- [ ] Verify indication chart shows correct hierarchy

---

## Completion Criteria

All tasks marked `[x]` AND:
- [ ] App compiles without errors (`reflex compile` succeeds)
- [ ] Both chart types generate pathway data
- [ ] Indication charts show drug-specific indication matching
- [ ] Drugs under the same indication for the same patient are in one pathway
- [ ] Drugs under different indications for the same patient create separate pathways
- [ ] Fallback works for drugs with no indication match
- [ ] Full refresh completes successfully
- [ ] Existing directory charts are unaffected

---

## Reference

### DimSearchTerm.csv Structure
```
Search_Term,CleanedDrugName,PrimaryDirectorate
rheumatoid arthritis,ABATACEPT|ADALIMUMAB|ANAKINRA|BARICITINIB|...,RHEUMATOLOGY
asthma,BENRALIZUMAB|DUPILUMAB|INHALED|MEPOLIZUMAB|OMALIZUMAB|RESLIZUMAB,THORACIC MEDICINE
```

### Modified UPID Format
```
Original:  RMV12345
Modified:  RMV12345|rheumatoid arthritis
Fallback:  RMV12345|RHEUMATOLOGY (no GP dx)
```

### Current vs New Indication Flow
```
CURRENT:
  Patient ÔåÆ GP dx (most recent) ÔåÆ single Search_Term ÔåÆ one pathway

NEW:
  Patient + Drug A ÔåÆ GP dx matching Drug A ÔåÆ Search_Term X
  Patient + Drug B ÔåÆ GP dx matching Drug B ÔåÆ Search_Term Y
  ÔåÆ If X == Y: one pathway under X
  ÔåÆ If X != Y: two pathways (modified UPIDs)
```

### Key Files

| File | Changes |
|------|---------|
| `data_processing/diagnosis_lookup.py` | Update query, add drug mapping functions |
| `data_processing/pathway_pipeline.py` | Possibly minor changes for modified UPIDs |
| `cli/refresh_pathways.py` | Integrate drug-aware matching into pipeline |
| `data/DimSearchTerm.csv` | Reference data (read-only) |
| `analysis/pathway_analyzer.py` | No changes expected (UPID changes are transparent) |
| `pathways_app/pathways_app.py` | No changes expected |

-- [ ] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`

+- [~] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`


TodoWrite

Run dry run test: python -m cli.refresh_pathways --chart-type indication --dry-run -v

Running indication dry run test with Snowflake

Validate dry run output: modified UPIDs, match rates, pathway nodes

Mark Task 3.2 complete, commit, update progress.txt


Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable

Run dry run test: python -m cli.refresh_pathways --chart-type indication --dry-run -v

Running indication dry run test with Snowflake

Validate dry run output: modified UPIDs, match rates, pathway nodes

Mark Task 3.2 complete, commit, update progress.txt


Running the dry run test. This will fetch data from Snowflake and process indication charts without inserting to the database.


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && python -m cli.refresh_pathways --chart-type indication --dry-run -v 2>&1

Run indication chart dry run test with verbose output


Command running in background with ID: b361350. Output is being written to: C:\Users\CHARLW~1\AppData\Local\Temp\claude\C--Users-charlwoodand-Ralph-local-Tasks-UI-Redesign\tasks\b361350.output


The command is running in the background. Let me check the output.