     1ÔåÆ# Progress Log - Drug-Aware Indication Matching
     2ÔåÆ
     3ÔåÆ## Project Context
     4ÔåÆ
     5ÔåÆThis project extends the indication-based pathway charts (Phase 1-5 complete) with drug-aware matching.
     6ÔåÆ
     7ÔåÆ**Previous state**: Patients get ONE indication based on their most recent GP diagnosis match (SNOMED cluster codes). This ignores which drugs the patient is taking.
     8ÔåÆ
     9ÔåÆ**New goal**: Match each drug to an indication by cross-referencing the patient's GP diagnoses AND the drug's Search_Term mapping from DimSearchTerm.csv.
    10ÔåÆ
    11ÔåÆ## Key Data/Patterns
    12ÔåÆ
    13ÔåÆ### DimSearchTerm.csv
    14ÔåÆ- Located at `data/DimSearchTerm.csv`
    15ÔåÆ- Columns: Search_Term, CleanedDrugName (pipe-separated), PrimaryDirectorate
    16ÔåÆ- ~165 rows mapping clinical conditions to drug name fragments
    17ÔåÆ- Drug fragments are substrings that match standardized drug names from HCD data
    18ÔåÆ- Some entries have generic fragments: INHALED, CONTINUOUS, STANDARD-DOSE, PEGYLATED
    19ÔåÆ
    20ÔåÆ### Current get_patient_indication_groups() in diagnosis_lookup.py
    21ÔåÆ- Uses CLUSTER_MAPPING_SQL as CTE in Snowflake query
    22ÔåÆ- Returns ONLY the most recent match per patient (QUALIFY ROW_NUMBER() = 1)
    23ÔåÆ- Needs to return ALL matching Search_Terms per patient (remove QUALIFY)
    24ÔåÆ- Batches 500 patients per query
    25ÔåÆ
    26ÔåÆ### Modified UPID approach
    27ÔåÆ- Current: UPID = Provider Code[:3] + PersonKey (e.g., "RMV12345")
    28ÔåÆ- New: UPID = original + "|" + search_term (e.g., "RMV12345|rheumatoid arthritis")
    29ÔåÆ- The pipe delimiter "|" is safe because existing UPIDs are alphanumeric
    30ÔåÆ- generate_icicle_chart_indication() treats UPID as an opaque identifier ÔÇö modified UPIDs work transparently
    31ÔåÆ- The " - " delimiter in pathway ids is used for hierarchy levels, not within UPIDs
    32ÔåÆ
    33ÔåÆ### PseudoNHSNoLinked mapping
    34ÔåÆ- HCD data has PseudoNHSNoLinked column that matches PatientPseudonym in GP records
    35ÔåÆ- PersonKey is provider-specific local ID ÔÇö do NOT use for GP matching
    36ÔåÆ- One PseudoNHSNoLinked can map to multiple UPIDs (multi-provider patients)
    37ÔåÆ- GP match lookup: PseudoNHSNoLinked ÔåÆ list of matched Search_Terms
    38ÔåÆ
    39ÔåÆ### Drug matching logic
    40ÔåÆ- For each HCD row (UPID + Drug Name):
    41ÔåÆ  1. Get patient's GP-matched Search_Terms with code_frequency (via PseudoNHSNoLinked)
    42ÔåÆ  2. Get which Search_Terms list this drug (from DimSearchTerm.csv)
    43ÔåÆ  3. Intersection = valid indications
    44ÔåÆ  4. If 1: use it. If multiple: pick highest code_frequency (most GP coding = most likely indication). If 0: fallback to directory.
    45ÔåÆ- Modified UPID groups drugs under same indication together naturally
    46ÔåÆ- code_frequency = COUNT(*) of matching SNOMED codes per Search_Term per patient in GP records
    47ÔåÆ- GP code time range: only count codes from MIN(Intervention Date) onwards (the HCD data window)
    48ÔåÆ  - Reduces noise from old/irrelevant diagnoses, makes frequency more meaningful
    49ÔåÆ  - Pass earliest_hcd_date as parameter to get_patient_indication_groups()
    50ÔåÆ- Tiebreaker rationale: 47 RA codes vs 2 crohn's codes ÔåÆ RA is clearly the active condition
    51ÔåÆ
    52ÔåÆ### Known edge cases
    53ÔåÆ- Some DimSearchTerm drug fragments are generic (INHALED, ORAL, CONTINUOUS)
    54ÔåÆ  - These could match broadly but are constrained by GP diagnosis requirement
    55ÔåÆ- A patient visiting multiple providers has multiple UPIDs
    56ÔåÆ  - Each UPID gets its own drug-indication matching independently
    57ÔåÆ- Same Search_Term appears twice in DimSearchTerm.csv with different directorates
    58ÔåÆ  - e.g., "diabetes" ÔåÆ DIABETIC MEDICINE and OPHTHALMOLOGY
    59ÔåÆ  - For indication charts, we use Search_Term not directorate, so this is fine
    60ÔåÆ
    61ÔåÆ## Iteration Log
    62ÔåÆ
    63ÔåÆ## Iteration 1 ÔÇö 2026-02-05
    64ÔåÆ### Task: 1.3 ÔÇö Build drug-to-Search_Term lookup from DimSearchTerm.csv
    65ÔåÆ### Why this task:
    66ÔåÆ- First iteration, chose Phase 1 foundations. Task 1.2 (CSV loading) is self-contained and testable locally without Snowflake.
    67ÔåÆ- Task 1.1 (Snowflake query update) can't be verified without a live connection ÔÇö better to do 1.2 first.
    68ÔåÆ- Both 1.1 and 1.2 are independent, so order doesn't matter for dependencies.
    69ÔåÆ### Status: COMPLETE
    70ÔåÆ### What was done:
    71ÔåÆ- Added `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
    72ÔåÆ  - Loads `data/DimSearchTerm.csv`, builds two dicts:
    73ÔåÆ    - `fragment_to_search_terms`: drug fragment (UPPER) ÔåÆ list of Search_Terms
    74ÔåÆ    - `search_term_to_fragments`: search_term ÔåÆ list of drug fragments (UPPER)
    75ÔåÆ  - Handles duplicate Search_Terms (e.g., "diabetes" rows combined)
    76ÔåÆ  - Result: 164 Search_Terms, 346 drug fragments
    77ÔåÆ- Added `get_search_terms_for_drug()` to `diagnosis_lookup.py`:
    78ÔåÆ  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
    79ÔåÆ  - Named differently from plan's `drug_matches_search_term()` ÔÇö returns all matches at once rather than single boolean, more practical for Phase 2
    80ÔåÆ- Updated `__all__` exports
    81ÔåÆ### Validation results:
    82ÔåÆ- Tier 1 (Code): py_compile passed, import check passed
    83ÔåÆ- Tier 2 (Data): ADALIMUMAB ÔåÆ 7 indications (including axial spondyloarthritis, rheumatoid arthritis), OMALIZUMAB ÔåÆ 4 indications (asthma, allergic asthma, etc.), PEGYLATED LIPOSOMAL DOXORUBICIN ÔåÆ 4 matches via substring, "ADALIMUMAB 40MG" matches correctly with dosage info, diabetes fragments combined from 2 CSV rows
    84ÔåÆ- Tier 3 (Functional): N/A (no UI changes)
    85ÔåÆ### Files changed:
    86ÔåÆ- data_processing/diagnosis_lookup.py (added load_drug_indication_mapping, get_search_terms_for_drug)
    87ÔåÆ- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
    88ÔåÆ### Committed: 0779df7 "feat: add drug-to-indication mapping from DimSearchTerm.csv (Task 1.3)"
    89ÔåÆ### Patterns discovered:
    90ÔåÆ- DimSearchTerm.csv has 164 unique Search_Terms (not 165 as noted) because diabetes appears twice with different directorates but same Search_Term
    91ÔåÆ- Some drug fragments are very generic: INHALED, CONTINUOUS, ORAL, STANDARD-DOSE, INTRAVENOUS, PEGYLATED, ROUTINE, INDUCTION ÔÇö these will match broadly but are constrained by the GP diagnosis requirement in Phase 2
    92ÔåÆ- Function signatures for Phase 2: `get_search_terms_for_drug(drug_name, search_term_to_fragments)` returns list[str] ÔÇö use this to get candidate indications per drug
    93ÔåÆ### Next iteration should:
    94ÔåÆ- Work on Task 1.2: Merge asthma Search_Terms in CLUSTER_MAPPING_SQL and load_drug_indication_mapping()
    95ÔåÆ  - Merge "allergic asthma", "asthma", "severe persistent allergic asthma" ÔåÆ "asthma"
    96ÔåÆ  - Keep "urticaria" separate
    97ÔåÆ  - This is self-contained and testable locally
    98ÔåÆ- OR work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
    99ÔåÆ  - The current query at line ~1352 of diagnosis_lookup.py uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1` ÔÇö this must be replaced with GROUP BY + COUNT(*)
   100ÔåÆ  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
   101ÔåÆ  - Return columns: PatientPseudonym, Search_Term, code_frequency (not EventDateTime)
   102ÔåÆ- OR if Snowflake isn't available to test 1.1, skip to Task 2.1 (assign_drug_indications function) which can be built and tested with mock data
   103ÔåÆ### Blocked items:
   104ÔåÆ- None
   105ÔåÆ
   106ÔåÆ## Iteration 2 ÔÇö 2026-02-05
   107ÔåÆ### Task: 1.2 ÔÇö Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
   108ÔåÆ### Why this task:
   109ÔåÆ- Previous iteration recommended this as the next task (self-contained, testable locally)
   110ÔåÆ- Both CLUSTER_MAPPING_SQL and load_drug_indication_mapping() need consistent Search_Term names
   111ÔåÆ- Must be done before Task 1.1 (Snowflake query) to ensure GP lookups return "asthma" not "allergic asthma"
   112ÔåÆ### Status: COMPLETE
   113ÔåÆ### What was done:
   114ÔåÆ- Updated CLUSTER_MAPPING_SQL: changed 'allergic asthma' ÔåÆ 'asthma' (AST_COD) and 'severe persistent allergic asthma' ÔåÆ 'asthma' (SEVAST_COD)
   115ÔåÆ  - Now 3 rows for 'asthma': AST_COD, eFI2_Asthma, SEVAST_COD
   116ÔåÆ  - urticaria (XSAL_COD) stays separate
   117ÔåÆ- Added SEARCH_TERM_MERGE_MAP constant: {"allergic asthma": "asthma", "severe persistent allergic asthma": "asthma"}
   118ÔåÆ- Updated load_drug_indication_mapping() to apply SEARCH_TERM_MERGE_MAP when loading CSV
   119ÔåÆ  - Normalizes Search_Term before accumulating fragments
   120ÔåÆ  - Drug fragments from all 3 original rows combined under "asthma" key
   121ÔåÆ- Exported SEARCH_TERM_MERGE_MAP in __all__
   122ÔåÆ### Validation results:
   123ÔåÆ- Tier 1 (Code): py_compile passed, import check passed
   124ÔåÆ- Tier 2 (Data):
   125ÔåÆ  - "asthma" fragments: OMALIZUMAB, BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, RESLIZUMAB (complete combined list)
   126ÔåÆ  - "allergic asthma" no longer exists as separate key
   127ÔåÆ  - "severe persistent allergic asthma" no longer exists as separate key
   128ÔåÆ  - "urticaria" ÔåÆ ['OMALIZUMAB'] ÔÇö correctly separate
   129ÔåÆ  - OMALIZUMAB maps to: ['asthma', 'urticaria'] ÔÇö correct
   130ÔåÆ  - Total Search_Terms: 162 (was 164, 3 asthma entries ÔåÆ 1)
   131ÔåÆ  - Total fragments: 346 (unchanged)
   132ÔåÆ- Tier 3 (Functional): N/A (no UI changes)
   133ÔåÆ### Files changed:
   134ÔåÆ- data_processing/diagnosis_lookup.py (CLUSTER_MAPPING_SQL, SEARCH_TERM_MERGE_MAP, load_drug_indication_mapping)
   135ÔåÆ- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
   136ÔåÆ### Committed: b0a8a9d "feat: merge asthma Search_Term variants in CLUSTER_MAPPING_SQL and drug mapping (Task 1.2)"
   137ÔåÆ### Patterns discovered:
   138ÔåÆ- SEARCH_TERM_MERGE_MAP is reusable: any future module that receives Search_Terms from Snowflake can apply the same normalization
   139ÔåÆ- The merge approach (normalize at load time) is cleaner than post-hoc deduplication
   140ÔåÆ### Next iteration should:
   141ÔåÆ- Work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
   142ÔåÆ  - The current query at ~line 1467 uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
   143ÔåÆ  - Replace with GROUP BY + COUNT(*) for code_frequency
   144ÔåÆ  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
   145ÔåÆ  - Return columns: PatientPseudonym, Search_Term, code_frequency
   146ÔåÆ  - Empty DataFrame columns should match new return type
   147ÔåÆ  - This requires Snowflake connectivity to fully test, but code changes can be verified with py_compile and import checks
   148ÔåÆ- OR work on Task 2.1: Create assign_drug_indications() ÔÇö can be built and tested with mock data
   149ÔåÆ  - This is independent of Task 1.1 if you mock the gp_matches_df input
   150ÔåÆ### Blocked items:
   151ÔåÆ- None
   152ÔåÆ
   153ÔåÆ## Iteration 3 ÔÇö 2026-02-05
   154ÔåÆ### Task: 1.1 ÔÇö Update get_patient_indication_groups() to return ALL matches with code_frequency
   155ÔåÆ### Why this task:
   156ÔåÆ- Previous iteration recommended this as the next task
   157ÔåÆ- This is the last remaining Phase 1 dependency ÔÇö Tasks 2.1/2.2 need the GP matches data in the right format (multiple rows per patient with code_frequency)
   158ÔåÆ- Task 1.1 changes the query fundamentals; better to do this before building assign_drug_indications() which consumes the output
   159ÔåÆ### Status: COMPLETE
   160ÔåÆ### What was done:
   161ÔåÆ- Modified `get_patient_indication_groups()` in `diagnosis_lookup.py`:
   162ÔåÆ  1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
   163ÔåÆ  2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"`
   164ÔåÆ  3. **Added** `earliest_hcd_date: Optional[str] = None` parameter
   165ÔåÆ  4. **Added** optional `AND pc."EventDateTime" >= %s` when earliest_hcd_date is provided
   166ÔåÆ  5. **Updated** return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`
   167ÔåÆ  6. **Updated** all empty DataFrame returns to use new column names
   168ÔåÆ  7. **Updated** logging to show multiple-rows-per-patient stats (avg indications per patient)
   169ÔåÆ  8. **Updated** docstring to describe new behavior and parameters
   170ÔåÆ- Backward compatible: `earliest_hcd_date` defaults to `None`, existing callers still work
   171ÔåÆ- Note: caller in `refresh_pathways.py` (line 424-428) does `dict(zip(...))` which will only keep last match per patient with new multi-row format ÔÇö this will be updated in Task 3.1
   172ÔåÆ### Validation results:
   173ÔåÆ- Tier 1 (Code): py_compile PASSED, import check PASSED, function signature verified
   174ÔåÆ- Tier 2 (Data): Empty DataFrame returns correct columns ['PatientPseudonym', 'Search_Term', 'code_frequency']; live Snowflake test deferred to Phase 3/4
   175ÔåÆ- Tier 3 (Functional): N/A (no UI changes)
   176ÔåÆ### Files changed:
   177ÔåÆ- data_processing/diagnosis_lookup.py (modified get_patient_indication_groups function)
   178ÔåÆ- IMPLEMENTATION_PLAN.md (marked 1.1 subtasks [x])
   179ÔåÆ### Committed: c93417f "feat: return ALL GP matches with code_frequency in get_patient_indication_groups (Task 1.1)"
   180ÔåÆ### Patterns discovered:
   181ÔåÆ- The `earliest_hcd_date` parameter is passed as a string in ISO format (YYYY-MM-DD) via Snowflake %s placeholder ÔÇö Snowflake handles string-to-timestamp comparison implicitly
   182ÔåÆ- The GROUP BY approach naturally deduplicates SNOMED codes within the same Search_Term ÔÇö a patient with the same SNOMED code recorded 5 times gets code_frequency=5 (reflecting clinical activity intensity)
   183ÔåÆ- params list is built dynamically: `batch_pseudonyms + [earliest_hcd_date]` only when date filter is active
   184ÔåÆ### Next iteration should:
   185ÔåÆ- Work on Task 2.1: Create `assign_drug_indications()` function
   186ÔåÆ  - This is now unblocked since 1.1 is complete (return format is known)
   187ÔåÆ  - Input: HCD df, gp_matches_df (PatientPseudonym, Search_Term, code_frequency), drug_mapping from load_drug_indication_mapping()
   188ÔåÆ  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
   189ÔåÆ  - Can be built and tested with mock data (no Snowflake needed)
   190ÔåÆ  - Key logic: for each UPID+Drug pair, intersect drug's Search_Terms with patient's GP matches, pick highest code_frequency as tiebreaker
   191ÔåÆ  - The function needs PseudoNHSNoLinked to look up GP matches, so the df must have that column
   192ÔåÆ  - Task 2.2 (tiebreaker logic) can be done within 2.1 or as a follow-up
   193ÔåÆ- The final Phase 1 subtask (1.1 verify with live Snowflake) will be tested during Phase 3/4 integration
   194ÔåÆ### Blocked items:
   195ÔåÆ- Task 1.1 final subtask "Verify: Query returns more rows" requires live Snowflake ÔÇö deferred to Phase 3/4
   196ÔåÆ
   197ÔåÆ## Iteration 4 ÔÇö 2026-02-05
   198ÔåÆ### Task: 2.1 + 2.2 ÔÇö Create assign_drug_indications() with tiebreaker logic
   199ÔåÆ### Why this task:
   200ÔåÆ- All Phase 1 dependencies complete (1.1 query returns ALL matches, 1.2 asthma merged, 1.3 drug mapping loaded)
   201ÔåÆ- Task 2.1 is the core matching function needed before Phase 3 pipeline integration
   202ÔåÆ- Task 2.2 (tiebreaker) is naturally part of 2.1 ÔÇö implemented together
   203ÔåÆ- Can be built and tested with mock data (no Snowflake needed)
   204ÔåÆ### Status: COMPLETE
   205ÔåÆ### What was done:
   206ÔåÆ- Added `assign_drug_indications()` to `diagnosis_lookup.py`:
   207ÔåÆ  - Input: HCD df (with UPID, Drug Name, PseudoNHSNoLinked, Directory), gp_matches_df, search_term_to_fragments
   208ÔåÆ  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
   209ÔåÆ  - Builds GP lookup: PseudoNHSNoLinked ÔåÆ {Search_Term: code_frequency}
   210ÔåÆ  - Caches drugÔåÆSearch_Term lookups to avoid recomputing per row
   211ÔåÆ  - For each (UPID, Drug Name) pair:
   212ÔåÆ    - Intersects drug's Search_Terms with patient's GP matches
   213ÔåÆ    - Single match: use it
   214ÔåÆ    - Multiple matches: highest code_frequency wins, alphabetical tiebreak
   215ÔåÆ    - No match: fallback to "{Directory} (no GP dx)"
   216ÔåÆ  - Applies modified UPIDs via df.apply() (vectorized lookup from cache)
   217ÔåÆ  - Builds indication_df with unique modified UPID ÔåÆ Directory column
   218ÔåÆ  - Comprehensive logging: match rate, tiebreaker count, fallback count, top 5 indications
   219ÔåÆ- Updated __all__ exports
   220ÔåÆ### Validation results:
   221ÔåÆ- Tier 1 (Code): py_compile PASSED, import check PASSED
   222ÔåÆ- Tier 2 (Data): Mock data tests ALL PASSED:
   223ÔåÆ  - ADALIMUMAB + GP dx (RA + asthma) ÔåÆ matched to RA (drug mapping intersection)
   224ÔåÆ  - OMALIZUMAB + GP dx (RA + asthma) ÔåÆ matched to asthma (drug mapping intersection)
   225ÔåÆ  - ADALIMUMAB + GP dx (RA 3 freq + crohn's 2 freq) ÔåÆ tiebreaker picks RA
   226ÔåÆ  - ADALIMUMAB + GP dx (psoriatic 5 freq + RA 5 freq) ÔåÆ alphabetical tiebreak picks psoriatic arthritis
   227ÔåÆ  - Higher frequency (47 RA vs 3 psoriatic) ÔåÆ RA wins
   228ÔåÆ  - No GP match ÔåÆ fallback to directory
   229ÔåÆ  - Empty GP DataFrame ÔåÆ all fallback
   230ÔåÆ  - Different drugs with different indications ÔåÆ different modified UPIDs
   231ÔåÆ- Tier 3 (Functional): N/A (no UI changes yet)
   232ÔåÆ### Files changed:
   233ÔåÆ- data_processing/diagnosis_lookup.py (added assign_drug_indications, updated __all__)
   234ÔåÆ- IMPLEMENTATION_PLAN.md (marked 2.1 and 2.2 subtasks [x])
   235ÔåÆ### Committed: 408976e "feat: add assign_drug_indications() for drug-aware indication matching (Task 2.1 + 2.2)"
   236ÔåÆ### Patterns discovered:
   237ÔåÆ- Function signature takes `search_term_to_fragments` (the second element from load_drug_indication_mapping()) ÔÇö NOT the full tuple. Callers must destructure: `_, st_to_frags = load_drug_indication_mapping()`
   238ÔåÆ- The function uses df.apply() to set modified UPIDs ÔÇö for large DataFrames (656K rows), this could be slow. If performance is an issue in Phase 3, could vectorize with merge operations instead. But apply with cached lookup dict should be OK.
   239ÔåÆ- "crohn's disease" is NOT in ADALIMUMAB's DimSearchTerm mapping (ADALIMUMAB maps to: ankylosing spondylitis, axial spondyloarthritis, plaque psoriasis, psoriatic arthritis, rheumatoid arthritis, ulcerative colitis, uveitis). Initial test assumption was wrong.
   240ÔåÆ- indication_df has 'Directory' column (not 'Search_Term') for compatibility with generate_icicle_chart_indication() which expects indication_df.loc[upid, 'Directory']
   241ÔåÆ### Next iteration should:
   242ÔåÆ- Work on Task 3.1: Update `refresh_pathways.py` indication processing to use assign_drug_indications()
   243ÔåÆ  - The current code at lines 424-428 uses `dict(zip(...))` which only keeps LAST match per patient ÔÇö this must be replaced
   244ÔåÆ  - Key changes in the `elif current_chart_type == "indication":` block:
   245ÔåÆ    1. Load drug mapping: `_, st_to_frags = load_drug_indication_mapping()`
   246ÔåÆ    2. Pass `earliest_hcd_date=df['Intervention Date'].min().strftime('%Y-%m-%d')` to get_patient_indication_groups()
   247ÔåÆ    3. Call `assign_drug_indications(df, gp_matches_df, st_to_frags)` to get (modified_df, indication_df)
   248ÔåÆ    4. Use modified_df (not original df) for pathway processing
   249ÔåÆ    5. indication_df is already in the right format (indexed by modified UPID, 'Directory' column)
   250ÔåÆ    6. Remove the old match_lookup/dict(zip) code and the manual indication_df building
   251ÔåÆ  - Import assign_drug_indications and load_drug_indication_mapping at top of file
   252ÔåÆ  - This replaces ~50 lines of the old approach with ~10 lines using the new function
   253ÔåÆ  - Can verify with py_compile; full Snowflake test via --dry-run
   254ÔåÆ### Blocked items:
   255ÔåÆ- None
   256ÔåÆ
   257ÔåÆ## Iteration 5 ÔÇö 2026-02-05
   258ÔåÆ### Task: 3.1 ÔÇö Update refresh_pathways.py indication processing to use assign_drug_indications()
   259ÔåÆ### Why this task:
   260ÔåÆ- All Phase 1 & 2 dependencies complete (query returns all matches, drug mapping loaded, assign_drug_indications() exists)
   261ÔåÆ- Task 3.1 is the pipeline integration step ÔÇö wires the new drug-aware matching into the actual refresh pipeline
   262ÔåÆ- Must be done before Task 3.2 (dry run test) which validates the integrated pipeline
   263ÔåÆ### Status: COMPLETE
   264ÔåÆ### What was done:
   265ÔåÆ- Updated imports at top of `cli/refresh_pathways.py`:
   266ÔåÆ  - Added `assign_drug_indications` and `load_drug_indication_mapping` from `data_processing.diagnosis_lookup`
   267ÔåÆ- Replaced the entire indication processing block (old ~90 lines ÔåÆ new ~60 lines):
   268ÔåÆ  - **Old approach**: `dict(zip(gp_matches_df['PatientPseudonym'], gp_matches_df['Search_Term']))` ÔÇö only kept LAST match per patient, no drug awareness
   269ÔåÆ  - **New approach**:
   270ÔåÆ    1. `load_drug_indication_mapping()` ÔåÆ `search_term_to_fragments`
   271ÔåÆ    2. Compute `earliest_hcd_date` from `df['Intervention Date'].min()` as ISO string
   272ÔåÆ    3. `get_patient_indication_groups(earliest_hcd_date=earliest_hcd_date_str)` ÔåÆ all GP matches with code_frequency
   273ÔåÆ    4. `assign_drug_indications(df, gp_matches_df, search_term_to_fragments)` ÔåÆ `(modified_df, indication_df)`
   274ÔåÆ    5. Pass `modified_df` (not original `df`) to `process_indication_pathway_for_date_filter()`
   275ÔåÆ    6. `indication_df` already indexed by modified UPID with 'Directory' column ÔÇö directly compatible
   276ÔåÆ- Removed: old `match_lookup`, `upid_lookup`, manual `indication_records` building, `indication_df_for_chart` renaming
   277ÔåÆ- Kept: Snowflake availability check, PseudoNHSNoLinked column check, error handling, date filter loop
   278ÔåÆ### Validation results:
   279ÔåÆ- Tier 1 (Code): py_compile PASSED, individual imports PASSED, full module import PASSED
   280ÔåÆ- Tier 2 (Data): N/A ÔÇö requires live Snowflake for dry run test (Task 3.2)
   281ÔåÆ- Tier 3 (Functional): N/A ÔÇö no UI changes
   282ÔåÆ### Files changed:
   283ÔåÆ- cli/refresh_pathways.py (updated imports, replaced indication processing block)
   284ÔåÆ- IMPLEMENTATION_PLAN.md (marked 3.1 subtasks [x])
   285ÔåÆ### Committed: 920570b "feat: integrate drug-aware indication matching into refresh pipeline (Task 3.1)"
   286ÔåÆ### Patterns discovered:
   287ÔåÆ- `assign_drug_indications()` returns `indication_df` already indexed by modified UPID with 'Directory' column ÔÇö no need for intermediate renaming/reindexing steps that the old code required
   288ÔåÆ- `earliest_hcd_date` must be converted via `pd.Timestamp(...).strftime('%Y-%m-%d')` because `df['Intervention Date'].min()` may return a Timestamp or string depending on data source
   289ÔåÆ- The old code had a "stats['diagnosis_coverage']" tracking block ÔÇö this is now handled internally by `assign_drug_indications()` logging. If stats tracking in the return dict is needed later, can add it back.
   290ÔåÆ### Next iteration should:
   291ÔåÆ- Work on Task 3.2: Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`
   292ÔåÆ  - This requires a live Snowflake connection
   293ÔåÆ  - Verify: modified UPIDs appear in logs, match rates logged, pathway nodes generated
   294ÔåÆ  - If dry run passes, move to Phase 4 (full refresh + validation)
   295ÔåÆ- Key things to check in dry run output:
   296ÔåÆ  - "Drug-aware indication matching complete" log message with match/fallback counts
   297ÔåÆ  - "Modified UPIDs" count should be HIGHER than unique patient count (patients with multiple drugs for different indications)
   298ÔåÆ  - Pathway node counts for indication charts should be in same ballpark as before (~300 per date filter)
   299ÔåÆ  - No errors in indication pathway processing
   300ÔåÆ### Blocked items:
   301ÔåÆ- None
   302ÔåÆ
   303ÔåÆ## Iteration 6 ÔÇö 2026-02-05
   304ÔåÆ### Task: 3.2 ÔÇö Test with dry run
   305ÔåÆ### Why this task:
   306ÔåÆ- All Phase 1-3.1 dependencies complete (query, drug mapping, matching function, pipeline integration)
   307ÔåÆ- 3.2 validates the integrated pipeline end-to-end before Phase 4 (full refresh)
   308ÔåÆ- Must pass before moving to production refresh
   309ÔåÆ### Status: COMPLETE
   310ÔåÆ### What was done:
   311ÔåÆ- **Discovered**: GP lookup queries were timing out at 30 seconds ÔÇö every batch failed
   312ÔåÆ  - Root cause: `connection_timeout=30` in config/snowflake.toml sets Snowflake Python client `network_timeout`
   313ÔåÆ  - This kills any query taking >30s, regardless of server-side STATEMENT_TIMEOUT (300s)
   314ÔåÆ  - The GROUP BY + COUNT(*) query takes ~40s per batch (even for 5 patients)
   315ÔåÆ  - The old QUALIFY ROW_NUMBER() query took ~20s (borderline but usually OK with caching)
   316ÔåÆ- **Fixed timeout**: Changed `connection_timeout` from 30 ÔåÆ 600 in snowflake.toml and config/__init__.py fallback
   317ÔåÆ  - Safe because query_timeout (300s) still controls server-side statement limits
   318ÔåÆ  - All existing queries still work fine (activity data fetch: 7s, chunked)
   319ÔåÆ- **Optimized batch size**: Changed from 500 ÔåÆ 5000 patients per batch
   320ÔåÆ  - Query time is ~constant regardless of batch size (~40s) ÔÇö bottleneck is CTE compilation, not data volume
   321ÔåÆ  - 500-patient batches: 74 batches ├ù 40s = ~50 minutes for GP lookup
   322ÔåÆ  - 5000-patient batches: 8 batches ├ù 45s = ~6 minutes for GP lookup
   323ÔåÆ  - Updated both default in get_patient_indication_groups() and caller in refresh_pathways.py
   324ÔåÆ- **Dry run results** (successful):
   325ÔåÆ  - GP Lookup: 36,628 patients, 33,642 matched (91.8%), 8 batches in ~5.5 min
   326ÔåÆ  - Drug-Indication Matching: 50,797 UPID-Drug pairs ÔåÆ 25,059 matched (49.3%), 15,238 tiebreakers, 25,738 fallback
   327ÔåÆ  - Modified UPIDs: 42,072 (up from 36,628 original patients ÔÇö some patients split across indications)
   328ÔåÆ  - Pathway nodes per date filter: all_6mo=438, all_12mo=484, 1yr_6mo=181, 1yr_12mo=199, 2yr_6mo=257, 2yr_12mo=287
   329ÔåÆ  - Total: 1,846 indication nodes across 6 date filters
   330ÔåÆ  - No errors during pathway processing
   331ÔåÆ### Validation results:
   332ÔåÆ- Tier 1 (Code): py_compile PASSED for diagnosis_lookup.py, refresh_pathways.py, config/__init__.py
   333ÔåÆ- Tier 2 (Data): Dry run completed successfully with correct log output:
   334ÔåÆ  - Modified UPIDs appear (42,072 unique)
   335ÔåÆ  - Match/fallback rates logged (49.3% / 50.7%)
   336ÔåÆ  - Tiebreaker count logged (15,238)
   337ÔåÆ  - Top indications: macular degeneration, diabetes, rheumatoid arthritis
   338ÔåÆ  - Pathway node counts reasonable (181-484 per date filter)
   339ÔåÆ- Tier 3 (Functional): Dry run completed, no insertion (as expected)
   340ÔåÆ### Files changed:
   341ÔåÆ- config/snowflake.toml (connection_timeout 30 ÔåÆ 600)
   342ÔåÆ- config/__init__.py (fallback connection_timeout 30 ÔåÆ 600)
   343ÔåÆ- data_processing/diagnosis_lookup.py (batch_size default 500 ÔåÆ 5000)
   344ÔåÆ- cli/refresh_pathways.py (batch_size 500 ÔåÆ 5000)
   345ÔåÆ- IMPLEMENTATION_PLAN.md (marked 3.2 subtasks [x])
   346ÔåÆ### Committed: c6e426e "fix: increase network timeout and batch size for GP lookup queries (Task 3.2)"
   347ÔåÆ### Patterns discovered:
   348ÔåÆ- Snowflake Python connector `network_timeout` (set via connection_timeout in config) controls client-side wait time for ALL query responses, not just connection establishment. Must be high enough for slow queries.
   349ÔåÆ- PrimaryCareClinicalCoding query performance is dominated by CTE compilation (~40s fixed cost), not by patient count. Larger batches (5000 vs 500) are dramatically more efficient.
   350ÔåÆ- 49.3% match rate means about half of UPID-Drug pairs have both a drug mapping in DimSearchTerm AND matching GP diagnosis. The 50.7% fallback is expected since not all HCD drugs are in DimSearchTerm.csv.
   351ÔåÆ### Next iteration should:
   352ÔåÆ- Work on Task 4.1: Full refresh with both chart types
   353ÔåÆ  - Run `python -m cli.refresh_pathways --chart-type all` (no --dry-run)
   354ÔåÆ  - This will insert ~1,846 indication nodes + ~1,800 directory nodes into pathway_nodes table
   355ÔåÆ  - Verify both chart types generate data, directory charts unchanged
   356ÔåÆ  - Takes ~15 minutes total (7s Snowflake + 6min transforms + 6min GP lookup + 2min pathways)
   357ÔåÆ- After 4.1, Tasks 4.2 and 4.3 can be done together:
   358ÔåÆ  - 4.2: Validate indication chart correctness (spot-check drug grouping)
   359ÔåÆ  - 4.3: Validate Reflex UI compiles and chart type toggle works
   360ÔåÆ### Blocked items:
   361ÔåÆ- None
   362ÔåÆ

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

# Progress Log - Drug-Aware Indication Matching

## Project Context

This project extends the indication-based pathway charts (Phase 1-5 complete) with drug-aware matching.

**Previous state**: Patients get ONE indication based on their most recent GP diagnosis match (SNOMED cluster codes). This ignores which drugs the patient is taking.

**New goal**: Match each drug to an indication by cross-referencing the patient's GP diagnoses AND the drug's Search_Term mapping from DimSearchTerm.csv.

## Key Data/Patterns

### DimSearchTerm.csv
- Located at `data/DimSearchTerm.csv`
- Columns: Search_Term, CleanedDrugName (pipe-separated), PrimaryDirectorate
- ~165 rows mapping clinical conditions to drug name fragments
- Drug fragments are substrings that match standardized drug names from HCD data
- Some entries have generic fragments: INHALED, CONTINUOUS, STANDARD-DOSE, PEGYLATED

### Current get_patient_indication_groups() in diagnosis_lookup.py
- Uses CLUSTER_MAPPING_SQL as CTE in Snowflake query
- Returns ONLY the most recent match per patient (QUALIFY ROW_NUMBER() = 1)
- Needs to return ALL matching Search_Terms per patient (remove QUALIFY)
- Batches 500 patients per query

### Modified UPID approach
- Current: UPID = Provider Code[:3] + PersonKey (e.g., "RMV12345")
- New: UPID = original + "|" + search_term (e.g., "RMV12345|rheumatoid arthritis")
- The pipe delimiter "|" is safe because existing UPIDs are alphanumeric
- generate_icicle_chart_indication() treats UPID as an opaque identifier ÔÇö modified UPIDs work transparently
- The " - " delimiter in pathway ids is used for hierarchy levels, not within UPIDs

### PseudoNHSNoLinked mapping
- HCD data has PseudoNHSNoLinked column that matches PatientPseudonym in GP records
- PersonKey is provider-specific local ID ÔÇö do NOT use for GP matching
- One PseudoNHSNoLinked can map to multiple UPIDs (multi-provider patients)
- GP match lookup: PseudoNHSNoLinked ÔåÆ list of matched Search_Terms

### Drug matching logic
- For each HCD row (UPID + Drug Name):
  1. Get patient's GP-matched Search_Terms with code_frequency (via PseudoNHSNoLinked)
  2. Get which Search_Terms list this drug (from DimSearchTerm.csv)
  3. Intersection = valid indications
  4. If 1: use it. If multiple: pick highest code_frequency (most GP coding = most likely indication). If 0: fallback to directory.
- Modified UPID groups drugs under same indication together naturally
- code_frequency = COUNT(*) of matching SNOMED codes per Search_Term per patient in GP records
- GP code time range: only count codes from MIN(Intervention Date) onwards (the HCD data window)
  - Reduces noise from old/irrelevant diagnoses, makes frequency more meaningful
  - Pass earliest_hcd_date as parameter to get_patient_indication_groups()
- Tiebreaker rationale: 47 RA codes vs 2 crohn's codes ÔåÆ RA is clearly the active condition

### Known edge cases
- Some DimSearchTerm drug fragments are generic (INHALED, ORAL, CONTINUOUS)
  - These could match broadly but are constrained by GP diagnosis requirement
- A patient visiting multiple providers has multiple UPIDs
  - Each UPID gets its own drug-indication matching independently
- Same Search_Term appears twice in DimSearchTerm.csv with different directorates
  - e.g., "diabetes" ÔåÆ DIABETIC MEDICINE and OPHTHALMOLOGY
  - For indication charts, we use Search_Term not directorate, so this is fine

## Iteration Log

## Iteration 1 ÔÇö 2026-02-05
### Task: 1.3 ÔÇö Build drug-to-Search_Term lookup from DimSearchTerm.csv
### Why this task:
- First iteration, chose Phase 1 foundations. Task 1.2 (CSV loading) is self-contained and testable locally without Snowflake.
- Task 1.1 (Snowflake query update) can't be verified without a live connection ÔÇö better to do 1.2 first.
- Both 1.1 and 1.2 are independent, so order doesn't matter for dependencies.
### Status: COMPLETE
### What was done:
- Added `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
  - Loads `data/DimSearchTerm.csv`, builds two dicts:
    - `fragment_to_search_terms`: drug fragment (UPPER) ÔåÆ list of Search_Terms
    - `search_term_to_fragments`: search_term ÔåÆ list of drug fragments (UPPER)
  - Handles duplicate Search_Terms (e.g., "diabetes" rows combined)
  - Result: 164 Search_Terms, 346 drug fragments
- Added `get_search_terms_for_drug()` to `diagnosis_lookup.py`:
  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
  - Named differently from plan's `drug_matches_search_term()` ÔÇö returns all matches at once rather than single boolean, more practical for Phase 2
- Updated `__all__` exports
### Validation results:
- Tier 1 (Code): py_compile passed, import check passed
- Tier 2 (Data): ADALIMUMAB ÔåÆ 7 indications (including axial spondyloarthritis, rheumatoid arthritis), OMALIZUMAB ÔåÆ 4 indications (asthma, allergic asthma, etc.), PEGYLATED LIPOSOMAL DOXORUBICIN ÔåÆ 4 matches via substring, "ADALIMUMAB 40MG" matches correctly with dosage info, diabetes fragments combined from 2 CSV rows
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (added load_drug_indication_mapping, get_search_terms_for_drug)
- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
### Committed: 0779df7 "feat: add drug-to-indication mapping from DimSearchTerm.csv (Task 1.3)"
### Patterns discovered:
- DimSearchTerm.csv has 164 unique Search_Terms (not 165 as noted) because diabetes appears twice with different directorates but same Search_Term
- Some drug fragments are very generic: INHALED, CONTINUOUS, ORAL, STANDARD-DOSE, INTRAVENOUS, PEGYLATED, ROUTINE, INDUCTION ÔÇö these will match broadly but are constrained by the GP diagnosis requirement in Phase 2
- Function signatures for Phase 2: `get_search_terms_for_drug(drug_name, search_term_to_fragments)` returns list[str] ÔÇö use this to get candidate indications per drug
### Next iteration should:
- Work on Task 1.2: Merge asthma Search_Terms in CLUSTER_MAPPING_SQL and load_drug_indication_mapping()
  - Merge "allergic asthma", "asthma", "severe persistent allergic asthma" ÔåÆ "asthma"
  - Keep "urticaria" separate
  - This is self-contained and testable locally
- OR work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
  - The current query at line ~1352 of diagnosis_lookup.py uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1` ÔÇö this must be replaced with GROUP BY + COUNT(*)
  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
  - Return columns: PatientPseudonym, Search_Term, code_frequency (not EventDateTime)
- OR if Snowflake isn't available to test 1.1, skip to Task 2.1 (assign_drug_indications function) which can be built and tested with mock data
### Blocked items:
- None

## Iteration 2 ÔÇö 2026-02-05
### Task: 1.2 ÔÇö Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
### Why this task:
- Previous iteration recommended this as the next task (self-contained, testable locally)
- Both CLUSTER_MAPPING_SQL and load_drug_indication_mapping() need consistent Search_Term names
- Must be done before Task 1.1 (Snowflake query) to ensure GP lookups return "asthma" not "allergic asthma"
### Status: COMPLETE
### What was done:
- Updated CLUSTER_MAPPING_SQL: changed 'allergic asthma' ÔåÆ 'asthma' (AST_COD) and 'severe persistent allergic asthma' ÔåÆ 'asthma' (SEVAST_COD)
  - Now 3 rows for 'asthma': AST_COD, eFI2_Asthma, SEVAST_COD
  - urticaria (XSAL_COD) stays separate
- Added SEARCH_TERM_MERGE_MAP constant: {"allergic asthma": "asthma", "severe persistent allergic asthma": "asthma"}
- Updated load_drug_indication_mapping() to apply SEARCH_TERM_MERGE_MAP when loading CSV
  - Normalizes Search_Term before accumulating fragments
  - Drug fragments from all 3 original rows combined under "asthma" key
- Exported SEARCH_TERM_MERGE_MAP in __all__
### Validation results:
- Tier 1 (Code): py_compile passed, import check passed
- Tier 2 (Data):
  - "asthma" fragments: OMALIZUMAB, BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, RESLIZUMAB (complete combined list)
  - "allergic asthma" no longer exists as separate key
  - "severe persistent allergic asthma" no longer exists as separate key
  - "urticaria" ÔåÆ ['OMALIZUMAB'] ÔÇö correctly separate
  - OMALIZUMAB maps to: ['asthma', 'urticaria'] ÔÇö correct
  - Total Search_Terms: 162 (was 164, 3 asthma entries ÔåÆ 1)
  - Total fragments: 346 (unchanged)
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (CLUSTER_MAPPING_SQL, SEARCH_TERM_MERGE_MAP, load_drug_indication_mapping)
- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
### Committed: b0a8a9d "feat: merge asthma Search_Term variants in CLUSTER_MAPPING_SQL and drug mapping (Task 1.2)"
### Patterns discovered:
- SEARCH_TERM_MERGE_MAP is reusable: any future module that receives Search_Terms from Snowflake can apply the same normalization
- The merge approach (normalize at load time) is cleaner than post-hoc deduplication
### Next iteration should:
- Work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
  - The current query at ~line 1467 uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  - Replace with GROUP BY + COUNT(*) for code_frequency
  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
  - Return columns: PatientPseudonym, Search_Term, code_frequency
  - Empty DataFrame columns should match new return type
  - This requires Snowflake connectivity to fully test, but code changes can be verified with py_compile and import checks
- OR work on Task 2.1: Create assign_drug_indications() ÔÇö can be built and tested with mock data
  - This is independent of Task 1.1 if you mock the gp_matches_df input
### Blocked items:
- None

## Iteration 3 ÔÇö 2026-02-05
### Task: 1.1 ÔÇö Update get_patient_indication_groups() to return ALL matches with code_frequency
### Why this task:
- Previous iteration recommended this as the next task
- This is the last remaining Phase 1 dependency ÔÇö Tasks 2.1/2.2 need the GP matches data in the right format (multiple rows per patient with code_frequency)
- Task 1.1 changes the query fundamentals; better to do this before building assign_drug_indications() which consumes the output
### Status: COMPLETE
### What was done:
- Modified `get_patient_indication_groups()` in `diagnosis_lookup.py`:
  1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"`
  3. **Added** `earliest_hcd_date: Optional[str] = None` parameter
  4. **Added** optional `AND pc."EventDateTime" >= %s` when earliest_hcd_date is provided
  5. **Updated** return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`
  6. **Updated** all empty DataFrame returns to use new column names
  7. **Updated** logging to show multiple-rows-per-patient stats (avg indications per patient)
  8. **Updated** docstring to describe new behavior and parameters
- Backward compatible: `earliest_hcd_date` defaults to `None`, existing callers still work
- Note: caller in `refresh_pathways.py` (line 424-428) does `dict(zip(...))` which will only keep last match per patient with new multi-row format ÔÇö this will be updated in Task 3.1
### Validation results:
- Tier 1 (Code): py_compile PASSED, import check PASSED, function signature verified
- Tier 2 (Data): Empty DataFrame returns correct columns ['PatientPseudonym', 'Search_Term', 'code_frequency']; live Snowflake test deferred to Phase 3/4
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (modified get_patient_indication_groups function)
- IMPLEMENTATION_PLAN.md (marked 1.1 subtasks [x])
### Committed: c93417f "feat: return ALL GP matches with code_frequency in get_patient_indication_groups (Task 1.1)"
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format (YYYY-MM-DD) via Snowflake %s placeholder ÔÇö Snowflake handles string-to-timestamp comparison implicitly
- The GROUP BY approach naturally deduplicates SNOMED codes within the same Search_Term ÔÇö a patient with the same SNOMED code recorded 5 times gets code_frequency=5 (reflecting clinical activity intensity)
- params list is built dynamically: `batch_pseudonyms + [earliest_hcd_date]` only when date filter is active
### Next iteration should:
- Work on Task 2.1: Create `assign_drug_indications()` function
  - This is now unblocked since 1.1 is complete (return format is known)
  - Input: HCD df, gp_matches_df (PatientPseudonym, Search_Term, code_frequency), drug_mapping from load_drug_indication_mapping()
  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
  - Can be built and tested with mock data (no Snowflake needed)
  - Key logic: for each UPID+Drug pair, intersect drug's Search_Terms with patient's GP matches, pick highest code_frequency as tiebreaker
  - The function needs PseudoNHSNoLinked to look up GP matches, so the df must have that column
  - Task 2.2 (tiebreaker logic) can be done within 2.1 or as a follow-up
- The final Phase 1 subtask (1.1 verify with live Snowflake) will be tested during Phase 3/4 integration
### Blocked items:
- Task 1.1 final subtask "Verify: Query returns more rows" requires live Snowflake ÔÇö deferred to Phase 3/4

## Iteration 4 ÔÇö 2026-02-05
### Task: 2.1 + 2.2 ÔÇö Create assign_drug_indications() with tiebreaker logic
### Why this task:
- All Phase 1 dependencies complete (1.1 query returns ALL matches, 1.2 asthma merged, 1.3 drug mapping loaded)
- Task 2.1 is the core matching function needed before Phase 3 pipeline integration
- Task 2.2 (tiebreaker) is naturally part of 2.1 ÔÇö implemented together
- Can be built and tested with mock data (no Snowflake needed)
### Status: COMPLETE
### What was done:
- Added `assign_drug_indications()` to `diagnosis_lookup.py`:
  - Input: HCD df (with UPID, Drug Name, PseudoNHSNoLinked, Directory), gp_matches_df, search_term_to_fragments
  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
  - Builds GP lookup: PseudoNHSNoLinked ÔåÆ {Search_Term: code_frequency}
  - Caches drugÔåÆSearch_Term lookups to avoid recomputing per row
  - For each (UPID, Drug Name) pair:
    - Intersects drug's Search_Terms with patient's GP matches
    - Single match: use it
    - Multiple matches: highest code_frequency wins, alphabetical tiebreak
    - No match: fallback to "{Directory} (no GP dx)"
  - Applies modified UPIDs via df.apply() (vectorized lookup from cache)
  - Builds indication_df with unique modified UPID ÔåÆ Directory column
  - Comprehensive logging: match rate, tiebreaker count, fallback count, top 5 indications
- Updated __all__ exports
### Validation results:
- Tier 1 (Code): py_compile PASSED, import check PASSED
- Tier 2 (Data): Mock data tests ALL PASSED:
  - ADALIMUMAB + GP dx (RA + asthma) ÔåÆ matched to RA (drug mapping intersection)
  - OMALIZUMAB + GP dx (RA + asthma) ÔåÆ matched to asthma (drug mapping intersection)
  - ADALIMUMAB + GP dx (RA 3 freq + crohn's 2 freq) ÔåÆ tiebreaker picks RA
  - ADALIMUMAB + GP dx (psoriatic 5 freq + RA 5 freq) ÔåÆ alphabetical tiebreak picks psoriatic arthritis
  - Higher frequency (47 RA vs 3 psoriatic) ÔåÆ RA wins
  - No GP match ÔåÆ fallback to directory
  - Empty GP DataFrame ÔåÆ all fallback
  - Different drugs with different indications ÔåÆ different modified UPIDs
- Tier 3 (Functional): N/A (no UI changes yet)
### Files changed:
- data_processing/diagnosis_lookup.py (added assign_drug_indications, updated __all__)
- IMPLEMENTATION_PLAN.md (marked 2.1 and 2.2 subtasks [x])
### Committed: 408976e "feat: add assign_drug_indications() for drug-aware indication matching (Task 2.1 + 2.2)"
### Patterns discovered:
- Function signature takes `search_term_to_fragments` (the second element from load_drug_indication_mapping()) ÔÇö NOT the full tuple. Callers must destructure: `_, st_to_frags = load_drug_indication_mapping()`
- The function uses df.apply() to set modified UPIDs ÔÇö for large DataFrames (656K rows), this could be slow. If performance is an issue in Phase 3, could vectorize with merge operations instead. But apply with cached lookup dict should be OK.
- "crohn's disease" is NOT in ADALIMUMAB's DimSearchTerm mapping (ADALIMUMAB maps to: ankylosing spondylitis, axial spondyloarthritis, plaque psoriasis, psoriatic arthritis, rheumatoid arthritis, ulcerative colitis, uveitis). Initial test assumption was wrong.
- indication_df has 'Directory' column (not 'Search_Term') for compatibility with generate_icicle_chart_indication() which expects indication_df.loc[upid, 'Directory']
### Next iteration should:
- Work on Task 3.1: Update `refresh_pathways.py` indication processing to use assign_drug_indications()
  - The current code at lines 424-428 uses `dict(zip(...))` which only keeps LAST match per patient ÔÇö this must be replaced
  - Key changes in the `elif current_chart_type == "indication":` block:
    1. Load drug mapping: `_, st_to_frags = load_drug_indication_mapping()`
    2. Pass `earliest_hcd_date=df['Intervention Date'].min().strftime('%Y-%m-%d')` to get_patient_indication_groups()
    3. Call `assign_drug_indications(df, gp_matches_df, st_to_frags)` to get (modified_df, indication_df)
    4. Use modified_df (not original df) for pathway processing
    5. indication_df is already in the right format (indexed by modified UPID, 'Directory' column)
    6. Remove the old match_lookup/dict(zip) code and the manual indication_df building
  - Import assign_drug_indications and load_drug_indication_mapping at top of file
  - This replaces ~50 lines of the old approach with ~10 lines using the new function
  - Can verify with py_compile; full Snowflake test via --dry-run
### Blocked items:
- None

## Iteration 5 ÔÇö 2026-02-05
### Task: 3.1 ÔÇö Update refresh_pathways.py indication processing to use assign_drug_indications()
### Why this task:
- All Phase 1 & 2 dependencies complete (query returns all matches, drug mapping loaded, assign_drug_indications() exists)
- Task 3.1 is the pipeline integration step ÔÇö wires the new drug-aware matching into the actual refresh pipeline
- Must be done before Task 3.2 (dry run test) which validates the integrated pipeline
### Status: COMPLETE
### What was done:
- Updated imports at top of `cli/refresh_pathways.py`:
  - Added `assign_drug_indications` and `load_drug_indication_mapping` from `data_processing.diagnosis_lookup`
- Replaced the entire indication processing block (old ~90 lines ÔåÆ new ~60 lines):
  - **Old approach**: `dict(zip(gp_matches_df['PatientPseudonym'], gp_matches_df['Search_Term']))` ÔÇö only kept LAST match per patient, no drug awareness
  - **New approach**:
    1. `load_drug_indication_mapping()` ÔåÆ `search_term_to_fragments`
    2. Compute `earliest_hcd_date` from `df['Intervention Date'].min()` as ISO string
    3. `get_patient_indication_groups(earliest_hcd_date=earliest_hcd_date_str)` ÔåÆ all GP matches with code_frequency
    4. `assign_drug_indications(df, gp_matches_df, search_term_to_fragments)` ÔåÆ `(modified_df, indication_df)`
    5. Pass `modified_df` (not original `df`) to `process_indication_pathway_for_date_filter()`
    6. `indication_df` already indexed by modified UPID with 'Directory' column ÔÇö directly compatible
- Removed: old `match_lookup`, `upid_lookup`, manual `indication_records` building, `indication_df_for_chart` renaming
- Kept: Snowflake availability check, PseudoNHSNoLinked column check, error handling, date filter loop
### Validation results:
- Tier 1 (Code): py_compile PASSED, individual imports PASSED, full module import PASSED
- Tier 2 (Data): N/A ÔÇö requires live Snowflake for dry run test (Task 3.2)
- Tier 3 (Functional): N/A ÔÇö no UI changes
### Files changed:
- cli/refresh_pathways.py (updated imports, replaced indication processing block)
- IMPLEMENTATION_PLAN.md (marked 3.1 subtasks [x])
### Committed: 920570b "feat: integrate drug-aware indication matching into refresh pipeline (Task 3.1)"
### Patterns discovered:
- `assign_drug_indications()` returns `indication_df` already indexed by modified UPID with 'Directory' column ÔÇö no need for intermediate renaming/reindexing steps that the old code required
- `earliest_hcd_date` must be converted via `pd.Timestamp(...).strftime('%Y-%m-%d')` because `df['Intervention Date'].min()` may return a Timestamp or string depending on data source
- The old code had a "stats['diagnosis_coverage']" tracking block ÔÇö this is now handled internally by `assign_drug_indications()` logging. If stats tracking in the return dict is needed later, can add it back.
### Next iteration should:
- Work on Task 3.2: Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`
  - This requires a live Snowflake connection
  - Verify: modified UPIDs appear in logs, match rates logged, pathway nodes generated
  - If dry run passes, move to Phase 4 (full refresh + validation)
- Key things to check in dry run output:
  - "Drug-aware indication matching complete" log message with match/fallback counts
  - "Modified UPIDs" count should be HIGHER than unique patient count (patients with multiple drugs for different indications)
  - Pathway node counts for indication charts should be in same ballpark as before (~300 per date filter)
  - No errors in indication pathway processing
### Blocked items:
- None

## Iteration 6 ÔÇö 2026-02-05
### Task: 3.2 ÔÇö Test with dry run
### Why this task:
- All Phase 1-3.1 dependencies complete (query, drug mapping, matching function, pipeline integration)
- 3.2 validates the integrated pipeline end-to-end before Phase 4 (full refresh)
- Must pass before moving to production refresh
### Status: COMPLETE
### What was done:
- **Discovered**: GP lookup queries were timing out at 30 seconds ÔÇö every batch failed
  - Root cause: `connection_timeout=30` in config/snowflake.toml sets Snowflake Python client `network_timeout`
  - This kills any query taking >30s, regardless of server-side STATEMENT_TIMEOUT (300s)
  - The GROUP BY + COUNT(*) query takes ~40s per batch (even for 5 patients)
  - The old QUALIFY ROW_NUMBER() query took ~20s (borderline but usually OK with caching)
- **Fixed timeout**: Changed `connection_timeout` from 30 ÔåÆ 600 in snowflake.toml and config/__init__.py fallback
  - Safe because query_timeout (300s) still controls server-side statement limits
  - All existing queries still work fine (activity data fetch: 7s, chunked)
- **Optimized batch size**: Changed from 500 ÔåÆ 5000 patients per batch
  - Query time is ~constant regardless of batch size (~40s) ÔÇö bottleneck is CTE compilation, not data volume
  - 500-patient batches: 74 batches ├ù 40s = ~50 minutes for GP lookup
  - 5000-patient batches: 8 batches ├ù 45s = ~6 minutes for GP lookup
  - Updated both default in get_patient_indication_groups() and caller in refresh_pathways.py
- **Dry run results** (successful):
  - GP Lookup: 36,628 patients, 33,642 matched (91.8%), 8 batches in ~5.5 min
  - Drug-Indication Matching: 50,797 UPID-Drug pairs ÔåÆ 25,059 matched (49.3%), 15,238 tiebreakers, 25,738 fallback
  - Modified UPIDs: 42,072 (up from 36,628 original patients ÔÇö some patients split across indications)
  - Pathway nodes per date filter: all_6mo=438, all_12mo=484, 1yr_6mo=181, 1yr_12mo=199, 2yr_6mo=257, 2yr_12mo=287
  - Total: 1,846 indication nodes across 6 date filters
  - No errors during pathway processing
### Validation results:
- Tier 1 (Code): py_compile PASSED for diagnosis_lookup.py, refresh_pathways.py, config/__init__.py
- Tier 2 (Data): Dry run completed successfully with correct log output:
  - Modified UPIDs appear (42,072 unique)
  - Match/fallback rates logged (49.3% / 50.7%)
  - Tiebreaker count logged (15,238)
  - Top indications: macular degeneration, diabetes, rheumatoid arthritis
  - Pathway node counts reasonable (181-484 per date filter)
- Tier 3 (Functional): Dry run completed, no insertion (as expected)
### Files changed:
- config/snowflake.toml (connection_timeout 30 ÔåÆ 600)
- config/__init__.py (fallback connection_timeout 30 ÔåÆ 600)
- data_processing/diagnosis_lookup.py (batch_size default 500 ÔåÆ 5000)
- cli/refresh_pathways.py (batch_size 500 ÔåÆ 5000)
- IMPLEMENTATION_PLAN.md (marked 3.2 subtasks [x])
### Committed: c6e426e "fix: increase network timeout and batch size for GP lookup queries (Task 3.2)"
### Patterns discovered:
- Snowflake Python connector `network_timeout` (set via connection_timeout in config) controls client-side wait time for ALL query responses, not just connection establishment. Must be high enough for slow queries.
- PrimaryCareClinicalCoding query performance is dominated by CTE compilation (~40s fixed cost), not by patient count. Larger batches (5000 vs 500) are dramatically more efficient.
- 49.3% match rate means about half of UPID-Drug pairs have both a drug mapping in DimSearchTerm AND matching GP diagnosis. The 50.7% fallback is expected since not all HCD drugs are in DimSearchTerm.csv.
### Next iteration should:
- Work on Task 4.1: Full refresh with both chart types
  - Run `python -m cli.refresh_pathways --chart-type all` (no --dry-run)
  - This will insert ~1,846 indication nodes + ~1,800 directory nodes into pathway_nodes table
  - Verify both chart types generate data, directory charts unchanged
  - Takes ~15 minutes total (7s Snowflake + 6min transforms + 6min GP lookup + 2min pathways)
- After 4.1, Tasks 4.2 and 4.3 can be done together:
  - 4.2: Validate indication chart correctness (spot-check drug grouping)
  - 4.3: Validate Reflex UI compiles and chart type toggle works
### Blocked items:
- None


Good. I've read progress.txt, IMPLEMENTATION_PLAN.md, and guardrails.md. Now let me check recent git history.


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && git log --oneline -5

Show recent git commits


b674543 docs: update progress.txt with Iteration 6 results (Task 3.2)
c6e426e fix: increase network timeout and batch size for GP lookup queries (Task 3.2)
73088b0 docs: update progress.txt with Iteration 5 results (Task 3.1)
920570b feat: integrate drug-aware indication matching into refresh pipeline (Task 3.1)
d9891c8 docs: update progress.txt with Iteration 4 results (Task 2.1 + 2.2)


Now I have the full picture. Let me assess the state:

**Completed**: Phase 1 (1.1, 1.2, 1.3), Phase 2 (2.1, 2.2), Phase 3 (3.1, 3.2)
**Remaining**: Phase 4 ÔÇö Tasks 4.1, 4.2, 4.3

The previous iteration (6) recommended working on **Task 4.1: Full refresh with both chart types**. This requires running `python -m cli.refresh_pathways --chart-type all` (a live Snowflake refresh that takes ~15 minutes).

**Why Task 4.1 next**: It's the next logical step ÔÇö dry run passed, now we do the actual insert. Tasks 4.2 and 4.3 depend on 4.1 completing (they validate the results).

Let me mark Task 4.1 as in-progress in the implementation plan, then run the full refresh.


Edit

### 4.1 Full refresh with both chart types
- [ ] Run `python -m cli.refresh_pathways --chart-type all`

### 4.1 Full refresh with both chart types
- [~] Run `python -m cli.refresh_pathways --chart-type all`


The file C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\IMPLEMENTATION_PLAN.md has been updated successfully.

### 4.1 Full refresh with both chart types
- [~] Run `python -m cli.refresh_pathways --chart-type all`

# Implementation Plan - Drug-Aware Indication Matching

## Project Overview

Update the indication-based pathway charts so that patient indications are matched **per drug**, not just per patient. Currently, each patient gets ONE indication (most recent GP diagnosis match). This ignores which drugs the patient is actually taking.

### The Problem

A patient on ADALIMUMAB + OMALIZUMAB currently gets assigned a single indication (e.g., "rheumatoid arthritis" ÔÇö the most recent GP match). But:
- ADALIMUMAB is used for rheumatoid arthritis, axial spondyloarthritis, crohn's disease, etc.
- OMALIZUMAB is used for asthma, allergic asthma, urticaria

These are different clinical pathways and should be treated as separate treatment journeys.

### The Solution

Match each drug to an indication by cross-referencing:
1. **GP diagnosis** ÔÇö which Search_Terms the patient has matching SNOMED codes for
2. **Drug mapping** ÔÇö which Search_Terms list each drug (from `DimSearchTerm.csv`)

Only assign a drug to an indication if BOTH conditions are met. If a patient's drugs map to different indications, they become separate pathways (via modified UPID).

### Key Design Decisions

| Aspect | Decision |
|--------|----------|
| Drug-indication source | `data/DimSearchTerm.csv` ÔÇö Search_Term ÔåÆ CleanedDrugName mapping |
| UPID modification | `{original_UPID}\|{search_term}` for drugs with matched indication |
| GP diagnosis matching | Return ALL matches per patient (not just most recent) |
| Drug matching | Substring match: HCD drug name contains DimSearchTerm fragment |
| Multiple indication matches per drug | Use highest GP code frequency as tiebreaker (COUNT of matching SNOMED codes per Search_Term) |
| GP code time range | Only codes from MIN(Intervention Date) onwards ÔÇö restricts to HCD data window |
| No indication match | Fallback to directory (same as current behavior) |
| Same patient, different indications | Separate pathways via different modified UPIDs |

### Examples

**Patient on ADALIMUMAB + GOLIMUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis drug list includes both ADALIMUMAB and GOLIMUMAB
- ÔåÆ Both drugs grouped under "axial spondyloarthritis", single pathway
- Modified UPID: `RMV12345|axial spondyloarthritis`

**Patient on ADALIMUMAB + OMALIZUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis lists ADALIMUMAB but not OMALIZUMAB
- asthma lists OMALIZUMAB but not ADALIMUMAB
- ÔåÆ Two separate pathways:
  - `RMV12345|axial spondyloarthritis` with ADALIMUMAB
  - `RMV12345|asthma` with OMALIZUMAB

**Patient on ADALIMUMAB, GP dx: rheumatoid arthritis (47 codes) + crohn's disease (2 codes)**
- Both Search_Terms list ADALIMUMAB AND patient has GP dx for both
- ÔåÆ Tiebreaker: highest code frequency ÔÇö rheumatoid arthritis has 47 matching SNOMED codes vs 2 for crohn's
- ÔåÆ Single pathway under rheumatoid arthritis (more clinical activity = more likely the treatment indication)

---

## Phase 1: Update Snowflake Query & Drug Mapping

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [x] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [x] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [x] Keep batch processing (500 patients per query)
- [x] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses) *(requires live Snowflake ÔÇö will be verified in Phase 3/4)*

### 1.2 Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
- [x] In `CLUSTER_MAPPING_SQL` (diagnosis_lookup.py), merge these 3 Search_Terms into one `"asthma"` entry:
  - `allergic asthma` (Cluster: OMALIZUMAB only)
  - `asthma` (Cluster: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB)
  - `severe persistent allergic asthma` (Cluster: OMALIZUMAB only)
- [x] Map all 3 Cluster_IDs to `Search_Term = 'asthma'` in the CTE VALUES
- [x] `urticaria` (OMALIZUMAB, DERMATOLOGY) stays SEPARATE ÔÇö do NOT merge with asthma
- [x] Also update `load_drug_indication_mapping()` to apply the same merge when loading DimSearchTerm.csv:
  - Combine drug lists from all 3 entries under a single `"asthma"` key
  - Deduplicate drug fragments (OMALIZUMAB appears in all 3)
- [x] Verify: GP code lookup returns `"asthma"` (not `"allergic asthma"` or `"severe persistent allergic asthma"`)
- [x] Verify: Drug mapping for `"asthma"` includes full combined drug list: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB

### 1.3 Build drug-to-Search_Term lookup from DimSearchTerm.csv
- [x] Add function `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
  - Loads `data/DimSearchTerm.csv`
  - Builds dict: `drug_fragment (uppercase) ÔåÆ list[Search_Term]`
  - Also builds reverse: `search_term ÔåÆ list[drug_fragments]`
  - CleanedDrugName is pipe-separated (e.g., "ADALIMUMAB|GOLIMUMAB|IXEKIZUMAB")
- [x] Add function `get_search_terms_for_drug(drug_name, search_term_to_fragments) -> list[str]`:
  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
  - More practical than per-term boolean check ÔÇö returns all matches at once for Phase 2 use
- [x] Verify: ADALIMUMAB matches "axial spondyloarthritis", OMALIZUMAB matches "asthma"

---

## Phase 2: Drug-Aware Indication Matching Logic

### 2.1 Create `assign_drug_indications()` function
- [x] Add to `diagnosis_lookup.py` or `pathway_pipeline.py`:
  ```
  def assign_drug_indications(
      df: pd.DataFrame,              # HCD data with UPID, Drug Name columns
      gp_matches_df: pd.DataFrame,   # PatientPseudonym ÔåÆ list of matched Search_Terms
      drug_mapping: dict,             # From load_drug_indication_mapping()
  ) -> tuple[pd.DataFrame, pd.DataFrame]:
      Returns: (modified_df, indication_df)
      - modified_df: HCD data with UPID replaced by {UPID}|{indication}
      - indication_df: mapping modified_UPID ÔåÆ Search_Term
  ```
- [x] Logic per UPID + Drug Name pair:
  1. Get patient's GP-matched Search_Terms with code_frequency (from gp_matches_df via PseudoNHSNoLinked)
  2. Get which Search_Terms include this drug (from drug_mapping)
  3. Intersection = valid indications for this drug-patient pair
  4. If 1 match: use it
  5. If multiple matches: use highest code_frequency as tiebreaker (most GP coding activity = most likely treatment indication)
  6. If 0 matches: use fallback directory
- [x] Modify UPID in df rows: `{original_UPID}|{matched_search_term}`
- [x] Build indication_df: `{modified_UPID}` ÔåÆ `Search_Term` (or fallback label)
- [x] Verify: Function compiles, handles edge cases (no GP match, no drug match)

### 2.2 Handle tiebreaker for multiple indication matches
- [x] When a drug matches multiple Search_Terms AND patient has GP dx for multiple:
  - Use `code_frequency` from the GP query (COUNT of matching SNOMED codes per Search_Term)
  - Higher code_frequency = more clinical activity for that condition = more likely treatment indication
  - E.g., patient with 47 RA codes and 2 crohn's codes ÔåÆ ADALIMUMAB assigned to RA
  - code_frequency is already returned by the updated query in Task 1.1
- [x] Verify: Tiebreaker logic correctly picks highest-frequency diagnosis
- [x] Verify: Tie on frequency (rare but possible) falls back to alphabetical Search_Term for determinism

---

## Phase 3: Pipeline Integration

### 3.1 Update `refresh_pathways.py` indication processing
- [x] In the `elif current_chart_type == "indication":` block:
  1. Call `get_patient_indication_groups()` as before (but now returns ALL matches)
  2. Load drug mapping: `drug_mapping = load_drug_indication_mapping()`
  3. Call `assign_drug_indications(df, gp_matches_df, drug_mapping)`
  4. Use modified_df (with indication-aware UPIDs) for pathway processing
  5. Use indication_df for the indication mapping
- [x] Pass modified_df (not original df) to `process_indication_pathway_for_date_filter()`
- [x] Verify: Pipeline compiles, `python -m py_compile cli/refresh_pathways.py`

### 3.2 Test with dry run
- [x] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`
- [x] Verify:
  - Modified UPIDs appear in pipeline log (42,072 unique modified UPIDs)
  - Patient counts are reasonable (42,072 modified UPIDs vs 36,628 original patients)
  - Drug-indication matching is logged (49.3% match, 50.7% fallback, 15,238 tiebreakers)
  - Pathway hierarchy shows drug-specific grouping under correct indications (1,846 total nodes)
- [x] Fixed: network_timeout increased from 30ÔåÆ600 (was killing GP lookup queries)
- [x] Fixed: batch_size increased from 500ÔåÆ5000 (reduces CTE compilation overhead from 74 to 8 batches)

---

## Phase 4: Full Refresh & Validation

### 4.1 Full refresh with both chart types
- [ ] Run `python -m cli.refresh_pathways --chart-type all`
- [ ] Verify:
  - Both chart types generate data
  - Directory charts unchanged (no modified UPIDs)
  - Indication charts reflect drug-aware matching

### 4.2 Validate indication chart correctness
- [ ] Check that drugs under an indication all appear in that Search_Term's drug list
- [ ] Verify that a patient on drugs for different indications creates separate pathway branches
- [ ] Verify that drugs sharing an indication are grouped in the same pathway
- [ ] Log: patient count comparison (old vs new approach)

### 4.3 Validate Reflex UI
- [ ] Run `python -m reflex compile` to verify app compiles
- [ ] Verify chart type toggle still works
- [ ] Verify indication chart shows correct hierarchy

---

## Completion Criteria

All tasks marked `[x]` AND:
- [ ] App compiles without errors (`reflex compile` succeeds)
- [ ] Both chart types generate pathway data
- [ ] Indication charts show drug-specific indication matching
- [ ] Drugs under the same indication for the same patient are in one pathway
- [ ] Drugs under different indications for the same patient create separate pathways
- [ ] Fallback works for drugs with no indication match
- [ ] Full refresh completes successfully
- [ ] Existing directory charts are unaffected

---

## Reference

### DimSearchTerm.csv Structure
```
Search_Term,CleanedDrugName,PrimaryDirectorate
rheumatoid arthritis,ABATACEPT|ADALIMUMAB|ANAKINRA|BARICITINIB|...,RHEUMATOLOGY
asthma,BENRALIZUMAB|DUPILUMAB|INHALED|MEPOLIZUMAB|OMALIZUMAB|RESLIZUMAB,THORACIC MEDICINE
```

### Modified UPID Format
```
Original:  RMV12345
Modified:  RMV12345|rheumatoid arthritis
Fallback:  RMV12345|RHEUMATOLOGY (no GP dx)
```

### Current vs New Indication Flow
```
CURRENT:
  Patient ÔåÆ GP dx (most recent) ÔåÆ single Search_Term ÔåÆ one pathway

NEW:
  Patient + Drug A ÔåÆ GP dx matching Drug A ÔåÆ Search_Term X
  Patient + Drug B ÔåÆ GP dx matching Drug B ÔåÆ Search_Term Y
  ÔåÆ If X == Y: one pathway under X
  ÔåÆ If X != Y: two pathways (modified UPIDs)
```

### Key Files

| File | Changes |
|------|---------|
| `data_processing/diagnosis_lookup.py` | Update query, add drug mapping functions |
| `data_processing/pathway_pipeline.py` | Possibly minor changes for modified UPIDs |
| `cli/refresh_pathways.py` | Integrate drug-aware matching into pipeline |
| `data/DimSearchTerm.csv` | Reference data (read-only) |
| `analysis/pathway_analyzer.py` | No changes expected (UPID changes are transparent) |
| `pathways_app/pathways_app.py` | No changes expected |

-- [ ] Run `python -m cli.refresh_pathways --chart-type all`

+- [~] Run `python -m cli.refresh_pathways --chart-type all`


Running the full refresh now. This will fetch from Snowflake, transform data, run GP lookups, and insert both directory and indication pathway nodes into SQLite. Expected time: ~15 minutes.


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && python -m cli.refresh_pathways --chart-type all -v 2>&1

Full pathway refresh ÔÇö both chart types with verbose output


Command running in background with ID: bb39d2e. Output is being written to: C:\Users\CHARLW~1\AppData\Local\Temp\claude\C--Users-charlwoodand-Ralph-local-Tasks-UI-Redesign\tasks\bb39d2e.output


The refresh is running in the background. Let me check the output so far.


TodoWrite

Run full pathway refresh (--chart-type all)

Validate indication chart correctness (Task 4.2)

Validate Reflex UI compilation (Task 4.3)

Update progress.txt and commit


Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable

Run full pathway refresh (--chart-type all)

Validate indication chart correctness (Task 4.2)

Validate Reflex UI compilation (Task 4.3)

Update progress.txt and commit