C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\CLAUDE.md:218:- `get_patient_indication_groups(patient_pseudonyms)` - Batch queries Snowflake to match patients to GP diagnoses:
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\CLAUDE.md:300:           Ôöé    Ôöé   ÔåÆ get_patient_indication_groups()      Ôöé           Ôöé
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\cli\refresh_pathways.py:51:from data_processing.diagnosis_lookup import get_patient_indication_groups
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\cli\refresh_pathways.py:363:                # using the new Snowflake-direct approach via get_patient_indication_groups()
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\cli\refresh_pathways.py:393:                    gp_matches_df = get_patient_indication_groups(
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\guardrails.md:37:- **Rule**: Add `AND pc."EventDateTime" >= :earliest_hcd_date` where `earliest_hcd_date` is `MIN(Intervention Date)` from the HCD DataFrame. Pass this as a parameter to `get_patient_indication_groups()`.
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\IMPLEMENTATION_PLAN.md:59:### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\IMPLEMENTATION_PLAN.md:60:- [~] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\IMPLEMENTATION_PLAN.md:76:- [ ] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\IMPLEMENTATION_PLAN.md:147:  1. Call `get_patient_indication_groups()` as before (but now returns ALL matches)
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\diagnosis_lookup.py:1399:def get_patient_indication_groups(
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\diagnosis_lookup.py:1571:    "get_patient_indication_groups",
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\progress.txt:20:### Current get_patient_indication_groups() in diagnosis_lookup.py
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\progress.txt:49:  - Pass earliest_hcd_date as parameter to get_patient_indication_groups()
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\progress.txt:98:- OR work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\progress.txt:141:- Work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency


The main caller is in `cli/refresh_pathways.py`. Let me check how it uses the result to understand if the caller needs updating (though per the plan, caller changes are Task 3.1, not 1.1).


   355ÔåÆ                )
   356ÔåÆ                # Add results with chart_type suffix
   357ÔåÆ                for filter_id, records in dir_results.items():
   358ÔåÆ                    # Records already have chart_type set by convert_to_records
   359ÔåÆ                    results[f"{filter_id}:directory"] = records
   360ÔåÆ
   361ÔåÆ            elif current_chart_type == "indication":
   362ÔåÆ                # For indication charts, we need to look up GP diagnoses for all patients
   363ÔåÆ                # using the new Snowflake-direct approach via get_patient_indication_groups()
   364ÔåÆ                logger.info("Building indication groups from GP diagnosis lookups (Snowflake-direct)...")
   365ÔåÆ
   366ÔåÆ                # Check Snowflake availability
   367ÔåÆ                from data_processing.snowflake_connector import get_connector, is_snowflake_available
   368ÔåÆ
   369ÔåÆ                if not is_snowflake_available():
   370ÔåÆ                    logger.warning("Snowflake not available - cannot process indication charts")
   371ÔåÆ                    for config in DATE_FILTER_CONFIGS:
   372ÔåÆ                        results[f"{config.id}:indication"] = []
   373ÔåÆ                    continue
   374ÔåÆ
   375ÔåÆ                try:
   376ÔåÆ                    import pandas as pd
   377ÔåÆ                    connector = get_connector()
   378ÔåÆ
   379ÔåÆ                    # Step 1: Extract unique PseudoNHSNoLinked values from df
   380ÔåÆ                    # This is the patient identifier that matches PatientPseudonym in GP records
   381ÔåÆ                    if 'PseudoNHSNoLinked' not in df.columns:
   382ÔåÆ                        logger.error("DataFrame missing 'PseudoNHSNoLinked' column - cannot lookup GP records")
   383ÔåÆ                        for config in DATE_FILTER_CONFIGS:
   384ÔåÆ                            results[f"{config.id}:indication"] = []
   385ÔåÆ                        continue
   386ÔåÆ
   387ÔåÆ                    # Get unique patient pseudonyms for GP lookup (avoid redundant queries)
   388ÔåÆ                    patient_pseudonyms = df['PseudoNHSNoLinked'].dropna().unique().tolist()
   389ÔåÆ
   390ÔåÆ                    logger.info(f"Looking up GP diagnoses for {len(patient_pseudonyms)} unique patients...")
   391ÔåÆ
   392ÔåÆ                    # Step 2: Call the new Snowflake-direct indication lookup
   393ÔåÆ                    gp_matches_df = get_patient_indication_groups(
   394ÔåÆ                        patient_pseudonyms=patient_pseudonyms,
   395ÔåÆ                        connector=connector,
   396ÔåÆ                        batch_size=500,
   397ÔåÆ                    )
   398ÔåÆ
   399ÔåÆ                    # Step 3: Build indication_df mapping UPID -> Indication_Group
   400ÔåÆ                    # For matched patients: Indication_Group = Search_Term
   401ÔåÆ                    # For unmatched patients: Indication_Group = Directory + " (no GP dx)"
   402ÔåÆ                    #
   403ÔåÆ                    # IMPORTANT: We need ALL unique UPIDs, not just unique PseudoNHSNoLinked.
   404ÔåÆ                    # A patient can have multiple UPIDs if they visited multiple providers.
   405ÔåÆ
   406ÔåÆ                    # Get all unique UPID records with their PseudoNHSNoLinked and Directory
   407ÔåÆ                    upid_lookup = df[['UPID', 'PseudoNHSNoLinked', 'Directory']].drop_duplicates(
   408ÔåÆ                        subset=['UPID']
   409ÔåÆ                    ).copy()
   410ÔåÆ
   411ÔåÆ                    if gp_matches_df.empty:
   412ÔåÆ                        logger.warning("No GP matches found - all patients will use fallback directory")
   413ÔåÆ                        # All patients use fallback
   414ÔåÆ                        indication_records = []
   415ÔåÆ                        for _, row in upid_lookup.iterrows():
   416ÔåÆ                            directory = row['Directory']
   417ÔåÆ                            indication_records.append({
   418ÔåÆ                                'UPID': row['UPID'],
   419ÔåÆ                                'Indication_Group': str(directory) + " (no GP dx)" if pd.notna(directory) else "UNKNOWN (no GP dx)",
   420ÔåÆ                                'Source': 'FALLBACK',
   421ÔåÆ                            })
   422ÔåÆ                        indication_df = pd.DataFrame(indication_records)
   423ÔåÆ                    else:
   424ÔåÆ                        # Create lookup dict: PseudoNHSNoLinked -> Search_Term
   425ÔåÆ                        match_lookup = dict(zip(
   426ÔåÆ                            gp_matches_df['PatientPseudonym'],
   427ÔåÆ                            gp_matches_df['Search_Term']
   428ÔåÆ                        ))
   429ÔåÆ
   430ÔåÆ                        # Build indication records for each unique UPID
   431ÔåÆ                        indication_records = []
   432ÔåÆ                        for _, row in upid_lookup.iterrows():
   433ÔåÆ                            pseudo = row['PseudoNHSNoLinked']
   434ÔåÆ                            upid = row['UPID']
   435ÔåÆ                            directory = row['Directory']
   436ÔåÆ
   437ÔåÆ                            if pd.notna(pseudo) and pseudo in match_lookup:
   438ÔåÆ                                indication_records.append({
   439ÔåÆ                                    'UPID': upid,
   440ÔåÆ                                    'Indication_Group': match_lookup[pseudo],
   441ÔåÆ                                    'Source': 'DIAGNOSIS',
   442ÔåÆ                                })
   443ÔåÆ                            else:
   444ÔåÆ                                # Use fallback: Directory + " (no GP dx)"
   445ÔåÆ                                fallback_label = str(directory) + " (no GP dx)" if pd.notna(directory) else "UNKNOWN (no GP dx)"
   446ÔåÆ                                indication_records.append({
   447ÔåÆ                                    'UPID': upid,
   448ÔåÆ                                    'Indication_Group': fallback_label,
   449ÔåÆ                                    'Source': 'FALLBACK',
   450ÔåÆ                                })
   451ÔåÆ
   452ÔåÆ                        indication_df = pd.DataFrame(indication_records)
   453ÔåÆ
   454ÔåÆ                    # Log coverage statistics

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

                )
                # Add results with chart_type suffix
                for filter_id, records in dir_results.items():
                    # Records already have chart_type set by convert_to_records
                    results[f"{filter_id}:directory"] = records

            elif current_chart_type == "indication":
                # For indication charts, we need to look up GP diagnoses for all patients
                # using the new Snowflake-direct approach via get_patient_indication_groups()
                logger.info("Building indication groups from GP diagnosis lookups (Snowflake-direct)...")

                # Check Snowflake availability
                from data_processing.snowflake_connector import get_connector, is_snowflake_available

                if not is_snowflake_available():
                    logger.warning("Snowflake not available - cannot process indication charts")
                    for config in DATE_FILTER_CONFIGS:
                        results[f"{config.id}:indication"] = []
                    continue

                try:
                    import pandas as pd
                    connector = get_connector()

                    # Step 1: Extract unique PseudoNHSNoLinked values from df
                    # This is the patient identifier that matches PatientPseudonym in GP records
                    if 'PseudoNHSNoLinked' not in df.columns:
                        logger.error("DataFrame missing 'PseudoNHSNoLinked' column - cannot lookup GP records")
                        for config in DATE_FILTER_CONFIGS:
                            results[f"{config.id}:indication"] = []
                        continue

                    # Get unique patient pseudonyms for GP lookup (avoid redundant queries)
                    patient_pseudonyms = df['PseudoNHSNoLinked'].dropna().unique().tolist()

                    logger.info(f"Looking up GP diagnoses for {len(patient_pseudonyms)} unique patients...")

                    # Step 2: Call the new Snowflake-direct indication lookup
                    gp_matches_df = get_patient_indication_groups(
                        patient_pseudonyms=patient_pseudonyms,
                        connector=connector,
                        batch_size=500,
                    )

                    # Step 3: Build indication_df mapping UPID -> Indication_Group
                    # For matched patients: Indication_Group = Search_Term
                    # For unmatched patients: Indication_Group = Directory + " (no GP dx)"
                    #
                    # IMPORTANT: We need ALL unique UPIDs, not just unique PseudoNHSNoLinked.
                    # A patient can have multiple UPIDs if they visited multiple providers.

                    # Get all unique UPID records with their PseudoNHSNoLinked and Directory
                    upid_lookup = df[['UPID', 'PseudoNHSNoLinked', 'Directory']].drop_duplicates(
                        subset=['UPID']
                    ).copy()

                    if gp_matches_df.empty:
                        logger.warning("No GP matches found - all patients will use fallback directory")
                        # All patients use fallback
                        indication_records = []
                        for _, row in upid_lookup.iterrows():
                            directory = row['Directory']
                            indication_records.append({
                                'UPID': row['UPID'],
                                'Indication_Group': str(directory) + " (no GP dx)" if pd.notna(directory) else "UNKNOWN (no GP dx)",
                                'Source': 'FALLBACK',
                            })
                        indication_df = pd.DataFrame(indication_records)
                    else:
                        # Create lookup dict: PseudoNHSNoLinked -> Search_Term
                        match_lookup = dict(zip(
                            gp_matches_df['PatientPseudonym'],
                            gp_matches_df['Search_Term']
                        ))

                        # Build indication records for each unique UPID
                        indication_records = []
                        for _, row in upid_lookup.iterrows():
                            pseudo = row['PseudoNHSNoLinked']
                            upid = row['UPID']
                            directory = row['Directory']

                            if pd.notna(pseudo) and pseudo in match_lookup:
                                indication_records.append({
                                    'UPID': upid,
                                    'Indication_Group': match_lookup[pseudo],
                                    'Source': 'DIAGNOSIS',
                                })
                            else:
                                # Use fallback: Directory + " (no GP dx)"
                                fallback_label = str(directory) + " (no GP dx)" if pd.notna(directory) else "UNKNOWN (no GP dx)"
                                indication_records.append({
                                    'UPID': upid,
                                    'Indication_Group': fallback_label,
                                    'Source': 'FALLBACK',
                                })

                        indication_df = pd.DataFrame(indication_records)

                    # Log coverage statistics


I can see the caller at line 424-428 does `dict(zip(gp_matches_df['PatientPseudonym'], gp_matches_df['Search_Term']))`. This creates a 1:1 mapping ÔÇö with the old query this was fine (one row per patient), but with the new query returning multiple rows per patient, `dict(zip(...))` would only keep the last match.

However, this is the **current** caller code and updating it is **Task 3.1** (Pipeline Integration), not Task 1.1. Task 1.1 only covers updating the function itself. The caller changes will be done in a future iteration when we integrate `assign_drug_indications()`.

The key thing is that the function change is backward-compatible ÔÇö the new parameter `earliest_hcd_date` defaults to `None`, so the existing caller still works. The return format change (multiple rows per patient, `code_frequency` instead of `EventDateTime`) means the caller needs updating, but that's explicitly Task 3.1.

Now let me also verify the empty DataFrame columns match the new return type throughout.


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && python -c "
from data_processing.diagnosis_lookup import get_patient_indication_groups
# Test with empty list - should return DataFrame with correct columns
result = get_patient_indication_groups([])
print(f'Columns: {list(result.columns)}')
print(f'Empty: {result.empty}')
print(f'Shape: {result.shape}')
assert list(result.columns) == ['PatientPseudonym', 'Search_Term', 'code_frequency'], f'Wrong columns: {list(result.columns)}'
print('Empty DataFrame column check PASSED')
"

Verify empty DataFrame has correct columns


Columns: ['PatientPseudonym', 'Search_Term', 'code_frequency']
Empty: True
Shape: (0, 3)
Empty DataFrame column check PASSED
Empty patient list provided


All validation passes. Let me update the IMPLEMENTATION_PLAN.md to mark Task 1.1 subtasks complete.


TodoWrite

Update get_patient_indication_groups() to return ALL matches with code_frequency (Task 1.1)

Updating get_patient_indication_groups() Snowflake query

Validate code compiles and imports work

Mark task complete in IMPLEMENTATION_PLAN.md

Update progress.txt with iteration results

Commit changes


Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable

Update get_patient_indication_groups() to return ALL matches with code_frequency (Task 1.1)

Updating get_patient_indication_groups() Snowflake query

Validate code compiles and imports work

Mark task complete in IMPLEMENTATION_PLAN.md

Update progress.txt with iteration results

Commit changes

Validate code compiles and imports work

Commit changes


    58ÔåÆ
    59ÔåÆ### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
    60ÔåÆ- [~] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
    61ÔåÆ  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
    62ÔåÆ  - Return ALL matching Search_Terms per patient with code frequency:
    63ÔåÆ    ```sql
    64ÔåÆ    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
    65ÔåÆ           aic.Search_Term AS "Search_Term",
    66ÔåÆ           COUNT(*) AS "code_frequency"
    67ÔåÆ    FROM PrimaryCareClinicalCoding pc
    68ÔåÆ    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    69ÔåÆ    WHERE pc."PatientPseudonym" IN (...)
    70ÔåÆ      AND pc."EventDateTime" >= :earliest_hcd_date
    71ÔåÆ    GROUP BY pc."PatientPseudonym", aic.Search_Term
    72ÔåÆ    ```
    73ÔåÆ  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
    74ÔåÆ  - Higher frequency = more clinical activity = stronger signal for tiebreaker
    75ÔåÆ  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
    76ÔåÆ- [ ] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
    77ÔåÆ- [ ] Keep batch processing (500 patients per query)
    78ÔåÆ- [ ] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
    79ÔåÆ- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [~] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [ ] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [ ] Keep batch processing (500 patients per query)
- [ ] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)


Edit

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [~] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [ ] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [ ] Keep batch processing (500 patients per query)
- [ ] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [x] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [x] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [x] Keep batch processing (500 patients per query)
- [x] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)


The file C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\IMPLEMENTATION_PLAN.md has been updated successfully.

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [x] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [x] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [x] Keep batch processing (500 patients per query)
- [x] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)

# Implementation Plan - Drug-Aware Indication Matching

## Project Overview

Update the indication-based pathway charts so that patient indications are matched **per drug**, not just per patient. Currently, each patient gets ONE indication (most recent GP diagnosis match). This ignores which drugs the patient is actually taking.

### The Problem

A patient on ADALIMUMAB + OMALIZUMAB currently gets assigned a single indication (e.g., "rheumatoid arthritis" ÔÇö the most recent GP match). But:
- ADALIMUMAB is used for rheumatoid arthritis, axial spondyloarthritis, crohn's disease, etc.
- OMALIZUMAB is used for asthma, allergic asthma, urticaria

These are different clinical pathways and should be treated as separate treatment journeys.

### The Solution

Match each drug to an indication by cross-referencing:
1. **GP diagnosis** ÔÇö which Search_Terms the patient has matching SNOMED codes for
2. **Drug mapping** ÔÇö which Search_Terms list each drug (from `DimSearchTerm.csv`)

Only assign a drug to an indication if BOTH conditions are met. If a patient's drugs map to different indications, they become separate pathways (via modified UPID).

### Key Design Decisions

| Aspect | Decision |
|--------|----------|
| Drug-indication source | `data/DimSearchTerm.csv` ÔÇö Search_Term ÔåÆ CleanedDrugName mapping |
| UPID modification | `{original_UPID}\|{search_term}` for drugs with matched indication |
| GP diagnosis matching | Return ALL matches per patient (not just most recent) |
| Drug matching | Substring match: HCD drug name contains DimSearchTerm fragment |
| Multiple indication matches per drug | Use highest GP code frequency as tiebreaker (COUNT of matching SNOMED codes per Search_Term) |
| GP code time range | Only codes from MIN(Intervention Date) onwards ÔÇö restricts to HCD data window |
| No indication match | Fallback to directory (same as current behavior) |
| Same patient, different indications | Separate pathways via different modified UPIDs |

### Examples

**Patient on ADALIMUMAB + GOLIMUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis drug list includes both ADALIMUMAB and GOLIMUMAB
- ÔåÆ Both drugs grouped under "axial spondyloarthritis", single pathway
- Modified UPID: `RMV12345|axial spondyloarthritis`

**Patient on ADALIMUMAB + OMALIZUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis lists ADALIMUMAB but not OMALIZUMAB
- asthma lists OMALIZUMAB but not ADALIMUMAB
- ÔåÆ Two separate pathways:
  - `RMV12345|axial spondyloarthritis` with ADALIMUMAB
  - `RMV12345|asthma` with OMALIZUMAB

**Patient on ADALIMUMAB, GP dx: rheumatoid arthritis (47 codes) + crohn's disease (2 codes)**
- Both Search_Terms list ADALIMUMAB AND patient has GP dx for both
- ÔåÆ Tiebreaker: highest code frequency ÔÇö rheumatoid arthritis has 47 matching SNOMED codes vs 2 for crohn's
- ÔåÆ Single pathway under rheumatoid arthritis (more clinical activity = more likely the treatment indication)

---

## Phase 1: Update Snowflake Query & Drug Mapping

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [~] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [ ] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [ ] Keep batch processing (500 patients per query)
- [ ] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)

### 1.2 Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
- [x] In `CLUSTER_MAPPING_SQL` (diagnosis_lookup.py), merge these 3 Search_Terms into one `"asthma"` entry:
  - `allergic asthma` (Cluster: OMALIZUMAB only)
  - `asthma` (Cluster: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB)
  - `severe persistent allergic asthma` (Cluster: OMALIZUMAB only)
- [x] Map all 3 Cluster_IDs to `Search_Term = 'asthma'` in the CTE VALUES
- [x] `urticaria` (OMALIZUMAB, DERMATOLOGY) stays SEPARATE ÔÇö do NOT merge with asthma
- [x] Also update `load_drug_indication_mapping()` to apply the same merge when loading DimSearchTerm.csv:
  - Combine drug lists from all 3 entries under a single `"asthma"` key
  - Deduplicate drug fragments (OMALIZUMAB appears in all 3)
- [x] Verify: GP code lookup returns `"asthma"` (not `"allergic asthma"` or `"severe persistent allergic asthma"`)
- [x] Verify: Drug mapping for `"asthma"` includes full combined drug list: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB

### 1.3 Build drug-to-Search_Term lookup from DimSearchTerm.csv
- [x] Add function `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
  - Loads `data/DimSearchTerm.csv`
  - Builds dict: `drug_fragment (uppercase) ÔåÆ list[Search_Term]`
  - Also builds reverse: `search_term ÔåÆ list[drug_fragments]`
  - CleanedDrugName is pipe-separated (e.g., "ADALIMUMAB|GOLIMUMAB|IXEKIZUMAB")
- [x] Add function `get_search_terms_for_drug(drug_name, search_term_to_fragments) -> list[str]`:
  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
  - More practical than per-term boolean check ÔÇö returns all matches at once for Phase 2 use
- [x] Verify: ADALIMUMAB matches "axial spondyloarthritis", OMALIZUMAB matches "asthma"

---

## Phase 2: Drug-Aware Indication Matching Logic

### 2.1 Create `assign_drug_indications()` function
- [ ] Add to `diagnosis_lookup.py` or `pathway_pipeline.py`:
  ```
  def assign_drug_indications(
      df: pd.DataFrame,              # HCD data with UPID, Drug Name columns
      gp_matches_df: pd.DataFrame,   # PatientPseudonym ÔåÆ list of matched Search_Terms
      drug_mapping: dict,             # From load_drug_indication_mapping()
  ) -> tuple[pd.DataFrame, pd.DataFrame]:
      Returns: (modified_df, indication_df)
      - modified_df: HCD data with UPID replaced by {UPID}|{indication}
      - indication_df: mapping modified_UPID ÔåÆ Search_Term
  ```
- [ ] Logic per UPID + Drug Name pair:
  1. Get patient's GP-matched Search_Terms with code_frequency (from gp_matches_df via PseudoNHSNoLinked)
  2. Get which Search_Terms include this drug (from drug_mapping)
  3. Intersection = valid indications for this drug-patient pair
  4. If 1 match: use it
  5. If multiple matches: use highest code_frequency as tiebreaker (most GP coding activity = most likely treatment indication)
  6. If 0 matches: use fallback directory
- [ ] Modify UPID in df rows: `{original_UPID}|{matched_search_term}`
- [ ] Build indication_df: `{modified_UPID}` ÔåÆ `Search_Term` (or fallback label)
- [ ] Verify: Function compiles, handles edge cases (no GP match, no drug match)

### 2.2 Handle tiebreaker for multiple indication matches
- [ ] When a drug matches multiple Search_Terms AND patient has GP dx for multiple:
  - Use `code_frequency` from the GP query (COUNT of matching SNOMED codes per Search_Term)
  - Higher code_frequency = more clinical activity for that condition = more likely treatment indication
  - E.g., patient with 47 RA codes and 2 crohn's codes ÔåÆ ADALIMUMAB assigned to RA
  - code_frequency is already returned by the updated query in Task 1.1
- [ ] Verify: Tiebreaker logic correctly picks highest-frequency diagnosis
- [ ] Verify: Tie on frequency (rare but possible) falls back to alphabetical Search_Term for determinism

---

## Phase 3: Pipeline Integration

### 3.1 Update `refresh_pathways.py` indication processing
- [ ] In the `elif current_chart_type == "indication":` block:
  1. Call `get_patient_indication_groups()` as before (but now returns ALL matches)
  2. Load drug mapping: `drug_mapping = load_drug_indication_mapping()`
  3. Call `assign_drug_indications(df, gp_matches_df, drug_mapping)`
  4. Use modified_df (with indication-aware UPIDs) for pathway processing
  5. Use indication_df for the indication mapping
- [ ] Pass modified_df (not original df) to `process_indication_pathway_for_date_filter()`
- [ ] Verify: Pipeline compiles, `python -m py_compile cli/refresh_pathways.py`

### 3.2 Test with dry run
- [ ] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`
- [ ] Verify:
  - Modified UPIDs appear in pipeline log (e.g., `RMV12345|rheumatoid arthritis`)
  - Patient counts are reasonable (will be higher than before since same patient can appear under multiple indications)
  - Drug-indication matching is logged (match rate, fallback rate)
  - Pathway hierarchy shows drug-specific grouping under correct indications

---

## Phase 4: Full Refresh & Validation

### 4.1 Full refresh with both chart types
- [ ] Run `python -m cli.refresh_pathways --chart-type all`
- [ ] Verify:
  - Both chart types generate data
  - Directory charts unchanged (no modified UPIDs)
  - Indication charts reflect drug-aware matching

### 4.2 Validate indication chart correctness
- [ ] Check that drugs under an indication all appear in that Search_Term's drug list
- [ ] Verify that a patient on drugs for different indications creates separate pathway branches
- [ ] Verify that drugs sharing an indication are grouped in the same pathway
- [ ] Log: patient count comparison (old vs new approach)

### 4.3 Validate Reflex UI
- [ ] Run `python -m reflex compile` to verify app compiles
- [ ] Verify chart type toggle still works
- [ ] Verify indication chart shows correct hierarchy

---

## Completion Criteria

All tasks marked `[x]` AND:
- [ ] App compiles without errors (`reflex compile` succeeds)
- [ ] Both chart types generate pathway data
- [ ] Indication charts show drug-specific indication matching
- [ ] Drugs under the same indication for the same patient are in one pathway
- [ ] Drugs under different indications for the same patient create separate pathways
- [ ] Fallback works for drugs with no indication match
- [ ] Full refresh completes successfully
- [ ] Existing directory charts are unaffected

---

## Reference

### DimSearchTerm.csv Structure
```
Search_Term,CleanedDrugName,PrimaryDirectorate
rheumatoid arthritis,ABATACEPT|ADALIMUMAB|ANAKINRA|BARICITINIB|...,RHEUMATOLOGY
asthma,BENRALIZUMAB|DUPILUMAB|INHALED|MEPOLIZUMAB|OMALIZUMAB|RESLIZUMAB,THORACIC MEDICINE
```

### Modified UPID Format
```
Original:  RMV12345
Modified:  RMV12345|rheumatoid arthritis
Fallback:  RMV12345|RHEUMATOLOGY (no GP dx)
```

### Current vs New Indication Flow
```
CURRENT:
  Patient ÔåÆ GP dx (most recent) ÔåÆ single Search_Term ÔåÆ one pathway

NEW:
  Patient + Drug A ÔåÆ GP dx matching Drug A ÔåÆ Search_Term X
  Patient + Drug B ÔåÆ GP dx matching Drug B ÔåÆ Search_Term Y
  ÔåÆ If X == Y: one pathway under X
  ÔåÆ If X != Y: two pathways (modified UPIDs)
```

### Key Files

| File | Changes |
|------|---------|
| `data_processing/diagnosis_lookup.py` | Update query, add drug mapping functions |
| `data_processing/pathway_pipeline.py` | Possibly minor changes for modified UPIDs |
| `cli/refresh_pathways.py` | Integrate drug-aware matching into pipeline |
| `data/DimSearchTerm.csv` | Reference data (read-only) |
| `analysis/pathway_analyzer.py` | No changes expected (UPID changes are transparent) |
| `pathways_app/pathways_app.py` | No changes expected |

-- [~] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):

+- [x] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):

-- [ ] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query

-- [ ] Keep batch processing (500 patients per query)

-- [ ] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)

+- [x] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query

+- [x] Keep batch processing (500 patients per query)

+- [x] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)


The last subtask "Verify: Query returns more rows than before" requires a live Snowflake connection ÔÇö that will be verified during Phase 3/4 integration testing. The code change itself is complete and validated locally. Let me note this in the implementation plan.


- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)

- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses) *(requires live Snowflake ÔÇö will be verified in Phase 3/4)*


# Implementation Plan - Drug-Aware Indication Matching

## Project Overview

Update the indication-based pathway charts so that patient indications are matched **per drug**, not just per patient. Currently, each patient gets ONE indication (most recent GP diagnosis match). This ignores which drugs the patient is actually taking.

### The Problem

A patient on ADALIMUMAB + OMALIZUMAB currently gets assigned a single indication (e.g., "rheumatoid arthritis" ÔÇö the most recent GP match). But:
- ADALIMUMAB is used for rheumatoid arthritis, axial spondyloarthritis, crohn's disease, etc.
- OMALIZUMAB is used for asthma, allergic asthma, urticaria

These are different clinical pathways and should be treated as separate treatment journeys.

### The Solution

Match each drug to an indication by cross-referencing:
1. **GP diagnosis** ÔÇö which Search_Terms the patient has matching SNOMED codes for
2. **Drug mapping** ÔÇö which Search_Terms list each drug (from `DimSearchTerm.csv`)

Only assign a drug to an indication if BOTH conditions are met. If a patient's drugs map to different indications, they become separate pathways (via modified UPID).

### Key Design Decisions

| Aspect | Decision |
|--------|----------|
| Drug-indication source | `data/DimSearchTerm.csv` ÔÇö Search_Term ÔåÆ CleanedDrugName mapping |
| UPID modification | `{original_UPID}\|{search_term}` for drugs with matched indication |
| GP diagnosis matching | Return ALL matches per patient (not just most recent) |
| Drug matching | Substring match: HCD drug name contains DimSearchTerm fragment |
| Multiple indication matches per drug | Use highest GP code frequency as tiebreaker (COUNT of matching SNOMED codes per Search_Term) |
| GP code time range | Only codes from MIN(Intervention Date) onwards ÔÇö restricts to HCD data window |
| No indication match | Fallback to directory (same as current behavior) |
| Same patient, different indications | Separate pathways via different modified UPIDs |

### Examples

**Patient on ADALIMUMAB + GOLIMUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis drug list includes both ADALIMUMAB and GOLIMUMAB
- ÔåÆ Both drugs grouped under "axial spondyloarthritis", single pathway
- Modified UPID: `RMV12345|axial spondyloarthritis`

**Patient on ADALIMUMAB + OMALIZUMAB, GP dx: axial spondyloarthritis + asthma**
- axial spondyloarthritis lists ADALIMUMAB but not OMALIZUMAB
- asthma lists OMALIZUMAB but not ADALIMUMAB
- ÔåÆ Two separate pathways:
  - `RMV12345|axial spondyloarthritis` with ADALIMUMAB
  - `RMV12345|asthma` with OMALIZUMAB

**Patient on ADALIMUMAB, GP dx: rheumatoid arthritis (47 codes) + crohn's disease (2 codes)**
- Both Search_Terms list ADALIMUMAB AND patient has GP dx for both
- ÔåÆ Tiebreaker: highest code frequency ÔÇö rheumatoid arthritis has 47 matching SNOMED codes vs 2 for crohn's
- ÔåÆ Single pathway under rheumatoid arthritis (more clinical activity = more likely the treatment indication)

---

## Phase 1: Update Snowflake Query & Drug Mapping

### 1.1 Update `get_patient_indication_groups()` to return ALL matches with frequency
- [x] Modify the Snowflake query in `get_patient_indication_groups()` (diagnosis_lookup.py):
  - Remove `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1`
  - Return ALL matching Search_Terms per patient with code frequency:
    ```sql
    SELECT pc."PatientPseudonym" AS "PatientPseudonym",
           aic.Search_Term AS "Search_Term",
           COUNT(*) AS "code_frequency"
    FROM PrimaryCareClinicalCoding pc
    JOIN AllIndicationCodes aic ON pc."SNOMEDCode" = aic.SNOMEDCode
    WHERE pc."PatientPseudonym" IN (...)
      AND pc."EventDateTime" >= :earliest_hcd_date
    GROUP BY pc."PatientPseudonym", aic.Search_Term
    ```
  - `code_frequency` = number of matching SNOMED codes per Search_Term per patient
  - Higher frequency = more clinical activity = stronger signal for tiebreaker
  - `earliest_hcd_date` = `MIN(Intervention Date)` from the HCD DataFrame ÔÇö restricts GP codes to the HCD data window, reducing noise from old/irrelevant diagnoses
- [x] Accept `earliest_hcd_date` parameter in `get_patient_indication_groups()` and pass to query
- [x] Keep batch processing (500 patients per query)
- [x] Update return type: DataFrame now has multiple rows per patient (PatientPseudonym, Search_Term, code_frequency)
- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)

### 1.2 Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
- [x] In `CLUSTER_MAPPING_SQL` (diagnosis_lookup.py), merge these 3 Search_Terms into one `"asthma"` entry:
  - `allergic asthma` (Cluster: OMALIZUMAB only)
  - `asthma` (Cluster: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB)
  - `severe persistent allergic asthma` (Cluster: OMALIZUMAB only)
- [x] Map all 3 Cluster_IDs to `Search_Term = 'asthma'` in the CTE VALUES
- [x] `urticaria` (OMALIZUMAB, DERMATOLOGY) stays SEPARATE ÔÇö do NOT merge with asthma
- [x] Also update `load_drug_indication_mapping()` to apply the same merge when loading DimSearchTerm.csv:
  - Combine drug lists from all 3 entries under a single `"asthma"` key
  - Deduplicate drug fragments (OMALIZUMAB appears in all 3)
- [x] Verify: GP code lookup returns `"asthma"` (not `"allergic asthma"` or `"severe persistent allergic asthma"`)
- [x] Verify: Drug mapping for `"asthma"` includes full combined drug list: BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, OMALIZUMAB, RESLIZUMAB

### 1.3 Build drug-to-Search_Term lookup from DimSearchTerm.csv
- [x] Add function `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
  - Loads `data/DimSearchTerm.csv`
  - Builds dict: `drug_fragment (uppercase) ÔåÆ list[Search_Term]`
  - Also builds reverse: `search_term ÔåÆ list[drug_fragments]`
  - CleanedDrugName is pipe-separated (e.g., "ADALIMUMAB|GOLIMUMAB|IXEKIZUMAB")
- [x] Add function `get_search_terms_for_drug(drug_name, search_term_to_fragments) -> list[str]`:
  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
  - More practical than per-term boolean check ÔÇö returns all matches at once for Phase 2 use
- [x] Verify: ADALIMUMAB matches "axial spondyloarthritis", OMALIZUMAB matches "asthma"

---

## Phase 2: Drug-Aware Indication Matching Logic

### 2.1 Create `assign_drug_indications()` function
- [ ] Add to `diagnosis_lookup.py` or `pathway_pipeline.py`:
  ```
  def assign_drug_indications(
      df: pd.DataFrame,              # HCD data with UPID, Drug Name columns
      gp_matches_df: pd.DataFrame,   # PatientPseudonym ÔåÆ list of matched Search_Terms
      drug_mapping: dict,             # From load_drug_indication_mapping()
  ) -> tuple[pd.DataFrame, pd.DataFrame]:
      Returns: (modified_df, indication_df)
      - modified_df: HCD data with UPID replaced by {UPID}|{indication}
      - indication_df: mapping modified_UPID ÔåÆ Search_Term
  ```
- [ ] Logic per UPID + Drug Name pair:
  1. Get patient's GP-matched Search_Terms with code_frequency (from gp_matches_df via PseudoNHSNoLinked)
  2. Get which Search_Terms include this drug (from drug_mapping)
  3. Intersection = valid indications for this drug-patient pair
  4. If 1 match: use it
  5. If multiple matches: use highest code_frequency as tiebreaker (most GP coding activity = most likely treatment indication)
  6. If 0 matches: use fallback directory
- [ ] Modify UPID in df rows: `{original_UPID}|{matched_search_term}`
- [ ] Build indication_df: `{modified_UPID}` ÔåÆ `Search_Term` (or fallback label)
- [ ] Verify: Function compiles, handles edge cases (no GP match, no drug match)

### 2.2 Handle tiebreaker for multiple indication matches
- [ ] When a drug matches multiple Search_Terms AND patient has GP dx for multiple:
  - Use `code_frequency` from the GP query (COUNT of matching SNOMED codes per Search_Term)
  - Higher code_frequency = more clinical activity for that condition = more likely treatment indication
  - E.g., patient with 47 RA codes and 2 crohn's codes ÔåÆ ADALIMUMAB assigned to RA
  - code_frequency is already returned by the updated query in Task 1.1
- [ ] Verify: Tiebreaker logic correctly picks highest-frequency diagnosis
- [ ] Verify: Tie on frequency (rare but possible) falls back to alphabetical Search_Term for determinism

---

## Phase 3: Pipeline Integration

### 3.1 Update `refresh_pathways.py` indication processing
- [ ] In the `elif current_chart_type == "indication":` block:
  1. Call `get_patient_indication_groups()` as before (but now returns ALL matches)
  2. Load drug mapping: `drug_mapping = load_drug_indication_mapping()`
  3. Call `assign_drug_indications(df, gp_matches_df, drug_mapping)`
  4. Use modified_df (with indication-aware UPIDs) for pathway processing
  5. Use indication_df for the indication mapping
- [ ] Pass modified_df (not original df) to `process_indication_pathway_for_date_filter()`
- [ ] Verify: Pipeline compiles, `python -m py_compile cli/refresh_pathways.py`

### 3.2 Test with dry run
- [ ] Run `python -m cli.refresh_pathways --chart-type indication --dry-run -v`
- [ ] Verify:
  - Modified UPIDs appear in pipeline log (e.g., `RMV12345|rheumatoid arthritis`)
  - Patient counts are reasonable (will be higher than before since same patient can appear under multiple indications)
  - Drug-indication matching is logged (match rate, fallback rate)
  - Pathway hierarchy shows drug-specific grouping under correct indications

---

## Phase 4: Full Refresh & Validation

### 4.1 Full refresh with both chart types
- [ ] Run `python -m cli.refresh_pathways --chart-type all`
- [ ] Verify:
  - Both chart types generate data
  - Directory charts unchanged (no modified UPIDs)
  - Indication charts reflect drug-aware matching

### 4.2 Validate indication chart correctness
- [ ] Check that drugs under an indication all appear in that Search_Term's drug list
- [ ] Verify that a patient on drugs for different indications creates separate pathway branches
- [ ] Verify that drugs sharing an indication are grouped in the same pathway
- [ ] Log: patient count comparison (old vs new approach)

### 4.3 Validate Reflex UI
- [ ] Run `python -m reflex compile` to verify app compiles
- [ ] Verify chart type toggle still works
- [ ] Verify indication chart shows correct hierarchy

---

## Completion Criteria

All tasks marked `[x]` AND:
- [ ] App compiles without errors (`reflex compile` succeeds)
- [ ] Both chart types generate pathway data
- [ ] Indication charts show drug-specific indication matching
- [ ] Drugs under the same indication for the same patient are in one pathway
- [ ] Drugs under different indications for the same patient create separate pathways
- [ ] Fallback works for drugs with no indication match
- [ ] Full refresh completes successfully
- [ ] Existing directory charts are unaffected

---

## Reference

### DimSearchTerm.csv Structure
```
Search_Term,CleanedDrugName,PrimaryDirectorate
rheumatoid arthritis,ABATACEPT|ADALIMUMAB|ANAKINRA|BARICITINIB|...,RHEUMATOLOGY
asthma,BENRALIZUMAB|DUPILUMAB|INHALED|MEPOLIZUMAB|OMALIZUMAB|RESLIZUMAB,THORACIC MEDICINE
```

### Modified UPID Format
```
Original:  RMV12345
Modified:  RMV12345|rheumatoid arthritis
Fallback:  RMV12345|RHEUMATOLOGY (no GP dx)
```

### Current vs New Indication Flow
```
CURRENT:
  Patient ÔåÆ GP dx (most recent) ÔåÆ single Search_Term ÔåÆ one pathway

NEW:
  Patient + Drug A ÔåÆ GP dx matching Drug A ÔåÆ Search_Term X
  Patient + Drug B ÔåÆ GP dx matching Drug B ÔåÆ Search_Term Y
  ÔåÆ If X == Y: one pathway under X
  ÔåÆ If X != Y: two pathways (modified UPIDs)
```

### Key Files

| File | Changes |
|------|---------|
| `data_processing/diagnosis_lookup.py` | Update query, add drug mapping functions |
| `data_processing/pathway_pipeline.py` | Possibly minor changes for modified UPIDs |
| `cli/refresh_pathways.py` | Integrate drug-aware matching into pipeline |
| `data/DimSearchTerm.csv` | Reference data (read-only) |
| `analysis/pathway_analyzer.py` | No changes expected (UPID changes are transparent) |
| `pathways_app/pathways_app.py` | No changes expected |

-- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses)

+- [ ] Verify: Query returns more rows than before (patients with multiple matching diagnoses) *(requires live Snowflake ÔÇö will be verified in Phase 3/4)*


Now let me update progress.txt with the iteration results.


TodoWrite

Update get_patient_indication_groups() to return ALL matches with code_frequency (Task 1.1)

Updating get_patient_indication_groups() Snowflake query

Validate code compiles and imports work

Mark task complete in IMPLEMENTATION_PLAN.md

Update progress.txt with iteration results

Commit changes


Validate code compiles and imports work

Commit changes

Validate code compiles and imports work

Commit changes


Edit

### Blocked items:
- None

## Iteration 3 ÔÇö 2026-02-05
### Task: 1.1 ÔÇö Update get_patient_indication_groups() to return ALL matches with code_frequency
### Why this task:
- Previous iteration recommended this as the next task
- This is the last remaining Phase 1 dependency ÔÇö Tasks 2.1/2.2 need the GP matches data in the right format (multiple rows per patient with code_frequency)
- Task 1.1 changes the query fundamentals; better to do this before building assign_drug_indications() which consumes the output
### Status: COMPLETE
### What was done:
- Modified `get_patient_indication_groups()` in `diagnosis_lookup.py`:
  1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"`
  3. **Added** `earliest_hcd_date: Optional[str] = None` parameter
  4. **Added** optional `AND pc."EventDateTime" >= %s` when earliest_hcd_date is provided
  5. **Updated** return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`
  6. **Updated** all empty DataFrame returns to use new column names
  7. **Updated** logging to show multiple-rows-per-patient stats (avg indications per patient)
  8. **Updated** docstring to describe new behavior and parameters
- Backward compatible: `earliest_hcd_date` defaults to `None`, existing callers still work
- Note: caller in `refresh_pathways.py` (line 424-428) does `dict(zip(...))` which will only keep last match per patient with new multi-row format ÔÇö this will be updated in Task 3.1
### Validation results:
- Tier 1 (Code): py_compile PASSED, import check PASSED, function signature verified
- Tier 2 (Data): Empty DataFrame returns correct columns ['PatientPseudonym', 'Search_Term', 'code_frequency']; live Snowflake test deferred to Phase 3/4
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (modified get_patient_indication_groups function)
- IMPLEMENTATION_PLAN.md (marked 1.1 subtasks [x])
### Committed: [pending]
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format (YYYY-MM-DD) via Snowflake %s placeholder ÔÇö Snowflake handles string-to-timestamp comparison implicitly
- The GROUP BY approach naturally deduplicates SNOMED codes within the same Search_Term ÔÇö a patient with the same SNOMED code recorded 5 times gets code_frequency=5 (reflecting clinical activity intensity)
- params list is built dynamically: `batch_pseudonyms + [earliest_hcd_date]` only when date filter is active
### Next iteration should:
- Work on Task 2.1: Create `assign_drug_indications()` function
  - This is now unblocked since 1.1 is complete (return format is known)
  - Input: HCD df, gp_matches_df (PatientPseudonym, Search_Term, code_frequency), drug_mapping from load_drug_indication_mapping()
  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
  - Can be built and tested with mock data (no Snowflake needed)
  - Key logic: for each UPID+Drug pair, intersect drug's Search_Terms with patient's GP matches, pick highest code_frequency as tiebreaker
  - The function needs PseudoNHSNoLinked to look up GP matches, so the df must have that column
  - Task 2.2 (tiebreaker logic) can be done within 2.1 or as a follow-up
- The final Phase 1 subtask (1.1 verify with live Snowflake) will be tested during Phase 3/4 integration
### Blocked items:
- Task 1.1 final subtask "Verify: Query returns more rows" requires live Snowflake ÔÇö deferred to Phase 3/4


<tool_use_error>Found 2 matches of the string to replace, but replace_all is false. To replace all occurrences, set replace_all to true. To replace only one occurrence, please provide more context to uniquely identify the instance.
String: ### Blocked items:
- None
</tool_use_error>

Error: Found 2 matches of the string to replace, but replace_all is false. To replace all occurrences, set replace_all to true. To replace only one occurrence, please provide more context to uniquely identify the instance.
String: ### Blocked items:
- None


There are two instances of "### Blocked items: - None". Let me use more context to uniquely identify the last one.


- OR work on Task 2.1: Create assign_drug_indications() ÔÇö can be built and tested with mock data
  - This is independent of Task 1.1 if you mock the gp_matches_df input
### Blocked items:
- None

- OR work on Task 2.1: Create assign_drug_indications() ÔÇö can be built and tested with mock data
  - This is independent of Task 1.1 if you mock the gp_matches_df input
### Blocked items:
- None

## Iteration 3 ÔÇö 2026-02-05
### Task: 1.1 ÔÇö Update get_patient_indication_groups() to return ALL matches with code_frequency
### Why this task:
- Previous iteration recommended this as the next task
- This is the last remaining Phase 1 dependency ÔÇö Tasks 2.1/2.2 need the GP matches data in the right format (multiple rows per patient with code_frequency)
- Task 1.1 changes the query fundamentals; better to do this before building assign_drug_indications() which consumes the output
### Status: COMPLETE
### What was done:
- Modified `get_patient_indication_groups()` in `diagnosis_lookup.py`:
  1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"`
  3. **Added** `earliest_hcd_date: Optional[str] = None` parameter
  4. **Added** optional `AND pc."EventDateTime" >= %s` when earliest_hcd_date is provided
  5. **Updated** return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`
  6. **Updated** all empty DataFrame returns to use new column names
  7. **Updated** logging to show multiple-rows-per-patient stats (avg indications per patient)
  8. **Updated** docstring to describe new behavior and parameters
- Backward compatible: `earliest_hcd_date` defaults to `None`, existing callers still work
- Note: caller in `refresh_pathways.py` (line 424-428) does `dict(zip(...))` which will only keep last match per patient with new multi-row format ÔÇö this will be updated in Task 3.1
### Validation results:
- Tier 1 (Code): py_compile PASSED, import check PASSED, function signature verified
- Tier 2 (Data): Empty DataFrame returns correct columns ['PatientPseudonym', 'Search_Term', 'code_frequency']; live Snowflake test deferred to Phase 3/4
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (modified get_patient_indication_groups function)
- IMPLEMENTATION_PLAN.md (marked 1.1 subtasks [x])
### Committed: [pending]
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format (YYYY-MM-DD) via Snowflake %s placeholder ÔÇö Snowflake handles string-to-timestamp comparison implicitly
- The GROUP BY approach naturally deduplicates SNOMED codes within the same Search_Term ÔÇö a patient with the same SNOMED code recorded 5 times gets code_frequency=5 (reflecting clinical activity intensity)
- params list is built dynamically: `batch_pseudonyms + [earliest_hcd_date]` only when date filter is active
### Next iteration should:
- Work on Task 2.1: Create `assign_drug_indications()` function
  - This is now unblocked since 1.1 is complete (return format is known)
  - Input: HCD df, gp_matches_df (PatientPseudonym, Search_Term, code_frequency), drug_mapping from load_drug_indication_mapping()
  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
  - Can be built and tested with mock data (no Snowflake needed)
  - Key logic: for each UPID+Drug pair, intersect drug's Search_Terms with patient's GP matches, pick highest code_frequency as tiebreaker
  - The function needs PseudoNHSNoLinked to look up GP matches, so the df must have that column
  - Task 2.2 (tiebreaker logic) can be done within 2.1 or as a follow-up
- The final Phase 1 subtask (1.1 verify with live Snowflake) will be tested during Phase 3/4 integration
### Blocked items:
- Task 1.1 final subtask "Verify: Query returns more rows" requires live Snowflake ÔÇö deferred to Phase 3/4


The file C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\progress.txt has been updated successfully.

- OR work on Task 2.1: Create assign_drug_indications() ÔÇö can be built and tested with mock data
  - This is independent of Task 1.1 if you mock the gp_matches_df input
### Blocked items:
- None

## Iteration 3 ÔÇö 2026-02-05
### Task: 1.1 ÔÇö Update get_patient_indication_groups() to return ALL matches with code_frequency
### Why this task:
- Previous iteration recommended this as the next task
- This is the last remaining Phase 1 dependency ÔÇö Tasks 2.1/2.2 need the GP matches data in the right format (multiple rows per patient with code_frequency)
- Task 1.1 changes the query fundamentals; better to do this before building assign_drug_indications() which consumes the output
### Status: COMPLETE
### What was done:
- Modified `get_patient_indication_groups()` in `diagnosis_lookup.py`:
  1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"`
  3. **Added** `earliest_hcd_date: Optional[str] = None` parameter
  4. **Added** optional `AND pc."EventDateTime" >= %s` when earliest_hcd_date is provided
  5. **Updated** return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`
  6. **Updated** all empty DataFrame returns to use new column names
  7. **Updated** logging to show multiple-rows-per-patient stats (avg indications per patient)
  8. **Updated** docstring to describe new behavior and parameters
- Backward compatible: `earliest_hcd_date` defaults to `None`, existing callers still work
- Note: caller in `refresh_pathways.py` (line 424-428) does `dict(zip(...))` which will only keep last match per patient with new multi-row format ÔÇö this will be updated in Task 3.1
### Validation results:
- Tier 1 (Code): py_compile PASSED, import check PASSED, function signature verified
- Tier 2 (Data): Empty DataFrame returns correct columns ['PatientPseudonym', 'Search_Term', 'code_frequency']; live Snowflake test deferred to Phase 3/4
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (modified get_patient_indication_groups function)
- IMPLEMENTATION_PLAN.md (marked 1.1 subtasks [x])
### Committed: [pending]
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format (YYYY-MM-DD) via Snowflake %s placeholder ÔÇö Snowflake handles string-to-timestamp comparison implicitly
- The GROUP BY approach naturally deduplicates SNOMED codes within the same Search_Term ÔÇö a patient with the same SNOMED code recorded 5 times gets code_frequency=5 (reflecting clinical activity intensity)
- params list is built dynamically: `batch_pseudonyms + [earliest_hcd_date]` only when date filter is active
### Next iteration should:
- Work on Task 2.1: Create `assign_drug_indications()` function
  - This is now unblocked since 1.1 is complete (return format is known)
  - Input: HCD df, gp_matches_df (PatientPseudonym, Search_Term, code_frequency), drug_mapping from load_drug_indication_mapping()
  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
  - Can be built and tested with mock data (no Snowflake needed)
  - Key logic: for each UPID+Drug pair, intersect drug's Search_Terms with patient's GP matches, pick highest code_frequency as tiebreaker
  - The function needs PseudoNHSNoLinked to look up GP matches, so the df must have that column
  - Task 2.2 (tiebreaker logic) can be done within 2.1 or as a follow-up
- The final Phase 1 subtask (1.1 verify with live Snowflake) will be tested during Phase 3/4 integration
### Blocked items:
- Task 1.1 final subtask "Verify: Query returns more rows" requires live Snowflake ÔÇö deferred to Phase 3/4

# Progress Log - Drug-Aware Indication Matching

## Project Context

This project extends the indication-based pathway charts (Phase 1-5 complete) with drug-aware matching.

**Previous state**: Patients get ONE indication based on their most recent GP diagnosis match (SNOMED cluster codes). This ignores which drugs the patient is taking.

**New goal**: Match each drug to an indication by cross-referencing the patient's GP diagnoses AND the drug's Search_Term mapping from DimSearchTerm.csv.

## Key Data/Patterns

### DimSearchTerm.csv
- Located at `data/DimSearchTerm.csv`
- Columns: Search_Term, CleanedDrugName (pipe-separated), PrimaryDirectorate
- ~165 rows mapping clinical conditions to drug name fragments
- Drug fragments are substrings that match standardized drug names from HCD data
- Some entries have generic fragments: INHALED, CONTINUOUS, STANDARD-DOSE, PEGYLATED

### Current get_patient_indication_groups() in diagnosis_lookup.py
- Uses CLUSTER_MAPPING_SQL as CTE in Snowflake query
- Returns ONLY the most recent match per patient (QUALIFY ROW_NUMBER() = 1)
- Needs to return ALL matching Search_Terms per patient (remove QUALIFY)
- Batches 500 patients per query

### Modified UPID approach
- Current: UPID = Provider Code[:3] + PersonKey (e.g., "RMV12345")
- New: UPID = original + "|" + search_term (e.g., "RMV12345|rheumatoid arthritis")
- The pipe delimiter "|" is safe because existing UPIDs are alphanumeric
- generate_icicle_chart_indication() treats UPID as an opaque identifier ÔÇö modified UPIDs work transparently
- The " - " delimiter in pathway ids is used for hierarchy levels, not within UPIDs

### PseudoNHSNoLinked mapping
- HCD data has PseudoNHSNoLinked column that matches PatientPseudonym in GP records
- PersonKey is provider-specific local ID ÔÇö do NOT use for GP matching
- One PseudoNHSNoLinked can map to multiple UPIDs (multi-provider patients)
- GP match lookup: PseudoNHSNoLinked ÔåÆ list of matched Search_Terms

### Drug matching logic
- For each HCD row (UPID + Drug Name):
  1. Get patient's GP-matched Search_Terms with code_frequency (via PseudoNHSNoLinked)
  2. Get which Search_Terms list this drug (from DimSearchTerm.csv)
  3. Intersection = valid indications
  4. If 1: use it. If multiple: pick highest code_frequency (most GP coding = most likely indication). If 0: fallback to directory.
- Modified UPID groups drugs under same indication together naturally
- code_frequency = COUNT(*) of matching SNOMED codes per Search_Term per patient in GP records
- GP code time range: only count codes from MIN(Intervention Date) onwards (the HCD data window)
  - Reduces noise from old/irrelevant diagnoses, makes frequency more meaningful
  - Pass earliest_hcd_date as parameter to get_patient_indication_groups()
- Tiebreaker rationale: 47 RA codes vs 2 crohn's codes ÔåÆ RA is clearly the active condition

### Known edge cases
- Some DimSearchTerm drug fragments are generic (INHALED, ORAL, CONTINUOUS)
  - These could match broadly but are constrained by GP diagnosis requirement
- A patient visiting multiple providers has multiple UPIDs
  - Each UPID gets its own drug-indication matching independently
- Same Search_Term appears twice in DimSearchTerm.csv with different directorates
  - e.g., "diabetes" ÔåÆ DIABETIC MEDICINE and OPHTHALMOLOGY
  - For indication charts, we use Search_Term not directorate, so this is fine

## Iteration Log

## Iteration 1 ÔÇö 2026-02-05
### Task: 1.3 ÔÇö Build drug-to-Search_Term lookup from DimSearchTerm.csv
### Why this task:
- First iteration, chose Phase 1 foundations. Task 1.2 (CSV loading) is self-contained and testable locally without Snowflake.
- Task 1.1 (Snowflake query update) can't be verified without a live connection ÔÇö better to do 1.2 first.
- Both 1.1 and 1.2 are independent, so order doesn't matter for dependencies.
### Status: COMPLETE
### What was done:
- Added `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
  - Loads `data/DimSearchTerm.csv`, builds two dicts:
    - `fragment_to_search_terms`: drug fragment (UPPER) ÔåÆ list of Search_Terms
    - `search_term_to_fragments`: search_term ÔåÆ list of drug fragments (UPPER)
  - Handles duplicate Search_Terms (e.g., "diabetes" rows combined)
  - Result: 164 Search_Terms, 346 drug fragments
- Added `get_search_terms_for_drug()` to `diagnosis_lookup.py`:
  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
  - Named differently from plan's `drug_matches_search_term()` ÔÇö returns all matches at once rather than single boolean, more practical for Phase 2
- Updated `__all__` exports
### Validation results:
- Tier 1 (Code): py_compile passed, import check passed
- Tier 2 (Data): ADALIMUMAB ÔåÆ 7 indications (including axial spondyloarthritis, rheumatoid arthritis), OMALIZUMAB ÔåÆ 4 indications (asthma, allergic asthma, etc.), PEGYLATED LIPOSOMAL DOXORUBICIN ÔåÆ 4 matches via substring, "ADALIMUMAB 40MG" matches correctly with dosage info, diabetes fragments combined from 2 CSV rows
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (added load_drug_indication_mapping, get_search_terms_for_drug)
- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
### Committed: 0779df7 "feat: add drug-to-indication mapping from DimSearchTerm.csv (Task 1.3)"
### Patterns discovered:
- DimSearchTerm.csv has 164 unique Search_Terms (not 165 as noted) because diabetes appears twice with different directorates but same Search_Term
- Some drug fragments are very generic: INHALED, CONTINUOUS, ORAL, STANDARD-DOSE, INTRAVENOUS, PEGYLATED, ROUTINE, INDUCTION ÔÇö these will match broadly but are constrained by the GP diagnosis requirement in Phase 2
- Function signatures for Phase 2: `get_search_terms_for_drug(drug_name, search_term_to_fragments)` returns list[str] ÔÇö use this to get candidate indications per drug
### Next iteration should:
- Work on Task 1.2: Merge asthma Search_Terms in CLUSTER_MAPPING_SQL and load_drug_indication_mapping()
  - Merge "allergic asthma", "asthma", "severe persistent allergic asthma" ÔåÆ "asthma"
  - Keep "urticaria" separate
  - This is self-contained and testable locally
- OR work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
  - The current query at line ~1352 of diagnosis_lookup.py uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1` ÔÇö this must be replaced with GROUP BY + COUNT(*)
  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
  - Return columns: PatientPseudonym, Search_Term, code_frequency (not EventDateTime)
- OR if Snowflake isn't available to test 1.1, skip to Task 2.1 (assign_drug_indications function) which can be built and tested with mock data
### Blocked items:
- None

## Iteration 2 ÔÇö 2026-02-05
### Task: 1.2 ÔÇö Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
### Why this task:
- Previous iteration recommended this as the next task (self-contained, testable locally)
- Both CLUSTER_MAPPING_SQL and load_drug_indication_mapping() need consistent Search_Term names
- Must be done before Task 1.1 (Snowflake query) to ensure GP lookups return "asthma" not "allergic asthma"
### Status: COMPLETE
### What was done:
- Updated CLUSTER_MAPPING_SQL: changed 'allergic asthma' ÔåÆ 'asthma' (AST_COD) and 'severe persistent allergic asthma' ÔåÆ 'asthma' (SEVAST_COD)
  - Now 3 rows for 'asthma': AST_COD, eFI2_Asthma, SEVAST_COD
  - urticaria (XSAL_COD) stays separate
- Added SEARCH_TERM_MERGE_MAP constant: {"allergic asthma": "asthma", "severe persistent allergic asthma": "asthma"}
- Updated load_drug_indication_mapping() to apply SEARCH_TERM_MERGE_MAP when loading CSV
  - Normalizes Search_Term before accumulating fragments
  - Drug fragments from all 3 original rows combined under "asthma" key
- Exported SEARCH_TERM_MERGE_MAP in __all__
### Validation results:
- Tier 1 (Code): py_compile passed, import check passed
- Tier 2 (Data):
  - "asthma" fragments: OMALIZUMAB, BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, RESLIZUMAB (complete combined list)
  - "allergic asthma" no longer exists as separate key
  - "severe persistent allergic asthma" no longer exists as separate key
  - "urticaria" ÔåÆ ['OMALIZUMAB'] ÔÇö correctly separate
  - OMALIZUMAB maps to: ['asthma', 'urticaria'] ÔÇö correct
  - Total Search_Terms: 162 (was 164, 3 asthma entries ÔåÆ 1)
  - Total fragments: 346 (unchanged)
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (CLUSTER_MAPPING_SQL, SEARCH_TERM_MERGE_MAP, load_drug_indication_mapping)
- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
### Committed: b0a8a9d "feat: merge asthma Search_Term variants in CLUSTER_MAPPING_SQL and drug mapping (Task 1.2)"
### Patterns discovered:
- SEARCH_TERM_MERGE_MAP is reusable: any future module that receives Search_Terms from Snowflake can apply the same normalization
- The merge approach (normalize at load time) is cleaner than post-hoc deduplication
### Next iteration should:
- Work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
  - The current query at ~line 1467 uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  - Replace with GROUP BY + COUNT(*) for code_frequency
  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
  - Return columns: PatientPseudonym, Search_Term, code_frequency
  - Empty DataFrame columns should match new return type
  - This requires Snowflake connectivity to fully test, but code changes can be verified with py_compile and import checks
- OR work on Task 2.1: Create assign_drug_indications() ÔÇö can be built and tested with mock data
  - This is independent of Task 1.1 if you mock the gp_matches_df input
### Blocked items:
- None

+### Task: 1.1 ÔÇö Update get_patient_indication_groups() to return ALL matches with code_frequency

+- Previous iteration recommended this as the next task

+- This is the last remaining Phase 1 dependency ÔÇö Tasks 2.1/2.2 need the GP matches data in the right format (multiple rows per patient with code_frequency)

+- Task 1.1 changes the query fundamentals; better to do this before building assign_drug_indications() which consumes the output

+- Modified `get_patient_indication_groups()` in `diagnosis_lookup.py`:

+  1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`

+  2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"`

+  3. **Added** `earliest_hcd_date: Optional[str] = None` parameter

+  4. **Added** optional `AND pc."EventDateTime" >= %s` when earliest_hcd_date is provided

+  5. **Updated** return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`

+  6. **Updated** all empty DataFrame returns to use new column names

+  7. **Updated** logging to show multiple-rows-per-patient stats (avg indications per patient)

+  8. **Updated** docstring to describe new behavior and parameters

+- Backward compatible: `earliest_hcd_date` defaults to `None`, existing callers still work

+- Note: caller in `refresh_pathways.py` (line 424-428) does `dict(zip(...))` which will only keep last match per patient with new multi-row format ÔÇö this will be updated in Task 3.1

+- Tier 1 (Code): py_compile PASSED, import check PASSED, function signature verified

+- Tier 2 (Data): Empty DataFrame returns correct columns ['PatientPseudonym', 'Search_Term', 'code_frequency']; live Snowflake test deferred to Phase 3/4

+- Tier 3 (Functional): N/A (no UI changes)

+- data_processing/diagnosis_lookup.py (modified get_patient_indication_groups function)

+- IMPLEMENTATION_PLAN.md (marked 1.1 subtasks [x])

+- The `earliest_hcd_date` parameter is passed as a string in ISO format (YYYY-MM-DD) via Snowflake %s placeholder ÔÇö Snowflake handles string-to-timestamp comparison implicitly

+- The GROUP BY approach naturally deduplicates SNOMED codes within the same Search_Term ÔÇö a patient with the same SNOMED code recorded 5 times gets code_frequency=5 (reflecting clinical activity intensity)

+- params list is built dynamically: `batch_pseudonyms + [earliest_hcd_date]` only when date filter is active

+- Work on Task 2.1: Create `assign_drug_indications()` function

+  - This is now unblocked since 1.1 is complete (return format is known)

+  - Input: HCD df, gp_matches_df (PatientPseudonym, Search_Term, code_frequency), drug_mapping from load_drug_indication_mapping()

+  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)

+  - Can be built and tested with mock data (no Snowflake needed)

+  - Key logic: for each UPID+Drug pair, intersect drug's Search_Terms with patient's GP matches, pick highest code_frequency as tiebreaker

+  - The function needs PseudoNHSNoLinked to look up GP matches, so the df must have that column

+  - Task 2.2 (tiebreaker logic) can be done within 2.1 or as a follow-up

+- The final Phase 1 subtask (1.1 verify with live Snowflake) will be tested during Phase 3/4 integration

+- Task 1.1 final subtask "Verify: Query returns more rows" requires live Snowflake ÔÇö deferred to Phase 3/4


Now let me commit the changes.


TodoWrite

Update get_patient_indication_groups() to return ALL matches with code_frequency (Task 1.1)

Updating get_patient_indication_groups() Snowflake query

Validate code compiles and imports work

Mark task complete in IMPLEMENTATION_PLAN.md

Update progress.txt with iteration results

Commit changes


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && git status

Check git status before commit


On branch feature/snomed-indication-mapping
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   .claude/settings.local.json
	modified:   DESIGN_SYSTEM.md
	modified:   IMPLEMENTATION_PLAN.md
	modified:   RALPH_PROMPT.md
	modified:   data_processing/diagnosis_lookup.py
	modified:   guardrails.md
	modified:   pathways_app/__init__.py
	deleted:    pathways_app/app_v2.py
	modified:   progress.txt
	modified:   ralph.ps1
	modified:   uv.lock

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	data/DimSearchTerm.csv
	data/LookupSearchTermCleanedDrugName.csv
	data/condition_directorate_mapping.csv
	data/drug_indication_clusters.csv
	data/drug_snomed_mapping_enriched.csv
	data/na_directory_rows.csv
	pathways_app/pathways_app_old.py.bak
	snomed_indication_mapping_query.sql

no changes added to commit (use "git add" and/or "git commit -a")


I need to commit only the files I changed in this iteration: `data_processing/diagnosis_lookup.py`, `IMPLEMENTATION_PLAN.md`, and `progress.txt`.


cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && git add data_processing/diagnosis_lookup.py IMPLEMENTATION_PLAN.md progress.txt && git commit -m "$(cat <<'EOF'
feat: return ALL GP matches with code_frequency in get_patient_indication_groups (Task 1.1)

- Replace QUALIFY ROW_NUMBER()=1 with GROUP BY + COUNT(*) to return all matching
  Search_Terms per patient instead of just the most recent
- Add earliest_hcd_date parameter to restrict GP codes to HCD data window
- Return code_frequency column (count of matching SNOMED codes per Search_Term)
  for use as tiebreaker in drug-aware indication matching
- Update empty DataFrame returns to match new column format
EOF
)"

Commit Task 1.1 changes


[feature/snomed-indication-mapping c93417f] feat: return ALL GP matches with code_frequency in get_patient_indication_groups (Task 1.1)
 3 files changed, 89 insertions(+), 27 deletions(-)


Commit successful. Let me update progress.txt with the commit hash.


Edit

### Committed: [pending]
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format

### Committed: c93417f "feat: return ALL GP matches with code_frequency in get_patient_indication_groups (Task 1.1)"
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format


### Committed: c93417f "feat: return ALL GP matches with code_frequency in get_patient_indication_groups (Task 1.1)"
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format

# Progress Log - Drug-Aware Indication Matching

## Project Context

This project extends the indication-based pathway charts (Phase 1-5 complete) with drug-aware matching.

**Previous state**: Patients get ONE indication based on their most recent GP diagnosis match (SNOMED cluster codes). This ignores which drugs the patient is taking.

**New goal**: Match each drug to an indication by cross-referencing the patient's GP diagnoses AND the drug's Search_Term mapping from DimSearchTerm.csv.

## Key Data/Patterns

### DimSearchTerm.csv
- Located at `data/DimSearchTerm.csv`
- Columns: Search_Term, CleanedDrugName (pipe-separated), PrimaryDirectorate
- ~165 rows mapping clinical conditions to drug name fragments
- Drug fragments are substrings that match standardized drug names from HCD data
- Some entries have generic fragments: INHALED, CONTINUOUS, STANDARD-DOSE, PEGYLATED

### Current get_patient_indication_groups() in diagnosis_lookup.py
- Uses CLUSTER_MAPPING_SQL as CTE in Snowflake query
- Returns ONLY the most recent match per patient (QUALIFY ROW_NUMBER() = 1)
- Needs to return ALL matching Search_Terms per patient (remove QUALIFY)
- Batches 500 patients per query

### Modified UPID approach
- Current: UPID = Provider Code[:3] + PersonKey (e.g., "RMV12345")
- New: UPID = original + "|" + search_term (e.g., "RMV12345|rheumatoid arthritis")
- The pipe delimiter "|" is safe because existing UPIDs are alphanumeric
- generate_icicle_chart_indication() treats UPID as an opaque identifier ÔÇö modified UPIDs work transparently
- The " - " delimiter in pathway ids is used for hierarchy levels, not within UPIDs

### PseudoNHSNoLinked mapping
- HCD data has PseudoNHSNoLinked column that matches PatientPseudonym in GP records
- PersonKey is provider-specific local ID ÔÇö do NOT use for GP matching
- One PseudoNHSNoLinked can map to multiple UPIDs (multi-provider patients)
- GP match lookup: PseudoNHSNoLinked ÔåÆ list of matched Search_Terms

### Drug matching logic
- For each HCD row (UPID + Drug Name):
  1. Get patient's GP-matched Search_Terms with code_frequency (via PseudoNHSNoLinked)
  2. Get which Search_Terms list this drug (from DimSearchTerm.csv)
  3. Intersection = valid indications
  4. If 1: use it. If multiple: pick highest code_frequency (most GP coding = most likely indication). If 0: fallback to directory.
- Modified UPID groups drugs under same indication together naturally
- code_frequency = COUNT(*) of matching SNOMED codes per Search_Term per patient in GP records
- GP code time range: only count codes from MIN(Intervention Date) onwards (the HCD data window)
  - Reduces noise from old/irrelevant diagnoses, makes frequency more meaningful
  - Pass earliest_hcd_date as parameter to get_patient_indication_groups()
- Tiebreaker rationale: 47 RA codes vs 2 crohn's codes ÔåÆ RA is clearly the active condition

### Known edge cases
- Some DimSearchTerm drug fragments are generic (INHALED, ORAL, CONTINUOUS)
  - These could match broadly but are constrained by GP diagnosis requirement
- A patient visiting multiple providers has multiple UPIDs
  - Each UPID gets its own drug-indication matching independently
- Same Search_Term appears twice in DimSearchTerm.csv with different directorates
  - e.g., "diabetes" ÔåÆ DIABETIC MEDICINE and OPHTHALMOLOGY
  - For indication charts, we use Search_Term not directorate, so this is fine

## Iteration Log

## Iteration 1 ÔÇö 2026-02-05
### Task: 1.3 ÔÇö Build drug-to-Search_Term lookup from DimSearchTerm.csv
### Why this task:
- First iteration, chose Phase 1 foundations. Task 1.2 (CSV loading) is self-contained and testable locally without Snowflake.
- Task 1.1 (Snowflake query update) can't be verified without a live connection ÔÇö better to do 1.2 first.
- Both 1.1 and 1.2 are independent, so order doesn't matter for dependencies.
### Status: COMPLETE
### What was done:
- Added `load_drug_indication_mapping()` to `diagnosis_lookup.py`:
  - Loads `data/DimSearchTerm.csv`, builds two dicts:
    - `fragment_to_search_terms`: drug fragment (UPPER) ÔåÆ list of Search_Terms
    - `search_term_to_fragments`: search_term ÔåÆ list of drug fragments (UPPER)
  - Handles duplicate Search_Terms (e.g., "diabetes" rows combined)
  - Result: 164 Search_Terms, 346 drug fragments
- Added `get_search_terms_for_drug()` to `diagnosis_lookup.py`:
  - Returns all Search_Terms whose drug fragments are substrings of the drug name (case-insensitive)
  - Named differently from plan's `drug_matches_search_term()` ÔÇö returns all matches at once rather than single boolean, more practical for Phase 2
- Updated `__all__` exports
### Validation results:
- Tier 1 (Code): py_compile passed, import check passed
- Tier 2 (Data): ADALIMUMAB ÔåÆ 7 indications (including axial spondyloarthritis, rheumatoid arthritis), OMALIZUMAB ÔåÆ 4 indications (asthma, allergic asthma, etc.), PEGYLATED LIPOSOMAL DOXORUBICIN ÔåÆ 4 matches via substring, "ADALIMUMAB 40MG" matches correctly with dosage info, diabetes fragments combined from 2 CSV rows
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (added load_drug_indication_mapping, get_search_terms_for_drug)
- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
### Committed: 0779df7 "feat: add drug-to-indication mapping from DimSearchTerm.csv (Task 1.3)"
### Patterns discovered:
- DimSearchTerm.csv has 164 unique Search_Terms (not 165 as noted) because diabetes appears twice with different directorates but same Search_Term
- Some drug fragments are very generic: INHALED, CONTINUOUS, ORAL, STANDARD-DOSE, INTRAVENOUS, PEGYLATED, ROUTINE, INDUCTION ÔÇö these will match broadly but are constrained by the GP diagnosis requirement in Phase 2
- Function signatures for Phase 2: `get_search_terms_for_drug(drug_name, search_term_to_fragments)` returns list[str] ÔÇö use this to get candidate indications per drug
### Next iteration should:
- Work on Task 1.2: Merge asthma Search_Terms in CLUSTER_MAPPING_SQL and load_drug_indication_mapping()
  - Merge "allergic asthma", "asthma", "severe persistent allergic asthma" ÔåÆ "asthma"
  - Keep "urticaria" separate
  - This is self-contained and testable locally
- OR work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
  - The current query at line ~1352 of diagnosis_lookup.py uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1` ÔÇö this must be replaced with GROUP BY + COUNT(*)
  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
  - Return columns: PatientPseudonym, Search_Term, code_frequency (not EventDateTime)
- OR if Snowflake isn't available to test 1.1, skip to Task 2.1 (assign_drug_indications function) which can be built and tested with mock data
### Blocked items:
- None

## Iteration 2 ÔÇö 2026-02-05
### Task: 1.2 ÔÇö Merge related asthma Search_Terms in CLUSTER_MAPPING_SQL
### Why this task:
- Previous iteration recommended this as the next task (self-contained, testable locally)
- Both CLUSTER_MAPPING_SQL and load_drug_indication_mapping() need consistent Search_Term names
- Must be done before Task 1.1 (Snowflake query) to ensure GP lookups return "asthma" not "allergic asthma"
### Status: COMPLETE
### What was done:
- Updated CLUSTER_MAPPING_SQL: changed 'allergic asthma' ÔåÆ 'asthma' (AST_COD) and 'severe persistent allergic asthma' ÔåÆ 'asthma' (SEVAST_COD)
  - Now 3 rows for 'asthma': AST_COD, eFI2_Asthma, SEVAST_COD
  - urticaria (XSAL_COD) stays separate
- Added SEARCH_TERM_MERGE_MAP constant: {"allergic asthma": "asthma", "severe persistent allergic asthma": "asthma"}
- Updated load_drug_indication_mapping() to apply SEARCH_TERM_MERGE_MAP when loading CSV
  - Normalizes Search_Term before accumulating fragments
  - Drug fragments from all 3 original rows combined under "asthma" key
- Exported SEARCH_TERM_MERGE_MAP in __all__
### Validation results:
- Tier 1 (Code): py_compile passed, import check passed
- Tier 2 (Data):
  - "asthma" fragments: OMALIZUMAB, BENRALIZUMAB, DUPILUMAB, INHALED, MEPOLIZUMAB, RESLIZUMAB (complete combined list)
  - "allergic asthma" no longer exists as separate key
  - "severe persistent allergic asthma" no longer exists as separate key
  - "urticaria" ÔåÆ ['OMALIZUMAB'] ÔÇö correctly separate
  - OMALIZUMAB maps to: ['asthma', 'urticaria'] ÔÇö correct
  - Total Search_Terms: 162 (was 164, 3 asthma entries ÔåÆ 1)
  - Total fragments: 346 (unchanged)
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (CLUSTER_MAPPING_SQL, SEARCH_TERM_MERGE_MAP, load_drug_indication_mapping)
- IMPLEMENTATION_PLAN.md (marked 1.2 subtasks [x])
### Committed: b0a8a9d "feat: merge asthma Search_Term variants in CLUSTER_MAPPING_SQL and drug mapping (Task 1.2)"
### Patterns discovered:
- SEARCH_TERM_MERGE_MAP is reusable: any future module that receives Search_Terms from Snowflake can apply the same normalization
- The merge approach (normalize at load time) is cleaner than post-hoc deduplication
### Next iteration should:
- Work on Task 1.1: Update `get_patient_indication_groups()` to return ALL matches with code_frequency
  - The current query at ~line 1467 uses `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  - Replace with GROUP BY + COUNT(*) for code_frequency
  - Add `earliest_hcd_date` parameter to restrict GP codes to HCD data window
  - Return columns: PatientPseudonym, Search_Term, code_frequency
  - Empty DataFrame columns should match new return type
  - This requires Snowflake connectivity to fully test, but code changes can be verified with py_compile and import checks
- OR work on Task 2.1: Create assign_drug_indications() ÔÇö can be built and tested with mock data
  - This is independent of Task 1.1 if you mock the gp_matches_df input
### Blocked items:
- None

## Iteration 3 ÔÇö 2026-02-05
### Task: 1.1 ÔÇö Update get_patient_indication_groups() to return ALL matches with code_frequency
### Why this task:
- Previous iteration recommended this as the next task
- This is the last remaining Phase 1 dependency ÔÇö Tasks 2.1/2.2 need the GP matches data in the right format (multiple rows per patient with code_frequency)
- Task 1.1 changes the query fundamentals; better to do this before building assign_drug_indications() which consumes the output
### Status: COMPLETE
### What was done:
- Modified `get_patient_indication_groups()` in `diagnosis_lookup.py`:
  1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY pc."PatientPseudonym" ORDER BY pc."EventDateTime" DESC) = 1`
  2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"`
  3. **Added** `earliest_hcd_date: Optional[str] = None` parameter
  4. **Added** optional `AND pc."EventDateTime" >= %s` when earliest_hcd_date is provided
  5. **Updated** return columns from `(PatientPseudonym, Search_Term, EventDateTime)` to `(PatientPseudonym, Search_Term, code_frequency)`
  6. **Updated** all empty DataFrame returns to use new column names
  7. **Updated** logging to show multiple-rows-per-patient stats (avg indications per patient)
  8. **Updated** docstring to describe new behavior and parameters
- Backward compatible: `earliest_hcd_date` defaults to `None`, existing callers still work
- Note: caller in `refresh_pathways.py` (line 424-428) does `dict(zip(...))` which will only keep last match per patient with new multi-row format ÔÇö this will be updated in Task 3.1
### Validation results:
- Tier 1 (Code): py_compile PASSED, import check PASSED, function signature verified
- Tier 2 (Data): Empty DataFrame returns correct columns ['PatientPseudonym', 'Search_Term', 'code_frequency']; live Snowflake test deferred to Phase 3/4
- Tier 3 (Functional): N/A (no UI changes)
### Files changed:
- data_processing/diagnosis_lookup.py (modified get_patient_indication_groups function)
- IMPLEMENTATION_PLAN.md (marked 1.1 subtasks [x])
### Committed: [pending]
### Patterns discovered:
- The `earliest_hcd_date` parameter is passed as a string in ISO format (YYYY-MM-DD) via Snowflake %s placeholder ÔÇö Snowflake handles string-to-timestamp comparison implicitly
- The GROUP BY approach naturally deduplicates SNOMED codes within the same Search_Term ÔÇö a patient with the same SNOMED code recorded 5 times gets code_frequency=5 (reflecting clinical activity intensity)
- params list is built dynamically: `batch_pseudonyms + [earliest_hcd_date]` only when date filter is active
### Next iteration should:
- Work on Task 2.1: Create `assign_drug_indications()` function
  - This is now unblocked since 1.1 is complete (return format is known)
  - Input: HCD df, gp_matches_df (PatientPseudonym, Search_Term, code_frequency), drug_mapping from load_drug_indication_mapping()
  - Output: (modified_df with UPID|search_term, indication_df mapping modified_UPID ÔåÆ Search_Term)
  - Can be built and tested with mock data (no Snowflake needed)
  - Key logic: for each UPID+Drug pair, intersect drug's Search_Terms with patient's GP matches, pick highest code_frequency as tiebreaker
  - The function needs PseudoNHSNoLinked to look up GP matches, so the df must have that column
  - Task 2.2 (tiebreaker logic) can be done within 2.1 or as a follow-up
- The final Phase 1 subtask (1.1 verify with live Snowflake) will be tested during Phase 3/4 integration
### Blocked items:
- Task 1.1 final subtask "Verify: Query returns more rows" requires live Snowflake ÔÇö deferred to Phase 3/4

+### Committed: c93417f "feat: return ALL GP matches with code_frequency in get_patient_indication_groups (Task 1.1)"


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && git add progress.txt && git commit -m "$(cat <<'EOF'
docs: update progress.txt with Iteration 3 results (Task 1.1)
EOF
)"

Commit progress.txt update


[feature/snomed-indication-mapping 947b87a] docs: update progress.txt with Iteration 3 results (Task 1.1)
 1 file changed, 1 insertion(+), 1 deletion(-)


TodoWrite

Update get_patient_indication_groups() to return ALL matches with code_frequency (Task 1.1)

Updating get_patient_indication_groups() Snowflake query

Validate code compiles and imports work

Mark task complete in IMPLEMENTATION_PLAN.md

Update progress.txt with iteration results

Commit changes


## Iteration 3 Summary

**Task completed: 1.1 ÔÇö Update `get_patient_indication_groups()` to return ALL matches with code_frequency**

**Changes made to `data_processing/diagnosis_lookup.py`:**
1. **Removed** `QUALIFY ROW_NUMBER() OVER (PARTITION BY ... ORDER BY EventDateTime DESC) = 1` ÔÇö was returning only the single most recent match per patient
2. **Replaced** with `GROUP BY pc."PatientPseudonym", aic.Search_Term` + `COUNT(*) AS "code_frequency"` ÔÇö now returns ALL matching Search_Terms per patient with a frequency count
3. **Added** `earliest_hcd_date: Optional[str] = None` parameter ÔÇö restricts GP codes to the HCD data window when provided
4. **Updated** return columns: `code_frequency` replaces `EventDateTime`
5. **Updated** all empty DataFrame returns and docstring

**Validation:**
- py_compile: PASSED
- Import check: PASSED
- Function signature: Verified (4 params, new `earliest_hcd_date` defaults to None)
- Empty DataFrame columns: `['PatientPseudonym', 'Search_Term', 'code_frequency']` verified

**Commits:**
- `c93417f` ÔÇö feat: return ALL GP matches with code_frequency
- `947b87a` ÔÇö docs: update progress.txt

**Next iteration should work on Task 2.1:** Create `assign_drug_indications()` function ÔÇö the core logic that cross-references GP matches with drug mappings to produce modified UPIDs.