     1ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Loaded 8 default trusts
     2ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Loaded 54 default drugs
     3ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Loaded 57 default directories
     4ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: ============================================================
     5ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Pathway Data Refresh Starting
     6ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: ============================================================
     7ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Minimum patients: 5
     8ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Trust filter: 8 trusts
     9ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Drug filter: 54 drugs
    10ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Directory filter: 57 directories
    11ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Provider codes: All
    12ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Chart type(s): indication
    13ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Database: C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data\pathways.db
    14ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Dry run: True
    15ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: ============================================================
    16ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__:
    17ÔåÆ2026-02-05 23:12:33 [INFO] pathways.__main__: Step 1/4: Fetching data from Snowflake...
    18ÔåÆ2026-02-05 23:12:33 [INFO] pathways.data_processing.pathway_pipeline: Fetching activity data from Snowflake...
    19ÔåÆ2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Connecting to Snowflake account: ZK91403.uk-south.azure
    20ÔåÆ2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Using warehouse: WH__XSMALL, database: DATA_HUB
    21ÔåÆ2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Authenticator: externalbrowser
    22ÔåÆ2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Browser window will open for NHS SSO authentication
    23ÔåÆInitiating login request with your identity provider. Press CTRL+C to abort and try again...
    24ÔåÆGoing to open: https://login.microsoftonline.com/f789227e-d70c-4029-b108-98ba67202f84/saml2?SAMLRequest=lZJRb9owGEX%2FSuQ9J3ECNMQCKgaCRms3BpRpfXNiAxaOnflzSPvvZ5IitQ%2Bt1LfIOdc%2B9v1Gt8%2Bl9M7cgNBqjKIAI4%2BrQjOhDmP0uF34Q%2BSBpYpRqRUfoxcO6HYyAlrKikxre1Rr%2Fq%2FmYD23kQLS%2Fhij2iiiKQggipYciC3IZvpwT%2BIAk8poqwst0ZvI5wkKwI11htcIA%2BH0jtZWJAybpgmaXqDNIYwxxiFOQ0ddkG9X%2Ftnd6QM%2BCnH%2FwjvC4atXt%2B9CdU%2FwmVbeQUDuttuVv%2Fq12SJvelWdaQV1yc2Gm7Mo%2BOP6vhMAZyAKYNTSY537lWEBKN3sJT3xQpdVbd2WgfsK95yFUh%2BEu3U2H6PqJNjvU%2FZE6%2BVyuT7j%2BQkG%2BbI%2FE7tFdsgfgKUH%2FCeJfsgd%2FL0z0wJ5u2ut8aXWDKDmmbqUad0Sjm98HPt4sI17JIpJLwkGvfQJeXNXplDUtsmrcesRlKIwGvTeaiWF4p1lMkzjOOE%2BS3Dh93Gc%2BnmEh346zOlN4o7ZD%2FvhpeIYdWNDWhEz%2BdJjjMK30dfx%2B%2BkayeYrLUXx4i20Kan9uLAoiNoVwfx9ixJeUiGnjBkO4IqTUjczw6l1U25NzVE46U59P%2BeT%2Fw%3D%3D&RelayState=ver%3A3-hint%3A468572343738374-ETMsDgAAAZwwFDijABRBRVMvQ0JDL1BLQ1M1UGFkZGluZwEAABAAED05RPF890YqBQqcwLrB0pYAAACgVIjTU9269PDQyOeYhZfmlyd563cAOHmEKccn0NHV4VI9rHYGo%2B3%2F7zXzA0XzJVQCJEV3IELxLr5vjYz823zFAKqREtztaWmSfHvWaY474wJhsY1hzIm4THN1hs6XaIAOAzkuJHfe2zjLEyH6YHm6po6o1Ggq45Td3%2B2%2FsRqSVGgA%2FEIfbNuw3wIRTCZK7RfrcS%2BkqH9DIh4rIU%2BrZT0x8wAUCXacrWs8rI8GDj6J7vBDU6g0JUo%3D to authenticate...
    25ÔåÆA browser window should have opened for you to complete the login. If you can't see it, check existing browser windows, or your OS settings.
    26ÔåÆ2026-02-05 23:12:42 [INFO] pathways.data_processing.snowflake_connector: Connected to Snowflake successfully in 9.3s
    27ÔåÆ2026-02-05 23:12:42 [INFO] pathways.data_processing.snowflake_connector: Fetching activity data from Snowflake
    28ÔåÆ2026-02-05 23:12:42 [INFO] pathways.data_processing.snowflake_connector: Executing chunked dict query (chunk_size=10000, timeout=300s)
    29ÔåÆ2026-02-05 23:12:42 [DEBUG] pathways.data_processing.snowflake_connector: Query:
    30ÔåÆ            SELECT
    31ÔåÆ                "PseudoNHSNoLinked",
    32ÔåÆ                "ProviderCode" AS "Provider Code",
    33ÔåÆ                "LocalPatientID" AS "PersonKey",
    34ÔåÆ                "DrugName" AS "Drug Name",
    35ÔåÆ...
    36ÔåÆ2026-02-05 23:12:44 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 1: 10000 rows (total: 10000)
    37ÔåÆ2026-02-05 23:12:44 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 10000 rows so far...
    38ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 2: 10000 rows (total: 20000)
    39ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 20000 rows so far...
    40ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 3: 10000 rows (total: 30000)
    41ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 30000 rows so far...
    42ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 4: 10000 rows (total: 40000)
    43ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 40000 rows so far...
    44ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 5: 10000 rows (total: 50000)
    45ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 50000 rows so far...
    46ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 6: 10000 rows (total: 60000)
    47ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 60000 rows so far...
    48ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 7: 10000 rows (total: 70000)
    49ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 70000 rows so far...
    50ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 8: 10000 rows (total: 80000)
    51ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 80000 rows so far...
    52ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 9: 10000 rows (total: 90000)
    53ÔåÆ2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 90000 rows so far...
    54ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 10: 10000 rows (total: 100000)
    55ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 100000 rows so far...
    56ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 11: 10000 rows (total: 110000)
    57ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 110000 rows so far...
    58ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 12: 10000 rows (total: 120000)
    59ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 120000 rows so far...
    60ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 13: 10000 rows (total: 130000)
    61ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 130000 rows so far...
    62ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 14: 10000 rows (total: 140000)
    63ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 140000 rows so far...
    64ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 15: 10000 rows (total: 150000)
    65ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 150000 rows so far...
    66ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 16: 10000 rows (total: 160000)
    67ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 160000 rows so far...
    68ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 17: 10000 rows (total: 170000)
    69ÔåÆ2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 170000 rows so far...
    70ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 18: 10000 rows (total: 180000)
    71ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 180000 rows so far...
    72ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 19: 10000 rows (total: 190000)
    73ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 190000 rows so far...
    74ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 20: 10000 rows (total: 200000)
    75ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 200000 rows so far...
    76ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 21: 10000 rows (total: 210000)
    77ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 210000 rows so far...
    78ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 22: 10000 rows (total: 220000)
    79ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 220000 rows so far...
    80ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 23: 10000 rows (total: 230000)
    81ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 230000 rows so far...
    82ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 24: 10000 rows (total: 240000)
    83ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 240000 rows so far...
    84ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 25: 10000 rows (total: 250000)
    85ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 250000 rows so far...
    86ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 26: 10000 rows (total: 260000)
    87ÔåÆ2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 260000 rows so far...
    88ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 27: 10000 rows (total: 270000)
    89ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 270000 rows so far...
    90ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 28: 10000 rows (total: 280000)
    91ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 280000 rows so far...
    92ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 29: 10000 rows (total: 290000)
    93ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 290000 rows so far...
    94ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 30: 10000 rows (total: 300000)
    95ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 300000 rows so far...
    96ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 31: 10000 rows (total: 310000)
    97ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 310000 rows so far...
    98ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 32: 10000 rows (total: 320000)
    99ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 320000 rows so far...
   100ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 33: 10000 rows (total: 330000)
   101ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 330000 rows so far...
   102ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 34: 10000 rows (total: 340000)
   103ÔåÆ2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 340000 rows so far...
   104ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 35: 10000 rows (total: 350000)
   105ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 350000 rows so far...
   106ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 36: 10000 rows (total: 360000)
   107ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 360000 rows so far...
   108ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 37: 10000 rows (total: 370000)
   109ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 370000 rows so far...
   110ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 38: 10000 rows (total: 380000)
   111ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 380000 rows so far...
   112ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 39: 10000 rows (total: 390000)
   113ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 390000 rows so far...
   114ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 40: 10000 rows (total: 400000)
   115ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 400000 rows so far...
   116ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 41: 10000 rows (total: 410000)
   117ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 410000 rows so far...
   118ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 42: 10000 rows (total: 420000)
   119ÔåÆ2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 420000 rows so far...
   120ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 43: 10000 rows (total: 430000)
   121ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 430000 rows so far...
   122ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 44: 10000 rows (total: 440000)
   123ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 440000 rows so far...
   124ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 45: 10000 rows (total: 450000)
   125ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 450000 rows so far...
   126ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 46: 10000 rows (total: 460000)
   127ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 460000 rows so far...
   128ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 47: 10000 rows (total: 470000)
   129ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 470000 rows so far...
   130ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 48: 10000 rows (total: 480000)
   131ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 480000 rows so far...
   132ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 49: 10000 rows (total: 490000)
   133ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 490000 rows so far...
   134ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 50: 10000 rows (total: 500000)
   135ÔåÆ2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 500000 rows so far...
   136ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 51: 10000 rows (total: 510000)
   137ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 510000 rows so far...
   138ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 52: 10000 rows (total: 520000)
   139ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 520000 rows so far...
   140ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 53: 10000 rows (total: 530000)
   141ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 530000 rows so far...
   142ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 54: 10000 rows (total: 540000)
   143ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 540000 rows so far...
   144ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 55: 10000 rows (total: 550000)
   145ÔåÆ2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 550000 rows so far...
   146ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 56: 10000 rows (total: 560000)
   147ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 560000 rows so far...
   148ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 57: 10000 rows (total: 570000)
   149ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 570000 rows so far...
   150ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 58: 10000 rows (total: 580000)
   151ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 580000 rows so far...
   152ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 59: 10000 rows (total: 590000)
   153ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 590000 rows so far...
   154ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 60: 10000 rows (total: 600000)
   155ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 600000 rows so far...
   156ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 61: 10000 rows (total: 610000)
   157ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 610000 rows so far...
   158ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 62: 10000 rows (total: 620000)
   159ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 620000 rows so far...
   160ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 63: 10000 rows (total: 630000)
   161ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 630000 rows so far...
   162ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 64: 10000 rows (total: 640000)
   163ÔåÆ2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 640000 rows so far...
   164ÔåÆ2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 65: 10000 rows (total: 650000)
   165ÔåÆ2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 650000 rows so far...
   166ÔåÆ2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 66: 6695 rows (total: 656695)
   167ÔåÆ2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 656695 rows so far...
   168ÔåÆ2026-02-05 23:12:53 [INFO] pathways.data_processing.snowflake_connector: Chunked dict query returned 656695 rows in 66 chunks (10.05s)
   169ÔåÆ2026-02-05 23:12:53 [INFO] pathways.data_processing.snowflake_connector: Fetched 656695 activity records from Snowflake
   170ÔåÆ2026-02-05 23:12:53 [INFO] pathways.data_processing.pathway_pipeline: Fetched 656695 records from Snowflake
   171ÔåÆ2026-02-05 23:12:55 [INFO] pathways.data_processing.pathway_pipeline: Applying data transformations...
   172ÔåÆ2026-02-05 23:12:55 [INFO] pathways.data_processing.pathway_pipeline: Generated UPID for 42769 unique patients
   173ÔåÆ2026-02-05 23:12:56 [INFO] pathways.data_processing.pathway_pipeline: Removed 136847 rows with unmapped drug names
   174ÔåÆ2026-02-05 23:12:57 [ERROR] pathways.tools.data: Error processing column NCDR Treatment Function Name: 'NCDR Treatment Function Name'
   175ÔåÆC:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:139: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
   176ÔåÆThe behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.
   177ÔåÆ
   178ÔåÆFor example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.
   179ÔåÆ
   180ÔåÆ
   181ÔåÆ  df["Treatment Function Code"].replace(np.nan, 0, inplace=True)
   182ÔåÆC:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:241: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
   183ÔåÆ  calculated_dirs = valid_groups.apply(lambda grp: find_first_valid_dir(grp, drug_to_valid_dirs))
   184ÔåÆC:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:263: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
   185ÔåÆThe behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.
   186ÔåÆ
   187ÔåÆFor example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.
   188ÔåÆ
   189ÔåÆ
   190ÔåÆ  df['Directory'].fillna(df['Calculated_Directory'], inplace=True)
   191ÔåÆC:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:273: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
   192ÔåÆThe behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.
   193ÔåÆ
   194ÔåÆFor example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.
   195ÔåÆ
   196ÔåÆ
   197ÔåÆ  df['UPID'].replace('', np.nan, inplace=True) # Ensure empty strings are NaN
   198ÔåÆC:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:304: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
   199ÔåÆThe behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.
   200ÔåÆ
   201ÔåÆFor example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.
   202ÔåÆ
   203ÔåÆ
   204ÔåÆ  df['Directory'].fillna("Undefined", inplace=True)
   205ÔåÆ2026-02-05 23:18:58 [INFO] pathways.data_processing.pathway_pipeline: Assigned directories to 519848 records
   206ÔåÆ2026-02-05 23:18:58 [INFO] pathways.data_processing.pathway_pipeline: Data transformation complete. Final record count: 519848
   207ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__: Fetched 519848 records from Snowflake
   208ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__:
   209ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__: Step 2/4: Processing pathway data for 6 datasets (6 date filters x 1 chart types)...
   210ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__:
   211ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__: Processing chart type: indication
   212ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__: Building drug-aware indication groups...
   213ÔåÆ2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup: Loaded drug-indication mapping: 162 Search_Terms, 346 drug fragments
   214ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__: Loaded drug mapping: 162 Search_Terms
   215ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__: Looking up GP diagnoses for 36628 unique patients...
   216ÔåÆ2026-02-05 23:18:58 [INFO] pathways.__main__: Restricting GP codes to HCD window: >= 2019-04-01
   217ÔåÆ2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup: Starting Snowflake-direct indication lookup for 36628 patients...
   218ÔåÆ2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup:   Restricting GP codes to >= 2019-04-01
   219ÔåÆ2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup: Batch 1/74: patients 1 to 500
   220ÔåÆ2026-02-05 23:18:58 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   221ÔåÆ2026-02-05 23:18:58 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   222ÔåÆ
   223ÔåÆWITH SearchTermClusters AS (
   224ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   225ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   226ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   227ÔåÆ   ...
   228ÔåÆ2026-02-05 23:19:29 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 1: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
   229ÔåÆ2026-02-05 23:19:29 [INFO] pathways.data_processing.diagnosis_lookup: Batch 2/74: patients 501 to 1000
   230ÔåÆ2026-02-05 23:19:29 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   231ÔåÆ2026-02-05 23:19:29 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   232ÔåÆ
   233ÔåÆWITH SearchTermClusters AS (
   234ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   235ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   236ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   237ÔåÆ   ...
   238ÔåÆ2026-02-05 23:20:00 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 2: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
   239ÔåÆ2026-02-05 23:20:00 [INFO] pathways.data_processing.diagnosis_lookup: Batch 3/74: patients 1001 to 1500
   240ÔåÆ2026-02-05 23:20:00 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   241ÔåÆ2026-02-05 23:20:00 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   242ÔåÆ
   243ÔåÆWITH SearchTermClusters AS (
   244ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   245ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   246ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   247ÔåÆ   ...
   248ÔåÆ2026-02-05 23:20:30 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 3: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
   249ÔåÆ2026-02-05 23:20:30 [INFO] pathways.data_processing.diagnosis_lookup: Batch 4/74: patients 1501 to 2000
   250ÔåÆ2026-02-05 23:20:30 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   251ÔåÆ2026-02-05 23:20:30 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   252ÔåÆ
   253ÔåÆWITH SearchTermClusters AS (
   254ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   255ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   256ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   257ÔåÆ   ...
   258ÔåÆ2026-02-05 23:21:00 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 4: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
   259ÔåÆ2026-02-05 23:21:00 [INFO] pathways.data_processing.diagnosis_lookup: Batch 5/74: patients 2001 to 2500
   260ÔåÆ2026-02-05 23:21:00 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   261ÔåÆ2026-02-05 23:21:00 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   262ÔåÆ
   263ÔåÆWITH SearchTermClusters AS (
   264ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   265ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   266ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   267ÔåÆ   ...
   268ÔåÆ2026-02-05 23:21:30 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 5: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
   269ÔåÆ2026-02-05 23:21:30 [INFO] pathways.data_processing.diagnosis_lookup: Batch 6/74: patients 2501 to 3000
   270ÔåÆ2026-02-05 23:21:30 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   271ÔåÆ2026-02-05 23:21:30 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   272ÔåÆ
   273ÔåÆWITH SearchTermClusters AS (
   274ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   275ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   276ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   277ÔåÆ   ...
   278ÔåÆ2026-02-05 23:22:00 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 6: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
   279ÔåÆ2026-02-05 23:22:00 [INFO] pathways.data_processing.diagnosis_lookup: Batch 7/74: patients 3001 to 3500
   280ÔåÆ2026-02-05 23:22:00 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   281ÔåÆ2026-02-05 23:22:00 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   282ÔåÆ
   283ÔåÆWITH SearchTermClusters AS (
   284ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   285ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   286ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   287ÔåÆ   ...
   288ÔåÆ2026-02-05 23:22:31 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 7: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
   289ÔåÆ2026-02-05 23:22:31 [INFO] pathways.data_processing.diagnosis_lookup: Batch 8/74: patients 3501 to 4000
   290ÔåÆ2026-02-05 23:22:31 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
   291ÔåÆ2026-02-05 23:22:31 [DEBUG] pathways.data_processing.snowflake_connector: Query:
   292ÔåÆ
   293ÔåÆWITH SearchTermClusters AS (
   294ÔåÆ    SELECT Search_Term, Cluster_ID FROM (VALUES
   295ÔåÆ        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
   296ÔåÆ        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   297ÔåÆ   ...
   298ÔåÆ

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

2026-02-05 23:12:33 [INFO] pathways.__main__: Loaded 8 default trusts
2026-02-05 23:12:33 [INFO] pathways.__main__: Loaded 54 default drugs
2026-02-05 23:12:33 [INFO] pathways.__main__: Loaded 57 default directories
2026-02-05 23:12:33 [INFO] pathways.__main__: ============================================================
2026-02-05 23:12:33 [INFO] pathways.__main__: Pathway Data Refresh Starting
2026-02-05 23:12:33 [INFO] pathways.__main__: ============================================================
2026-02-05 23:12:33 [INFO] pathways.__main__: Minimum patients: 5
2026-02-05 23:12:33 [INFO] pathways.__main__: Trust filter: 8 trusts
2026-02-05 23:12:33 [INFO] pathways.__main__: Drug filter: 54 drugs
2026-02-05 23:12:33 [INFO] pathways.__main__: Directory filter: 57 directories
2026-02-05 23:12:33 [INFO] pathways.__main__: Provider codes: All
2026-02-05 23:12:33 [INFO] pathways.__main__: Chart type(s): indication
2026-02-05 23:12:33 [INFO] pathways.__main__: Database: C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data\pathways.db
2026-02-05 23:12:33 [INFO] pathways.__main__: Dry run: True
2026-02-05 23:12:33 [INFO] pathways.__main__: ============================================================
2026-02-05 23:12:33 [INFO] pathways.__main__:
2026-02-05 23:12:33 [INFO] pathways.__main__: Step 1/4: Fetching data from Snowflake...
2026-02-05 23:12:33 [INFO] pathways.data_processing.pathway_pipeline: Fetching activity data from Snowflake...
2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Connecting to Snowflake account: ZK91403.uk-south.azure
2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Using warehouse: WH__XSMALL, database: DATA_HUB
2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Authenticator: externalbrowser
2026-02-05 23:12:33 [INFO] pathways.data_processing.snowflake_connector: Browser window will open for NHS SSO authentication
Initiating login request with your identity provider. Press CTRL+C to abort and try again...
Going to open: https://login.microsoftonline.com/f789227e-d70c-4029-b108-98ba67202f84/saml2?SAMLRequest=lZJRb9owGEX%2FSuQ9J3ECNMQCKgaCRms3BpRpfXNiAxaOnflzSPvvZ5IitQ%2Bt1LfIOdc%2B9v1Gt8%2Bl9M7cgNBqjKIAI4%2BrQjOhDmP0uF34Q%2BSBpYpRqRUfoxcO6HYyAlrKikxre1Rr%2Fq%2FmYD23kQLS%2Fhij2iiiKQggipYciC3IZvpwT%2BIAk8poqwst0ZvI5wkKwI11htcIA%2BH0jtZWJAybpgmaXqDNIYwxxiFOQ0ddkG9X%2Ftnd6QM%2BCnH%2FwjvC4atXt%2B9CdU%2FwmVbeQUDuttuVv%2Fq12SJvelWdaQV1yc2Gm7Mo%2BOP6vhMAZyAKYNTSY537lWEBKN3sJT3xQpdVbd2WgfsK95yFUh%2BEu3U2H6PqJNjvU%2FZE6%2BVyuT7j%2BQkG%2BbI%2FE7tFdsgfgKUH%2FCeJfsgd%2FL0z0wJ5u2ut8aXWDKDmmbqUad0Sjm98HPt4sI17JIpJLwkGvfQJeXNXplDUtsmrcesRlKIwGvTeaiWF4p1lMkzjOOE%2BS3Dh93Gc%2BnmEh346zOlN4o7ZD%2FvhpeIYdWNDWhEz%2BdJjjMK30dfx%2B%2BkayeYrLUXx4i20Kan9uLAoiNoVwfx9ixJeUiGnjBkO4IqTUjczw6l1U25NzVE46U59P%2BeT%2Fw%3D%3D&RelayState=ver%3A3-hint%3A468572343738374-ETMsDgAAAZwwFDijABRBRVMvQ0JDL1BLQ1M1UGFkZGluZwEAABAAED05RPF890YqBQqcwLrB0pYAAACgVIjTU9269PDQyOeYhZfmlyd563cAOHmEKccn0NHV4VI9rHYGo%2B3%2F7zXzA0XzJVQCJEV3IELxLr5vjYz823zFAKqREtztaWmSfHvWaY474wJhsY1hzIm4THN1hs6XaIAOAzkuJHfe2zjLEyH6YHm6po6o1Ggq45Td3%2B2%2FsRqSVGgA%2FEIfbNuw3wIRTCZK7RfrcS%2BkqH9DIh4rIU%2BrZT0x8wAUCXacrWs8rI8GDj6J7vBDU6g0JUo%3D to authenticate...
A browser window should have opened for you to complete the login. If you can't see it, check existing browser windows, or your OS settings.
2026-02-05 23:12:42 [INFO] pathways.data_processing.snowflake_connector: Connected to Snowflake successfully in 9.3s
2026-02-05 23:12:42 [INFO] pathways.data_processing.snowflake_connector: Fetching activity data from Snowflake
2026-02-05 23:12:42 [INFO] pathways.data_processing.snowflake_connector: Executing chunked dict query (chunk_size=10000, timeout=300s)
2026-02-05 23:12:42 [DEBUG] pathways.data_processing.snowflake_connector: Query:
            SELECT
                "PseudoNHSNoLinked",
                "ProviderCode" AS "Provider Code",
                "LocalPatientID" AS "PersonKey",
                "DrugName" AS "Drug Name",
...
2026-02-05 23:12:44 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 1: 10000 rows (total: 10000)
2026-02-05 23:12:44 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 10000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 2: 10000 rows (total: 20000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 20000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 3: 10000 rows (total: 30000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 30000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 4: 10000 rows (total: 40000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 40000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 5: 10000 rows (total: 50000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 50000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 6: 10000 rows (total: 60000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 60000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 7: 10000 rows (total: 70000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 70000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 8: 10000 rows (total: 80000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 80000 rows so far...
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 9: 10000 rows (total: 90000)
2026-02-05 23:12:45 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 90000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 10: 10000 rows (total: 100000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 100000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 11: 10000 rows (total: 110000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 110000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 12: 10000 rows (total: 120000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 120000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 13: 10000 rows (total: 130000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 130000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 14: 10000 rows (total: 140000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 140000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 15: 10000 rows (total: 150000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 150000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 16: 10000 rows (total: 160000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 160000 rows so far...
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 17: 10000 rows (total: 170000)
2026-02-05 23:12:46 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 170000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 18: 10000 rows (total: 180000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 180000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 19: 10000 rows (total: 190000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 190000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 20: 10000 rows (total: 200000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 200000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 21: 10000 rows (total: 210000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 210000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 22: 10000 rows (total: 220000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 220000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 23: 10000 rows (total: 230000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 230000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 24: 10000 rows (total: 240000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 240000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 25: 10000 rows (total: 250000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 250000 rows so far...
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 26: 10000 rows (total: 260000)
2026-02-05 23:12:47 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 260000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 27: 10000 rows (total: 270000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 270000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 28: 10000 rows (total: 280000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 280000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 29: 10000 rows (total: 290000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 290000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 30: 10000 rows (total: 300000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 300000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 31: 10000 rows (total: 310000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 310000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 32: 10000 rows (total: 320000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 320000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 33: 10000 rows (total: 330000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 330000 rows so far...
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 34: 10000 rows (total: 340000)
2026-02-05 23:12:48 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 340000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 35: 10000 rows (total: 350000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 350000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 36: 10000 rows (total: 360000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 360000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 37: 10000 rows (total: 370000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 370000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 38: 10000 rows (total: 380000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 380000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 39: 10000 rows (total: 390000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 390000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 40: 10000 rows (total: 400000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 400000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 41: 10000 rows (total: 410000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 410000 rows so far...
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 42: 10000 rows (total: 420000)
2026-02-05 23:12:49 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 420000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 43: 10000 rows (total: 430000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 430000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 44: 10000 rows (total: 440000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 440000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 45: 10000 rows (total: 450000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 450000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 46: 10000 rows (total: 460000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 460000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 47: 10000 rows (total: 470000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 470000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 48: 10000 rows (total: 480000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 480000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 49: 10000 rows (total: 490000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 490000 rows so far...
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 50: 10000 rows (total: 500000)
2026-02-05 23:12:50 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 500000 rows so far...
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 51: 10000 rows (total: 510000)
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 510000 rows so far...
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 52: 10000 rows (total: 520000)
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 520000 rows so far...
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 53: 10000 rows (total: 530000)
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 530000 rows so far...
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 54: 10000 rows (total: 540000)
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 540000 rows so far...
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 55: 10000 rows (total: 550000)
2026-02-05 23:12:51 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 550000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 56: 10000 rows (total: 560000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 560000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 57: 10000 rows (total: 570000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 570000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 58: 10000 rows (total: 580000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 580000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 59: 10000 rows (total: 590000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 590000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 60: 10000 rows (total: 600000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 600000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 61: 10000 rows (total: 610000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 610000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 62: 10000 rows (total: 620000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 620000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 63: 10000 rows (total: 630000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 630000 rows so far...
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 64: 10000 rows (total: 640000)
2026-02-05 23:12:52 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 640000 rows so far...
2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 65: 10000 rows (total: 650000)
2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 650000 rows so far...
2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Chunk 66: 6695 rows (total: 656695)
2026-02-05 23:12:53 [DEBUG] pathways.data_processing.snowflake_connector: Fetched 656695 rows so far...
2026-02-05 23:12:53 [INFO] pathways.data_processing.snowflake_connector: Chunked dict query returned 656695 rows in 66 chunks (10.05s)
2026-02-05 23:12:53 [INFO] pathways.data_processing.snowflake_connector: Fetched 656695 activity records from Snowflake
2026-02-05 23:12:53 [INFO] pathways.data_processing.pathway_pipeline: Fetched 656695 records from Snowflake
2026-02-05 23:12:55 [INFO] pathways.data_processing.pathway_pipeline: Applying data transformations...
2026-02-05 23:12:55 [INFO] pathways.data_processing.pathway_pipeline: Generated UPID for 42769 unique patients
2026-02-05 23:12:56 [INFO] pathways.data_processing.pathway_pipeline: Removed 136847 rows with unmapped drug names
2026-02-05 23:12:57 [ERROR] pathways.tools.data: Error processing column NCDR Treatment Function Name: 'NCDR Treatment Function Name'
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:139: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.

  df["Treatment Function Code"].replace(np.nan, 0, inplace=True)
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:241: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  calculated_dirs = valid_groups.apply(lambda grp: find_first_valid_dir(grp, drug_to_valid_dirs))
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:263: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.

  df['Directory'].fillna(df['Calculated_Directory'], inplace=True)
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:273: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.

  df['UPID'].replace('', np.nan, inplace=True) # Ensure empty strings are NaN
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\tools\data.py:304: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.

  df['Directory'].fillna("Undefined", inplace=True)
2026-02-05 23:18:58 [INFO] pathways.data_processing.pathway_pipeline: Assigned directories to 519848 records
2026-02-05 23:18:58 [INFO] pathways.data_processing.pathway_pipeline: Data transformation complete. Final record count: 519848
2026-02-05 23:18:58 [INFO] pathways.__main__: Fetched 519848 records from Snowflake
2026-02-05 23:18:58 [INFO] pathways.__main__:
2026-02-05 23:18:58 [INFO] pathways.__main__: Step 2/4: Processing pathway data for 6 datasets (6 date filters x 1 chart types)...
2026-02-05 23:18:58 [INFO] pathways.__main__:
2026-02-05 23:18:58 [INFO] pathways.__main__: Processing chart type: indication
2026-02-05 23:18:58 [INFO] pathways.__main__: Building drug-aware indication groups...
2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup: Loaded drug-indication mapping: 162 Search_Terms, 346 drug fragments
2026-02-05 23:18:58 [INFO] pathways.__main__: Loaded drug mapping: 162 Search_Terms
2026-02-05 23:18:58 [INFO] pathways.__main__: Looking up GP diagnoses for 36628 unique patients...
2026-02-05 23:18:58 [INFO] pathways.__main__: Restricting GP codes to HCD window: >= 2019-04-01
2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup: Starting Snowflake-direct indication lookup for 36628 patients...
2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup:   Restricting GP codes to >= 2019-04-01
2026-02-05 23:18:58 [INFO] pathways.data_processing.diagnosis_lookup: Batch 1/74: patients 1 to 500
2026-02-05 23:18:58 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:18:58 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...
2026-02-05 23:19:29 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 1: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
2026-02-05 23:19:29 [INFO] pathways.data_processing.diagnosis_lookup: Batch 2/74: patients 501 to 1000
2026-02-05 23:19:29 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:19:29 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...
2026-02-05 23:20:00 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 2: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
2026-02-05 23:20:00 [INFO] pathways.data_processing.diagnosis_lookup: Batch 3/74: patients 1001 to 1500
2026-02-05 23:20:00 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:20:00 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...
2026-02-05 23:20:30 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 3: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
2026-02-05 23:20:30 [INFO] pathways.data_processing.diagnosis_lookup: Batch 4/74: patients 1501 to 2000
2026-02-05 23:20:30 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:20:30 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...
2026-02-05 23:21:00 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 4: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
2026-02-05 23:21:00 [INFO] pathways.data_processing.diagnosis_lookup: Batch 5/74: patients 2001 to 2500
2026-02-05 23:21:00 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:21:00 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...
2026-02-05 23:21:30 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 5: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
2026-02-05 23:21:30 [INFO] pathways.data_processing.diagnosis_lookup: Batch 6/74: patients 2501 to 3000
2026-02-05 23:21:30 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:21:30 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...
2026-02-05 23:22:00 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 6: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
2026-02-05 23:22:00 [INFO] pathways.data_processing.diagnosis_lookup: Batch 7/74: patients 3001 to 3500
2026-02-05 23:22:00 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:22:00 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...
2026-02-05 23:22:31 [ERROR] pathways.data_processing.diagnosis_lookup: Error querying GP records for batch 7: 000604 (57014): SQL execution was cancelled by the client due to a timeout. Error message received from the server: SQL execution canceled
2026-02-05 23:22:31 [INFO] pathways.data_processing.diagnosis_lookup: Batch 8/74: patients 3501 to 4000
2026-02-05 23:22:31 [INFO] pathways.data_processing.snowflake_connector: Executing query (timeout=300s)
2026-02-05 23:22:31 [DEBUG] pathways.data_processing.snowflake_connector: Query:

WITH SearchTermClusters AS (
    SELECT Search_Term, Cluster_ID FROM (VALUES
        ('acute lymphoblastic leukaemia', 'HAEMCANMORPH_COD'),
        ('acute myeloid leukaemia', 'C19HAEMCAN_COD'),
   ...


Every batch is timing out after ~30 seconds. The query is too slow with the `GROUP BY + COUNT(*)` change. The previous version (with `QUALIFY ROW_NUMBER()`) worked because it returned 1 row per patient. Now it's doing `GROUP BY + COUNT(*)` which is much more expensive on the large GP records table.

Let me stop the background task since it's going to keep failing, and investigate the query.


Successfully stopped task: b361350 (cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && python -m cli.refresh_pathways --chart-type indication --dry-run -v 2>&1)


The issue is clear: the updated query with `GROUP BY + COUNT(*)` is timing out at 30 seconds per batch. The original `QUALIFY ROW_NUMBER()` version worked within the timeout. I need to understand the query structure and the timeout setting.

Let me look at the actual query and the timeout configuration.


170:        timeout_cfg = self._config.timeouts
188:                "login_timeout": timeout_cfg.login_timeout,
189:                "network_timeout": timeout_cfg.connection_timeout,
329:        timeout: Optional[int] = None
337:            timeout: Optional query timeout in seconds (overrides config)
349:        effective_timeout = timeout or self._config.timeouts.query_timeout
352:            logger.info(f"Executing query (timeout={effective_timeout}s)")
355:            if effective_timeout > 0:
356:                cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
370:        timeout: Optional[int] = None
378:            timeout: Optional query timeout in seconds
386:        effective_timeout = timeout or self._config.timeouts.query_timeout
389:            logger.info(f"Executing query (timeout={effective_timeout}s)")
392:            if effective_timeout > 0:
393:                cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
408:        timeout: Optional[int] = None,
421:            timeout: Optional query timeout in seconds (overrides config)
434:        effective_timeout = timeout or self._config.timeouts.query_timeout
439:            logger.info(f"Executing chunked query (chunk_size={effective_chunk_size}, timeout={effective_timeout}s)")
442:            if effective_timeout > 0:
443:                cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
478:        timeout: Optional[int] = None,
490:            timeout: Optional query timeout in seconds
499:        effective_timeout = timeout or self._config.timeouts.query_timeout
504:            logger.info(f"Executing chunked dict query (chunk_size={effective_chunk_size}, timeout={effective_timeout}s)")
507:            if effective_timeout > 0:
508:                cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
543:        timeout: Optional[int] = None
554:            timeout: Optional query timeout in seconds
564:        effective_timeout = timeout or self._config.timeouts.query_timeout
568:            logger.info(f"Executing query with limit (max_rows={effective_max_rows}, timeout={effective_timeout}s)")
571:            if effective_timeout > 0:
572:                cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
594:        timeout: Optional[int] = None,
607:            timeout: Query timeout in seconds (default from config)
681:        effective_timeout = timeout or self._config.timeouts.query_timeout
690:            timeout=effective_timeout,


1568:def get_patient_indication_groups(

1568:def get_patient_indication_groups(


  1568ÔåÆdef get_patient_indication_groups(
  1569ÔåÆ    patient_pseudonyms: list[str],
  1570ÔåÆ    connector: Optional[SnowflakeConnector] = None,
  1571ÔåÆ    batch_size: int = 500,
  1572ÔåÆ    earliest_hcd_date: Optional[str] = None,
  1573ÔåÆ) -> "pd.DataFrame":
  1574ÔåÆ    """
  1575ÔåÆ    Batch lookup GP diagnosis-based indication groups using Snowflake cluster query.
  1576ÔåÆ
  1577ÔåÆ    Returns ALL matching Search_Terms per patient with code_frequency (count of
  1578ÔåÆ    matching SNOMED codes). This enables drug-aware indication matching where
  1579ÔåÆ    each drug is cross-referenced against the patient's GP diagnoses.
  1580ÔåÆ
  1581ÔåÆ    The query:
  1582ÔåÆ    1. Uses the cluster mapping CTE to get all Search_Term -> SNOMED code mappings
  1583ÔåÆ    2. Joins with PrimaryCareClinicalCoding to find patients with matching codes
  1584ÔåÆ    3. Groups by patient + Search_Term and counts matching codes (code_frequency)
  1585ÔåÆ    4. Optionally restricts to GP codes from earliest_hcd_date onwards
  1586ÔåÆ
  1587ÔåÆ    Args:
  1588ÔåÆ        patient_pseudonyms: List of PseudoNHSNoLinked values (matches PatientPseudonym in GP records)
  1589ÔåÆ        connector: Optional SnowflakeConnector (defaults to singleton)
  1590ÔåÆ        batch_size: Number of patients per Snowflake query batch (default 500)
  1591ÔåÆ        earliest_hcd_date: Optional ISO date string (YYYY-MM-DD). If provided, only
  1592ÔåÆ            counts GP codes from this date onwards. Should be MIN(Intervention Date)
  1593ÔåÆ            from the HCD DataFrame to restrict to the HCD data window.
  1594ÔåÆ
  1595ÔåÆ    Returns:
  1596ÔåÆ        DataFrame with columns:
  1597ÔåÆ        - PatientPseudonym: The patient identifier (PseudoNHSNoLinked value)
  1598ÔåÆ        - Search_Term: The matched indication (e.g., "rheumatoid arthritis")
  1599ÔåÆ        - code_frequency: Count of matching SNOMED codes for this Search_Term
  1600ÔåÆ
  1601ÔåÆ        Multiple rows per patient (one per matched Search_Term).
  1602ÔåÆ        Patients not found in results have no matching GP diagnosis.
  1603ÔåÆ    """
  1604ÔåÆ    import pandas as pd
  1605ÔåÆ
  1606ÔåÆ    logger.info(f"Starting Snowflake-direct indication lookup for {len(patient_pseudonyms)} patients...")
  1607ÔåÆ    if earliest_hcd_date:
  1608ÔåÆ        logger.info(f"  Restricting GP codes to >= {earliest_hcd_date}")
  1609ÔåÆ
  1610ÔåÆ    # Handle edge case: empty patient list
  1611ÔåÆ    if not patient_pseudonyms:
  1612ÔåÆ        logger.warning("Empty patient list provided")
  1613ÔåÆ        return pd.DataFrame(columns=['PatientPseudonym', 'Search_Term', 'code_frequency'])
  1614ÔåÆ
  1615ÔåÆ    # Check Snowflake availability
  1616ÔåÆ    if not SNOWFLAKE_AVAILABLE:
  1617ÔåÆ        logger.error("Snowflake connector not available - cannot lookup GP records")
  1618ÔåÆ        return pd.DataFrame(columns=['PatientPseudonym', 'Search_Term', 'code_frequency'])
  1619ÔåÆ
  1620ÔåÆ    if not is_snowflake_configured():
  1621ÔåÆ        logger.error("Snowflake not configured - cannot lookup GP records")
  1622ÔåÆ        return pd.DataFrame(columns=['PatientPseudonym', 'Search_Term', 'code_frequency'])
  1623ÔåÆ
  1624ÔåÆ    if connector is None:
  1625ÔåÆ        connector = get_connector()
  1626ÔåÆ
  1627ÔåÆ    # Results list to collect all matches
  1628ÔåÆ    all_results: list[dict] = []
  1629ÔåÆ
  1630ÔåÆ    # Process patients in batches
  1631ÔåÆ    total_patients = len(patient_pseudonyms)
  1632ÔåÆ    for batch_start in range(0, total_patients, batch_size):
  1633ÔåÆ        batch_end = min(batch_start + batch_size, total_patients)
  1634ÔåÆ        batch_pseudonyms = patient_pseudonyms[batch_start:batch_end]
  1635ÔåÆ        batch_num = batch_start // batch_size + 1
  1636ÔåÆ        total_batches = (total_patients + batch_size - 1) // batch_size
  1637ÔåÆ
  1638ÔåÆ        logger.info(f"Batch {batch_num}/{total_batches}: patients {batch_start + 1} to {batch_end}")
  1639ÔåÆ
  1640ÔåÆ        # Build patient IN clause placeholders
  1641ÔåÆ        patient_placeholders = ", ".join(["%s"] * len(batch_pseudonyms))
  1642ÔåÆ
  1643ÔåÆ        # Build WHERE clause with optional date filter
  1644ÔåÆ        date_filter = ""
  1645ÔåÆ        if earliest_hcd_date:
  1646ÔåÆ            date_filter = f"\n    AND pc.\"EventDateTime\" >= %s"
  1647ÔåÆ
  1648ÔåÆ        # Build the full query with cluster CTE
  1649ÔåÆ        # Returns ALL matching Search_Terms per patient with code_frequency
  1650ÔåÆ        # code_frequency = COUNT of matching SNOMED codes per Search_Term per patient
  1651ÔåÆ        query = f"""
  1652ÔåÆ{CLUSTER_MAPPING_SQL}
  1653ÔåÆSELECT
  1654ÔåÆ    pc."PatientPseudonym" AS "PatientPseudonym",
  1655ÔåÆ    aic.Search_Term AS "Search_Term",
  1656ÔåÆ    COUNT(*) AS "code_frequency"
  1657ÔåÆFROM DATA_HUB.PHM."PrimaryCareClinicalCoding" pc
  1658ÔåÆINNER JOIN AllIndicationCodes aic
  1659ÔåÆ    ON pc."SNOMEDCode" = aic.SNOMEDCode
  1660ÔåÆWHERE pc."PatientPseudonym" IN ({patient_placeholders}){date_filter}
  1661ÔåÆGROUP BY pc."PatientPseudonym", aic.Search_Term
  1662ÔåÆ"""
  1663ÔåÆ
  1664ÔåÆ        # Build params: patient pseudonyms + optional date
  1665ÔåÆ        params = list(batch_pseudonyms)
  1666ÔåÆ        if earliest_hcd_date:
  1667ÔåÆ            params.append(earliest_hcd_date)
  1668ÔåÆ
  1669ÔåÆ        try:
  1670ÔåÆ            results = connector.execute_dict(query, tuple(params))
  1671ÔåÆ
  1672ÔåÆ            for row in results:
  1673ÔåÆ                all_results.append({
  1674ÔåÆ                    'PatientPseudonym': row.get('PatientPseudonym'),
  1675ÔåÆ                    'Search_Term': row.get('Search_Term'),
  1676ÔåÆ                    'code_frequency': row.get('code_frequency', 0),
  1677ÔåÆ                })
  1678ÔåÆ
  1679ÔåÆ            logger.debug(f"Batch {batch_num}: found {len(results)} patient-indication matches")
  1680ÔåÆ
  1681ÔåÆ        except Exception as e:
  1682ÔåÆ            logger.error(f"Error querying GP records for batch {batch_num}: {e}")
  1683ÔåÆ            # Continue with other batches - partial results are better than none
  1684ÔåÆ
  1685ÔåÆ    # Build result DataFrame
  1686ÔåÆ    result_df = pd.DataFrame(all_results)
  1687ÔåÆ
  1688ÔåÆ    # Log summary statistics
  1689ÔåÆ    if len(result_df) > 0:
  1690ÔåÆ        unique_patients = result_df['PatientPseudonym'].nunique()
  1691ÔåÆ        total_rows = len(result_df)
  1692ÔåÆ        match_rate = 100 * unique_patients / total_patients
  1693ÔåÆ        unique_terms = result_df['Search_Term'].nunique()
  1694ÔåÆ        avg_indications = total_rows / unique_patients if unique_patients > 0 else 0
  1695ÔåÆ        logger.info(f"Indication lookup complete:")
  1696ÔåÆ        logger.info(f"  Total patients queried: {total_patients}")
  1697ÔåÆ        logger.info(f"  Patients with GP match: {unique_patients} ({match_rate:.1f}%)")
  1698ÔåÆ        logger.info(f"  Total patient-indication rows: {total_rows} (avg {avg_indications:.1f} per patient)")
  1699ÔåÆ        logger.info(f"  Unique Search_Terms found: {unique_terms}")
  1700ÔåÆ
  1701ÔåÆ        # Log top Search_Terms
  1702ÔåÆ        top_terms = result_df['Search_Term'].value_counts().head(5)
  1703ÔåÆ        logger.info(f"  Top 5 indications: {dict(top_terms)}")
  1704ÔåÆ    else:
  1705ÔåÆ        logger.info(f"Indication lookup complete: 0 matches from {total_patients} patients")
  1706ÔåÆ
  1707ÔåÆ    return result_df
  1708ÔåÆ
  1709ÔåÆ
  1710ÔåÆ# Export public API
  1711ÔåÆ__all__ = [
  1712ÔåÆ    # Dataclasses
  1713ÔåÆ    "ClusterSnomedCodes",
  1714ÔåÆ    "IndicationValidationResult",
  1715ÔåÆ    "DrugIndicationMatchRate",
  1716ÔåÆ    "DrugSnomedMapping",
  1717ÔåÆ    "DirectSnomedMatchResult",
  1718ÔåÆ    "DirectorateAssignment",
  1719ÔåÆ    # Cluster-based lookup functions (existing)
  1720ÔåÆ    "get_drug_clusters",
  1721ÔåÆ    "get_drug_cluster_ids",
  1722ÔåÆ    "get_cluster_snomed_codes",
  1723ÔåÆ    "patient_has_indication",
  1724ÔåÆ    "validate_indication",
  1725ÔåÆ    "get_indication_match_rate",
  1726ÔåÆ    "batch_validate_indications",
  1727ÔåÆ    "get_available_clusters",
  1728ÔåÆ    # Direct SNOMED lookup functions (new)
  1729ÔåÆ    "get_drug_snomed_codes",
  1730ÔåÆ    "patient_has_indication_direct",
  1731ÔåÆ    # Diagnosis-based directorate assignment
  1732ÔåÆ    "get_directorate_from_diagnosis",
  1733ÔåÆ    # Batch lookup for indication groups
  1734ÔåÆ    "batch_lookup_indication_groups",
  1735ÔåÆ    # Drug-indication mapping from DimSearchTerm.csv
  1736ÔåÆ    "SEARCH_TERM_MERGE_MAP",
  1737ÔåÆ    "load_drug_indication_mapping",
  1738ÔåÆ    "get_search_terms_for_drug",
  1739ÔåÆ    # Drug-aware indication assignment
  1740ÔåÆ    "assign_drug_indications",
  1741ÔåÆ    # Snowflake-direct indication lookup (new approach)
  1742ÔåÆ    "get_patient_indication_groups",
  1743ÔåÆ    "CLUSTER_MAPPING_SQL",
  1744ÔåÆ]
  1745ÔåÆ

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

def get_patient_indication_groups(
    patient_pseudonyms: list[str],
    connector: Optional[SnowflakeConnector] = None,
    batch_size: int = 500,
    earliest_hcd_date: Optional[str] = None,
) -> "pd.DataFrame":
    """
    Batch lookup GP diagnosis-based indication groups using Snowflake cluster query.

    Returns ALL matching Search_Terms per patient with code_frequency (count of
    matching SNOMED codes). This enables drug-aware indication matching where
    each drug is cross-referenced against the patient's GP diagnoses.

    The query:
    1. Uses the cluster mapping CTE to get all Search_Term -> SNOMED code mappings
    2. Joins with PrimaryCareClinicalCoding to find patients with matching codes
    3. Groups by patient + Search_Term and counts matching codes (code_frequency)
    4. Optionally restricts to GP codes from earliest_hcd_date onwards

    Args:
        patient_pseudonyms: List of PseudoNHSNoLinked values (matches PatientPseudonym in GP records)
        connector: Optional SnowflakeConnector (defaults to singleton)
        batch_size: Number of patients per Snowflake query batch (default 500)
        earliest_hcd_date: Optional ISO date string (YYYY-MM-DD). If provided, only
            counts GP codes from this date onwards. Should be MIN(Intervention Date)
            from the HCD DataFrame to restrict to the HCD data window.

    Returns:
        DataFrame with columns:
        - PatientPseudonym: The patient identifier (PseudoNHSNoLinked value)
        - Search_Term: The matched indication (e.g., "rheumatoid arthritis")
        - code_frequency: Count of matching SNOMED codes for this Search_Term

        Multiple rows per patient (one per matched Search_Term).
        Patients not found in results have no matching GP diagnosis.
    """
    import pandas as pd

    logger.info(f"Starting Snowflake-direct indication lookup for {len(patient_pseudonyms)} patients...")
    if earliest_hcd_date:
        logger.info(f"  Restricting GP codes to >= {earliest_hcd_date}")

    # Handle edge case: empty patient list
    if not patient_pseudonyms:
        logger.warning("Empty patient list provided")
        return pd.DataFrame(columns=['PatientPseudonym', 'Search_Term', 'code_frequency'])

    # Check Snowflake availability
    if not SNOWFLAKE_AVAILABLE:
        logger.error("Snowflake connector not available - cannot lookup GP records")
        return pd.DataFrame(columns=['PatientPseudonym', 'Search_Term', 'code_frequency'])

    if not is_snowflake_configured():
        logger.error("Snowflake not configured - cannot lookup GP records")
        return pd.DataFrame(columns=['PatientPseudonym', 'Search_Term', 'code_frequency'])

    if connector is None:
        connector = get_connector()

    # Results list to collect all matches
    all_results: list[dict] = []

    # Process patients in batches
    total_patients = len(patient_pseudonyms)
    for batch_start in range(0, total_patients, batch_size):
        batch_end = min(batch_start + batch_size, total_patients)
        batch_pseudonyms = patient_pseudonyms[batch_start:batch_end]
        batch_num = batch_start // batch_size + 1
        total_batches = (total_patients + batch_size - 1) // batch_size

        logger.info(f"Batch {batch_num}/{total_batches}: patients {batch_start + 1} to {batch_end}")

        # Build patient IN clause placeholders
        patient_placeholders = ", ".join(["%s"] * len(batch_pseudonyms))

        # Build WHERE clause with optional date filter
        date_filter = ""
        if earliest_hcd_date:
            date_filter = f"\n    AND pc.\"EventDateTime\" >= %s"

        # Build the full query with cluster CTE
        # Returns ALL matching Search_Terms per patient with code_frequency
        # code_frequency = COUNT of matching SNOMED codes per Search_Term per patient
        query = f"""
{CLUSTER_MAPPING_SQL}
SELECT
    pc."PatientPseudonym" AS "PatientPseudonym",
    aic.Search_Term AS "Search_Term",
    COUNT(*) AS "code_frequency"
FROM DATA_HUB.PHM."PrimaryCareClinicalCoding" pc
INNER JOIN AllIndicationCodes aic
    ON pc."SNOMEDCode" = aic.SNOMEDCode
WHERE pc."PatientPseudonym" IN ({patient_placeholders}){date_filter}
GROUP BY pc."PatientPseudonym", aic.Search_Term
"""

        # Build params: patient pseudonyms + optional date
        params = list(batch_pseudonyms)
        if earliest_hcd_date:
            params.append(earliest_hcd_date)

        try:
            results = connector.execute_dict(query, tuple(params))

            for row in results:
                all_results.append({
                    'PatientPseudonym': row.get('PatientPseudonym'),
                    'Search_Term': row.get('Search_Term'),
                    'code_frequency': row.get('code_frequency', 0),
                })

            logger.debug(f"Batch {batch_num}: found {len(results)} patient-indication matches")

        except Exception as e:
            logger.error(f"Error querying GP records for batch {batch_num}: {e}")
            # Continue with other batches - partial results are better than none

    # Build result DataFrame
    result_df = pd.DataFrame(all_results)

    # Log summary statistics
    if len(result_df) > 0:
        unique_patients = result_df['PatientPseudonym'].nunique()
        total_rows = len(result_df)
        match_rate = 100 * unique_patients / total_patients
        unique_terms = result_df['Search_Term'].nunique()
        avg_indications = total_rows / unique_patients if unique_patients > 0 else 0
        logger.info(f"Indication lookup complete:")
        logger.info(f"  Total patients queried: {total_patients}")
        logger.info(f"  Patients with GP match: {unique_patients} ({match_rate:.1f}%)")
        logger.info(f"  Total patient-indication rows: {total_rows} (avg {avg_indications:.1f} per patient)")
        logger.info(f"  Unique Search_Terms found: {unique_terms}")

        # Log top Search_Terms
        top_terms = result_df['Search_Term'].value_counts().head(5)
        logger.info(f"  Top 5 indications: {dict(top_terms)}")
    else:
        logger.info(f"Indication lookup complete: 0 matches from {total_patients} patients")

    return result_df

# Export public API
__all__ = [
    # Dataclasses
    "ClusterSnomedCodes",
    "IndicationValidationResult",
    "DrugIndicationMatchRate",
    "DrugSnomedMapping",
    "DirectSnomedMatchResult",
    "DirectorateAssignment",
    # Cluster-based lookup functions (existing)
    "get_drug_clusters",
    "get_drug_cluster_ids",
    "get_cluster_snomed_codes",
    "patient_has_indication",
    "validate_indication",
    "get_indication_match_rate",
    "batch_validate_indications",
    "get_available_clusters",
    # Direct SNOMED lookup functions (new)
    "get_drug_snomed_codes",
    "patient_has_indication_direct",
    # Diagnosis-based directorate assignment
    "get_directorate_from_diagnosis",
    # Batch lookup for indication groups
    "batch_lookup_indication_groups",
    # Drug-indication mapping from DimSearchTerm.csv
    "SEARCH_TERM_MERGE_MAP",
    "load_drug_indication_mapping",
    "get_search_terms_for_drug",
    # Drug-aware indication assignment
    "assign_drug_indications",
    # Snowflake-direct indication lookup (new approach)
    "get_patient_indication_groups",
    "CLUSTER_MAPPING_SQL",
]


349:        effective_timeout = timeout or self._config.timeouts.query_timeout
386:        effective_timeout = timeout or self._config.timeouts.query_timeout
434:        effective_timeout = timeout or self._config.timeouts.query_timeout
499:        effective_timeout = timeout or self._config.timeouts.query_timeout
564:        effective_timeout = timeout or self._config.timeouts.query_timeout
681:        effective_timeout = timeout or self._config.timeouts.query_timeout


Let me check the default query timeout and the `execute_dict` method to understand the 30s timeout.


No matches found


57-    pass
58-
59-
60:class SnowflakeNotConfiguredError(Exception):
61-    """Raised when Snowflake is not configured (no account)."""
62-    pass
63-
--
167-            self.close()
168-
169-        conn_cfg = self._config.connection
170:        timeout_cfg = self._config.timeouts
171-
172-        logger.info(f"Connecting to Snowflake account: {conn_cfg.account}")
173-        logger.info(f"Using warehouse: {conn_cfg.warehouse}, database: {conn_cfg.database}")
--
346-        if not self.is_connected:
347-            self.connect()
348-
349:        effective_timeout = timeout or self._config.timeouts.query_timeout
350-
351-        with self.get_cursor() as cursor:
352-            logger.info(f"Executing query (timeout={effective_timeout}s)")
--
383-        if not self.is_connected:
384-            self.connect()
385-
386:        effective_timeout = timeout or self._config.timeouts.query_timeout
387-
388-        with self.get_cursor(dict_cursor=True) as cursor:
389-            logger.info(f"Executing query (timeout={effective_timeout}s)")
--
431-        if not self.is_connected:
432-            self.connect()
433-
434:        effective_timeout = timeout or self._config.timeouts.query_timeout
435-        effective_chunk_size = chunk_size or self._config.query.chunk_size
436-        effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
437-
--
496-        if not self.is_connected:
497-            self.connect()
498-
499:        effective_timeout = timeout or self._config.timeouts.query_timeout
500-        effective_chunk_size = chunk_size or self._config.query.chunk_size
501-        effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
502-
--
561-        if not self.is_connected:
562-            self.connect()
563-
564:        effective_timeout = timeout or self._config.timeouts.query_timeout
565-        effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
566-
567-        with self.get_cursor(dict_cursor=True) as cursor:
--
678-            logger.info(f"  Providers: {provider_codes}")
679-
680-        effective_max_rows = max_rows if max_rows is not None else self._config.query.max_rows
681:        effective_timeout = timeout or self._config.timeouts.query_timeout
682-
683-        # Execute with chunked results for large datasets
684-        all_results = []


60:class SnowflakeNotConfiguredError(Exception):


The config is not defined in this file. Let me find where the timeout defaults are.


C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\config\__init__.py:34:class TimeoutConfig:
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\config\__init__.py:37:    query_timeout: int = 300
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\config\__init__.py:96:    timeouts: TimeoutConfig = field(default_factory=TimeoutConfig)
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\config\__init__.py:184:    timeouts = TimeoutConfig(
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\config\__init__.py:186:        query_timeout=timeout_data.get("query_timeout", 300),
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\config\__init__.py:260:    "TimeoutConfig",
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\snowflake_connector.py:349:        effective_timeout = timeout or self._config.timeouts.query_timeout
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\snowflake_connector.py:386:        effective_timeout = timeout or self._config.timeouts.query_timeout
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\snowflake_connector.py:434:        effective_timeout = timeout or self._config.timeouts.query_timeout
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\snowflake_connector.py:499:        effective_timeout = timeout or self._config.timeouts.query_timeout
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\snowflake_connector.py:564:        effective_timeout = timeout or self._config.timeouts.query_timeout
C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\snowflake_connector.py:681:        effective_timeout = timeout or self._config.timeouts.query_timeout


The default query_timeout is 300 seconds. But the logs show queries being cancelled after ~30 seconds: `000604 (57014): SQL execution was cancelled by the client due to a timeout`. Wait ÔÇö the error says "SQL execution cancelled by the client due to a timeout" but the Snowflake session timeout is set to 300s. Let me look more carefully. The `execute_dict` method is what's being called.


   370ÔåÆ        timeout: Optional[int] = None
   371ÔåÆ    ) -> list[dict]:
   372ÔåÆ        """
   373ÔåÆ        Execute a query and return results as list of dictionaries.
   374ÔåÆ
   375ÔåÆ        Args:
   376ÔåÆ            query: SQL query to execute
   377ÔåÆ            params: Optional query parameters
   378ÔåÆ            timeout: Optional query timeout in seconds
   379ÔåÆ
   380ÔåÆ        Returns:
   381ÔåÆ            List of result rows as dictionaries
   382ÔåÆ        """
   383ÔåÆ        if not self.is_connected:
   384ÔåÆ            self.connect()
   385ÔåÆ
   386ÔåÆ        effective_timeout = timeout or self._config.timeouts.query_timeout
   387ÔåÆ
   388ÔåÆ        with self.get_cursor(dict_cursor=True) as cursor:
   389ÔåÆ            logger.info(f"Executing query (timeout={effective_timeout}s)")
   390ÔåÆ            logger.debug(f"Query: {query[:200]}...")
   391ÔåÆ
   392ÔåÆ            if effective_timeout > 0:
   393ÔåÆ                cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")
   394ÔåÆ
   395ÔåÆ            start_time = time.time()
   396ÔåÆ            cursor.execute(query, params)
   397ÔåÆ            results = cursor.fetchall()
   398ÔåÆ            elapsed = time.time() - start_time
   399ÔåÆ
   400ÔåÆ            logger.info(f"Query returned {len(results)} rows in {elapsed:.2f}s")
   401ÔåÆ            return results  # type: ignore[return-value]
   402ÔåÆ
   403ÔåÆ    def execute_chunked(
   404ÔåÆ        self,
   405ÔåÆ        query: str,
   406ÔåÆ        params: Optional[tuple] = None,
   407ÔåÆ        chunk_size: Optional[int] = None,
   408ÔåÆ        timeout: Optional[int] = None,
   409ÔåÆ        max_rows: Optional[int] = None,

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>

        timeout: Optional[int] = None
    ) -> list[dict]:
        """
        Execute a query and return results as list of dictionaries.

        Args:
            query: SQL query to execute
            params: Optional query parameters
            timeout: Optional query timeout in seconds

        Returns:
            List of result rows as dictionaries
        """
        if not self.is_connected:
            self.connect()

        effective_timeout = timeout or self._config.timeouts.query_timeout

        with self.get_cursor(dict_cursor=True) as cursor:
            logger.info(f"Executing query (timeout={effective_timeout}s)")
            logger.debug(f"Query: {query[:200]}...")

            if effective_timeout > 0:
                cursor.execute(f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {effective_timeout}")

            start_time = time.time()
            cursor.execute(query, params)
            results = cursor.fetchall()
            elapsed = time.time() - start_time

            logger.info(f"Query returned {len(results)} rows in {elapsed:.2f}s")
            return results  # type: ignore[return-value]

    def execute_chunked(
        self,
        query: str,
        params: Optional[tuple] = None,
        chunk_size: Optional[int] = None,
        timeout: Optional[int] = None,
        max_rows: Optional[int] = None,


The log says `Executing query (timeout=300s)` but the queries are dying at ~30s. Wait ÔÇö let me re-read the error timestamps more carefully:

- Batch 1: 23:18:58 ÔåÆ 23:19:29 = 31s
- Batch 2: 23:19:29 ÔåÆ 23:20:00 = 31s

Each batch is timing out at ~30s. The error code is `000604 (57014)` which is Snowflake's SQL cancellation. But the timeout is set to 300s. This suggests something else is cancelling ÔÇö maybe the warehouse has a statement timeout configured at the Snowflake level (30s), which would override the session setting.

The real issue is the query performance. The `GROUP BY + COUNT(*)` with the large CTE and join on `PrimaryCareClinicalCoding` is expensive. The original `QUALIFY ROW_NUMBER()` query worked because it could short-circuit once it found one match per patient.

Let me understand the problem better. The key difference is:
- **Old query**: `QUALIFY ROW_NUMBER() = 1` ÔåÆ returns 1 row per patient (efficient, limits result set)
- **New query**: `GROUP BY + COUNT(*)` ÔåÆ scans ALL matching rows, groups them, counts ÔåÆ much heavier

The solution is to optimize the query. Let me think about approaches:

1. **Increase timeout** ÔÇö Won't help if the query is inherently slow for 500-patient batches
2. **Reduce batch size** ÔÇö smaller batches might complete faster
3. **Optimize query structure** ÔÇö maybe restructure to be more efficient

Let me test with a direct Snowflake query to understand the actual query plan. First, let me check if we can run a smaller test batch.


TodoWrite

Diagnose query timeout: GROUP BY + COUNT(*) too slow for GP lookup batches

Fix query performance or timeout configuration

Re-run dry run test after fix

Mark Task 3.2 complete, commit, update progress.txt


Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable

Run dry run test: python -m cli.refresh_pathways --chart-type indication --dry-run -v

Running indication dry run test with Snowflake

Validate dry run output: modified UPIDs, match rates, pathway nodes

Mark Task 3.2 complete, commit, update progress.txt

Diagnose query timeout: GROUP BY + COUNT(*) too slow for GP lookup batches

Fix query performance or timeout configuration

Re-run dry run test after fix


Let me test the query directly against Snowflake to understand the performance characteristics. I'll run a small batch (10 patients) to see if the query works at all, and also test the current timeout behavior.


Bash

cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && python -c "
import time
from data_processing.snowflake_connector import get_connector, is_snowflake_available

if not is_snowflake_available():
    print('Snowflake not available')
    exit(1)

connector = get_connector()
print(f'Connected: {connector.is_connected}')

# Check the current session timeout
try:
    result = connector.execute_dict('SHOW PARAMETERS LIKE \\'STATEMENT_TIMEOUT_IN_SECONDS\\' IN SESSION')
    print(f'Session timeout params: {result}')
except Exception as e:
    print(f'Error checking timeout: {e}')

# Get a small batch of patient pseudonyms
result = connector.execute_dict('''
    SELECT DISTINCT \"PseudoNHSNoLinked\"
    FROM DataWarehouse.DWH.DimHighCostDrug
    WHERE \"PseudoNHSNoLinked\" IS NOT NULL
    LIMIT 10
''')
patients = [r['PseudoNHSNoLinked'] for r in result]
print(f'Got {len(patients)} test patients')

# Now try the GP lookup query with GROUP BY for just 10 patients
from data_processing.diagnosis_lookup import CLUSTER_MAPPING_SQL, get_patient_indication_groups

start = time.time()
df = get_patient_indication_groups(patients, connector=connector, batch_size=10, earliest_hcd_date='2019-04-01')
elapsed = time.time() - start
print(f'Query completed in {elapsed:.1f}s, rows: {len(df)}')
if not df.empty:
    print(df.head(10))
" 2>&1

Test GP lookup query with small batch of 10 patients


Error: Exit code 1
Traceback (most recent call last):
  File "<string>", line 20, in <module>
    result = connector.execute_dict('''
        SELECT DISTINCT "PseudoNHSNoLinked"
    ...<2 lines>...
        LIMIT 10
    ''')
  File "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign\data_processing\snowflake_connector.py", line 396, in execute_dict
    cursor.execute(query, params)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Users\charlwoodand\scoop\apps\python\current\Lib\site-packages\snowflake\connector\cursor.py", line 1121, in execute
    Error.errorhandler_wrapper(self.connection, self, error_class, errvalue)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\charlwoodand\scoop\apps\python\current\Lib\site-packages\snowflake\connector\errors.py", line 286, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
        connection,
    ...<2 lines>...
        error_value,
    )
  File "C:\Users\charlwoodand\scoop\apps\python\current\Lib\site-packages\snowflake\connector\errors.py", line 341, in hand_to_other_handler
    cursor.errorhandler(connection, cursor, error_class, error_value)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\charlwoodand\scoop\apps\python\current\Lib\site-packages\snowflake\connector\errors.py", line 217, in default_errorhandler
    raise error_class(
    ...<10 lines>...
    )
snowflake.connector.errors.ProgrammingError: 002003 (02000): SQL compilation error:
Database 'DATAWAREHOUSE' does not exist or not authorized.
Connected: False
Session timeout params: [{'key': 'STATEMENT_TIMEOUT_IN_SECONDS', 'value': '300', 'default': '172800', 'level': 'SESSION', 'description': 'Timeout in seconds for statements: statements are automatically canceled if they run for longer; if set to zero, max value (604800) is enforced.', 'type': 'NUMBER'}]


Good progress ÔÇö I can see two things:
1. The session timeout IS set to 300s, but queries still fail at ~30s. This suggests the warehouse itself has a timeout (e.g., XSMALL warehouse with its own statement timeout).
2. The query test failed because I used the wrong database name. Let me fix that and test properly.

Let me check the warehouse statement timeout and also run the actual test with the correct database.


cd "C:\Users\charlwoodand\Ralph local\Tasks\UI Redesign" && python -c "
import time
from data_processing.snowflake_connector import get_connector

connector = get_connector()

# Check warehouse-level timeout
try:
    result = connector.execute_dict('SHOW PARAMETERS LIKE \\'STATEMENT_TIMEOUT_IN_SECONDS\\' IN WAREHOUSE')
    print(f'Warehouse timeout: {result}')
except Exception as e:
    print(f'Error checking warehouse timeout: {e}')

# Check account-level timeout
try:
    result = connector.execute_dict('SHOW PARAMETERS LIKE \\'STATEMENT_TIMEOUT_IN_SECONDS\\' IN ACCOUNT')
    print(f'Account timeout: {result}')
except Exception as e:
    print(f'Error checking account timeout: {e}')
" 2>&1

Check warehouse and account statement timeout settings


Warehouse timeout: [{'key': 'STATEMENT_TIMEOUT_IN_SECONDS', 'value': '7200', 'default': '172800', 'level': 'WAREHOUSE', 'description': 'Timeout in seconds for statements: statements are automatically canceled if they run for longer; if set to zero, max value (604800) is enforced.', 'type': 'NUMBER'}]
Account timeout: [{'key': 'STATEMENT_TIMEOUT_IN_SECONDS', 'value': '172800', 'default': '172800', 'level': '', 'description': 'Timeout in seconds for statements: statements are automatically canceled if they run for longer; if set to zero, max value (604800) is enforced.', 'type': 'NUMBER'}]


The warehouse timeout is 7200s (2 hours) and the session timeout is 300s. Neither would cause a 30s timeout. Let me re-read the error messages more carefully ÔÇö maybe it's a network/connection timeout rather than a query execution timeout.

Actually, looking at the log more carefully: the session sets `STATEMENT_TIMEOUT_IN_SECONDS = 300` but the queries fail at 30s. The error is `000604 (57014): SQL execution was cancelled by the client due to a timeout`. The key phrase is "**by the client**" ÔÇö this is the Python client (snowflake-connector-python) timing out, not Snowflake server.

Let me check if there's a Python-side network timeout or client-side configuration.