Andrew Charlwood c6e426e36c fix: increase network timeout and batch size for GP lookup queries (Task 3.2) ...

Dry run test revealed GP lookup queries timing out at 30s (connection_timeout
in snowflake.toml). Increased to 600s. Also increased batch_size from 500 to
5000 — query time is ~40s regardless of batch size (CTE compilation overhead),
so larger batches reduce total time from ~50min to ~6min for 36K patients.

Dry run results: 91.8% GP match rate, 49.3% drug-indication match rate,
42,072 modified UPIDs, 1,846 pathway nodes across 6 date filters.

2026-02-05 23:55:12 +00:00

66 KiB

Raw Blame History

View Raw