feat: complete Task 2.2 - test refresh pipeline with Snowflake data

Tested full refresh pipeline end-to-end with real Snowflake data:
- Fixed trust filter to read Name column from defaultTrusts.csv
- Fixed Decimal type handling in calculate_cost_per_patient_per_annum
- Fixed array handling in convert_to_records for average_administered
- Added required reference CSV files to data/ directory
- Configured Snowflake connection (account, warehouse, user)

Results:
- Snowflake fetch: 656,695 records in ~7s
- Transformations: 519,848 records after UPID/drug/directory
- Pathway nodes: 293 for all_6mo (8 trusts, 14 directories)
- Total processing time: ~6.2 minutes
This commit is contained in:
Andrew Charlwood
2026-02-05 00:20:12 +00:00
parent 8b65dfd9a8
commit adc1dbfc58
12 changed files with 1708 additions and 21 deletions
+175
View File
@@ -0,0 +1,175 @@
Drug,Include
ABATACEPT,1
ACALABRUTINIB,1
ADALIMUMAB,1
AFLIBERCEPT,1
ALIROCUMAB,1
ALITRETINOIN,0
AMIKACIN,0
AMPHOTERICIN B LIPOSOMAL,0
ANAKINRA,0
ANDEXANET ALFA,0
ANIDULAFUNGIN,0
APALUTAMIDE,0
APOMORPHINE,0
APREMILAST,1
ATEZOLIZUMAB,1
AVELUMAB,0
AXITINIB,0
AZACITIDINE,0
AZATHIOPRINE,0
BACILLUS CALMETTE-GUERIN,0
BARICITINIB,1
BASILIXIMAB,0
BENDAMUSTINE ,0
BENRALIZUMAB,0
BEROTRALSTAT,0
BEVACIZUMAB,1
BEXAROTENE,0
BIMEKIZUMAB,1
BLEOMYCIN,0
BORTEZOMIB,0
BOTULINUM A TOXIN,1
BRENTUXIMAB VEDOTIN ,0
BRODALUMAB,1
BUSULFAN ,0
C1-ESTERASE INHIBITOR ,0
CABOZANTINIB,0
CAPECITABINE,0
CARBOPLATIN,0
CARFILZOMIB,0
CERTOLIZUMAB PEGOL,1
CETUXIMAB,0
CHLORAMBUCIL,0
CICLOSPORIN,0
CINACALCET,0
CISPLATIN,0
COLISTIMETHATE,0
COLISTIN,0
COPAXONE,0
CYCLOPHOSPHAMIDE,0
DARBEPOETIN ALFA,0
DAROLUTAMIDE,0
DASATINIB,0
DEFERASIROX,0
DENOSUMAB,1
DESFERRIOXAMINE,0
DEXAMETHASONE,1
DIGOXIN IMMUNE FAB,0
DIMETHYL FUMARATE,1
DOCETAXEL,1
DOXORUBICIN,0
DUPILUMAB,1
ECULIZUMAB,0
EFUDIX,0
ELTROMBOPAG,0
EMTRICITABINE,0
ENZALUTAMIDE,0
EPIRUBICIN,0
EPOETIN,0
EPOETIN ALFA,0
EPOETIN BETA,0
ERENUMAB,1
ETANERCEPT,1
EVOLOCUMAB,1
FARICIMAB,1
FIDAXOMICIN,0
FILGOTINIB,1
FILGRASTIM,0
FLUOCINOLONE,1
FLUOCINOLONE ACETONIDE,1
FLUOROURACIL,0
FOMEPIZOLE,0
FREMANEZUMAB,1
FULVESTRANT,0
GALCANEZUMAB,1
GEMCITABINE,0
GENERIC,0
GOLIMUMAB,1
GUSELKUMAB,1
HYDROXYCARBAMIDE,0
IBRUTINIB,0
IDARUBICIN,0
IDARUCIZUMAB,1
IFOSFAMIDE,0
ILOPROST,0
ILOPROST ,0
IMATINIB,1
IMMUNOGLOBULIN,1
INFLIXIMAB,1
INSULIN HUMAN,0
IRINOTECAN,0
ISATUXIMAB,0
IXEKIZUMAB,1
JYSELECA,0
KAFTRIO,0
LANREOTIDE,0
LENALIDOMIDE,0
LENOGRASTIM,0
LETERMOVIR,0
LIOTHYRONINE,0
LUCENTIS,1
MEPOLIZUMAB,0
METHOTREXATE,0
METRELEPTIN,0
MITOMYCIN,0
MOFETIL,0
MOLNUPIRAVIR,0
MYCOPHENOLATE,0
NERATINIB,0
NIRAPARIB,0
NORDITROPIN,0
NUSINERSEN,0
OBINUTUZUMAB,1
OCRIPLASMIN,0
OCTREOTIDE,1
OLAPARIB,0
OMALIZUMAB,1
OMNITROPE,0
OXALIPLATIN,0
PACLITAXEL,0
PALBOCICLIB,0
PALIVIZUMAB,0
PARENTERAL,0
PEGINTERFERON,0
RANIBIZUMAB,1
RASBURICASE,0
REMDESIVIR,0
REMICADE,1
RILUZOLE,1
RISANKIZUMAB,1
RITUXIMAB,1
RIXATHON,1
ROMIPLOSTIM,0
SACITUZUMAB,1
SARILUMAB,1
SECUKINUMAB,1
SERVICE,0
SEVELAMER,0
SILDENAFIL,0
SIMPONI,0
SIROLIMUS,0
SOLRIAMFETOL,0
SOMATROPIN,0
SOMATROPINŸ,0
SOTROVIMAB,0
STIRIPENTOL,0
TACROLIMUS,0
TERIPARATIDE,1
TILDRAKIZUMAB,1
TOCILIZUMAB,1
TOFACITINIB,1
TOLVAPTAN,0
TRASTUZUMAB,0
TRUXIMA,0
UNIT,0
UPADACITINIB,1
USTEKINUMAB,1
VABYSMO,0
VANCOMYCIN,0
VEDOLIZUMAB,1
VERTEPORFIN,0
VINBLASTINE,0
VISMODEGIB,0
VORICONAZOLE,0
ZANUBRUTINIB,0
1 Drug Include
2 ABATACEPT 1
3 ACALABRUTINIB 1
4 ADALIMUMAB 1
5 AFLIBERCEPT 1
6 ALIROCUMAB 1
7 ALITRETINOIN 0
8 AMIKACIN 0
9 AMPHOTERICIN B LIPOSOMAL 0
10 ANAKINRA 0
11 ANDEXANET ALFA 0
12 ANIDULAFUNGIN 0
13 APALUTAMIDE 0
14 APOMORPHINE 0
15 APREMILAST 1
16 ATEZOLIZUMAB 1
17 AVELUMAB 0
18 AXITINIB 0
19 AZACITIDINE 0
20 AZATHIOPRINE 0
21 BACILLUS CALMETTE-GUERIN 0
22 BARICITINIB 1
23 BASILIXIMAB 0
24 BENDAMUSTINE 0
25 BENRALIZUMAB 0
26 BEROTRALSTAT 0
27 BEVACIZUMAB 1
28 BEXAROTENE 0
29 BIMEKIZUMAB 1
30 BLEOMYCIN 0
31 BORTEZOMIB 0
32 BOTULINUM A TOXIN 1
33 BRENTUXIMAB VEDOTIN 0
34 BRODALUMAB 1
35 BUSULFAN 0
36 C1-ESTERASE INHIBITOR 0
37 CABOZANTINIB 0
38 CAPECITABINE 0
39 CARBOPLATIN 0
40 CARFILZOMIB 0
41 CERTOLIZUMAB PEGOL 1
42 CETUXIMAB 0
43 CHLORAMBUCIL 0
44 CICLOSPORIN 0
45 CINACALCET 0
46 CISPLATIN 0
47 COLISTIMETHATE 0
48 COLISTIN 0
49 COPAXONE 0
50 CYCLOPHOSPHAMIDE 0
51 DARBEPOETIN ALFA 0
52 DAROLUTAMIDE 0
53 DASATINIB 0
54 DEFERASIROX 0
55 DENOSUMAB 1
56 DESFERRIOXAMINE 0
57 DEXAMETHASONE 1
58 DIGOXIN IMMUNE FAB 0
59 DIMETHYL FUMARATE 1
60 DOCETAXEL 1
61 DOXORUBICIN 0
62 DUPILUMAB 1
63 ECULIZUMAB 0
64 EFUDIX 0
65 ELTROMBOPAG 0
66 EMTRICITABINE 0
67 ENZALUTAMIDE 0
68 EPIRUBICIN 0
69 EPOETIN 0
70 EPOETIN ALFA 0
71 EPOETIN BETA 0
72 ERENUMAB 1
73 ETANERCEPT 1
74 EVOLOCUMAB 1
75 FARICIMAB 1
76 FIDAXOMICIN 0
77 FILGOTINIB 1
78 FILGRASTIM 0
79 FLUOCINOLONE 1
80 FLUOCINOLONE ACETONIDE 1
81 FLUOROURACIL 0
82 FOMEPIZOLE 0
83 FREMANEZUMAB 1
84 FULVESTRANT 0
85 GALCANEZUMAB 1
86 GEMCITABINE 0
87 GENERIC 0
88 GOLIMUMAB 1
89 GUSELKUMAB 1
90 HYDROXYCARBAMIDE 0
91 IBRUTINIB 0
92 IDARUBICIN 0
93 IDARUCIZUMAB 1
94 IFOSFAMIDE 0
95 ILOPROST 0
96 ILOPROST 0
97 IMATINIB 1
98 IMMUNOGLOBULIN 1
99 INFLIXIMAB 1
100 INSULIN HUMAN 0
101 IRINOTECAN 0
102 ISATUXIMAB 0
103 IXEKIZUMAB 1
104 JYSELECA 0
105 KAFTRIO 0
106 LANREOTIDE 0
107 LENALIDOMIDE 0
108 LENOGRASTIM 0
109 LETERMOVIR 0
110 LIOTHYRONINE 0
111 LUCENTIS 1
112 MEPOLIZUMAB 0
113 METHOTREXATE 0
114 METRELEPTIN 0
115 MITOMYCIN 0
116 MOFETIL 0
117 MOLNUPIRAVIR 0
118 MYCOPHENOLATE 0
119 NERATINIB 0
120 NIRAPARIB 0
121 NORDITROPIN 0
122 NUSINERSEN 0
123 OBINUTUZUMAB 1
124 OCRIPLASMIN 0
125 OCTREOTIDE 1
126 OLAPARIB 0
127 OMALIZUMAB 1
128 OMNITROPE 0
129 OXALIPLATIN 0
130 PACLITAXEL 0
131 PALBOCICLIB 0
132 PALIVIZUMAB 0
133 PARENTERAL 0
134 PEGINTERFERON 0
135 RANIBIZUMAB 1
136 RASBURICASE 0
137 REMDESIVIR 0
138 REMICADE 1
139 RILUZOLE 1
140 RISANKIZUMAB 1
141 RITUXIMAB 1
142 RIXATHON 1
143 ROMIPLOSTIM 0
144 SACITUZUMAB 1
145 SARILUMAB 1
146 SECUKINUMAB 1
147 SERVICE 0
148 SEVELAMER 0
149 SILDENAFIL 0
150 SIMPONI 0
151 SIROLIMUS 0
152 SOLRIAMFETOL 0
153 SOMATROPIN 0
154 SOMATROPINŸ 0
155 SOTROVIMAB 0
156 STIRIPENTOL 0
157 TACROLIMUS 0
158 TERIPARATIDE 1
159 TILDRAKIZUMAB 1
160 TOCILIZUMAB 1
161 TOFACITINIB 1
162 TOLVAPTAN 0
163 TRASTUZUMAB 0
164 TRUXIMA 0
165 UNIT 0
166 UPADACITINIB 1
167 USTEKINUMAB 1
168 VABYSMO 0
169 VANCOMYCIN 0
170 VEDOLIZUMAB 1
171 VERTEPORFIN 0
172 VINBLASTINE 0
173 VISMODEGIB 0
174 VORICONAZOLE 0
175 ZANUBRUTINIB 0