feat: complete Task 2.2 - test refresh pipeline with Snowflake data
Tested full refresh pipeline end-to-end with real Snowflake data: - Fixed trust filter to read Name column from defaultTrusts.csv - Fixed Decimal type handling in calculate_cost_per_patient_per_annum - Fixed array handling in convert_to_records for average_administered - Added required reference CSV files to data/ directory - Configured Snowflake connection (account, warehouse, user) Results: - Snowflake fetch: 656,695 records in ~7s - Transformations: 519,848 records after UPID/drug/directory - Pathway nodes: 293 for all_6mo (8 trusts, 14 directories) - Total processing time: ~6.2 minutes
This commit is contained in:
@@ -0,0 +1,175 @@
|
||||
Drug,Include
|
||||
ABATACEPT,1
|
||||
ACALABRUTINIB,1
|
||||
ADALIMUMAB,1
|
||||
AFLIBERCEPT,1
|
||||
ALIROCUMAB,1
|
||||
ALITRETINOIN,0
|
||||
AMIKACIN,0
|
||||
AMPHOTERICIN B LIPOSOMAL,0
|
||||
ANAKINRA,0
|
||||
ANDEXANET ALFA,0
|
||||
ANIDULAFUNGIN,0
|
||||
APALUTAMIDE,0
|
||||
APOMORPHINE,0
|
||||
APREMILAST,1
|
||||
ATEZOLIZUMAB,1
|
||||
AVELUMAB,0
|
||||
AXITINIB,0
|
||||
AZACITIDINE,0
|
||||
AZATHIOPRINE,0
|
||||
BACILLUS CALMETTE-GUERIN,0
|
||||
BARICITINIB,1
|
||||
BASILIXIMAB,0
|
||||
BENDAMUSTINE ,0
|
||||
BENRALIZUMAB,0
|
||||
BEROTRALSTAT,0
|
||||
BEVACIZUMAB,1
|
||||
BEXAROTENE,0
|
||||
BIMEKIZUMAB,1
|
||||
BLEOMYCIN,0
|
||||
BORTEZOMIB,0
|
||||
BOTULINUM A TOXIN,1
|
||||
BRENTUXIMAB VEDOTIN ,0
|
||||
BRODALUMAB,1
|
||||
BUSULFAN ,0
|
||||
C1-ESTERASE INHIBITOR ,0
|
||||
CABOZANTINIB,0
|
||||
CAPECITABINE,0
|
||||
CARBOPLATIN,0
|
||||
CARFILZOMIB,0
|
||||
CERTOLIZUMAB PEGOL,1
|
||||
CETUXIMAB,0
|
||||
CHLORAMBUCIL,0
|
||||
CICLOSPORIN,0
|
||||
CINACALCET,0
|
||||
CISPLATIN,0
|
||||
COLISTIMETHATE,0
|
||||
COLISTIN,0
|
||||
COPAXONE,0
|
||||
CYCLOPHOSPHAMIDE,0
|
||||
DARBEPOETIN ALFA,0
|
||||
DAROLUTAMIDE,0
|
||||
DASATINIB,0
|
||||
DEFERASIROX,0
|
||||
DENOSUMAB,1
|
||||
DESFERRIOXAMINE,0
|
||||
DEXAMETHASONE,1
|
||||
DIGOXIN IMMUNE FAB,0
|
||||
DIMETHYL FUMARATE,1
|
||||
DOCETAXEL,1
|
||||
DOXORUBICIN,0
|
||||
DUPILUMAB,1
|
||||
ECULIZUMAB,0
|
||||
EFUDIX,0
|
||||
ELTROMBOPAG,0
|
||||
EMTRICITABINE,0
|
||||
ENZALUTAMIDE,0
|
||||
EPIRUBICIN,0
|
||||
EPOETIN,0
|
||||
EPOETIN ALFA,0
|
||||
EPOETIN BETA,0
|
||||
ERENUMAB,1
|
||||
ETANERCEPT,1
|
||||
EVOLOCUMAB,1
|
||||
FARICIMAB,1
|
||||
FIDAXOMICIN,0
|
||||
FILGOTINIB,1
|
||||
FILGRASTIM,0
|
||||
FLUOCINOLONE,1
|
||||
FLUOCINOLONE ACETONIDE,1
|
||||
FLUOROURACIL,0
|
||||
FOMEPIZOLE,0
|
||||
FREMANEZUMAB,1
|
||||
FULVESTRANT,0
|
||||
GALCANEZUMAB,1
|
||||
GEMCITABINE,0
|
||||
GENERIC,0
|
||||
GOLIMUMAB,1
|
||||
GUSELKUMAB,1
|
||||
HYDROXYCARBAMIDE,0
|
||||
IBRUTINIB,0
|
||||
IDARUBICIN,0
|
||||
IDARUCIZUMAB,1
|
||||
IFOSFAMIDE,0
|
||||
ILOPROST,0
|
||||
ILOPROST ,0
|
||||
IMATINIB,1
|
||||
IMMUNOGLOBULIN,1
|
||||
INFLIXIMAB,1
|
||||
INSULIN HUMAN,0
|
||||
IRINOTECAN,0
|
||||
ISATUXIMAB,0
|
||||
IXEKIZUMAB,1
|
||||
JYSELECA,0
|
||||
KAFTRIO,0
|
||||
LANREOTIDE,0
|
||||
LENALIDOMIDE,0
|
||||
LENOGRASTIM,0
|
||||
LETERMOVIR,0
|
||||
LIOTHYRONINE,0
|
||||
LUCENTIS,1
|
||||
MEPOLIZUMAB,0
|
||||
METHOTREXATE,0
|
||||
METRELEPTIN,0
|
||||
MITOMYCIN,0
|
||||
MOFETIL,0
|
||||
MOLNUPIRAVIR,0
|
||||
MYCOPHENOLATE,0
|
||||
NERATINIB,0
|
||||
NIRAPARIB,0
|
||||
NORDITROPIN,0
|
||||
NUSINERSEN,0
|
||||
OBINUTUZUMAB,1
|
||||
OCRIPLASMIN,0
|
||||
OCTREOTIDE,1
|
||||
OLAPARIB,0
|
||||
OMALIZUMAB,1
|
||||
OMNITROPE,0
|
||||
OXALIPLATIN,0
|
||||
PACLITAXEL,0
|
||||
PALBOCICLIB,0
|
||||
PALIVIZUMAB,0
|
||||
PARENTERAL,0
|
||||
PEGINTERFERON,0
|
||||
RANIBIZUMAB,1
|
||||
RASBURICASE,0
|
||||
REMDESIVIR,0
|
||||
REMICADE,1
|
||||
RILUZOLE,1
|
||||
RISANKIZUMAB,1
|
||||
RITUXIMAB,1
|
||||
RIXATHON,1
|
||||
ROMIPLOSTIM,0
|
||||
SACITUZUMAB,1
|
||||
SARILUMAB,1
|
||||
SECUKINUMAB,1
|
||||
SERVICE,0
|
||||
SEVELAMER,0
|
||||
SILDENAFIL,0
|
||||
SIMPONI,0
|
||||
SIROLIMUS,0
|
||||
SOLRIAMFETOL,0
|
||||
SOMATROPIN,0
|
||||
SOMATROPINŸ,0
|
||||
SOTROVIMAB,0
|
||||
STIRIPENTOL,0
|
||||
TACROLIMUS,0
|
||||
TERIPARATIDE,1
|
||||
TILDRAKIZUMAB,1
|
||||
TOCILIZUMAB,1
|
||||
TOFACITINIB,1
|
||||
TOLVAPTAN,0
|
||||
TRASTUZUMAB,0
|
||||
TRUXIMA,0
|
||||
UNIT,0
|
||||
UPADACITINIB,1
|
||||
USTEKINUMAB,1
|
||||
VABYSMO,0
|
||||
VANCOMYCIN,0
|
||||
VEDOLIZUMAB,1
|
||||
VERTEPORFIN,0
|
||||
VINBLASTINE,0
|
||||
VISMODEGIB,0
|
||||
VORICONAZOLE,0
|
||||
ZANUBRUTINIB,0
|
||||
|
Reference in New Issue
Block a user