feat: complete Task 2.2 - test refresh pipeline with Snowflake data

Tested full refresh pipeline end-to-end with real Snowflake data:
- Fixed trust filter to read Name column from defaultTrusts.csv
- Fixed Decimal type handling in calculate_cost_per_patient_per_annum
- Fixed array handling in convert_to_records for average_administered
- Added required reference CSV files to data/ directory
- Configured Snowflake connection (account, warehouse, user)

Results:
- Snowflake fetch: 656,695 records in ~7s
- Transformations: 519,848 records after UPID/drug/directory
- Pathway nodes: 293 for all_6mo (8 trusts, 14 directories)
- Total processing time: ~6.2 minutes

data/treatment_function_codes.csv

@@ -0,0 +1,185 @@
+Code,Service
+100,GENERAL SURGERY SERVICE
+101,UROLOGY SERVICE
+102,TRANSPLANT SURGERY SERVICE
+103,BREAST SURGERY SERVICE
+104,COLORECTAL SURGERY SERVICE
+105,HEPATOBILIARY AND PANCREATIC SURGERY SERVICE
+106,UPPER GASTROINTESTINAL SURGERY SERVICE
+107,VASCULAR SURGERY SERVICE
+108,SPINAL SURGERY SERVICE
+109,BARIATRIC SURGERY SERVICE
+110,TRAUMA AND ORTHOPAEDIC SERVICE
+111,ORTHOPAEDIC SERVICE
+113,ENDOCRINE SURGERY SERVICE
+115,TRAUMA SURGERY SERVICE
+120,EAR NOSE AND THROAT SERVICE
+130,OPHTHALMOLOGY SERVICE
+140,ORAL SURGERY SERVICE
+141,RESTORATIVE DENTISTRY SERVICE
+143,ORTHODONTIC SERVICE
+144,MAXILLOFACIAL SURGERY SERVICE
+145,ORAL AND MAXILLOFACIAL SURGERY SERVICE
+150,NEUROSURGICAL SERVICE
+160,PLASTIC SURGERY SERVICE
+161,BURNS CARE SERVICE
+170,CARDIOTHORACIC SURGERY SERVICE
+172,CARDIAC SURGERY SERVICE
+173,THORACIC SURGERY SERVICE
+174,CARDIOTHORACIC TRANSPLANTATION SERVICE
+191,PAIN MANAGEMENT SERVICE
+142,PAEDIATRIC DENTISTRY SERVICE
+171,PAEDIATRIC SURGERY SERVICE
+211,PAEDIATRIC UROLOGY SERVICE
+212,PAEDIATRIC TRANSPLANTATION SURGERY SERVICE
+213,PAEDIATRIC GASTROINTESTINAL SURGERY SERVICE
+214,PAEDIATRIC TRAUMA AND ORTHOPAEDIC SERVICE
+215,PAEDIATRIC EAR NOSE AND THROAT SERVICE
+216,PAEDIATRIC OPHTHALMOLOGY SERVICE
+217,PAEDIATRIC ORAL AND MAXILLOFACIAL SURGERY SERVICE
+218,PAEDIATRIC NEUROSURGERY SERVICE
+219,PAEDIATRIC PLASTIC SURGERY SERVICE
+220,PAEDIATRIC BURNS CARE SERVICE
+221,PAEDIATRIC CARDIAC SURGERY SERVICE
+222,PAEDIATRIC THORACIC SURGERY SERVICE
+223,PAEDIATRIC EPILEPSY SERVICE
+230,PAEDIATRIC CLINICAL PHARMACOLOGY SERVICE
+240,PAEDIATRIC PALLIATIVE MEDICINE SERVICE
+241,PAEDIATRIC PAIN MANAGEMENT SERVICE
+242,PAEDIATRIC INTENSIVE CARE SERVICE
+250,PAEDIATRIC HEPATOLOGY SERVICE
+251,PAEDIATRIC GASTROENTEROLOGY SERVICE
+252,PAEDIATRIC ENDOCRINOLOGY SERVICE
+253,PAEDIATRIC CLINICAL HAEMATOLOGY SERVICE
+254,PAEDIATRIC AUDIO VESTIBULAR MEDICINE SERVICE
+255,PAEDIATRIC CLINICAL IMMUNOLOGY AND ALLERGY SERVICE
+256,PAEDIATRIC INFECTIOUS DISEASES SERVICE
+257,PAEDIATRIC DERMATOLOGY SERVICE
+258,PAEDIATRIC RESPIRATORY MEDICINE SERVICE
+259,PAEDIATRIC NEPHROLOGY SERVICE
+260,PAEDIATRIC MEDICAL ONCOLOGY SERVICE
+261,PAEDIATRIC INHERITED METABOLIC MEDICINE SERVICE
+262,PAEDIATRIC RHEUMATOLOGY SERVICE
+263,PAEDIATRIC DIABETES SERVICE
+264,PAEDIATRIC CYSTIC FIBROSIS SERVICE
+270,PAEDIATRIC EMERGENCY MEDICINE SERVICE
+280,PAEDIATRIC INTERVENTIONAL RADIOLOGY SERVICE
+290,COMMUNITY PAEDIATRIC SERVICE
+291,PAEDIATRIC NEURODISABILITY SERVICE
+321,PAEDIATRIC CARDIOLOGY SERVICE
+421,PAEDIATRIC NEUROLOGY SERVICE
+180,EMERGENCY MEDICINE SERVICE
+190,ANAESTHETIC SERVICE
+192,INTENSIVE CARE MEDICINE SERVICE
+200,AVIATION AND SPACE MEDICINE SERVICE
+300,GENERAL INTERNAL MEDICINE SERVICE
+301,GASTROENTEROLOGY SERVICE
+302,ENDOCRINOLOGY SERVICE
+303,CLINICAL HAEMATOLOGY SERVICE
+304,CLINICAL PHYSIOLOGY SERVICE
+305,CLINICAL PHARMACOLOGY SERVICE
+306,HEPATOLOGY SERVICE
+307,DIABETES SERVICE
+308,BLOOD AND MARROW TRANSPLANTATION SERVICE
+309,HAEMOPHILIA SERVICE
+310,AUDIO VESTIBULAR MEDICINE SERVICE
+311,CLINICAL GENETICS SERVICE
+313,CLINICAL IMMUNOLOGY AND ALLERGY SERVICE
+314,REHABILITATION MEDICINE SERVICE
+315,PALLIATIVE MEDICINE SERVICE
+316,CLINICAL IMMUNOLOGY SERVICE
+317,ALLERGY SERVICE
+318,INTERMEDIATE CARE SERVICE
+319,RESPITE CARE SERVICE
+320,CARDIOLOGY SERVICE
+322,CLINICAL MICROBIOLOGY SERVICE
+323,SPINAL INJURIES SERVICE
+324,ANTICOAGULANT SERVICE
+325,SPORT AND EXERCISE MEDICINE SERVICE
+326,ACUTE INTERNAL MEDICINE SERVICE
+327,CARDIAC REHABILITATION SERVICE
+328,STROKE MEDICINE SERVICE
+329,TRANSIENT ISCHAEMIC ATTACK SERVICE
+330,DERMATOLOGY SERVICE
+331,CONGENITAL HEART DISEASE SERVICE
+333,RARE DISEASE SERVICE
+335,INHERITED METABOLIC MEDICINE SERVICE
+340,RESPIRATORY MEDICINE SERVICE
+341,RESPIRATORY PHYSIOLOGY SERVICE
+342,PULMONARY REHABILITATION SERVICE
+343,ADULT CYSTIC FIBROSIS SERVICE
+344,COMPLEX SPECIALISED REHABILITATION SERVICE
+345,SPECIALIST REHABILITATION SERVICE
+346,LOCAL SPECIALIST REHABILITATION SERVICE
+347,SLEEP MEDICINE SERVICE
+348,POST-COVID-19 SYNDROME SERVICE
+350,INFECTIOUS DISEASES SERVICE
+352,TROPICAL MEDICINE SERVICE
+360,GENITOURINARY MEDICINE SERVICE
+361,RENAL MEDICINE SERVICE
+370,MEDICAL ONCOLOGY SERVICE
+371,NUCLEAR MEDICINE SERVICE
+400,NEUROLOGY SERVICE
+401,CLINICAL NEUROPHYSIOLOGY SERVICE
+410,RHEUMATOLOGY SERVICE
+420,PAEDIATRIC SERVICE
+422,NEONATAL CRITICAL CARE SERVICE
+424,WELL BABY SERVICE
+430,ELDERLY MEDICINE SERVICE
+431,ORTHOGERIATRIC MEDICINE SERVICE
+450,DENTAL MEDICINE SERVICE
+451,SPECIAL CARE DENTISTRY SERVICE
+460,MEDICAL OPHTHALMOLOGY SERVICE
+461,OPHTHALMIC AND VISION SCIENCE SERVICE
+501,OBSTETRICS SERVICE
+502,GYNAECOLOGY SERVICE
+503,GYNAECOLOGICAL ONCOLOGY SERVICE
+504,COMMUNITY SEXUAL AND REPRODUCTIVE HEALTH SERVICE
+505,FETAL MEDICINE SERVICE
+510,RETIRED BUT RETAINED FOR HISTORICAL PURPOSES
+520,RETIRED BUT RETAINED FOR HISTORICAL PURPOSES
+560,MIDWIFERY SERVICE
+610,RETIRED BUT RETAINED FOR HISTORICAL PURPOSES
+620,RETIRED BUT RETAINED FOR HISTORICAL PURPOSES
+656,CLINICAL PSYCHOLOGY SERVICE
+700,LEARNING DISABILITY SERVICE
+710,ADULT MENTAL HEALTH SERVICE
+711,CHILD AND ADOLESCENT PSYCHIATRY SERVICE
+712,FORENSIC PSYCHIATRY SERVICE
+713,MEDICAL PSYCHOTHERAPY SERVICE
+715,OLD AGE PSYCHIATRY SERVICE
+720,EATING DISORDERS SERVICE
+721,ADDICTION SERVICE
+722,LIAISON PSYCHIATRY SERVICE
+723,PSYCHIATRIC INTENSIVE CARE SERVICE
+724,PERINATAL MENTAL HEALTH SERVICE
+725,MENTAL HEALTH RECOVERY AND REHABILITATION SERVICE
+726,MENTAL HEALTH DUAL DIAGNOSIS SERVICE
+727,DEMENTIA ASSESSMENT SERVICE
+730,NEUROPSYCHIATRY SERVICE
+800,CLINICAL ONCOLOGY SERVICE
+811,INTERVENTIONAL RADIOLOGY SERVICE
+812,DIAGNOSTIC IMAGING SERVICE
+822,CHEMICAL PATHOLOGY SERVICE
+832,RETIRED BUT RETAINED FOR HISTORICAL PURPOSES
+834,MEDICAL VIROLOGY SERVICE
+650,PHYSIOTHERAPY SERVICE
+651,OCCUPATIONAL THERAPY SERVICE
+652,SPEECH AND LANGUAGE THERAPY SERVICE
+653,PODIATRY SERVICE
+654,DIETETICS SERVICE
+655,ORTHOPTICS SERVICE
+657,PROSTHETICS SERVICE
+658,ORTHOTICS SERVICE
+659,DRAMATHERAPY SERVICE
+660,ART THERAPY SERVICE
+661,MUSIC THERAPY SERVICE
+662,OPTOMETRY SERVICE
+663,PODIATRIC SURGERY SERVICE
+670,UROLOGICAL PHYSIOLOGY SERVICE
+673,VASCULAR PHYSIOLOGY SERVICE
+675,CARDIAC PHYSIOLOGY SERVICE
+677,GASTROINTESTINAL PHYSIOLOGY SERVICE
+840,AUDIOLOGY SERVICE
+920,DIABETIC EDUCATION SERVICE
+990,RETIRED BUT RETAINED FOR HISTORICAL PURPOSES