feat: complete Task 2.2 - test refresh pipeline with Snowflake data

Tested full refresh pipeline end-to-end with real Snowflake data:
- Fixed trust filter to read Name column from defaultTrusts.csv
- Fixed Decimal type handling in calculate_cost_per_patient_per_annum
- Fixed array handling in convert_to_records for average_administered
- Added required reference CSV files to data/ directory
- Configured Snowflake connection (account, warehouse, user)

Results:
- Snowflake fetch: 656,695 records in ~7s
- Transformations: 519,848 records after UPID/drug/directory
- Pathway nodes: 293 for all_6mo (8 trusts, 14 directories)
- Total processing time: ~6.2 minutes
This commit is contained in:
Andrew Charlwood
2026-02-05 00:20:12 +00:00
parent 8b65dfd9a8
commit adc1dbfc58
12 changed files with 1708 additions and 21 deletions
+58
View File
@@ -0,0 +1,58 @@
directory
ACCIDENT & EMERGENCY
ANAESTHETICS
ANTICOAGULANT SERVICE
AUDIOLOGY
BLOOD AND MARROW TRANSPLANTATION
BREAST SURGERY
CARDIOLOGY
CARDIOTHORACIC TRANSPLANTATION
CHEMICAL PATHOLOGY
CLINICAL HAEMATOLOGY
CLINICAL IMMUNOLOGY
CLINICAL NEUROPHYSIOLOGY
CLINICAL ONCOLOGY (PREVIOUSLY RADIOTHERAPY)
COLORECTAL SURGERY
CRITICAL CARE MEDICINE
DERMATOLOGY
DIABETIC MEDICINE
DIETETICS
ENDOCRINOLOGY
ENT
GASTROENTEROLOGY
GENERAL MEDICINE
GENERAL SURGERY
GERIATRIC MEDICINE
GYNAECOLOGICAL ONCOLOGY
GYNAECOLOGY
INFECTIOUS DISEASES
MEDICAL ONCOLOGY
MEDICAL OPHTHALMOLOGY
MIDWIFE EPISODE
NEONATOLOGY
NEPHROLOGY
NEUROLOGY
NEUROSURGERY
OBSTETRICS
OPHTHALMOLOGY
ORAL SURGERY
ORTHOPTICS
ORTHOTICS
PAEDIATRIC CARDIOLOGY
PAEDIATRIC DIABETIC MEDICINE
PAEDIATRIC GASTROENTEROLOGY
PAEDIATRIC UROLOGY
PAEDIATRICS
PAIN MANAGEMENT
PLASTIC SURGERY
REHABILITATION
RHEUMATOLOGY
STROKE MEDICINE
THORACIC MEDICINE
THORACIC SURGERY
TRANSIENT ISCHAEMIC ATTACK
TRANSPLANTATION SURGERY
TRAUMA & ORTHOPAEDICS
UPPER GASTROINTESTINAL SURGERY
UROLOGY
VASCULAR SURGERY
1 directory
2 ACCIDENT & EMERGENCY
3 ANAESTHETICS
4 ANTICOAGULANT SERVICE
5 AUDIOLOGY
6 BLOOD AND MARROW TRANSPLANTATION
7 BREAST SURGERY
8 CARDIOLOGY
9 CARDIOTHORACIC TRANSPLANTATION
10 CHEMICAL PATHOLOGY
11 CLINICAL HAEMATOLOGY
12 CLINICAL IMMUNOLOGY
13 CLINICAL NEUROPHYSIOLOGY
14 CLINICAL ONCOLOGY (PREVIOUSLY RADIOTHERAPY)
15 COLORECTAL SURGERY
16 CRITICAL CARE MEDICINE
17 DERMATOLOGY
18 DIABETIC MEDICINE
19 DIETETICS
20 ENDOCRINOLOGY
21 ENT
22 GASTROENTEROLOGY
23 GENERAL MEDICINE
24 GENERAL SURGERY
25 GERIATRIC MEDICINE
26 GYNAECOLOGICAL ONCOLOGY
27 GYNAECOLOGY
28 INFECTIOUS DISEASES
29 MEDICAL ONCOLOGY
30 MEDICAL OPHTHALMOLOGY
31 MIDWIFE EPISODE
32 NEONATOLOGY
33 NEPHROLOGY
34 NEUROLOGY
35 NEUROSURGERY
36 OBSTETRICS
37 OPHTHALMOLOGY
38 ORAL SURGERY
39 ORTHOPTICS
40 ORTHOTICS
41 PAEDIATRIC CARDIOLOGY
42 PAEDIATRIC DIABETIC MEDICINE
43 PAEDIATRIC GASTROENTEROLOGY
44 PAEDIATRIC UROLOGY
45 PAEDIATRICS
46 PAIN MANAGEMENT
47 PLASTIC SURGERY
48 REHABILITATION
49 RHEUMATOLOGY
50 STROKE MEDICINE
51 THORACIC MEDICINE
52 THORACIC SURGERY
53 TRANSIENT ISCHAEMIC ATTACK
54 TRANSPLANTATION SURGERY
55 TRAUMA & ORTHOPAEDICS
56 UPPER GASTROINTESTINAL SURGERY
57 UROLOGY
58 VASCULAR SURGERY