feat: complete Task 2.2 - test refresh pipeline with Snowflake data
Tested full refresh pipeline end-to-end with real Snowflake data: - Fixed trust filter to read Name column from defaultTrusts.csv - Fixed Decimal type handling in calculate_cost_per_patient_per_annum - Fixed array handling in convert_to_records for average_administered - Added required reference CSV files to data/ directory - Configured Snowflake connection (account, warehouse, user) Results: - Snowflake fetch: 656,695 records in ~7s - Transformations: 519,848 records after UPID/drug/directory - Pathway nodes: 293 for all_6mo (8 trusts, 14 directories) - Total processing time: ~6.2 minutes
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
directory
|
||||
ACCIDENT & EMERGENCY
|
||||
ANAESTHETICS
|
||||
ANTICOAGULANT SERVICE
|
||||
AUDIOLOGY
|
||||
BLOOD AND MARROW TRANSPLANTATION
|
||||
BREAST SURGERY
|
||||
CARDIOLOGY
|
||||
CARDIOTHORACIC TRANSPLANTATION
|
||||
CHEMICAL PATHOLOGY
|
||||
CLINICAL HAEMATOLOGY
|
||||
CLINICAL IMMUNOLOGY
|
||||
CLINICAL NEUROPHYSIOLOGY
|
||||
CLINICAL ONCOLOGY (PREVIOUSLY RADIOTHERAPY)
|
||||
COLORECTAL SURGERY
|
||||
CRITICAL CARE MEDICINE
|
||||
DERMATOLOGY
|
||||
DIABETIC MEDICINE
|
||||
DIETETICS
|
||||
ENDOCRINOLOGY
|
||||
ENT
|
||||
GASTROENTEROLOGY
|
||||
GENERAL MEDICINE
|
||||
GENERAL SURGERY
|
||||
GERIATRIC MEDICINE
|
||||
GYNAECOLOGICAL ONCOLOGY
|
||||
GYNAECOLOGY
|
||||
INFECTIOUS DISEASES
|
||||
MEDICAL ONCOLOGY
|
||||
MEDICAL OPHTHALMOLOGY
|
||||
MIDWIFE EPISODE
|
||||
NEONATOLOGY
|
||||
NEPHROLOGY
|
||||
NEUROLOGY
|
||||
NEUROSURGERY
|
||||
OBSTETRICS
|
||||
OPHTHALMOLOGY
|
||||
ORAL SURGERY
|
||||
ORTHOPTICS
|
||||
ORTHOTICS
|
||||
PAEDIATRIC CARDIOLOGY
|
||||
PAEDIATRIC DIABETIC MEDICINE
|
||||
PAEDIATRIC GASTROENTEROLOGY
|
||||
PAEDIATRIC UROLOGY
|
||||
PAEDIATRICS
|
||||
PAIN MANAGEMENT
|
||||
PLASTIC SURGERY
|
||||
REHABILITATION
|
||||
RHEUMATOLOGY
|
||||
STROKE MEDICINE
|
||||
THORACIC MEDICINE
|
||||
THORACIC SURGERY
|
||||
TRANSIENT ISCHAEMIC ATTACK
|
||||
TRANSPLANTATION SURGERY
|
||||
TRAUMA & ORTHOPAEDICS
|
||||
UPPER GASTROINTESTINAL SURGERY
|
||||
UROLOGY
|
||||
VASCULAR SURGERY
|
||||
|
Reference in New Issue
Block a user