About HEOR

1. Machine Learning in Healthcare: Beyond PerformanceMichael Hurst (@mikehurst) – Principal Data Scientist HEOR Ltd

2. About HEOR Birmingham, UK • HEOR – Health Economics and Outcomes Research Ltd. • Health economics consultancy formed in Cardiff in 2012 • A team of 60 medical writers, analysts, modellers, data scientists and statisticians across three locations • www.heor.co.uk • bit.ly/HEOR_Ltd • @heor_ltd Cardiff, UK Cologne, Germany

3. Health Economics • Undertaken primarily once a technology has been approved for use (European medical agency (EMA), food and drug administration (FDA) etc.) • Demonstrating the value of an intervention (versus current practice) • Value ≠ just cost • Considers differences in life expectancy (LE) and quality of life (QoL) • Value derived from mathematical models, simulating patient cohorts and expected outcomes • Uptake in activities due to drive to strategy to improve diagnosis rates within the National Health Service (NHS)

4. Detect, Protect and Perfect [1] • One such scheme associated with atrial fibrillation (AF) • Irregular heart rate, if undiagnosed can lead to severe complications including stroke and heart failure • Significant value • Treatment can prevent down stream complications (e.g. myocardial infarctions (MI), strokes) • Find more (detect), treat more (protect) and treat better (perfect) [1] National Health Service. (2017) Atrial fibrillation toolkit, Detect, Protect and Perfect. Retrieved from: http://www.londonscn.nhs.uk/wp-content/uploads/2017/06/detect-protect-perfect-london-af-toolkit-062017.pdf

5. Detect, Protect and Perfect • Approximately 30% of patients who are living with AF are undiagnosed [2] • How do we detect these patients? • Combination of 12-lead electrocardiogram (ECG) and electroencephalogram (EEG) heart rate (pulse) monitor • Resource intensive • Three standard methods [3] • Systematic screening – Screen all patients • Targeted screening – Screen patients that are of a higher risk (i.e. aged >65 years) • Opportunistic screening – Screen patients who exhibited symptoms (i.e. irregular pulse) [2] Adderley N, Ryan R, Nirantharakumar K, et al. Prevalence and treatment of atrial fibrillation in the UK general practice from 2000 to 2016. Heart 2019;105:27-33 [3] Hobbs R, Fitzmaurice D, Jowett S, Mant J, Murray E. A randomised controlled trial and cost-effectiveness study of systematic screening (targeted and total population screening) versus routine practice for the detection of atrial fibrillation in people aged 65 and over: The SAFE study. Health Technol Assess 2005;9(40)

6. Detect, Protect and Perfect • Approximately 30% of patients who are living with AF are undiagnosed [2] • How do we detect these patients? • Combination of 12-lead electrocardiogram (ECG) and electroencephalogram (EEG) heart rate (pulse) monitor • Resource intensive • Three standard methods [3] • Systematic screening – No markable increase • Targeted screening – No markable increase • Opportunistic screening – Increase but not cost-effective [2] Adderley N, Ryan R, Nirantharakumar K, et al. Prevalence and treatment of atrial fibrillation in the UK general practice from 2000 to 2016. Heart 2019;105:27-33 [3] Hobbs R, Fitzmaurice D, Jowett S, Mant J, Murray E. A randomised controlled trial and cost-effectiveness study of systematic screening (targeted and total population screening) versus routine practice for the detection of atrial fibrillation in people aged 65 and over: The SAFE study. Health Technol Assess 2005;9(40)

7. Detecting AF – How? • How can we detect conditions better? • Traditionally risk prediction models were developed: • Regression models - aiming to find relationships between patient characteristics and outcomes • Formulaic or points-based measures based on clearly defined known risk factors • Well used in clinical practice due to: • Accurately determine a patient's risk whilst; • Understandable to clinicians • Mimicking well known clinical risk factors, allowing clinicians to validate their own insights and target their care

8. The role of machine learning • ML is becoming more prominent in healthcare • Real-world application of improving patient outcomes • Mathematical models are not new • Expand on traditional methods in the aim to benefit from complex non-linear associations within highly complex data • The ability to draw new inferences from a dataset • Could have profound implications on person-centred care • Large benefits and are truly life changing but there are unique challenges that need to be overcome

9. Challenges • Privacy concerns • Allowing the analysis of personal and sensitive data by third party organisations • Ethical concerns • Addressing the ‘ethical AI’ argument • Lack of ethics can have a significant impact on patients physical and mental well-being • Validity concerns • It needs to stack up • Clinicians need to be able to a) interpret the findings; b) confident in what it is predicting and c) understand what is driving the predictions

10. Challenges • Privacy concerns • Allowing the analysis of personal and sensitive data by third party organisations • Ethical concerns • Addressing the ‘ethical AI’ argument • Lack of ethics can have a significant impact on patients physical and mental well-being • Validity concerns • It needs to stack up • Clinicians need to be able to a) interpret the findings; b) confident in what it is predicting and c) understand what is driving the predictions

11. The pitfalls of performance Reducing interpretability Improving performance vs Each small increase leads to more people diagnosed Refinements may hamper inferences being made • Healthcare is highly governed and accountable • The need to avoid/address the ‘black-box’ problem is paramount • Question: What relationships were found? • Inability to answer this question will limit the application/rollout of the models

12. Beyond performance • Performance should not be ignored however; • The model development process should be considered carefully • All stages should be interpretable and allows for inferences to be made • Dimensionality reduction is typically problematic construction of a model • e.g. principal component analysis (PCA) • ‘Squashing’ down features are hard to unsquash • A feature set where the dimensions have been reduced will inevitably lead to difficulties in understanding what is driving model development

13. The ML pipeline Model testing Prediction Tuning hyperparameters Data preparation Feature development

14. The ML pipeline Model testing Prediction Tuning hyperparameters Data preparation Feature development Clinician insight • A recursive process • Taking inferences developed from prior models and expert clinical insight to further refine the model development process (e.g. depression) • Ability to use inferences across different types of model to drive feature set development

15. Methods • The simplest methods are often the best • Example: 2d heatmap indicating a diagnosis of AF • Heatmap identifies an inverse relationship between systolic (SBP) and diastolic (DBP) blood pressure at 3 months prior to a diagnosis • Patients with a high SBP and a low DBP indicative of AF • Further exploration into blood pressure (BP) found relationships between high and low SBP and DBP between 3 months prior and at time of AF.

16. Methods • The simplest methods are often the best • Example: Random forest; INR control whilst taking Warfarin • Mean decrease in GINI coefficient • Flag for depression driver of control • Further examination – Poor control • Further development based on clinical insight • Inclusion of anti-psychotics and substance abuse • Improved model performance

17. Conclusion • Within healthcare it’s a matter of life and death • Understandably cautious • A highly tuned model is only as good as the ability to apply it • In some cases new inferences (knowledge) can have more utility than a new model • The need to revise working practices and the ‘pipeline’ to learn from what the models are telling us is key • Multi-disciplinary approach to improving patient centred care

18. Questions