Evaluation of Domain Generalization and Adaptation on Improving Model Robustness to Temporal Dataset Shift in Clinical Medicine

Lin Lawrence Guo, Stephen R Pfohl, Jason Fries, Alistair Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, Lillian Sung

Evaluation of Domain Generalization and Adaptation on Improving Model Robustness to Temporal Dataset Shift in Clinical Medicine

Lin Lawrence Guo, Stephen R Pfohl, Jason Fries, Alistair Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, Lillian Sung

Abstract: Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization and unsupervised domain adaptation might be suitable to proactively mitigate dataset shift. In this study, we characterized the impact of temporal dataset shift on clinical prediction models learned from electronic health records and evaluated domain generalization and unsupervised domain adaptation algorithms on improving model robustness compared with the standard empirical risk minimization. Intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008Ð2010,2011Ð2013, 2014Ð2016 and 2017Ð2019). Feed-forward neural networks were trained using empirical risk minimization as well as variants of domain generalization and adaptation to predict mortality, long length of stay, sepsis, and invasive ventilation. We observed heterogeneity in the impact of temporal dataset shift across clinical prediction tasks, with the worst impact observed in sepsis prediction. When compared with empirical risk minimization, domain generalization and adaptation algorithms failed to produce more robust models. These findings highlight the difficulty of improving robustness to dataset shift with purely data-driven techniques that do not leverage prior knowledge of the nature of the shift and a need for alternate approaches to preserve model performance overtime in clinical medicine.

Poster

Abstract: Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization and unsupervised domain adaptation might be suitable to proactively mitigate dataset shift. In this study, we characterized the impact of temporal dataset shift on clinical prediction models learned from electronic health records and evaluated domain generalization and unsupervised domain adaptation algorithms on improving model robustness compared with the standard empirical risk minimization. Intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008Ð2010,2011Ð2013, 2014Ð2016 and 2017Ð2019). Feed-forward neural networks were trained using empirical risk minimization as well as variants of domain generalization and adaptation to predict mortality, long length of stay, sepsis, and invasive ventilation. We observed heterogeneity in the impact of temporal dataset shift across clinical prediction tasks, with the worst impact observed in sepsis prediction. When compared with empirical risk minimization, domain generalization and adaptation algorithms failed to produce more robust models. These findings highlight the difficulty of improving robustness to dataset shift with purely data-driven techniques that do not leverage prior knowledge of the nature of the shift and a need for alternate approaches to preserve model performance overtime in clinical medicine.