Generalization of Machine-Learning in Clinical Randomized Controlled Trials: Evaluation and Development

Abstract

In healthcare, machine learning (ML) shows significant promise in improving patient diagnostics, prognostics, and personalized care. However, its real-world deployment is often constrained by models' inconsistent performance on diverse and unseen patient data, a critical challenge known as generalization. Despite ongoing advancements, existing methodologies have shown only limited success in assessing and improving ML generalization, raising uncertainty in clinical deployment. This dissertation tackles this gap by presenting a robust evaluation framework and a predictive tool to cultivate more reliable healthcare AI. Applying Logistic Regression and XGBoost models on a dataset from nine double-blind, randomized, placebo-controlled trials investigating beta-blockers in heart failure. This study employs leave-one-trial-out, reverse leave- one-trial-out, and systematic evaluation to comprehensively assess generalization. The findings indicate that while generalization is often suboptimal, strategic selection of training cohorts markedly improves performance. Furthermore, a developed meta-learning framework effectively predicts model degradation. This research provides crucial insights into model generalizability across varied clinical datasets and introduces a practical pre-screening tool, essential for facilitating a safer and more effective integration of ML into clinical practice and promoting fair patient outcomes.