An Ensemble Approach to Emergency Medicine Skill Assessment

Objective evaluation and feedback are crucial for effective simulation-based training in emergency medicine. However, the current evaluation paradigm is highly subjective, consequently suffers from poor inter-rater reliability. Feedback is entirely based on the discrepancy between a trainee’s performance and the instructor’s mental model of the procedure. This work directly addresses these limitations by developing an ensemble approach for the formative assessment of emergency medicine skills. The approach leverages a multimodal dataset comprising brain imaging, eye tracking, and head-mounted video recordings—each capturing distinct aspects of the brain-behavior relationship. While videos effectively capture behavioral elements, the brain activation signals and pupillometry provide physiological biomarkers that can be correlated with the stages of learning. The multimodal data are processed using a suite of explainable deep learning models to distill out biomarkers or behavioral measures that are shown to correlate with the endotracheal intubation (ETI) expertise level. Although all 3 modalities are shown to differentiate between successful and unsuccessful ETI tasks with greater than 85% accuracy, the brain activation and pupil dilation reveal a much-pronounced difference with accuracy >90%. The models are not only accurate, but they also identify the task segments where novice learners may benefit from targeted interventions. These findings demonstrate the unique potential of an ensemble approach for the objective and formative assessment of emergency medicine skill level.