Teaching and evaluating first and second year medical students' practice of evidence-based medicine

PURPOSE: To implement an evidence-based medicine (EBM) curriculum for Year 1 and 2 medical students, and to develop a method to evaluate their practice of EBM in discrete and relevant worksteps. METHODS: For the 100 students entering Year 1 of their medical education in 2000, we implemented a curriculum with 25-30 student contact hours of EBM instruction which used a variety of teaching formats and spanned the first and second years of their training. We developed an evaluation module that assessed the following 5 steps in the practice of EBM: generating well built questions; searching for evidence; critical appraisal; applying the evidence, and self-evaluation. We tested 2 different versions of the test module 3-months apart with the same cohort of second year students, and correlated their scores on the second module with examination components of a comprehensive assessment. We obtained feedback from the students regarding the EBM curriculum and evaluation method. RESULTS: Each test module took 2-4 hours to complete and 5-8 minutes to grade. There was moderate test-retest reliability for the total test scores (r = 0.35, P < 0.001). Step 1 scores correlated with the mock board examination scores (r = 0.23, P = 0.05). Step 2 scores correlated with the peer assessment factor 'work habits' (r = 0.24, P = 0.02), and Step 3 scores correlated with clinical reasoning exercises (r = 0.31, P = 0.002). Step 4 scores lacked test-retest reliability and did not correlate with components of the comprehensive assessment. The majority of students felt there was too much focus on EBM during the first 2 years of the curriculum and they rated the EBM test module the lowest rated component of the comprehensive assessment. CONCLUSIONS: Although we have demonstrated preliminary reliability and validity of a new evaluation instrument that assess the domains of scientific knowledge, work habits and reasoning skills required in the practice of EBM, many of the correlations were weak, and we remain in the very early stages of determining if, when and how EBM instruction should occur in medical education.