Using ordered multiple-choice items to assess students' understanding of the structure and composition of matter

Helping students develop a sound understanding of scientific concepts can be a major challenge. Lately, learning progressions have received increasing attention as a means to support students in developing understanding of core scientific concepts. At the center of a learning progression is a sequence of developmental levels reflecting an idealized progression toward understanding a particular core concept. This sequence is supposed to serve as a basis for designing instruction that can foster learning as well as assessments that can monitor students' progression. So-called ordered multiple-choice (OMC) items have recently been suggested as a simple and effective way of assessing students' level of understanding of a core concept. This article details our efforts in developing an instrument for assessing students' understanding of the structure and composition of matter based on OMC items. Ten OMC items were developed and administered to a sample of N = 294 students in grades 6-12. Rasch analysis was used to investigate instrument functioning and to determine linear measures of person abilities and item difficulties. In addition to the OMC items, students were administered corresponding open-ended items in order to investigate the validity of the results obtained through the OMC items. Our findings suggest assessing students' understanding of scientific concepts through OMC items is indeed quite worthwhile and should be subject to further research.