Investigating the usage of Likert-style items within Computer Science Education Research Instruments

One of the most ubiquitous techniques for evaluating research, particularly in educational settings, has been Likert-style questionnaires. Having a participant rank their level of engagement, agreement, or interest on a scale can provide powerful insight into an individual's perspective and attitude. However, as computing education matures as a discipline it becomes important for us to examine our practices and ensure that we are employing the techniques of evaluation properly. Likert-style questionnaires can be prone to unintentional biases and noise which, from the perspective of the researcher, may affect the study in unknown, unexpected, and potentially undesirable ways. In this research we seek to aid the computing education researcher not only avoid biases and noise but also improve the reproducibility of their work. First, we establish best practices for these instruments by synthesizing recommendations from the original creator, Rensis Likert, the Center for Disease Control (CDC), the Association of American Medical Colleges (AAMC), and the American Association for Public Opinion Research (AAPOR). We then considered additional sources of unintentional biasing and noise resulting from the measurement scales/response options, specifically possible biasing resulting from how response options are presented to the study participant. With these recommendations, we then examined 121 evaluation instruments used in computer science education research and curated in the csedresearch.org online database to see how often researchers unintentionally fall victim to the pitfalls of ambiguity, awkward phrasing, use of conjunctions, leading or biased statements, and double negatives. We found that the occurrence of at least one of these problematic statements to be in 82.6% of all instruments. We also examine demographic information for the intended study participants, number of response options, and how those options are presented. Overall, while we see many instrument authors falling victim to a few of these common pitfalls, we believe that increased awareness of potentially problematic statements/measurement scales and their impact on research bias and reproducibility will help insulate computing education researchers from avoidable complications and strengthen the discipline throughout.