Investigating the Challenges of Adapting Quantitative Scales into an Engineering Education Context

Quantitative measures are important tools in engineering education research that often lack an ongoing and robust process of validation. Researchers may take a scale from one study and use it in another because it has been published as ‘‘validated’’ without gathering evidence for its use in the new context. Validation is particularly important when attempting to develop measures of abstract constructs. The purpose of this study is to illustrate the process and challenges of adapting quantitative scales into an engineering education context by presenting a case exploring the process of adapting a scale to measure engineering epistemic constructs. We explicitly map our work to The Standards for Educational and Psychological Testing with the goal of supporting other engineering education researchers as they use these standards. A sample of 3,711 responses from first-year engineering students from institutions across the United States was used to demonstrate a multistep, validation process, which included exploratory and confirmatory factor analysis. Our process provides an example of how to use the Standards while also demonstrating that a robust validation process does not automatically result in a trusted scale. The outcomes of this study provide a basis for better understanding the challenges associated with scale adaptation and validation. This work also adds to knowledge around quantitatively measuring epistemic constructs in engineering and encourages our field to critically consider its value.