A Work-Centered Approach to System User-Evaluation

New systems are often based on optimistic assumptions of how they will improve human performance. In the cognitive engineering tradition, these assumed benefits are regarded as hypotheses that need to be tested. An important element of a system user evaluation is to determine whether the hypothesized benefits are realized. Evaluation may also uncover unsupported aspects of performance or unanticipated side-effects of introducing the new technology that need to be addressed. We present a work-centered approach to user evaluation intended to meet these objectives, focusing specifically on design of tailored user-feedback questionnaires (work-centered questionnaires) that are intended to be diagnostic of how specific system elements do, or do not, support work. We summarize two recent evaluation studies we have conducted that illustrate our approach and the diagnostic power of work-centered questionnaires. We discuss how the goals and approach of a work-centered evaluation differ from more traditional approaches to usability evaluation that emphasize the use of standardized questionnaires and broad assessments of usability.