Digital Module 12: Think-aloud Interviews and Cognitive Labs

5 (6 votes)

Recorded On: 11/15/2021

In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze distinct types of verbal reports in the collection of evidence of test-taker response processes. The module includes details on (a) the different types of cognition that are tapped by different interviewer probes, (b) traditional interviewing methods and new automated tools for collecting verbal reports, and (c) options for analyses of verbal reports. This includes a discussion of reliability and validity issues such as potential bias in the collection of verbal reports, ways to mitigate bias, and inter-rater agreement to enhance credibility of analysis. A novel digital tool for data-collection called the ABC tool is presented via illustrative videos. As always, the module contains audio-narrated slides, quiz questions with feedback, a glossary, and curated resources. 

Keywords: ABC tool, cognitive laboratory, cog lab, cognitive model, interrater agreement, kappa, probe, rubric, thematic analysis, think-aloud interview, verbal report

Jacqueline P. Leighton

Professor

Jackie is a Registered Psychologist and Professor of School and Clinical Child Psychology at the University of Alberta. She completed her graduate degrees in Psychology at the University of Alberta and postdoctoral fellowship studies in Psychology at Yale University. Her research and teaching is driven by the overarching goal to enhance fairness in testing. In pursuit of this goal, she has increasingly focused her research on investigating the interplay between cognitive and emotional processes underlying learning and academic achievement. For example, she investigates variables that can cognitively or emotionally bias participants’ response processes in testing situations, leading to misrepresentations in performance and weaknesses in validity arguments of test inferences. Overall, she is interested in methods designed to enhance diagnosis of achievement, validation of test inferences, and theoretical understanding of human learning. 

Contact Jackie via jacqueline.leighton@ualberta.ca

Blair Lehman

Research Scientist

Blair is a research scientist at Educational Testing Service. She completed her graduate degrees in Cognitive Psychology and certificate in Cognitive Science at the University of Memphis. Her research focuses on understanding students’ emotional and motivational processes during educational activities to design activities that maximize the experience for all students. Her research has focused on the task design of learning and assessment activities as well as the design of adaptive systems that consider both student cognition and motivation. For example, she has explored specific design features of game-based assessments in an effort to understand how to maintain measurement validity while also maximizing student motivation. 

Contact Blair via blehman@ets.org

Key:

Complete
Failed
Available
Locked
DM12 (Verbal Reports, Version 1.1)
Recorded 11/15/2021
Recorded 11/15/2021 The full interactive digital module with all resources, quiz questions, videos, and other materials.
DM12 VIDEO (Introduction, Version 1.0)
Open to view video.
Open to view video. Video version of the introduction of the module [5 Minutes]
DM12 VIDEO (Section 1, Version 1.0)
Open to view video.
Open to view video. Video version of the first content section of the module [29 Minutes]
DM12 VIDEO (Section 2, Version 1.0)
Open to view video.
Open to view video. Video version of the second content section of the module [26 Minutes]
DM12 VIDEO (Section 3, Version 1.0)
Open to view video.
Open to view video. Video version of the third content section of the module [25 Minutes]
DM12 VIDEO (Section 4, Version 1.0)
Open to view video.
Open to view video. Video version of the fourth content section of the module [15 Minutes]