Digital Module 27: Hierarchical Rater Models

5 (2 votes)

Recorded On: 11/30/2021

In this digital ITEMS module, Dr. Jodi M. Casabianca provides a primer on the hierarchical rater model (HRM) and the recent expansions to the model for analyzing raters and ratings of constructed responses. In the first part of the module, she establishes an understanding of the nature of constructed responses, the rating process and the common rater errors or rater effects originating from the rating process. In the second part of the module, she compares traditional measures for analyzing raters to item response theory (IRT) measures and discusses various IRT rater models. In the third section, she discusses longitudinal and multidimensional extensions and their foundations in different combinations of IRT and signal detection models. The module contains audio-narrated slides, quizzes with feedback, and additional resources such as a glossary and reference documents.

Keywords: hierarchical rater model, item response theory, longitudinal models, rater effects, multidimensional models, signal detection models

Jodi M. Casabianca

Educational Testing Service

Jodi M. Casabianca is a measurement scientist at ETS. Jodi received a B.A. in both Statistics and Psychology (2001) and an M.S. in Applied and Mathematical Statistics (2004) from Rutgers University as well as an M.A. (2008) and a Ph.D. (2011) in Psychometrics and Quantitative Psychology from Fordham University. She was the 2009 recipient of the Harold Gulliksen Dissertation Research Fellow awarded by ETS. Before joining ETS in 2016, she was an assistant professor at the University of Texas at Austin’s Department of Educational Psychology where she taught graduate courses and led a research lab in the Quantitative Methods program. Dr. Casabianca was awarded a National Science Foundation grant in 2013 (as co-PI, with Professor Junker) to expand the HRM into a framework that is more flexible for different assessment scenarios and has provided two conference workshops on the HRM in 2017 and 2018. Her current research focuses on statistical and psychometric modeling for constructed responses, with a focus on the evaluation of automated scoring models. Her research has been published in several notable peer reviewed journals, including the Journal of Educational and Behavioral Statistics and the Journal of Educational Measurement.

Module Overview

Core Slides