Digital Module 27: Hierarchical Rater Models

Digital Module 27: Hierarchical Rater Models

5 (2 votes)

Recorded On: 11/30/2021

In this digital ITEMS module, Dr. Jodi M. Casabianca provides a primer on the hierarchical rater model (HRM) and the recent expansions to the model for analyzing raters and ratings of constructed responses. In the first part of the module, she establishes an understanding of the nature of constructed responses, the rating process and the common rater errors or rater effects originating from the rating process. In the second part of the module, she compares traditional measures for analyzing raters to item response theory (IRT) measures and discusses various IRT rater models. In the third section, she discusses longitudinal and multidimensional extensions and their foundations in different combinations of IRT and signal detection models. The module contains audio-narrated slides, quizzes with feedback, and additional resources such as a glossary and reference documents.

Keywords: hierarchical rater model, item response theory, longitudinal models, rater effects, multidimensional models, signal detection models 

Jodi M. Casabianca

Educational Testing Service

Jodi M. Casabianca is a measurement scientist at ETS. Jodi received a B.A. in both Statistics and Psychology (2001) and an M.S. in Applied and Mathematical Statistics (2004) from Rutgers University as well as an M.A. (2008) and a Ph.D. (2011) in Psychometrics and Quantitative Psychology from Fordham University. She was the 2009 recipient of the Harold Gulliksen Dissertation Research Fellow awarded by ETS. Before joining ETS in 2016, she was an assistant professor at the University of Texas at Austin’s Department of Educational Psychology where she taught graduate courses and led a research lab in the Quantitative Methods program. Dr. Casabianca was awarded a National Science Foundation grant in 2013 (as co-PI, with Professor  Junker) to expand the HRM into a framework that is more flexible for different assessment scenarios and has provided two conference workshops on the HRM in 2017 and 2018. Her current research focuses on statistical and psychometric modeling for constructed responses, with a focus on the evaluation of automated scoring models. Her research has been published in several notable peer reviewed journals, including the Journal of Educational and Behavioral Statistics and the Journal of Educational Measurement

Key:

Complete
Failed
Available
Locked
DM27 (Rater Models, Version 1.0)
Recorded 11/30/2021
Recorded 11/30/2021 Full interactive version of the digital module with all quiz questions and resource links.
DM27 VIDEO (Introduction, Version 1.0)
Open to view video.
Open to view video. Video version of the introduction section of the module [4 minutes]
DM27 VIDEO (Section 1, Version 1.0)
Open to view video.
Open to view video. Video version of the first content section of the module [18 minutes]
DM27 VIDEO (Section 2, Version 1.0)
Open to view video.
Open to view video. Video version of the second content section of the module [19 minutes]
DM27 VIDEO (Section 3, Version 1.0)
Open to view video.
Open to view video. Video version of the third content section of the module [30 minutes}
DM27 VIDEO (Section 4, Version 1.0)
Open to view video.
Open to view video. Video version of the fourth content section of the module. [35 Minutes]