Digital Module 08: Foundations of Operational Item Analysis

3.8 (5 votes)

Item analysis is an integral part of operational test development and is typically conducted within two popular statistical frameworks: classical test theory (CTT) and item response theory (IRT). In this digital ITEMS module, Dr. Hanwook Yoo and Dr. Ronald K. Hambleton provide an accessible overview of operational item analysis approaches for dichotomously scored items within these frameworks. They review the different stages of test development and associated item analyses to identify poorly performing items and effective item selection. Moreover, they walk through the computational and interpretational steps for CTT- and IRT-based evaluation statistics using simulated data examples and review various graphical displays such as distractor response curves, item characteristic curves, and item information curves. The digital module contains sample data, Excel sheets with various templates and examples, diagnostic quiz questions, data-based activities, curated resources, and a glossary.

Keywords: Classical test theory, CTT, corrections, difficulty, discrimination, distractors, item analysis, item response theory, R Shiny, TAP, test development

Hanwook (Henry) Yoo

Managing Senior Psychometrician

Henry is a managing senior psychometrician in the Psychometric Analysis and Research division at Educational Testing Service (ETS). At ETS, he manages operational psychometric work for graduate admissions programs. He received his Ed.D. in Research and Evaluation Methods Program from the University of Massachusetts, Amherst. His research interests include measurement invariance across subgroups, innovative score reporting, construct validity of English language proficiency assessment, and applications of IRT to computer-based testing. He is a co-author of a bibliography of research on test score reporting, which is available at the NCME website (https://ncme.connectedcommunity.org/ncmedev/viewdocument/score-reporting-bibliography). 


Contact Hanwook (Henry) via hyoo@ets.org 

Ronald K. Hambleton

Professor Emeritus

Ronald holds the titles of Distinguished University Professor and Executive Director of the Center for Educational Assessment at the University of Massachusetts, Amherst. He earned his Ph.D. in 1969 from the University of Toronto with specialties in psychometric methods and statistics. He is the co-author or co-editor of eight measurement books as well as author or co-author of many research papers, reports, and reviews spanning 50 years on topics such as standard-setting, score reporting, test adaptation, and applications of IRT. He is currently conducting research on a number of topics: computer-based testing, methods and guidelines for adapting tests from one language and culture to another, and design and field-testing of new approaches for reporting test scores.


Contact Ron via rkh@umass.edu

Key:

Complete
Failed
Available
Locked
Digital Module (Complete Interactive Version)
Recorded 07/17/2019
Recorded 07/17/2019 The full version of the digital module with all interactive components, video, and resources.
VIDEO DM08 (Sections 1-3)
Open to view video.
Open to view video. Video version of the three core content section of the module with audio and animations but without any interactive components such as quiz questions or data activities.
VIDEO R Shiny Intro
Open to view video.
Open to view video. Four-part video that provides an introduction to how to use the Shiny Item Analysis application with an embedded example.
VIDEO R Shiny Activity
Open to view video.
Open to view video. Five-part video walk-through of the data activity for the Shiny R app.
VIDEO TAP Intro
Open to view video.
Open to view video. Five-part video that provides an introduction to the TAP program with an embedded worked example.
VIDEO Tap Activity
Open to view video.
Open to view video. Six-part video providing a walk-through of the data activity for the TAP program.
Data Files
Open to download resource.
Open to download resource. All data files for all worked examples and the data activities.