All-access Pass

5 (10 votes)

This provides immediate access to ALL print and digital modules in the portal by "registering" you for each and displaying all modules as a single collection as part of this pass.

 

  • Digital Module 13: Simulation Studies in IRT

    Contains 8 Component(s) Recorded On: 04/03/2020

    In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Key words: bias, bi-factor model, estimation, graded response model, item response theory, mean squared error, Monte Carlo, simulation, standard error, two-parameter logistic model

    In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Dr. Leventhal and Dr. Ames review the conceptual foundation of MCSS in IRT and walk through the processes of simulating total scores as well as item responses using the two-parameter logistic, graded response, and bi-factor models. They provide guidance for how to implement MCSS using other item response models and best practices for efficient syntax and executing an MCSS. The digital module contains sample SAS code, diagnostic quiz questions, activities, curated resources, and a glossary.

    Key words: bias, bi-factor model, estimation, graded response model, item response theory, mean squared error, Monte Carlo, simulation, standard error, two-parameter logistic model

    Brian Leventhal

    Assistant Professor

    Brian is an assistant professor in the Assessment and Measurement PhD program in the Department of Graduate Psychology at James Madison University as well as an assistant assessment specialist in the Center for Assessment and Research Studies at James Madison University. There, he teaches courses in quantitative methods, including a course on Simulation Studies in Psychometrics. Brian received his Ph.D. from the University of Pittsburgh. His research interests include multidimensional item response models that account for response styles, response process models, and classification errors in testing. Brian is passionate about teaching and providing professional development for graduate students and early-career practitioners. He has thoroughly enjoyed collaborating with Allison Ames and the Instructional Design Team to develop this module. 

    Contact Brian via leventbc@jmu.edu

    Allison J. Ames

    Assistant Professor

    Allison is an assistant professor in the Educational Statistics and Research Methods program in the Department of Rehabilitation, Human Resources and Communication Disorders, Research Methodology, and Counseling at the University of Arkansas. There, she teaches courses in educational statistics, including a course on Bayesian inference. Allison received her Ph.D. from the University of North Carolina at Greensboro. Her research interests include Bayesian item response theory, with an emphasis on prior specification; model-data fit; and models for response processes. Her research has been published in prominent peer-reviewed journals. She enjoyed collaborating on this project with a graduate student, senior faculty member, and the Instructional Design Team.
    Contact Allison via boykin@uark.edu

  • Digital Module 23: Multidimensional Item Response Theory Graphics

    Contains 6 Component(s) Recorded On: 03/13/2021

    In this digital ITEMS module, Dr. Terry Ackerman and Dr. Qing Xie cover the underlying theory and application of multidimensional item response theory models from a visual perspective. Keywords: centroid plot, clamshell plot, contour plot, item information curve, item information surface, multidimensional item response theory, MIRT, response surface, RShiny, test characteristic curve, test characteristic surface, vector

    In this digital ITEMS module, Dr. Terry Ackerman and Dr. Qing Xie cover the underlying theory and application of multidimensional item response theory models from a visual perspective. They begin the module with a brief review of how to interpret evidence of dimensionality of test data. They then examine the basic characteristics of unidimensional IRT models and how these concepts change when the IRT model is expanded to two dimensions. This leads to a more in-depth discussion of how unidimensional item characteristic curves change in two-dimensional models and can be represented as a surface, as a contour plot, or collectively as a set of vectors. They then expanded this to the test level where test characteristic curves become test characteristic surfaces and with accompanying contours. They include additional discussions on how to compute information and represent it in terms of “clamshell”, number, or centroid plots. The module includes audio-narrated slides as well as the usual package of the usual package of curated resources, a glossary, data activities, and quiz questions with diagnostic feedback.

    Keywords: centroid plot, clamshell plot, contour plot, item information curve, item information surface, multidimensional item response theory, MIRT, response surface, RShiny, test characteristic curve, test characteristic surface, vector

  • Digital Module 16: Longitudinal Data Analysis

    Contains 9 Component(s) Recorded On: 01/21/2021

    In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically-defensible function that adequately summarizes both individual change as well as the average response. Keywords: fixed effect, linear mixed effects models, longitudinal data analysis, multilevel models, population-average, random effect, regression, subject-specific, trajectory

    In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically-defensible function that adequately summarizes both individual change as well as the average response. The module begins with a non-technical overview of longitudinal data analyses drawing distinctions with cross-sectional analyses in terms of research questions to be addressed. Nuances of longitudinal designs, timing of measurements, and the real possibility of missing data are then discussed. The three interconnected components of the LME model: (1) a model for individual and mean response profiles, (2) a model to characterize the covariation among the time-specific residuals, and (3) a set of models that summarize the extent that individual coefficients vary, are discussed in the context of the set of activities comprising an analysis. Finally, they demonstrate how to estimate the linear mixed effects model within an open-source environment (R). The digital module contains sample R code, diagnostic quiz questions, hands-on activities in R, curated resources, and a glossary.

     Keywords: fixed effect, linear mixed effects models, longitudinal data analysis, multilevel models, population-average, random effect, regression, subject-specific, trajectory

    Jeffrey R. Harring

    Professor

    Jeff is a Professor in the Measurement, Statistics and Evaluation Program within the Department of Human Development and Quantitative Methodology at the University of Maryland, College Park. There, he teaches introductory and intermediate graduate level statistics courses and advanced quantitative methods seminars in longitudinal data analysis, mixture modeling, simulation design and statistical computing. Jeff has taught several multi-day workshops on the application of longitudinal methods using R and SAS statistical software most recently at the National Center of Educational Statistics (NCES) in Washington D.C.  Prior to joining the program faculty in the fall of 2006, Jeff received an M.S. degree in Statistics and completed his Ph.D. in the Quantitative Methods in Education from the University of Minnesota. Before that, Jeff taught high school mathematics for 12 years. He has published nearly 100 articles and book chapters, co-edited three volumes and co-authored a book. His research focuses on linear and nonlinear models for repeated measures data, structural equation models, finite mixtures of both linear and nonlinear growth models and extensions of these methods to multilevel data structures.

    Contact Jeff via harring@umd.edu

    Tessa Johnson

    Doctoral Candidate

    Tessa is a Ph.D. candidate in the Measurement, Statistics and Evaluation Program within the Department of Human Development and Quantitative Methodology at the University of Maryland, College Park. She received her Master of Science in Educational Research from Georgia State University. Tessa currently serves as a project coordinator for the Synthetic Data Project (SDP) of the Maryland Longitudinal Data System Center, an Institute of Education Sciences (IES) funded project aimed at assessing the feasibility of and implementing a system for synthesizing statewide longitudinal data in order to increase data access for researchers and policy analysts while minimizing risk of data disclosure. For the SDP, Tessa conducts research on the feasibility of reproducing nested data structures in longitudinal synthetization models. Outside her work with the SDP, Tessa’s research has centered around creating and improving statistical methods for analyzing complex data structures in longitudinal contexts. This includes modeling time as an outcome in latent growth models, accounting for similarities among schools when modeling student mobility in longitudinal studies, and exploring the development of ensembles of social networks in the classroom over time.

    Contact Tessa via johnsont@umd.edu

  • Digital Module 22: Supporting Decisions with Assessment

    Contains 5 Component(s) Recorded On: 03/04/2021

    In this digital ITEMS module, Dr. Chad Gotch walks through different forms of assessment, from everyday actions that are almost invisible, to high-profile, annual, large-scale tests with an eye towards educational decision-making. Keywords: assessment literacy, classroom assessment, decision-making, formative assessment, in-the-moment assessment, interim assessment, large-scale assessment, major milepost, periodic check-in, unit test

    In this digital ITEMS module, Dr. Chad Gotch walks through different forms of assessment, from everyday actions that are almost invisible, to high-profile, annual, large-scale tests with an eye towards educational decision-making. At each stage, he illustrates the form of assessment with real-life examples, pairs it with ideal types of instructional or programmatic decisions, and notes common mismatches between certain decisions and forms of assessment. Teachers, administrators, and policymakers who complete the module will build a foundation to use assessment appropriately and effectively for the benefit of student learning. By going through they module, they will appreciate how assessment, when done well, empowers students and educators and, when done poorly, undermines foundational educational goals and sows anxiety and discord. The module contains audio-narrated slides, interactive exercises with illustrative videos, and a curated set of resources.

    Keywords: assessment literacy, classroom assessment, decision-making, formative assessment, in-the-moment assessment, interim assessment, large-scale assessment, major milepost, periodic check-in, unit test

    Chad Gotch

    Washington State University

    Chad Gotch is an Assistant Professor in the Educational Psychology program at Washington State University. His first experiences as a teacher came through environmental and informal science education for children during his undergraduate and Master’s degrees. After working in program assessment within university administration, Chad moved into the world of educational measurement and student assessment. Currently, he works to maximize appropriate and effective use of educational assessment. To this end, he studies the development of assessment literacy among both pre-service and in-service teachers, the communication of assessment results (e.g., score reporting), and the construction of validity arguments from both technical and non-technical evidence. These complementary lines of research inform the life cycle of assessment, from development to use and policy.

    Chad previously partnered with NCME on the development of the Fundamentals of Classroom Assessment video for NCME (https://vimeo.com/212410753). At the university level, he teaches courses in educational statistics, educational measurement, and classroom assessment. Chad has served in an advisory role with the Washington Office of Superintendent of Public Instruction’s consolidated plan submission for the Every Student Succeeds Act (ESSA) and as a consultant with the Oregon Department of Education in its teacher and administrator education efforts. He has worked with K-16 educators through both workshops and one-on-one consultation on various aspects of student assessment, and is the lead author of the chapter “Preparing Pre-Service Teachers for Assessment of, for, and as Learning” in the forthcoming book Teaching on Assessment from Information Age Publishing.

    Contact Chad via cgotch@wsu.edu

  • Digital Module 14: Planning and Conducting Standard Setting

    Contains 7 Component(s) Recorded On: 05/30/2020

    In this digital ITEMS module, Dr. Michael B. Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, body of work) but on the practical tasks that must be completed for any standard setting activity. Keywords: achievement level descriptor, certification and licensure, cut score, feedback, interquartile range, performance level descriptor, score reporting, standard setting, panelist, vertical articulation

    In this digital ITEMS module, Dr. Michael B. Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, body of work) but on the practical tasks that must be completed for any standard setting activity. Dr. Bunch carries the participant through every stage of the standard setting process, from developing a plan, through preparations for standard setting, conducting standard setting, and all the follow-up activities that must occur after standard setting in order to obtain approval of cut scores and translate those cut scores into score reports. The digital module includes a 120-page manual, various ancillary files (e.g., PowerPoint slides, Excel workbooks, sample documents and forms), links to datasets from the book Standard Setting (Cizek & Bunch, 2007), links to final reports from four recent large-scale standard setting events, quiz questions with formative feedback, and a glossary.

    Keywords: achievement level descriptor, certification and licensure, cut score, feedback, interquartile range, performance level descriptor, score reporting, standard setting, panelist, vertical articulation 

    Michael B. Bunch

    Senior Vice President, Measurement Incorporated

    Mike joined Measurement Incorporated (MI) as director of test development in 1982. Prior to that time, he was a research psychologist at ACT (1976-78) and senior professional at NTS Research (1978-82). At MI he was soon promoted to vice president for research and development and later to senior vice president. He currently serves on MI’s board of directors and continues to direct MI’s psychometric team. While still employed by MI, Mike taught graduate courses in statistics and research methods at North Carolina Central University for six years and a mini-course on standard setting through the University of Maryland. He has presented several half-day and full-day training sessions on standard setting at NCME and other national and international organizations. In addition, he co-authored of Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests with Gregory J. Cizek (2007) as well as various other articles and book chapters on standard setting. He has planned and carried out some of the largest and most complex standard setting events ever conceived.

    Contact Mike via MBunch@measinc.com 

  • Digital Module 15: Accessibility of Educational Assessments

    Contains 7 Component(s) Recorded On: 06/01/2020

    In this digital ITEMS module, Dr. Ketterlin Geller and her colleagues provide an introduction to accessibility of educational assessments. They discuss the legal basis for accessibility in K-12 and higher education organizations and describe how test and item design features as well as examinee characteristics affect the role that accessibility plays in evaluating test validity during test development operational deployment. Keywords: Accessibility, accommodations, examinee characteristics, fairness, higher education, K-12 education, item design, legal guidelines, test development, universal design

    In this digital ITEMS module, Dr. Ketterlin Geller and her colleagues provide an introduction to accessibility of educational assessments. They discuss the legal basis for accessibility in K-12 and higher education organizations and describe two key factors that interact to influence accessibility of educational assessments: (1) test and item design features and (2) examinee characteristics. They highlight the outcome of this interaction in various situated examples and discuss the role accessibility plays in evaluating test validity as well as the principles of universal design during test development and test accommodations during operational deployment. The module includes an interview with Dr. Martha Thurlow, an expert in accessibility, who provides an important perspective on the past, present, and future of accessibility for educational assessments. The module is designed to be relevant for students, test developers, and users of educational tests in K-12 and higher education settings. It contains audio-narrated slides, interactive activities, and quizzes as well as curated resources and a glossary.

    Keywords: Accessibility, accommodations, examinee characteristics, fairness, higher education, K-12 education, item design, legal guidelines, test development, universal design

    Leanne Ketterlin Geller

    Professor, Southern Methodist University

    Leanne is a professor at Southern Methodist University, specializing in applied measurement and assessment. She holds the Texas Instruments Endowed Chair in Education and directs the center on Research in Mathematics Education. Her scholarship focuses on supporting all students in mathematics education through application of instructional leadership principles and practices. She has served as Principal Investigator for federally and state funded research grants to develop and implement formative assessment procedures and valid decision-making systems for students with diverse needs in the general education curriculum. She has published numerous articles and book chapters, and presented original research findings at local, national, and international conferences, and serves on the editorial boards for the International Journal of Testing and Assessment for Effective Intervention. She works closely with teachers and administrators to support their application of measurement and assessment principles in school-based decision making. Dr. Ketterlin-Geller was a high school science teacher and trained as a K-12 administrator.

    Contact Leanne at lkgeller@mail.smu.edu

    Brooke Istas

    Doctoral Student, Southern Methodist University

    Brooke is a Graduate Research Assistant and doctoral student at the Simmons School of Education and Human Development at Southern Methodist University. Her research interests center on adult learners perceptions of mathematics. Brooke is also a consultant, an adult education mathematics subject matter expert, and Cowley College Mathematics Faculty. She is recognized nationally for her knowledge of mathematics and mathematical instructional strategies. She has given several presentations at state and national conferences on enhancing mathematical instruction, understanding higher level mathematical content, teaching math at a distance, and on-line instruction; always willing to share her learning with others. She is also the Subject Matter Expert (SME) for the LINCS (Literacy Information and Communication System) Math and Numeracy and Science Community of Practice; also a reviewer for the Math/Numeracy and Science online resource collection that is a part of the Basic Skills Collection.

    Contact Brooke at XXX

    Robyn K. Pinilla

    Doctoral Student, Southern Methodist University

    Robyn is a Ph.D. student and Graduate Research Assistant in Research in Mathematics Education at the Simmons School of Education and Human Development at Southern Methodist University. Her specific research interests are in early childhood spatial reasoning and problem solving, and equity and access within assessment for all students. She served as an elementary school assistant principal and early childhood Special Education teacher prior to beginning her Ph.D. studies. In this work, she developed an assets-based approach to first celebrate student and teacher success before triangulating their current development with next steps and targeted objectives. This informal application of the curriculum, instruction, and assessment framework prepared her to begin her doctoral studies under the advisement of Dr. Leanne Ketterlin-Geller. She continues advocating for authentic and practical instruction and assessment that is accessible to all through research and local, state, and national collaborations.

    Contact Robyn at XXX

    Ann Marie Wernick

    Doctoral Student, Southern Methodist University

    Ann Marie is a Graduate Research Assistant and doctoral student at the Simmons School of Education and Human Development at Southern Methodist University. Her research interests focus on teacher induction, coaching models, practice-based teacher education, teacher evaluation systems, and mixed-reality simulations. Before her time at SMU, Ann Marie earned her M.Ed. from the University of Notre Dame, where she served as an ACE Fellow. She then spent the next seven years as a classroom teacher and instructional coach. During that time, she taught middle school English Language Arts and History in public, private, and parochial schools both in the United States and abroad. In each of these settings, she gained valuable experience mentoring and coaching novice teachers, which fostered her interest in teacher induction and pre-service teacher coaching in both alternative certification and traditional teacher preparation programs.

    Contact Ann Marie at XXX

  • Digital Module 12: Think-aloud Interviews and Cognitive Labs

    Contains 6 Component(s)

    ​In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes and illustrate both traditional and modern data-collection methods. Keywords: ABC tool, cognitive laboratory, cog lab, cognition, cognitive model, interrater agreement, kappa, probe, rubric, thematic analysis, think-aloud interview, verbal report

    In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze distinct types of verbal reports in the collection of evidence of test-taker response processes. The module includes details on (a) the different types of cognition that are tapped by different interviewer probes, (b) traditional interviewing methods and new automated tools for collecting verbal reports, and (c) options for analyses of verbal reports. This includes a discussion of reliability and validity issues such as potential bias in the collection of verbal reports, ways to mitigate bias, and inter-rater agreement to enhance credibility of analysis. A novel digital tool for data-collection called the ABC tool is presented via illustrative videos. As always, the module contains audio-narrated slides, quiz questions with feedback, a glossary, and curated resources. 

    Keywords: ABC tool, cognitive laboratory, cog lab, cognitive model, interrater agreement, kappa, probe, rubric, thematic analysis, think-aloud interview, verbal report

    Jacqueline P. Leighton

    Professor

    Jackie is a Registered Psychologist and Professor of School and Clinical Child Psychology at the University of Alberta. She completed her graduate degrees in Psychology at the University of Alberta and postdoctoral fellowship studies in Psychology at Yale University. Her research and teaching is driven by the overarching goal to enhance fairness in testing. In pursuit of this goal, she has increasingly focused her research on investigating the interplay between cognitive and emotional processes underlying learning and academic achievement. For example, she investigates variables that can cognitively or emotionally bias participants’ response processes in testing situations, leading to misrepresentations in performance and weaknesses in validity arguments of test inferences. Overall, she is interested in methods designed to enhance diagnosis of achievement, validation of test inferences, and theoretical understanding of human learning. 

    Contact Jackie via jacqueline.leighton@ualberta.ca

    Blair Lehman

    Research Scientist

    Blair is a research scientist at Educational Testing Service. She completed her graduate degrees in Cognitive Psychology and certificate in Cognitive Science at the University of Memphis. Her research focuses on understanding students’ emotional and motivational processes during educational activities to design activities that maximize the experience for all students. Her research has focused on the task design of learning and assessment activities as well as the design of adaptive systems that consider both student cognition and motivation. For example, she has explored specific design features of game-based assessments in an effort to understand how to maintain measurement validity while also maximizing student motivation. 

    Contact Blair via blehman@ets.org

  • Digital Module 17: Data Visualizations

    Contains 9 Component(s) Recorded On: 08/20/2020

    In this digital module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. The instructors review key literature, discuss strategies for enhancing graphical presentation, and provide an introduction to the Graph Template Language (GTL) in SAS to illustrate how elementary components can be used to make efficient, effective and accurate graphics for a variety of audiences. Key words: data visualization, graphical excellence, graphical template language, SAS

    In this digital module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. Data visualizations are commonly used by measurement professionals to communicate results to examinees, the public, educators, and other stakeholders. To create effective visualizations, it is important that they communicate data effectively, efficiently, and accurately. Unfortunately, measurement and statistical software default graphics typically fail to uphold these standards and are therefore not necessarily suitable for publication or presentation to the public. The instructors review key literature, discuss strategies for enhancing graphical presentation, and provide an introduction to the Graph Template Language (GTL) in SAS to illustrate how elementary components can be used to make efficient, effective and accurate graphics for a variety of audiences. The module contains audio-narrated slides, embedded illustrative videos, quiz questions with diagnostic feedback, a glossary, sample SAS code, and other learning resources.

    Key words: data visualization, graphical excellence, graphical template language, SAS

    Nikole Gregg

    Doctoral Student

    Nikole Gregg is a doctoral student in the Assessment & Measurement program at James Madison University, where she has taken on many roles, including various assessment and measurement consulting experiences at JMU and in K-12 settings. Through her work, she has refined and developed skills necessary to present sophisticated data analyses to non-technical audiences. Her research interests include the application of multidimensional item response theory to account for response styles, fairness in testing, and validity theory. Nikole is passionate about improving fairness and equity within assessment, measurement, and policy.

    Contact Nikole via greggnl@dukes.jmu.edu

    Brian Leventhal

    Assistant Professor

    Brian is an assistant professor in the Assessment and Measurement PhD program in the Department of Graduate Psychology at James Madison University as well as an assistant assessment specialist in the Center for Assessment and Research Studies at James Madison University. There, he teaches courses in quantitative methods, including a course on Simulation Studies in Psychometrics. Brian received his Ph.D. from the University of Pittsburgh. His research interests include multidimensional item response models that account for response styles, response process models, and classification errors in testing. Brian is passionate about teaching and providing professional development for graduate students and early-career practitioners. He has thoroughly enjoyed collaborating with Allison Ames and the Instructional Design Team to develop this module. 

    Contact Brian via leventbc@jmu.edu

  • Digital Module 18: Automated Scoring

    Contains 8 Component(s) Recorded On: 12/04/2020

    In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. They discuss automated scoring from a number of perspectives and provide two data examples, one focused on training and evaluating an automated scoring engine and one focused on the impact of rater error on predicted scores. Key words: automated scoring, hand-scoring, machine learning, natural language processes, constructed response items

    In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R-Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.

    Key words: automated scoring, hand-scoring, machine learning, natural language processes, constructed response items

    Susan Lottridge

    Cambium Assessment

    Sue Lottridge, Ph.D. is a Senior Director of Automated Scoring at the Cambium Assessment, Inc. (CAI).  In this role, she leads CAI’s machine learning and scoring team on the research, development, and operation of CAI’s automated scoring software. This software includes automated essay scoring, short answer scoring, automated speech scoring, and an engine that detects disturbing content in student responses. Dr. Lottridge has worked in automated scoring for twelve years and has contributed to the design, research, and use of multiple automated scoring engines including equation scoring, essay scoring, short answer scoring, alert detection, and dialogue systems. She earned her Ph.D. from James Madison University in assessment and measurement (2006) and holds Masters’ degrees in Mathematics and in Computer Science from the University of Wisconsin-Madison (1997). 

    Contact Sue via susanlottridge@hotmail.com

    Amy Burkhardt

    University of Colorado - Boulder

    Amy Burkhardt is a PhD Candidate in Research and Evaluation Methodology with an emphasis in Human Language Technology at the University of Colorado, Boulder. She has been involved in the development of two automated scoring systems. Ongoing research projects include the automatic detection of students reporting harm within online tests, the use of machine learning to explore public discourse around educational policies, and considerations in psychometric modeling when making diagnostic inferences aligned to a learning progression. 

    Contact Amy via amy.burkhardt@colorado.edu

    Michelle Boyer

    Center for Assessment

    Michelle Boyer, Ph.D. is a Senior Associate at The National Center for the Improvement of Educational Assessment, Inc. Dr. Boyer consults with states and organizations on such issues as assessment systems, validity of score interpretations, scoring design and evaluation criteria for both human and automated scoring, assessment literacy, and score comparability. She is also a regular contributor to professional publications and the annual conferences of AERA, NCME, and CCSSO. Her most recent research focuses on evaluating the quality of automated scoring and its impact test score scales and test equating solutions. Dr. Boyer earned her Ph.D. from the University of Massachusetts, Amherst in Research, Educational Measurement, and Psychometrics (2018). 

    Contact Michelle via mboyer@nciea.org

  • Digital Module 19: Foundations of IRT Estimation

    Contains 5 Component(s) Recorded On: 11/29/2020

    In this digital ITEMS module, Dr. Zhuoran Wang and Dr. Nathan Thompson introduce the basic item response theory (IRT) item calibration and examinee scoring procedures as well as strategies to improve estimation accuracy. Keywords: calibration, EM algorithm, estimation accuracy, item response theory (IRT), maximum likelihood estimation (MLE), maximum a posteriori (MAP), expected a posteriori (EAP), marginal maximum likelihood estimation (MMLE), scoring

    In this digital ITEMS module, Dr. Zhuoran Wang and Dr. Nathan Thompson introduce the basic item response theory (IRT) item calibration and examinee scoring procedures as well as strategies to improve estimation accuracy. They begin the module with a conceptual review of IRT that includes core advantages of the IRT framework, commonly used IRT models, and essential components such as information and likelihood functions. In the second part of the module, they illustrate the structure and inner workings of calibration and scoring algorithms such as the MMLE/EM algorithm for item parameter calibration and the MLE, EAP, and MAP algorithms for examinee scoring. In part three, they demonstrate the influence of multiple factors on estimation accuracy and provide strategies for maximizing accuracy. In addition to audio-narrated slides, the digital module contains sample R code, quiz questions with diagnostic feedback, curated resources, and a glossary.

    Keywords: calibration, EM algorithm, estimation accuracy, item response theory (IRT), maximum likelihood estimation (MLE), maximum a posteriori (MAP), expected a posteriori (EAP), marginal maximum likelihood estimation (MMLE), scoring 

    Zhuoran Wang

    Prometric

    Zhuoran Wang is a psychometrician at Prometric. She provides psychometric expertise for a number of clients, mostly professional associations and state-based licensure exams. Tasks mainly includes classical test theory analysis, item response theory analysis, computerized adaptive testing development, equating, scaling, job analysis, and standard setting. Zhuoran graduated from University of Minnesota with a Ph.D. in Psychometrics/ Quantitative Methods. Her research interests span a wide range of topics in psychometrics including multidimensional item response theory, cognitive diagnostic modeling, differential item functioning, computerized adaptive testing, and multistage testing. When in graduate school, she taught multiple undergraduate level and graduate level statistic and data analysis exercise courses. She also provided R tutorial in two workshops on multidimensional IRT models in IACAT conference 2017 and 2019.

    Contact Zhuoran via wang5105@umn.edu

    Nathan Thompson

    Assessment Systems Corporation (ASC)

    Nathan Thompson currently serve as VP for ASC (www.assess.com). His focus is on bringing quality measurement to more organizations, either through software that automates the work, makes it easier to implement sophisticated psychometrics like item response theory, or directly consulting with organizations to ensure that they align with best practices. Nate earned his PhD in Psychometrics from the University of Minnesota, with a focus on computerized adaptive testing. His undergraduate degree was from Luther College with a triple major of Mathematics, Psychology, and Latin. He is primarily interested in the use of artificial intelligence and software automation to augment and replace the work done by psychometricians, which has provided extensive experience in software design and programming. He has published over 100 journal articles and conference presentations. In addition to research, Nate has abundant teaching experiences. He served as an Instructor for the Department of Psychology for two undergraduate statistic courses while in graduate school. When working as adjunct faculty in University of Cincinnati, he taught an online class on measurement and assessment as part of a Master's in Medical Education program.

    Contact Nate via nate@assess.com