All-access Pass

5 (10 votes)

This provides immediate access to ALL print and digital modules in the portal by "registering" you for each and displaying all modules as a single collection as part of this pass.

 

  • Digital Module 13: Simulation Studies in IRT

    Contains 8 Component(s) Recorded On: 04/03/2020

    In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Key words: bias, bi-factor model, estimation, graded response model, item response theory, mean squared error, Monte Carlo, simulation, standard error, two-parameter logistic model

    In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Dr. Leventhal and Dr. Ames review the conceptual foundation of MCSS in IRT and walk through the processes of simulating total scores as well as item responses using the two-parameter logistic, graded response, and bi-factor models. They provide guidance for how to implement MCSS using other item response models and best practices for efficient syntax and executing an MCSS. The digital module contains sample SAS code, diagnostic quiz questions, activities, curated resources, and a glossary.

    Key words: bias, bi-factor model, estimation, graded response model, item response theory, mean squared error, Monte Carlo, simulation, standard error, two-parameter logistic model

    Brian Leventhal

    Assistant Professor

    Brian is an assistant professor in the Assessment and Measurement PhD program in the Department of Graduate Psychology at James Madison University as well as an assistant assessment specialist in the Center for Assessment and Research Studies at James Madison University. There, he teaches courses in quantitative methods, including a course on Simulation Studies in Psychometrics. Brian received his Ph.D. from the University of Pittsburgh. His research interests include multidimensional item response models that account for response styles, response process models, and classification errors in testing. Brian is passionate about teaching and providing professional development for graduate students and early-career practitioners. He has thoroughly enjoyed collaborating with Allison Ames and the Instructional Design Team to develop this module. 

    Contact Brian via leventbc@jmu.edu

    Allison J. Ames

    Assistant Professor

    Allison is an assistant professor in the Educational Statistics and Research Methods program in the Department of Rehabilitation, Human Resources and Communication Disorders, Research Methodology, and Counseling at the University of Arkansas. There, she teaches courses in educational statistics, including a course on Bayesian inference. Allison received her Ph.D. from the University of North Carolina at Greensboro. Her research interests include Bayesian item response theory, with an emphasis on prior specification; model-data fit; and models for response processes. Her research has been published in prominent peer-reviewed journals. She enjoyed collaborating on this project with a graduate student, senior faculty member, and the Instructional Design Team.
    Contact Allison via boykin@uark.edu

  • Digital Module 17: Data Visualizations

    Contains 9 Component(s) Recorded On: 08/20/2020

    In this digital module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. The instructors review key literature, discuss strategies for enhancing graphical presentation, and provide an introduction to the Graph Template Language (GTL) in SAS to illustrate how elementary components can be used to make efficient, effective and accurate graphics for a variety of audiences. Key words: data visualization, graphical excellence, graphical template language, SAS

    In this digital module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. Data visualizations are commonly used by measurement professionals to communicate results to examinees, the public, educators, and other stakeholders. To create effective visualizations, it is important that they communicate data effectively, efficiently, and accurately. Unfortunately, measurement and statistical software default graphics typically fail to uphold these standards and are therefore not necessarily suitable for publication or presentation to the public. The instructors review key literature, discuss strategies for enhancing graphical presentation, and provide an introduction to the Graph Template Language (GTL) in SAS to illustrate how elementary components can be used to make efficient, effective and accurate graphics for a variety of audiences. The module contains audio-narrated slides, embedded illustrative videos, quiz questions with diagnostic feedback, a glossary, sample SAS code, and other learning resources.

    Key words: data visualization, graphical excellence, graphical template language, SAS

    Nikole Gregg

    Doctoral Student

    Nikole Gregg is a doctoral student in the Assessment & Measurement program at James Madison University, where she has taken on many roles, including various assessment and measurement consulting experiences at JMU and in K-12 settings. Through her work, she has refined and developed skills necessary to present sophisticated data analyses to non-technical audiences. Her research interests include the application of multidimensional item response theory to account for response styles, fairness in testing, and validity theory. Nikole is passionate about improving fairness and equity within assessment, measurement, and policy.

    Contact Nikole via greggnl@dukes.jmu.edu

    Brian Leventhal

    Assistant Professor

    Brian is an assistant professor in the Assessment and Measurement PhD program in the Department of Graduate Psychology at James Madison University as well as an assistant assessment specialist in the Center for Assessment and Research Studies at James Madison University. There, he teaches courses in quantitative methods, including a course on Simulation Studies in Psychometrics. Brian received his Ph.D. from the University of Pittsburgh. His research interests include multidimensional item response models that account for response styles, response process models, and classification errors in testing. Brian is passionate about teaching and providing professional development for graduate students and early-career practitioners. He has thoroughly enjoyed collaborating with Allison Ames and the Instructional Design Team to develop this module. 

    Contact Brian via leventbc@jmu.edu

  • Digital Module 12: Think-aloud Interviews and Cognitive Labs

    Contains 6 Component(s)

    ​In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes and illustrate both traditional and modern data-collection methods. Keywords: ABC tool, cognitive laboratory, cog lab, cognition, cognitive model, interrater agreement, kappa, probe, rubric, thematic analysis, think-aloud interview, verbal report

    In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze distinct types of verbal reports in the collection of evidence of test-taker response processes. The module includes details on (a) the different types of cognition that are tapped by different interviewer probes, (b) traditional interviewing methods and new automated tools for collecting verbal reports, and (c) options for analyses of verbal reports. This includes a discussion of reliability and validity issues such as potential bias in the collection of verbal reports, ways to mitigate bias, and inter-rater agreement to enhance credibility of analysis. A novel digital tool for data-collection called the ABC tool is presented via illustrative videos. As always, the module contains audio-narrated slides, quiz questions with feedback, a glossary, and curated resources. 

    Keywords: ABC tool, cognitive laboratory, cog lab, cognitive model, interrater agreement, kappa, probe, rubric, thematic analysis, think-aloud interview, verbal report

    Jacqueline P. Leighton

    Professor

    Jackie is a Registered Psychologist and Professor of School and Clinical Child Psychology at the University of Alberta. She completed her graduate degrees in Psychology at the University of Alberta and postdoctoral fellowship studies in Psychology at Yale University. Her research and teaching is driven by the overarching goal to enhance fairness in testing. In pursuit of this goal, she has increasingly focused her research on investigating the interplay between cognitive and emotional processes underlying learning and academic achievement. For example, she investigates variables that can cognitively or emotionally bias participants’ response processes in testing situations, leading to misrepresentations in performance and weaknesses in validity arguments of test inferences. Overall, she is interested in methods designed to enhance diagnosis of achievement, validation of test inferences, and theoretical understanding of human learning. 

    Contact Jackie via jacqueline.leighton@ualberta.ca

    Blair Lehman

    Research Scientist

    Blair is a research scientist at Educational Testing Service. She completed her graduate degrees in Cognitive Psychology and certificate in Cognitive Science at the University of Memphis. Her research focuses on understanding students’ emotional and motivational processes during educational activities to design activities that maximize the experience for all students. Her research has focused on the task design of learning and assessment activities as well as the design of adaptive systems that consider both student cognition and motivation. For example, she has explored specific design features of game-based assessments in an effort to understand how to maintain measurement validity while also maximizing student motivation. 

    Contact Blair via blehman@ets.org

  • Digital Module 25: Testlet Response Theory

    Product not yet rated Contains 4 Component(s)

    ​In this digital ITEMS module, Dr. Hong Jiao and Dr. Manqian Liao describe testlet response theory and associated measurement models for the construction and evaluation of new measures and scales when local item dependence is present. Key words: Bayesian estimation, innovative assessment, item response theory, local item dependence, multi-part items, paired passages, testlets, testlet response theory

    In this digital ITEMS module, Dr. Hong Jiao and Dr. Manqian Liao describe testlet response theory for the construction and evaluation of new measures and scales. They start with an introduction to the needs of testlets when local item dependence is present and then introduce the basic testlet response models proposed from different theoretical frameworks built upon standard item response theory models. Furthermore, they introduce different methods for model parameter estimation and related software programs that implement them.  Finally, they showcase further extensions of the extended testlet response models for more complex local item dependence issues in innovative assessment. The module is designed for students, researchers, and data scientists in various disciplines such as psychology, sociology, education, business, health and other social sciences in developing testlet-based assessment. It contains audio-narrated slides, sample data, syntax filesdiagnostic quiz questions, data-based activities, curated resources, and a glossary.  

    Key words: Bayesian estimation, innovative assessment, item response theorylocal item dependence, multi-part items, paired passages, testletstestlet response theory 

    Hong Jiao

    Professor

    University of Maryland, College Park

    Hong Jiao is currently a professor at the University of Maryland (UMD), College Park specializing in educational measurement and psychometrics in large-scale assessment. She received her doctoral degree from Florida State University. Prior to joining the faculty in Measurement, Statistics, and Evaluation at UMD, she worked as a psychometrician at Harcourt Assessment on different state assessment programs. Her methodological research is to improve the practice in educational and psychological assessment and develop solutions to emerging psychometric challenges. Many of these are due to the use of more complex innovative assessment formats. Two areas of her research include methodological research on local dependence due to the use of testlet and Bayesian model parameter estimation. Her methodological research has been recognized by a national award, academic work including numerous edited books, book chapters, refereed journal papers, and national and international invited and refereed presentations and different research grants and contracts on which she serve as PI or CO-PI. Hong Jiao proposed a multilevel testlet model for mixed-format tests that won the 2014 Bradley Hanson Award for Contributions to Educational Measurement by the National Council on Measurement in Education.

    Contact Hong at hjiao@umd.edu

    Manqian Liao

    Psychometrician

    Duolingo

    Manqian Liao is a psychometrician at Duolingo where she conducts validity and fairness research on the Duolingo English Test. Among other things, she has worked on investigating the differential item functioning in the Duolingo English Test items and evaluating the efficiency of the item selection algorithm. Manqian received her Ph.D. degree in Measurement, Statistics and Evaluation from University of Maryland, College Park. Her research interest focuses on item response theory (IRT) and diagnostic classification models (DCM). Her dissertation is on modeling multiple problem-solving strategies and strategy shift with DCM 

    Contact Manqian at mancyliao@gmail.com

  • Digital Module 14: Planning and Conducting Standard Setting

    Contains 7 Component(s) Recorded On: 05/30/2020

    In this digital ITEMS module, Dr. Michael B. Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, body of work) but on the practical tasks that must be completed for any standard setting activity. Keywords: achievement level descriptor, certification and licensure, cut score, feedback, interquartile range, performance level descriptor, score reporting, standard setting, panelist, vertical articulation

    In this digital ITEMS module, Dr. Michael B. Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, body of work) but on the practical tasks that must be completed for any standard setting activity. Dr. Bunch carries the participant through every stage of the standard setting process, from developing a plan, through preparations for standard setting, conducting standard setting, and all the follow-up activities that must occur after standard setting in order to obtain approval of cut scores and translate those cut scores into score reports. The digital module includes a 120-page manual, various ancillary files (e.g., PowerPoint slides, Excel workbooks, sample documents and forms), links to datasets from the book Standard Setting (Cizek & Bunch, 2007), links to final reports from four recent large-scale standard setting events, quiz questions with formative feedback, and a glossary.

    Keywords: achievement level descriptor, certification and licensure, cut score, feedback, interquartile range, performance level descriptor, score reporting, standard setting, panelist, vertical articulation 

    Michael B. Bunch

    Senior Vice President, Measurement Incorporated

    Mike joined Measurement Incorporated (MI) as director of test development in 1982. Prior to that time, he was a research psychologist at ACT (1976-78) and senior professional at NTS Research (1978-82). At MI he was soon promoted to vice president for research and development and later to senior vice president. He currently serves on MI’s board of directors and continues to direct MI’s psychometric team. While still employed by MI, Mike taught graduate courses in statistics and research methods at North Carolina Central University for six years and a mini-course on standard setting through the University of Maryland. He has presented several half-day and full-day training sessions on standard setting at NCME and other national and international organizations. In addition, he co-authored of Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests with Gregory J. Cizek (2007) as well as various other articles and book chapters on standard setting. He has planned and carried out some of the largest and most complex standard setting events ever conceived.

    Contact Mike via MBunch@measinc.com 

  • Digital Module 15: Accessibility of Educational Assessments

    Contains 7 Component(s) Recorded On: 06/01/2020

    In this digital ITEMS module, Dr. Ketterlin Geller and her colleagues provide an introduction to accessibility of educational assessments. They discuss the legal basis for accessibility in K-12 and higher education organizations and describe how test and item design features as well as examinee characteristics affect the role that accessibility plays in evaluating test validity during test development operational deployment. Keywords: Accessibility, accommodations, examinee characteristics, fairness, higher education, K-12 education, item design, legal guidelines, test development, universal design

    In this digital ITEMS module, Dr. Ketterlin Geller and her colleagues provide an introduction to accessibility of educational assessments. They discuss the legal basis for accessibility in K-12 and higher education organizations and describe two key factors that interact to influence accessibility of educational assessments: (1) test and item design features and (2) examinee characteristics. They highlight the outcome of this interaction in various situated examples and discuss the role accessibility plays in evaluating test validity as well as the principles of universal design during test development and test accommodations during operational deployment. The module includes an interview with Dr. Martha Thurlow, an expert in accessibility, who provides an important perspective on the past, present, and future of accessibility for educational assessments. The module is designed to be relevant for students, test developers, and users of educational tests in K-12 and higher education settings. It contains audio-narrated slides, interactive activities, and quizzes as well as curated resources and a glossary.

    Keywords: Accessibility, accommodations, examinee characteristics, fairness, higher education, K-12 education, item design, legal guidelines, test development, universal design

    Leanne Ketterlin Geller

    Professor, Southern Methodist University

    Leanne is a professor at Southern Methodist University, specializing in applied measurement and assessment. She holds the Texas Instruments Endowed Chair in Education and directs the center on Research in Mathematics Education. Her scholarship focuses on supporting all students in mathematics education through application of instructional leadership principles and practices. She has served as Principal Investigator for federally and state funded research grants to develop and implement formative assessment procedures and valid decision-making systems for students with diverse needs in the general education curriculum. She has published numerous articles and book chapters, and presented original research findings at local, national, and international conferences, and serves on the editorial boards for the International Journal of Testing and Assessment for Effective Intervention. She works closely with teachers and administrators to support their application of measurement and assessment principles in school-based decision making. Dr. Ketterlin-Geller was a high school science teacher and trained as a K-12 administrator.

    Contact Leanne at lkgeller@mail.smu.edu

    Brooke Istas

    Doctoral Student, Southern Methodist University

    Brooke is a Graduate Research Assistant and doctoral student at the Simmons School of Education and Human Development at Southern Methodist University. Her research interests center on adult learners perceptions of mathematics. Brooke is also a consultant, an adult education mathematics subject matter expert, and Cowley College Mathematics Faculty. She is recognized nationally for her knowledge of mathematics and mathematical instructional strategies. She has given several presentations at state and national conferences on enhancing mathematical instruction, understanding higher level mathematical content, teaching math at a distance, and on-line instruction; always willing to share her learning with others. She is also the Subject Matter Expert (SME) for the LINCS (Literacy Information and Communication System) Math and Numeracy and Science Community of Practice; also a reviewer for the Math/Numeracy and Science online resource collection that is a part of the Basic Skills Collection.

    Contact Brooke at XXX

    Robyn K. Pinilla

    Doctoral Student, Southern Methodist University

    Robyn is a Ph.D. student and Graduate Research Assistant in Research in Mathematics Education at the Simmons School of Education and Human Development at Southern Methodist University. Her specific research interests are in early childhood spatial reasoning and problem solving, and equity and access within assessment for all students. She served as an elementary school assistant principal and early childhood Special Education teacher prior to beginning her Ph.D. studies. In this work, she developed an assets-based approach to first celebrate student and teacher success before triangulating their current development with next steps and targeted objectives. This informal application of the curriculum, instruction, and assessment framework prepared her to begin her doctoral studies under the advisement of Dr. Leanne Ketterlin-Geller. She continues advocating for authentic and practical instruction and assessment that is accessible to all through research and local, state, and national collaborations.

    Contact Robyn at XXX

    Ann Marie Wernick

    Doctoral Student, Southern Methodist University

    Ann Marie is a Graduate Research Assistant and doctoral student at the Simmons School of Education and Human Development at Southern Methodist University. Her research interests focus on teacher induction, coaching models, practice-based teacher education, teacher evaluation systems, and mixed-reality simulations. Before her time at SMU, Ann Marie earned her M.Ed. from the University of Notre Dame, where she served as an ACE Fellow. She then spent the next seven years as a classroom teacher and instructional coach. During that time, she taught middle school English Language Arts and History in public, private, and parochial schools both in the United States and abroad. In each of these settings, she gained valuable experience mentoring and coaching novice teachers, which fostered her interest in teacher induction and pre-service teacher coaching in both alternative certification and traditional teacher preparation programs.

    Contact Ann Marie at XXX

  • Digital Module 16: Longitudinal Data Analysis

    Contains 9 Component(s) Recorded On: 01/21/2021

    In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically-defensible function that adequately summarizes both individual change as well as the average response. Keywords: fixed effect, linear mixed effects models, longitudinal data analysis, multilevel models, population-average, random effect, regression, subject-specific, trajectory

    In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically-defensible function that adequately summarizes both individual change as well as the average response. The module begins with a non-technical overview of longitudinal data analyses drawing distinctions with cross-sectional analyses in terms of research questions to be addressed. Nuances of longitudinal designs, timing of measurements, and the real possibility of missing data are then discussed. The three interconnected components of the LME model: (1) a model for individual and mean response profiles, (2) a model to characterize the covariation among the time-specific residuals, and (3) a set of models that summarize the extent that individual coefficients vary, are discussed in the context of the set of activities comprising an analysis. Finally, they demonstrate how to estimate the linear mixed effects model within an open-source environment (R). The digital module contains sample R code, diagnostic quiz questions, hands-on activities in R, curated resources, and a glossary.

     Keywords: fixed effect, linear mixed effects models, longitudinal data analysis, multilevel models, population-average, random effect, regression, subject-specific, trajectory

    Jeffrey R. Harring

    Professor

    Jeff is a Professor in the Measurement, Statistics and Evaluation Program within the Department of Human Development and Quantitative Methodology at the University of Maryland, College Park. There, he teaches introductory and intermediate graduate level statistics courses and advanced quantitative methods seminars in longitudinal data analysis, mixture modeling, simulation design and statistical computing. Jeff has taught several multi-day workshops on the application of longitudinal methods using R and SAS statistical software most recently at the National Center of Educational Statistics (NCES) in Washington D.C.  Prior to joining the program faculty in the fall of 2006, Jeff received an M.S. degree in Statistics and completed his Ph.D. in the Quantitative Methods in Education from the University of Minnesota. Before that, Jeff taught high school mathematics for 12 years. He has published nearly 100 articles and book chapters, co-edited three volumes and co-authored a book. His research focuses on linear and nonlinear models for repeated measures data, structural equation models, finite mixtures of both linear and nonlinear growth models and extensions of these methods to multilevel data structures.

    Contact Jeff via harring@umd.edu

    Tessa Johnson

    Doctoral Candidate

    Tessa is a Ph.D. candidate in the Measurement, Statistics and Evaluation Program within the Department of Human Development and Quantitative Methodology at the University of Maryland, College Park. She received her Master of Science in Educational Research from Georgia State University. Tessa currently serves as a project coordinator for the Synthetic Data Project (SDP) of the Maryland Longitudinal Data System Center, an Institute of Education Sciences (IES) funded project aimed at assessing the feasibility of and implementing a system for synthesizing statewide longitudinal data in order to increase data access for researchers and policy analysts while minimizing risk of data disclosure. For the SDP, Tessa conducts research on the feasibility of reproducing nested data structures in longitudinal synthetization models. Outside her work with the SDP, Tessa’s research has centered around creating and improving statistical methods for analyzing complex data structures in longitudinal contexts. This includes modeling time as an outcome in latent growth models, accounting for similarities among schools when modeling student mobility in longitudinal studies, and exploring the development of ensembles of social networks in the classroom over time.

    Contact Tessa via johnsont@umd.edu

  • Digital Module 26: Content Alignment in Standards-based Educational Assessment

    Contains 5 Component(s)

    In this digital ITEMS module, Dr. Katherine Reynolds and Dr. Sebastian Moncaleano discuss content alignment, its role in standards-based educational assessment, and popular methods for conducting alignment studies. Key words: Achieve methodology, content alignment, content area standards, content validity, standards-based assessment, Surveys of Enacted Curriculum, Webb methodology

    In this digital ITEMS module, Dr. Katherine Reynolds and Dr. Sebastian Moncaleano discuss content alignment, its role in standards-based educational assessment, and popular methods for conducting alignment studies. The module begins with an overview of content alignment and its relationship to validity before discussing common alignment study methods. Each method’s conceptualization of alignment is discussed, along with more practical issues such as the materials and participants required and steps for carrying out each method. The final component of this module includes a deeper exploration of the Webb method, one of the most popular content alignment study approaches. This includes a detailed and practical discussion of the steps followed in a Webb alignment study, as well as how to interpret the various alignment indices provided in the Webb framework.

    Key words: Achieve methodology, content alignment, content area standards, content validity,  standards-based assessment, Surveys of Enacted Curriculum, Webb methodology

    Katherine A. Reynolds

    Assistant Research Director

    Boston College

    Katherine completed her Ph.D. in Measurement, Evaluation, Statistics, and Assessment in 2020 at Boston College. Her work focused on scale development and other applications of psychometric research. She has taught graduate courses in research methods and undergraduate courses in the foundations of education. She has worked on several alignment-related projects, including facilitating panels for Webb alignment studies.

    Contact Katherine via katherir@bc.edu

    Sebastian Moncaleano

    Senior Research Specialist

    Boston College

    Sebastian completed his Ph.D. in Measurement, Evaluation, Statistics and Assessment at Boston College in 2021 where he focused his doctoral research on the value of technology-enhanced items in computer-based educational assessments. He has taught graduate courses in introductory statistics and assessment development. He has also conducted content alignment studies for the Department of Elementary and Secondary dedication in Massachusetts.

    Contact Sebastian via moncalea@bc.edu

  • Digital Module 18: Automated Scoring

    Contains 8 Component(s) Recorded On: 12/04/2020

    In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. They discuss automated scoring from a number of perspectives and provide two data examples, one focused on training and evaluating an automated scoring engine and one focused on the impact of rater error on predicted scores. Key words: automated scoring, hand-scoring, machine learning, natural language processes, constructed response items

    In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R-Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.

    Key words: automated scoring, hand-scoring, machine learning, natural language processes, constructed response items

    Susan Lottridge

    Cambium Assessment

    Sue Lottridge, Ph.D. is a Senior Director of Automated Scoring at the Cambium Assessment, Inc. (CAI).  In this role, she leads CAI’s machine learning and scoring team on the research, development, and operation of CAI’s automated scoring software. This software includes automated essay scoring, short answer scoring, automated speech scoring, and an engine that detects disturbing content in student responses. Dr. Lottridge has worked in automated scoring for twelve years and has contributed to the design, research, and use of multiple automated scoring engines including equation scoring, essay scoring, short answer scoring, alert detection, and dialogue systems. She earned her Ph.D. from James Madison University in assessment and measurement (2006) and holds Masters’ degrees in Mathematics and in Computer Science from the University of Wisconsin-Madison (1997). 

    Contact Sue via susanlottridge@hotmail.com

    Amy Burkhardt

    University of Colorado - Boulder

    Amy Burkhardt is a PhD Candidate in Research and Evaluation Methodology with an emphasis in Human Language Technology at the University of Colorado, Boulder. She has been involved in the development of two automated scoring systems. Ongoing research projects include the automatic detection of students reporting harm within online tests, the use of machine learning to explore public discourse around educational policies, and considerations in psychometric modeling when making diagnostic inferences aligned to a learning progression. 

    Contact Amy via amy.burkhardt@colorado.edu

    Michelle Boyer

    Center for Assessment

    Michelle Boyer, Ph.D. is a Senior Associate at The National Center for the Improvement of Educational Assessment, Inc. Dr. Boyer consults with states and organizations on such issues as assessment systems, validity of score interpretations, scoring design and evaluation criteria for both human and automated scoring, assessment literacy, and score comparability. She is also a regular contributor to professional publications and the annual conferences of AERA, NCME, and CCSSO. Her most recent research focuses on evaluating the quality of automated scoring and its impact test score scales and test equating solutions. Dr. Boyer earned her Ph.D. from the University of Massachusetts, Amherst in Research, Educational Measurement, and Psychometrics (2018). 

    Contact Michelle via mboyer@nciea.org

  • Digital Module 20: Classroom Assessment Standards

    Contains 7 Component(s) Recorded On: 11/30/2020

    In this digital ITEMS module, Dr. Caroline Wylie reviews the Classroom Assessment Standards with their three sets of standards: (1) Foundations (these six standards provide the basis for developing and implementing sound and fair classroom assessment); (2) Use (these five standards follow a logical progression from the selection and development of classroom assessments to the communication of the assessment results); and (3) Quality (these five standards guide teachers in providing accurate, reliable, and fair classroom assessment results for all students). Keywords: assessment design, classroom assessment, formative assessment, professional development

    In this digital ITEMS module, Dr. Caroline Wylie reviews the Classroom Assessment Standards developed under the auspices of the Joint Committee on Standards for Educational Evaluation and briefly contrasts their role with the Standards for Educational and Psychological Testing (2014) issued by APA, AERA, and NCME, which are commonly used to inform quality metrics for high stakes and large-scale assessments. She includes details on the three categories of standards: (1) Foundations (these six standards provide the basis for developing and implementing sound and fair classroom assessment); (2) Use (these five standards follow a logical progression from the selection and development of classroom assessments to the communication of the assessment results); and (3) Quality (these five standards guide teachers in providing accurate, reliable, and fair classroom assessment results for all students). The module contains audio-narrated slides, reflection questions, and a set of resources to support application of the Classroom Assessment Standards to classroom assessments within teams of teachers and/or curriculum and assessment developers.

    Keywords: assessment design, classroom assessment, formative assessment, professional development

    Caroline Wylie

    Educational Testing Service (ETS)

    Caroline Wylie is a Research Director in the Student and Teacher Research Center and Senior Research Scientist at ETS. Her current research centers on issues around balanced assessment systems, with a focus on the use of formative assessment to improve classroom teaching and learning. She has led studies related to the creation of effective, scalable and sustainable teacher professional development, focused on formative assessment, on the formative use of diagnostic questions for classroom-based assessment, assessment literacy and on the role of learning progressions to support formative assessment in mathematics and science. She is specifically interested in issues of rater quality as it relates to formative classroom observations, and the relationship between observations, feedback and changes to practice. She was one of the co-authors of the Classroom Assessment Standards, and currently serves as the co-chair for the NCME Classroom Assessment Task Force.


    Contact Caroline via ecwylie@ets.org