All-access Pass

5 (7 votes)

This provides immediate access to ALL print and digital modules in the portal by "registering" you for each and displaying all modules as a single collection as part of this pass.

 

  • Digital Module 13: Simulation Studies in IRT

    Contains 8 Component(s) Recorded On: 04/03/2020

    In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Key words: bias, bi-factor model, estimation, graded response model, item response theory, mean squared error, Monte Carlo, simulation, standard error, two-parameter logistic model

    In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Dr. Leventhal and Dr. Ames review the conceptual foundation of MCSS in IRT and walk through the processes of simulating total scores as well as item responses using the two-parameter logistic, graded response, and bi-factor models. They provide guidance for how to implement MCSS using other item response models and best practices for efficient syntax and executing an MCSS. The digital module contains sample SAS code, diagnostic quiz questions, activities, curated resources, and a glossary.

    Key words: bias, bi-factor model, estimation, graded response model, item response theory, mean squared error, Monte Carlo, simulation, standard error, two-parameter logistic model

    Brian Leventhal

    Assistant Professor

    Brian is an assistant professor in the Assessment and Measurement PhD program in the Department of Graduate Psychology at James Madison University as well as an assistant assessment specialist in the Center for Assessment and Research Studies at James Madison University. There, he teaches courses in quantitative methods, including a course on Simulation Studies in Psychometrics. Brian received his Ph.D. from the University of Pittsburgh. His research interests include multidimensional item response models that account for response styles, response process models, and classification errors in testing. Brian is passionate about teaching and providing professional development for graduate students and early-career practitioners. He has thoroughly enjoyed collaborating with Allison Ames and the Instructional Design Team to develop this module. 

    Contact Brian via leventbc@jmu.edu

    Allison J. Ames

    Assistant Professor

    Allison is an assistant professor in the Educational Statistics and Research Methods program in the Department of Rehabilitation, Human Resources and Communication Disorders, Research Methodology, and Counseling at the University of Arkansas. There, she teaches courses in educational statistics, including a course on Bayesian inference. Allison received her Ph.D. from the University of North Carolina at Greensboro. Her research interests include Bayesian item response theory, with an emphasis on prior specification; model-data fit; and models for response processes. Her research has been published in prominent peer-reviewed journals. She enjoyed collaborating on this project with a graduate student, senior faculty member, and the Instructional Design Team.
    Contact Allison via boykin@uark.edu

  • Digital Module 18: Automated Scoring

    Product not yet rated Contains 8 Component(s) Recorded On: 08/26/2020

    In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. They discuss automated scoring from a number of perspectives and provide two data examples, one focused on training and evaluating an automated scoring engine and one focused on the impact of rater error on predicted scores. Key words: automated scoring, hand-scoring, machine learning, natural language processes, constructed response items

    In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R-Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.

    Key words: automated scoring, hand-scoring, machine learning, natural language processes, constructed response items

    Susan Lottridge

    Cambium Assessment

    Sue Lottridge, Ph.D. is a Senior Director of Automated Scoring at the Cambium Assessment, Inc. (CAI).  In this role, she leads CAI’s machine learning and scoring team on the research, development, and operation of CAI’s automated scoring software. This software includes automated essay scoring, short answer scoring, automated speech scoring, and an engine that detects disturbing content in student responses. Dr. Lottridge has worked in automated scoring for twelve years and has contributed to the design, research, and use of multiple automated scoring engines including equation scoring, essay scoring, short answer scoring, alert detection, and dialogue systems. She earned her Ph.D. from James Madison University in assessment and measurement (2006) and holds Masters’ degrees in Mathematics and in Computer Science from the University of Wisconsin-Madison (1997). 

    Contact Sue via susanlottridge@hotmail.com

    Amy Burkhardt

    University of Colorado - Boulder

    Amy Burkhardt is a PhD Candidate in Research and Evaluation Methodology with an emphasis in Human Language Technology at the University of Colorado, Boulder. She has been involved in the development of two automated scoring systems. Ongoing research projects include the automatic detection of students reporting harm within online tests, the use of machine learning to explore public discourse around educational policies, and considerations in psychometric modeling when making diagnostic inferences aligned to a learning progression. 

    Contact Amy via amy.burkhardt@colorado.edu

    Michelle Boyer

    Center for Assessment

    Michelle Boyer, Ph.D. is a Senior Associate at The National Center for the Improvement of Educational Assessment, Inc. Dr. Boyer consults with states and organizations on such issues as assessment systems, validity of score interpretations, scoring design and evaluation criteria for both human and automated scoring, assessment literacy, and score comparability. She is also a regular contributor to professional publications and the annual conferences of AERA, NCME, and CCSSO. Her most recent research focuses on evaluating the quality of automated scoring and its impact test score scales and test equating solutions. Dr. Boyer earned her Ph.D. from the University of Massachusetts, Amherst in Research, Educational Measurement, and Psychometrics (2018). 

    Contact Michelle via mboyer@nciea.org

  • Digital Module 16: Longitudinal Data Analysis

    Contains 9 Component(s) Recorded On: 08/01/2020

    In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically-defensible function that adequately summarizes both individual change as well as the average response. Keywords: fixed effect, linear mixed effects models, longitudinal data analysis, multilevel models, population-average, random effect, regression, subject-specific, trajectory

    In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically-defensible function that adequately summarizes both individual change as well as the average response. The module begins with a non-technical overview of longitudinal data analyses drawing distinctions with cross-sectional analyses in terms of research questions to be addressed. Nuances of longitudinal designs, timing of measurements, and the real possibility of missing data are then discussed. The three interconnected components of the LME model: (1) a model for individual and mean response profiles, (2) a model to characterize the covariation among the time-specific residuals, and (3) a set of models that summarize the extent that individual coefficients vary, are discussed in the context of the set of activities comprising an analysis. Finally, they demonstrate how to estimate the linear mixed effects model within an open-source environment (R). The digital module contains sample R code, diagnostic quiz questions, hands-on activities in R, curated resources, and a glossary.

     Keywords: fixed effect, linear mixed effects models, longitudinal data analysis, multilevel models, population-average, random effect, regression, subject-specific, trajectory

    Jeffrey R. Harring

    Professor

    Jeff is a Professor in the Measurement, Statistics and Evaluation Program within the Department of Human Development and Quantitative Methodology at the University of Maryland, College Park. There, he teaches introductory and intermediate graduate level statistics courses and advanced quantitative methods seminars in longitudinal data analysis, mixture modeling, simulation design and statistical computing. Jeff has taught several multi-day workshops on the application of longitudinal methods using R and SAS statistical software most recently at the National Center of Educational Statistics (NCES) in Washington D.C.  Prior to joining the program faculty in the fall of 2006, Jeff received an M.S. degree in Statistics and completed his Ph.D. in the Quantitative Methods in Education from the University of Minnesota. Before that, Jeff taught high school mathematics for 12 years. He has published nearly 100 articles and book chapters, co-edited three volumes and co-authored a book. His research focuses on linear and nonlinear models for repeated measures data, structural equation models, finite mixtures of both linear and nonlinear growth models and extensions of these methods to multilevel data structures.

    Contact Jeff via harring@umd.edu

    Tessa Johnson

    Doctoral Candidate

    Tessa is a Ph.D. candidate in the Measurement, Statistics and Evaluation Program within the Department of Human Development and Quantitative Methodology at the University of Maryland, College Park. She received her Master of Science in Educational Research from Georgia State University. Tessa currently serves as a project coordinator for the Synthetic Data Project (SDP) of the Maryland Longitudinal Data System Center, an Institute of Education Sciences (IES) funded project aimed at assessing the feasibility of and implementing a system for synthesizing statewide longitudinal data in order to increase data access for researchers and policy analysts while minimizing risk of data disclosure. For the SDP, Tessa conducts research on the feasibility of reproducing nested data structures in longitudinal synthetization models. Outside her work with the SDP, Tessa’s research has centered around creating and improving statistical methods for analyzing complex data structures in longitudinal contexts. This includes modeling time as an outcome in latent growth models, accounting for similarities among schools when modeling student mobility in longitudinal studies, and exploring the development of ensembles of social networks in the classroom over time.

    Contact Tessa via johnsont@umd.edu

  • Digital Module 15: Accessibility of Educational Assessments

    Contains 7 Component(s) Recorded On: 06/01/2020

    In this digital ITEMS module, Dr. Ketterlin Geller and her colleagues provide an introduction to accessibility of educational assessments. They discuss the legal basis for accessibility in K-12 and higher education organizations and describe how test and item design features as well as examinee characteristics affect the role that accessibility plays in evaluating test validity during test development operational deployment. Keywords: Accessibility, accommodations, examinee characteristics, fairness, higher education, K-12 education, item design, legal guidelines, test development, universal design

    In this digital ITEMS module, Dr. Ketterlin Geller and her colleagues provide an introduction to accessibility of educational assessments. They discuss the legal basis for accessibility in K-12 and higher education organizations and describe two key factors that interact to influence accessibility of educational assessments: (1) test and item design features and (2) examinee characteristics. They highlight the outcome of this interaction in various situated examples and discuss the role accessibility plays in evaluating test validity as well as the principles of universal design during test development and test accommodations during operational deployment. The module includes an interview with Dr. Martha Thurlow, an expert in accessibility, who provides an important perspective on the past, present, and future of accessibility for educational assessments. The module is designed to be relevant for students, test developers, and users of educational tests in K-12 and higher education settings. It contains audio-narrated slides, interactive activities, and quizzes as well as curated resources and a glossary.

    Keywords: Accessibility, accommodations, examinee characteristics, fairness, higher education, K-12 education, item design, legal guidelines, test development, universal design

    Leanne Ketterlin Geller

    Professor, Southern Methodist University

    Leanne is a professor at Southern Methodist University, specializing in applied measurement and assessment. She holds the Texas Instruments Endowed Chair in Education and directs the center on Research in Mathematics Education. Her scholarship focuses on supporting all students in mathematics education through application of instructional leadership principles and practices. She has served as Principal Investigator for federally and state funded research grants to develop and implement formative assessment procedures and valid decision-making systems for students with diverse needs in the general education curriculum. She has published numerous articles and book chapters, and presented original research findings at local, national, and international conferences, and serves on the editorial boards for the International Journal of Testing and Assessment for Effective Intervention. She works closely with teachers and administrators to support their application of measurement and assessment principles in school-based decision making. Dr. Ketterlin-Geller was a high school science teacher and trained as a K-12 administrator.

    Contact Leanne at lkgeller@mail.smu.edu

    Brooke Istas

    Doctoral Student, Southern Methodist University

    Brooke is a Graduate Research Assistant and doctoral student at the Simmons School of Education and Human Development at Southern Methodist University. Her research interests center on adult learners perceptions of mathematics. Brooke is also a consultant, an adult education mathematics subject matter expert, and Cowley College Mathematics Faculty. She is recognized nationally for her knowledge of mathematics and mathematical instructional strategies. She has given several presentations at state and national conferences on enhancing mathematical instruction, understanding higher level mathematical content, teaching math at a distance, and on-line instruction; always willing to share her learning with others. She is also the Subject Matter Expert (SME) for the LINCS (Literacy Information and Communication System) Math and Numeracy and Science Community of Practice; also a reviewer for the Math/Numeracy and Science online resource collection that is a part of the Basic Skills Collection.

    Contact Brooke at XXX

    Robyn K. Pinilla

    Doctoral Student, Southern Methodist University

    Robyn is a Ph.D. student and Graduate Research Assistant in Research in Mathematics Education at the Simmons School of Education and Human Development at Southern Methodist University. Her specific research interests are in early childhood spatial reasoning and problem solving, and equity and access within assessment for all students. She served as an elementary school assistant principal and early childhood Special Education teacher prior to beginning her Ph.D. studies. In this work, she developed an assets-based approach to first celebrate student and teacher success before triangulating their current development with next steps and targeted objectives. This informal application of the curriculum, instruction, and assessment framework prepared her to begin her doctoral studies under the advisement of Dr. Leanne Ketterlin-Geller. She continues advocating for authentic and practical instruction and assessment that is accessible to all through research and local, state, and national collaborations.

    Contact Robyn at XXX

    Ann Marie Wernick

    Doctoral Student, Southern Methodist University

    Ann Marie is a Graduate Research Assistant and doctoral student at the Simmons School of Education and Human Development at Southern Methodist University. Her research interests focus on teacher induction, coaching models, practice-based teacher education, teacher evaluation systems, and mixed-reality simulations. Before her time at SMU, Ann Marie earned her M.Ed. from the University of Notre Dame, where she served as an ACE Fellow. She then spent the next seven years as a classroom teacher and instructional coach. During that time, she taught middle school English Language Arts and History in public, private, and parochial schools both in the United States and abroad. In each of these settings, she gained valuable experience mentoring and coaching novice teachers, which fostered her interest in teacher induction and pre-service teacher coaching in both alternative certification and traditional teacher preparation programs.

    Contact Ann Marie at XXX

  • Digital Module 14: Planning and Conducting Standard Setting

    Contains 7 Component(s) Recorded On: 05/30/2020

    In this digital ITEMS module, Dr. Michael B. Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, body of work) but on the practical tasks that must be completed for any standard setting activity. Keywords: achievement level descriptor, certification and licensure, cut score, feedback, interquartile range, performance level descriptor, score reporting, standard setting, panelist, vertical articulation

    In this digital ITEMS module, Dr. Michael B. Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, body of work) but on the practical tasks that must be completed for any standard setting activity. Dr. Bunch carries the participant through every stage of the standard setting process, from developing a plan, through preparations for standard setting, conducting standard setting, and all the follow-up activities that must occur after standard setting in order to obtain approval of cut scores and translate those cut scores into score reports. The digital module includes a 120-page manual, various ancillary files (e.g., PowerPoint slides, Excel workbooks, sample documents and forms), links to datasets from the book Standard Setting (Cizek & Bunch, 2007), links to final reports from four recent large-scale standard setting events, quiz questions with formative feedback, and a glossary.

    Keywords: achievement level descriptor, certification and licensure, cut score, feedback, interquartile range, performance level descriptor, score reporting, standard setting, panelist, vertical articulation 

    Michael B. Bunch

    Senior Vice President, Measurement Incorporated

    Mike joined Measurement Incorporated (MI) as director of test development in 1982. Prior to that time, he was a research psychologist at ACT (1976-78) and senior professional at NTS Research (1978-82). At MI he was soon promoted to vice president for research and development and later to senior vice president. He currently serves on MI’s board of directors and continues to direct MI’s psychometric team. While still employed by MI, Mike taught graduate courses in statistics and research methods at North Carolina Central University for six years and a mini-course on standard setting through the University of Maryland. He has presented several half-day and full-day training sessions on standard setting at NCME and other national and international organizations. In addition, he co-authored of Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests with Gregory J. Cizek (2007) as well as various other articles and book chapters on standard setting. He has planned and carried out some of the largest and most complex standard setting events ever conceived.

    Contact Mike via MBunch@measinc.com 

  • Digital Module 12: Think-aloud Interviews and Cognitive Labs

    Contains 6 Component(s)

    ​In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes and illustrate both traditional and modern data-collection methods. Keywords: ABC tool, cognitive laboratory, cog lab, cognition, cognitive model, interrater agreement, kappa, probe, rubric, thematic analysis, think-aloud interview, verbal report

    In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze distinct types of verbal reports in the collection of evidence of test-taker response processes. The module includes details on (a) the different types of cognition that are tapped by different interviewer probes, (b) traditional interviewing methods and new automated tools for collecting verbal reports, and (c) options for analyses of verbal reports. This includes a discussion of reliability and validity issues such as potential bias in the collection of verbal reports, ways to mitigate bias, and inter-rater agreement to enhance credibility of analysis. A novel digital tool for data-collection called the ABC tool is presented via illustrative videos. As always, the module contains audio-narrated slides, quiz questions with feedback, a glossary, and curated resources. 

    Keywords: ABC tool, cognitive laboratory, cog lab, cognitive model, interrater agreement, kappa, probe, rubric, thematic analysis, think-aloud interview, verbal report

    Jacqueline P. Leighton

    Professor

    Jackie is a Registered Psychologist and Professor of School and Clinical Child Psychology at the University of Alberta. She completed her graduate degrees in Psychology at the University of Alberta and postdoctoral fellowship studies in Psychology at Yale University. Her research and teaching is driven by the overarching goal to enhance fairness in testing. In pursuit of this goal, she has increasingly focused her research on investigating the interplay between cognitive and emotional processes underlying learning and academic achievement. For example, she investigates variables that can cognitively or emotionally bias participants’ response processes in testing situations, leading to misrepresentations in performance and weaknesses in validity arguments of test inferences. Overall, she is interested in methods designed to enhance diagnosis of achievement, validation of test inferences, and theoretical understanding of human learning. 

    Contact Jackie via jacqueline.leighton@ualberta.ca

    Blair Lehman

    Research Scientist

    Blair is a research scientist at Educational Testing Service. She completed her graduate degrees in Cognitive Psychology and certificate in Cognitive Science at the University of Memphis. Her research focuses on understanding students’ emotional and motivational processes during educational activities to design activities that maximize the experience for all students. Her research has focused on the task design of learning and assessment activities as well as the design of adaptive systems that consider both student cognition and motivation. For example, she has explored specific design features of game-based assessments in an effort to understand how to maintain measurement validity while also maximizing student motivation. 

    Contact Blair via blehman@ets.org

  • Digital Module 17: Data Visualizations

    Product not yet rated Contains 9 Component(s) Recorded On: 08/20/2020

    In this digital module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. The instructors review key literature, discuss strategies for enhancing graphical presentation, and provide an introduction to the Graph Template Language (GTL) in SAS to illustrate how elementary components can be used to make efficient, effective and accurate graphics for a variety of audiences. Key words: data visualization, graphical excellence, graphical template language, SAS

    In this digital module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. Data visualizations are commonly used by measurement professionals to communicate results to examinees, the public, educators, and other stakeholders. To create effective visualizations, it is important that they communicate data effectively, efficiently, and accurately. Unfortunately, measurement and statistical software default graphics typically fail to uphold these standards and are therefore not necessarily suitable for publication or presentation to the public. The instructors review key literature, discuss strategies for enhancing graphical presentation, and provide an introduction to the Graph Template Language (GTL) in SAS to illustrate how elementary components can be used to make efficient, effective and accurate graphics for a variety of audiences. The module contains audio-narrated slides, embedded illustrative videos, quiz questions with diagnostic feedback, a glossary, sample SAS code, and other learning resources.

    Key words: data visualization, graphical excellence, graphical template language, SAS

    Nikole Gregg

    Doctoral Student

    Nikole Gregg is a doctoral student in the Assessment & Measurement program at James Madison University, where she has taken on many roles, including various assessment and measurement consulting experiences at JMU and in K-12 settings. Through her work, she has refined and developed skills necessary to present sophisticated data analyses to non-technical audiences. Her research interests include the application of multidimensional item response theory to account for response styles, fairness in testing, and validity theory. Nikole is passionate about improving fairness and equity within assessment, measurement, and policy.

    Contact Nikole via greggnl@dukes.jmu.edu

    Brian Leventhal

    Assistant Professor

    Brian is an assistant professor in the Assessment and Measurement PhD program in the Department of Graduate Psychology at James Madison University as well as an assistant assessment specialist in the Center for Assessment and Research Studies at James Madison University. There, he teaches courses in quantitative methods, including a course on Simulation Studies in Psychometrics. Brian received his Ph.D. from the University of Pittsburgh. His research interests include multidimensional item response models that account for response styles, response process models, and classification errors in testing. Brian is passionate about teaching and providing professional development for graduate students and early-career practitioners. He has thoroughly enjoyed collaborating with Allison Ames and the Instructional Design Team to develop this module. 

    Contact Brian via leventbc@jmu.edu

  • Digital Module 11: Bayesian Psychometrics

    Contains 6 Component(s) Recorded On: 01/31/2020

    In this digital ITEMS module, Dr. Roy Levy discusses how Bayesian inference is a mechanism for reasoning in probability-modeling framework, describes how this plays out in a normal distribution model and unidimensional item response theory (IRT) models, and illustrates these steps using the JAGS software and R. Keywords: Bayesian psychometrics, Bayes theorem, dichotomous data, item response theory (IRT), JAGS, Markov-chain Monte Carlo (MCMC) estimation, normal distribution, R, unidimensional models

    In this digital ITEMS module, Dr. Roy Levy describes Bayesian approaches to psychometric modeling. He discusses how Bayesian inference is a mechanism for reasoning in probability-modeling framework and is well-suited to core problems in educational measurement: reasoning from student performances on an assessment to make inferences about their capabilities more broadly conceived as well as fitting models to characterize the psychometric properties of tasks. The approach is first developed in the general context of estimating a mean and variance of a normal distribution before turning to the context of unidimensional item response theory (IRT) models for dichotomously scored data. Dr. Levy illustrates the process of fitting Bayesian models using the JAGS software facilitated through the R statistical environment. The module is designed to be relevant for students, researchers, and data scientists in various disciplines such as education, psychology, sociology, political science, business, health and other social sciences. It contains audio-narrated slides, diagnostic quiz questions, and data-based activities with video solutions as well as curated resources and a glossary.

    Keywords: Bayesian psychometrics, Bayes theorem, dichotomous data, item response theory (IRT), JAGS, Markov-chain Monte Carlo (MCMC) estimation, normal distribution, R, unidimensional models

    Roy Levy

    Professor, Arizona State University

    Roy is a professor in the T. Denny Sanford School of Social & Family Dynamics at Arizona State University, specializing in Measurement and Statistical Analysis. He received his Ph.D. in Measurement, Statistics & Evaluation from the University of Maryland. His research and teaching interests include methodological investigations and applications in psychometrics and statistical modeling, focusing on item response theory, structural equation modeling, Bayesian networks, and Bayesian approaches to inference and modeling, as well as evidentiary principles and applications in complex assessments. He is the co-author of the book Bayesian Psychometric Modeling, and has published his work in a variety of leading methodological journals. For his work, he has received awards from the National Council on Measurement in Education, the American Educational Research Association, and the President of the United States. He currently serves on the editorial boards for Structural Equation Modeling: A Multidisciplinary Journal, Educational Measurement: Issues and Practice, Measurement: Interdisciplinary Research and Perspectives, and Educational Assessment.

    Contact Roy via email at roy.levy@asu.edu

  • Digital Module 10: Rasch Measurement Theory

    Contains 2 Component(s) Recorded On: 12/03/2019

    In this digital ITEMS module, Dr. Jue Wang and Dr. George Engelhard Jr. describe the Rasch measurement framework for the construction and evaluation of new measures and scales and demonstrate the estimation of core models with the Shiny_ERMA and Winsteps programs. Keywords: invariance, item fit, item response theory, IRT, person fit, model fit, multi-faceted Rasch model, objective measurement, R, Rasch measurement, Shiny_ERMA, Winsteps

    In this digital ITEMS module, Dr. Jue Wang and Dr. George Engelhard Jr. describe the Rasch measurement framework for the construction and evaluation of new measures and scales. From a theoretical perspective, they discuss the historical and philosophical perspectives on measurement with a focus on Rasch’s concept of specific objectivity and invariant measurement. Specifically, they introduce the origins of Rasch measurement theory, the development of model-data fit indices, as well as commonly used Rasch measurement models. From an applied perspective, they discuss best practices in constructing, estimating, evaluating, and interpreting a Rasch scale using empirical examples. They provide an overview of a specialized Rasch software program (Winsteps) and an R program embedded within Shiny (Shiny_ERMA) for conducting the Rasch model analyses. The module is designed to be relevant for students, researchers, and data scientists in various disciplines such as psychology, sociology, education, business, health and other social sciences. It contains audio-narrated slides, sample data, syntax files, access to Shiny_ERMA program, diagnostic quiz questions, data-based activities, curated resources, and a glossary.

    Keywords: invariance, item fit, item response theory, IRT, person fit, model fit, multi-faceted Rasch model, objective measurement, R, Rasch measurement, Shiny_ERMA, Winsteps

    Jue Wang

    Assistant Professor

    Jue is an assistant professor in Research, Measurement & Evaluation Program at the University of Miami. She received her Ph.D. from the University of Georgia in Quantitative Methodology under the Department of Educational Psychology. Her research focuses on examining rating quality and rater effects in rater-mediated assessments using a variety of measurement models including Rasch models, unfolding models, and multilevel item response models. She has published in leading journals related to measurement including Educational and Psychological Measurement, Journal of Educational Measurement, Assessing Writing, and Measurement: Interdisciplinary Research and Perspectives. She is currently completing a book with Professor George Engelhard entitled Rasch models for solving measurement problems: Invariant measurement in the social sciences that will be published by Sage as a part of their Quantitative Applications in the Social Sciences (QASS) series. 

    Contact Jue at jue.wang@miami.edu

    George Engelhard Jr.

    Professor

    George joined the faculty at The University of Georgia in the fall of 2013.  He is professor emeritus at Emory University (1985 to 2013).  Professor Engelhard received his Ph.D. in 1985 from The University of Chicago (MESA Program--measurement, evaluation, and statistical analysis).  Professor Engelhard is the author of two books:  Invariant measurement with raters and rating scales: Rasch models for rater-mediated assessments (2018 with Dr. Stefanie A. Wind) and Invariant measurement: Using Rasch models in the social, behavioral, and health sciences (2013). He is the co-editor of five books, and he has authored or co-authored over 200 journal articles, book chapters, and monographs. He serves on several national technical advisory committees on educational measurement and policy in several states in the United States.  In 2015, he received the first Qiyas Award for Excellence in International Educational Assessment recognizing his contributions to the improvement of educational measurement at the local, national and international levels. Professor Engelhard is currently a co-editor of the Journal of Educational Measurement. He is a fellow of the American Educational Research Association.

    Contact George at gengelh@uga.edu

  • Digital Module 09: Sociocognitive Assessment for Diverse Populations

    Contains 2 Component(s)

    In this digital ITEMS module, Dr. Robert Mislevy and Dr. Maria Elena Oliveri introduce and illustrate a sociocognitive perspective on educational measurement, which focuses on a variety of design and implementation considerations for creating fair and valid assessments for learners from diverse populations with diverse sociocultural experiences. Keywords: assessment design, Bayesian statistics, cross-cultural assessment, diverse populations, educational measurement, evidence-centered design, fairness, international assessments, prototype, reliability, sociocognitive assessment, validity

    In this digital ITEMS module, Dr. Robert [Bob] Mislevy and Dr. Maria Elena Oliveri introduce and illustrate a sociocognitive perspective on educational measurement, which focuses on a variety of design and implementation considerations for creating fair and valid assessments for learners from diverse populations with diverse sociocultural experiences. The first part of the module, narrated by Dr. Mislevy, contains a general overview section, a description of the sociocognitive framing of assessment issues, and a section on implications for assessment around key concepts such as reliability, validity, and fairness. The second part of the module, narrated by Dr. Oliveri, contains a section on frameworks for fairness investigations and principled assessment design as well as brief vignette-based illustrations of the principles using a prototype activity to support collaboration and communication skills in the workplace. The module is designed to provide a relatively high-level, conceptual, and non-statistical overview and is intended for interdisciplinary team members who need to create fair and equitable learning and assessment systems for diverse populations.

    Keywords: assessment design, Bayesian statistics, cross-cultural assessment, diverse populations, educational measurement, evidence-centered design, fairness, international assessments, prototype, reliability, sociocognitive assessment, validity 

    Bob Mislevy

    Lord Chair in Measurement and Statistics

    Dr. Robert [Bob] Mislevy is the Frederic M. Lord Chair in Measurement and Statistics at Educational Testing Service as well as Professor Emeritus of Measurement, Statistics, and Evaluation at the University of Maryland, with affiliations with Second Language Acquisition and Survey Methods. Dr. Mislevys research applies developments in statistics, technology, and cognitive science to practical problems in educational assessment. His work includes a multiple-imputation approach to integrate sampling and psychometric models in the National Assessment of Educational Progress (NAEP), an evidence-centered framework for assessment design, and simulation- and game-based assessment with the Cisco Networking Academy. Among his many awards are AERA’s Raymond B. Cattell Early Career Award for Programmatic Research, NCME’s Triennial Award for Technical Contributions to Educational Measurement (3 times), NCME’s Award for Career Contributions, AERA’s E.F. Lindquist Award for contributions to educational assessment, the International Language Testing Association's Messick Lecture Award, and AERA Division D’s inaugural Robert L. Linn Distinguished Address Award.  He is a member of the National Academy of Education and a past president of the Psychometric Society. He has served on projects for the National Research Council, the Spencer Foundation, and the MacArthur Foundation concerning assessment, learning, and cognitive psychology, and on the Gordon Commission on the Future of Educational Assessment. His most recent book is "Sociocognitive Foundations of Educational Assessment" for which he received the 2019 NCME Annual Award and on which this ITEMS module is based.

    Contact Bob via rmislevy@ets.org

    Maria Elena Oliveri

    Research Scientist

    Dr. María Elena Oliveri is a Research Scientist in the Academic to Career research center at the Educational Testing Service (ETS). Her research focuses on fairness, validity, diversity, equity, and innovative assessment design and development of competency-based digital formative assessments of 21st century skills. She has actively disseminated her research in numerous published articles in journals such as Applied Measurement in Education and the International Journal of Testing; she has led various professional development workshops at national and international conferences such as AERA, NCME, and ITC; and she has presented at numerous national and international conferences. In earlier stages of her career, she was a literacy mentor to second-language teachers in the Vancouver School District as well as a teacher of second language learners and students with disabilities and she has hosted workshops for educators on innovative approaches to assessing culturally and linguistically diverse learners. She also was a lecturer at the University of British Columbia, Vancouver, Canada where she taught courses on assessment and developmental psychology to students pursuing Bachelor of Education degrees in French Immersion programs.

    Contact Malena via moliveri@ets.org