In this digital ITEMS module, Drs. Richard Feinberg, Carol Morrison, and Mark R. Raymond discuss an overview of how credentialing testing programs operate and special considerations that need to be made when unusual things occur.  Formal graduate education in a measurement related field provides a solid foundation for professionals who work on credentialing examinations. Those foundational skills are then expanded and refined over time as practitioners encounter complex and nuanced challenges that were not covered by or go beyond the context described in textbooks. For instance, as most of us who work on operational testing programs are (sometimes) painfully aware, real data can be very messy. Often unanticipated situations arise that can create a range of problems from threats to score validity, to unexpected financial costs, and even longer-term reputational damage. In practice, solutions for these unanticipated situations are not always straightforward, often requiring a compromise between psychometric best practices, business resources, and needs of the customer. In this module we discuss some of these unusual challenges that usually occur in a credentialing program. First, we provide a high-level summary of the main components of the assessment lifecycle and the different roles within a testing organization. Next, we propose a framework for qualifying risk along with various considerations and potential actions for managing these challenges. Lastly, we integrate this information by presenting a few scenarios that can occur in practice that should help learners think though applicable team-based problem-solving and better align recommended action from a psychometric perspective given the context and magnitude of the challenge.

Keywords:Credential/Licensure Testing, Assessment Design, Assessment Challenges, Threats to Score Validity, Operational Psychometrics