Pre-Conference Workshops, 1 December
Pre-Conference Workshop 1, 9:30 - 12:30 1 December 2026, £70 registration
Basic Item Response Theory in Practice: From Item Statistics to Better Tests
Speaker: Carlos Collares
Basic Item Response Theory in Practice: From Item Statistics to Better Tests
Most assessment programmes still rely on classical test theory. It is useful, but its item statistics depend on the particular group of students who sat the test, and a single reliability coefficient summarises precision for the whole score range. Item Response Theory (IRT) addresses both limitations. It places items and students on a common scale, describes how well each item discriminates at different ability levels, and shows where a test measures precisely and where it does not. This makes IRT especially valuable for progress testing, item banking and standard setting, but many assessment professionals find it hard to move from the theory to routine use.
This workshop offers a practical, non-mathematical introduction to IRT for people who work with written assessments in medical education. Starting from familiar classical statistics, we introduce the Rasch/one-parameter and two-parameter logistic models, item characteristic curves, and item and test information functions. Throughout, we emphasise interpretation: what the output means and what decisions it supports.
Format: A [180-minute] interactive workshop that alternates short conceptual introductions with hands-on analysis of an anonymised dataset from a medical knowledge test, using [free software, e.g. R with the mirt package / jamovi]. Participants work in pairs and discuss how the results apply to their own assessment programmes. Participants should bring a laptop; no prior experience with IRT or programming is required.
Learning outcomes. By the end of the workshop, participants will be able to:
- explain the main differences
- between classical test theory and IRT, and when each is most useful;
- interpret item difficulty and
- discrimination parameters, item characteristic curves and test information
- functions;
- use IRT output to identify
- problematic items and to judge where on the ability scale a test is most
- and least precise;
- identify concrete opportunities
- to apply IRT in their own assessment practice, such as progress testing
- and item banking.
Pre-Conference Workshop 2, 14:00 - 17:00 1 December 2026, £70 registration
Generalizability Theory for Practical OSCE Analysis: How Many Stations and Examiners Do We Really Need?
Speaker: Carlos Collares
Generalizability Theory for Practical OSCE Analysis: How Many Stations and Examiners Do We Really Need?
OSCE scores are influenced by much more than student competence. Station content, examiner
stringency, simulated patients and parallel circuits all add measurement error. A single reliability coefficient such as Cronbach's alpha cannot separate these sources or show how to reduce them. Generalizability Theory (G-theory) can. It estimates how much each source contributes to score variance. Through decision (D) studies, it also shows how reliability would change if the number of
stations, examiners or circuits were changed. This makes G-theory one of the most useful tools for designing and defending OSCEs, yet it remains underused in practice.
This workshop introduces G-theory through realistic OSCE designs. We cover facets and variance components, crossed and nested designs (for example, examiners nested within stations, and students distributed across parallel circuits), generalizability and dependability coefficients for relative and absolute decisions, and D-studies to optimise OSCE design within practical and budgetary constraints. We also discuss common pitfalls, including unbalanced data and designs that do not match how the OSCE was actually run.
Format: A 180-minute interactive workshop that combines short conceptual segments with hands-on analysis of an anonymised OSCE dataset, using gtheory-app, a free, browser-based Shiny application for G-theory analysis (carloscollares.shinyapps.io/gtheory-app/). The workshop has three parts. The first introduces facets, variance components and OSCE designs. In the second, participants set up and run G-studies for crossed and nested designs and interpret the results. In the third, they run D-studies to compare alternative OSCE configurations and discuss how the findings apply to their own programmes. Participants should bring a laptop; no installation or programming experience is required, and they can continue using the application with their own data after the workshop.
Learning outcomes. By the end of the workshop, participants will be able to:
- explain why G-theory offers more
- insight into OSCE reliability than classical coefficients;
- identify the facets and the
- design (crossed or nested) of their own OSCEs;
- interpret variance components
- and generalizability and dependability coefficients;
- use D-studies to make
- evidence-based decisions about the number of stations, examiners and
- circuits needed for defensible pass/fail decisions.
Pre-Conference Workshop 3 TBA