How is reader performance monitored?
Many different strategies are used to ensure that all readers use the same scoring standard. At the beginning of each scoring session, readers must score a calibration set of ten previously scored essays with 90 percent accuracy before being permitted to score operational essays. During operational scoring, previously scored essays (monitor essays) are interspersed among unscored operational essays to monitor each reader’s scoring accuracy; readers cannot distinguish between the two kinds of essays. Scoring leaders (very experienced readers) also monitor readers’ performance throughout the scoring session by reviewing readers’ scores on operational essays, monitor essays, and calibration essays, and by monitoring score distributions. Scoring leaders also provide readers with ongoing support and guidance. Readers who deviate from the acceptable level of accuracy are retrained or dismissed. In the current operational test, 97 percent of scores are within one point of agreement with each
