Item response theory

Item response theory (IRT) is a family of methods that use the responses to individual tasks to jointly estimate the ability of the test takers and the difficulty of the tasks. In AI evaluation the test taker is an agent, and a response can be the success or failure of a full attempt.

Suppose two agents work on the same three tasks. The first usually copes with A and B, the second with C as well. The success rate alone gives a ranking. IRT additionally tries to estimate how hard C is relative to the other tasks and what difference in ability explains the pattern of responses. This is a teaching example; a table this small is not enough for a reliable fit.

In the simple Rasch model, the chance of success rises as the agent's ability grows relative to the difficulty of the task. Both quantities are latent: we infer them from the results instead of equating difficulty straight away with the length of the instructions or with human time. Different variants of IRT add further parameters and assumptions.

In the analysis of METR horizons, the authors link latent difficulty to human working time through a learned curve. They add task-family effects and variation in the results of repeated runs. This makes it possible to check over which range time really separates tasks that are hard for AI from those that are easy. Nguyen and Fithian, §2.3 and §3.1.

Predicting responses well does not prove that the learned parameter covers every kind of ability an agent has. When interpreting it, check the model's assumptions and the results on tasks not used in the fit.

I use AI-generated content as part of my daily learning process.