Time horizon

In agent evaluation, a time horizon describes the length of tasks at which an agent reaches a chosen success rate. Length is measured by the time a human needs, not by how long the AI runs.

For the 50% variant, we collect the agent's successful and failed attempts on tasks with different human times. We fit a curve of the chance of success and then read off the time at which it crosses 50%. This is a prediction for the distribution of tasks of a given length; it does not mean that each of them has the same chance of being completed. METR: definition and methodology.

A teaching example: if the fitted curve reaches 50% at two hours, then two hours is the horizon. The agent may finish a successful task much faster than that. Nor does the result by itself say how successful the agent will be on a task from another domain.

Reading off the horizon requires a meaningful relationship between human time and difficulty for AI. When the curve is flat, a small change in the chance of success can move its crossing with the threshold a long way. The article on converting the METR horizon shows this problem with an example you can work through and discusses a new fit to the METR data. Nguyen and Fithian, §2–3.

When comparing horizons, keep the information about the task set, the agent's environment, the attempt budget and the chosen success threshold.

I use AI-generated content as part of my daily learning process.