A longer METR horizon does not mean proportionally harder tasks
A team wants to find out whether the next version of an agent can be trusted with longer programming tasks. So it compares benchmark results expressed as the time a human needed to do the work. It is easy to treat such a result as a hint about how much work can already be handed over to the agent.
That interpretation, however, requires checking whether longer human work really means a correspondingly harder task for AI. If tasks within some range are similarly difficult for the agent, a slight improvement in its success rate can move the result across a wide span of minutes.
When choosing an agent, we then need one more piece of information: how the chance of completing the same tasks has changed. The time multiplier alone can overstate the difference between versions if we happen to be comparing a part of the benchmark where time separates difficulty poorly.
METR calls this kind of measurement a time horizon. In the variant with a 50% threshold, it is the human working time at which the fitted curve predicts that half of the agent's attempts succeed. This is an average for tasks of a given length, not a guarantee for each of them. The running time of the AI itself is not what is measured here.
In the preprint On the estimation and validity of AI time horizons—a statistical look at the METR plot, Drew T. Nguyen and William Fithian analyze precisely this conversion of time into difficulty. The paper, from the field of AI evaluation and statistics and first published on 8 October 2026, identifies an almost flat stretch in the METR data between 2 and 30 minutes of human work. Tasks in this range were, on average, of similar difficulty for the agents studied.
A small change in the chances can move the threshold a long way
The example below is our own teaching simplification, not a result of the study. We have three tasks, A, B and C, which take a human 3, 10 and 30 minutes respectively. Each of two agents attempts each task 100 times in the same environment. We record success or failure.
| Task | Human time | Successes of the older agent | Successes of the newer agent |
|---|---|---|---|
| A | 3 minutes | 50 of 100 | 52 of 100 |
| B | 10 minutes | 49 of 100 | 51 of 100 |
| C | 30 minutes | 48 of 100 | 50 of 100 |
For clarity, let us treat these frequencies as exactly known chances and join the points with decreasing curves. The older agent crosses 50% at 3 minutes, the newer one at 30. The horizon grows tenfold, although the success rate on each task rose by only 2 percentage points. Over 300 attempts per agent, the number of successes changed from 147 to 153.
All three tasks and all the failures stay in the measurement. We only choose the place where the curve crosses the threshold; we filter nothing and reward nothing. The cost of this notional test is 600 full agent attempts. In a real experiment, sample sizes like these do not allow a difference of 2 percentage points to be treated as certain. The curve has to be fitted and the uncertainty estimated.
The example shows the operation of reading a horizon off an almost flat curve. In a study, the shape of the curve first has to be established from many tasks and systems. If the curve falls much faster in another range, the same horizon multiplier requires a larger increase in the chance of success. That is why the steps from 3 to 30 minutes and from 30 minutes to 5 hours are not, by definition, equal portions of progress.
Human time has to be fitted to difficulty for AI
The starting point is a table of results: which task a given agent attempted, how many times and with what outcome. We also know the human time and the task's family. What we do not know is a single objective number describing its difficulty for AI.
The authors' baseline assumes that each successive doubling of human time raises difficulty by the same step. It uses a common slope of the curves for all agents; they differ in their level of ability. This is a stronger reference point than the separate regression for each system that was used originally.
The first proposal replaces the constant step with a learned, monotonic spline curve: smoothly joined segments that can rise faster or slower. A longer time still does not lower the predicted difficulty, but the increase need not be uniform.
The second proposal, called Model 2 in the paper, uses item response theory, IRT. From the pattern of successes and failures it jointly estimates the agents' abilities and the tasks' difficulty. It also accounts for the similarity of tasks belonging to one family and for additional variation in the results of attempts. As a result, many repetitions of the same agent–task pair are not treated as equally many new, independent pieces of information about the whole benchmark.
The first step at which the baseline and the new methods differ is the conversion of human time into difficulty. The agents' input results stay the same. A new fit can therefore change the horizon without changing a single AI response.
The same GPT-4o results gave a different horizon
The authors reanalyze the recorded results of METR Time Horizon 1.1: 228 tasks and 26 AI systems. In Figure 1, GPT-4o's horizon is 7.2 minutes according to the common-slope baseline and 4.4 minutes according to Model 2. These are two estimates from the same attempts, not a decline in GPT-4o's performance or a comparison of its inference cost.
A more flexible curve could simply be fitting the noise better. So the authors check the prediction of results on whole task families not used in the fit. In five-fold validation they split the 79 families, not individual attempts at the same task. Both new methods have better point estimates in all 12 combinations of metric and result-weighting scheme. This means a smaller error in predicting success, not a higher success rate for the agents themselves. §2.4–2.5 and Figure 2
The advantage is not evenly spread. The fit improves most for systems of the GPT-4 era, whose horizons pass through the flat range identified above. For the strongest systems the compared curves almost coincide. Estimates for individual systems are also far less conclusive than the aggregate result. Appendix A
Details of the experiment
Data and setup. This is a reanalysis of public METR results, not a new series of agent runs. The tasks involve software work and have automatic scoring of success. Human time is usually the geometric mean of the times of successful completions; some values come from expert estimates. AI inference time is not a variable in the measured horizon.
Model and checkpoint selection. The authors do not train new LLMs or select their best checkpoint. They use the recorded results of earlier METR evaluations. GPT-4o is an API model with an undisclosed number of parameters. This analysis does not establish its internal protocol for selecting weights. The statistical fits use maximum likelihood; Model 2 applies the EM algorithm and numerical approximation of integrals. The plots of curve shapes use the full data, while the comparison of predictive quality described above uses the split by families.
Metrics. The set includes the marginal log score, that is, binary cross-entropy, the Brier score and smoothed error measures near the 50% and 80% thresholds. Cross-entropy heavily penalizes confident, wrong predictions; the Brier score measures the squared difference between the chance and the success/failure outcome. A lower value is better. Each of the four metrics was computed under three weighting schemes: equal tasks, equal families and the intermediate variant used by METR. We do not add them up into a single percentage improvement.
Cost and uncertainty. The new analysis requires statistical fits, including separate ones for the validation splits, but no additional calls to the LLMs being evaluated. The paper does not report any savings of time or money from deploying the estimator. The uncertainty intervals of the main comparison are computed over task families. A better point result on every metric does not mean that each separate contrast is statistically conclusive.
Longer tasks call for rechecking the scale
The paper does not establish the cause of the flat stretch and does not show that it will appear in every benchmark. Nor does it prove that the METR tasks represent all of a programmer's work. The authors do not question the observed exponential trend of progress; they examine what the intervals on the human-working-time axis mean.
The practical application is to supplement the horizon with a plot of the chance of success for fixed ranges of tasks. This shows whether a jump in the horizon corresponds to a clear change in reliability. When adding longer tasks to your own benchmark, it is worth checking on held-out families whether their time still predicts difficulty for the agent before extending the existing curve.
Sources
- Drew T. Nguyen, William Fithian, On the estimation and validity of AI time horizons—a statistical look at the METR plot, preprint, 2026, v1. arXiv:2610.12466; DOI: 10.48550/arXiv.2610.12466. Full text.
- METR, Task-Completion Time Horizons of Frontier AI Models, definition of the measurement, evaluation procedure and limitations. The page notes that the plot is no longer actively updated.
I use AI-generated content as part of my daily learning process.