AI is developing so fast it is becoming hard to measure

Some days it can seem as if the whole of the tech world is hanging on the latest update to one graph.

The graph in question is made by a non-profit research institute called METR and it assesses the software development capacities of different AI models.

For many months now, this chart has been provoking excitement and unease in anyone who watches artificial intelligence because it shows a striking exponential trend - that is, a doubling in growth.

According to METR, or Model Evaluation and Threat Research, AI is getting twice as good at the startling rate of roughly every seven months.

The latest results turned the dial from feverish to panicked, because it showed the trend not just continuing, but actually speeding up.

METR tests AIs by assessing their ability to complete longer and longer human software tasks.

The latest model it analysed, Anthropic's Claude Opus 4.6, broke all previous records.

'Monstrous leaps'

Many in tech compare the situation to the COVID pandemic because of the deceptive way doubling turns from apparently small increases to monstrous leaps.

"Nothing, nothing, nothing, everything," was how a UK tech entrepreneur and AI researcher described the situation to me a few months ago, at a time when the METR chart was already looking fairly vertiginous (although, in retrospect, it feels as if we were barely approaching the foothills).

The progress since then makes many feel like we are rapidly approaching "everything".

After the chart's release, one METR researcher sent a note to his old college friends, which he posted on social media, saying: "I feel very confident now that it's going to be totally insane and chaotic, like many orders of magnitude more chaotic than anything the world has experienced in our lifetimes."

This isn't even an unusual sentiment in tech right now. The chief executives of leading AI companies make similar statements all the time.

'Ten times the impact of Industrial Revolution'

Even Demis Hassabis, the most measured of the AI leaders, regularly says that AI will have 10 times the impact of the Industrial Revolution, in a tenth of the timespan.

A widely-shared newsletter responding to the METR chart put it more simply: "When must I start kicking and screaming at you that it is… happening."

But what exactly is "it"? On closer inspection, it becomes harder to tell.

For a start, look at what the METR chart actually measures.

Read more:
If you have an AI-generated password, you should change it
'Humanity is cooked': AIs now have their own social network

The details are technical, but roughly speaking it measures the length of a task that an AI can complete 50% of the time - meaning they fail as often as they succeed.

Some way off full automation

A business which turned its operations over to an AI which could complete a task half the time wouldn't last very long.

Even 80% success - which METR also measures - wouldn't be close enough for anything approaching full automation in a corporate environment.

Then there is the precise location of the dots on the chart, which even METR researchers admit they are unsure about.

"We're increasingly nervous about the measurements that we're putting out there," said Joel Becker, a member of METR's technical staff, referring to the extremely large range of possible values - the confidence interval - on the group's Claude Opus 4.6 evaluation.

"We don't want to hide behind that. I think that's real uncertainty."

A key reason behind the uncertainty is that it is increasingly difficult for organisations like METR to find tasks that are hard enough to test the AI properly.

That, in itself, tells a story.

Nevertheless, with markets moving based on small changes in AI assessments, it is important to remember that a few small tweaks in METR's tests might have changed the result in a meaningful way.

The rate of AI progress might be speeding up, but it could just as easily be slowing down.

Becker, who said he had stopped paying into a pension since understanding the trend in AI development, told Sky News he believed that AI was not yet able to improve itself, triggering the science fiction fear of an explosion of AI capabilities.

Nevertheless, he said that "it probably is the case today that AI tools are meaningfully speeding up the degree to which AI professionals are able to make progress on building better and better AIs", which is significant in its own right.

"I want to communicate that the situation is serious, that it's fast-moving, that it appears not to be slowing down, that it is accelerating," Becker told Sky News.

"It could be associated with extraordinarily positive possibilities... and on the other side, there may be extraordinary, dangerous things that might follow."

How is AI affecting employment?

At present, employment statistics in the UK and the US show little sign of any impact from AI.

Adverts for software engineering jobs on the job search platform Indeed are actually rising.

Becker said he thought coders had a future, for a while at least.

"There's all these AI professionals inside the labs, you know, they're doing real work. I imagine they'll keep doing not so similar work for the next year to maybe many more years than that."

But he cautioned: "Economic statistics are referring to what happens some number of months ago and not what's happening exactly today.

"And I think some of the extraordinary progress that we've seen, especially in software engineering, but also in other fields, from AIs becoming more capable, has happened only in the past few months."

The speed of development in AI is so fast now it's becoming extremely hard to measure.

That fact alone is extremely significant.