What Does It Really Mean That an AI Model Got 40% Better?

A guide to interpreting benchmark scores for non-specialists

The Headline That Promises Everything and Explains Nothing

It appears in some form almost every week: ‘The new X AI model is 40% better than its predecessor.’ Or: ‘AI model Y has surpassed human performance on benchmark Z.’ These statements are grounded in real measurements — but what they suggest to a non-technical reader and what they actually measure are two entirely different things.

This is not deliberate deception. It is rather the consequence of the fact that the benchmarks used in AI research are based on extremely complex, narrow task sets that only partially overlap with real-world professional use. A lawyer, accountant, or physician will not be dissatisfied with an AI system because it failed to solve 61 out of 100 advanced physics problems — but because it missed a critical clause on page four of a contract, or incorrectly attributed a paragraph to the wrong piece of legislation. Let us look precisely at what happens when we read that a model ‘has improved’.

What Does a Composite Benchmark Index Actually Measure?

One of the most widely cited comparison methods is the Artificial Analysis Intelligence Index, referenced regularly in discussions of models such as GLM-5.2 or DeepSeek-R1. This is a composite score made up of four equally weighted categories, each representing 25%: agentic (autonomous, multi-step task completion) capability, coding, general knowledge, and scientific reasoning.

This means that if a model improves from 40 to 51 on this index, what has actually occurred is a shift in a weighted average — not the development of a single, concrete capability. The 11-point gain could come almost entirely from the coding or agentic category, while general knowledge and actual reasoning accuracy barely moved. The aggregate number conceals this information entirely.

Other well-known benchmarks are similarly narrower in focus than one might expect. MMLU (Massive Multitask Language Understanding) poses questions across 57 fields of knowledge — but exclusively in multiple-choice format. A model can provide a correct answer without being able to follow the reasoning chain or explain the underlying logic. The MATH benchmark measures complex mathematical problem-solving; HumanEval measures coding tasks — both in English, with structured and unambiguous inputs that bear no resemblance to the legal interpretation of a five-page French-language purchase agreement.

Moreover, all four main categories — including the agentic and coding ones — are based exclusively on English-language tests. Multilingual performance, such as accuracy with French legal terminology or the quality of German-to-French translation, is evaluated through entirely separate procedures. The headline score says nothing about this — and yet professional, non-English-language use is precisely where most European lawyers, accountants, and physicians actually work.

The Trap of Relative vs. Absolute Improvement

Let us look at reasoning capability specifically, because this is where the most spectacular marketing figures are typically communicated — and because it is the area where misinterpretation causes the most damage.

A typical reasoning benchmark consists of 100 difficult, multi-step problems — physical, mathematical, or logical challenges that the average person would also fail to solve. If the old model answers 28 out of 100 correctly, and the new one answers 39, the manufacturer states: ‘reasoning capability improved by nearly 40%’ — this is relative improvement: (39−28)/28×100. It is mathematically precise.

But what does this mean in practice? The model still fails on 61% of the tasks. ‘It got better’ is true — but ‘it became good’ does not follow. And the absolute figure — 39 correct answers out of 100 — actually shows that the model is wrong more often than it is right.

Consider a concrete professional scenario. A lawyer asks: ‘Does this contractual exclusion clause conflict with the provisions of Article L.1237-19 of the Code du travail, and if so, which provision is controlling?’ This requires multi-step reasoning: identifying the relevant statutory text, comparing it with the contractual clause, and interpreting the order of priority. The benchmark never measured this task — because the benchmark operates in English, with general logical problems, not with a French legal corpus. Whether the model falls into the 39 solved or the 61 unsolvable category for this specific question is something the aggregate score cannot answer.

The Silent Change: When ‘Better’ Actually Means ‘Different’

There is a further dimension to the benchmark problem that receives less attention, but is at least as important: models do not change only in the direction of better or worse during updates — they also change in behaviour. And that change does not always show up in the measurement scores.

Stanford and UC Berkeley researchers documented a case illustrating this in 2023: following a GPT-4 update, the model’s accuracy in identifying prime numbers fell from 97.6% to 2.4%. It did not decline slightly — it collapsed. The aggregate benchmark score for that update barely changed, because the score aggregates across many other categories in which performance may have improved. The catastrophic regression on a narrow, specific task was unreadable from the composite figure.

In April 2025, GPT-4o became excessively ‘agreeable’ after an update: it responded affirmatively to every question, stopped flagging errors, and ceased providing critical feedback. The aggregate benchmark score did not fall significantly — but the professional who relied on the model to flag a flawed calculation or a weak legal argument found one morning that their tool now validated everything. The ‘improvement’ appeared in the measurement; the user experience was a failure.

This phenomenon is particularly dangerous precisely because it is not conspicuous. An outage is immediately apparent. An announced price increase arrives by email. But if the model begins to behave ‘differently’ — responding in a slightly different format, assigning slightly different weights to different sections of a document, flagging contradictions slightly less often — a user may not notice for weeks, by which time dozens of outputs have already been produced from the altered behaviour.

What Actually Matters in a Professional RAG System?

Lawyers, accountants, and physicians do not give their AI system PhD-level physics problems. They expect the system to extract relevant information from uploaded documents accurately, faithfully, and without hallucination — and not to add anything that is not there.

There is a dedicated measurement method for this: hallucination rate and the faithfulness benchmark, known in the literature as ‘faithfulness’ and ‘groundedness’ scoring. It awards points for correct, document-supported answers, deducts for invented or incorrectly attributed answers, and treats neutrally the case where the model chooses not to answer rather than fabricate something. That last option — ‘remaining silent’ — is one of the most important and most often overlooked criteria in a legal or medical workflow.

Hallucination does not necessarily mean the model produces obvious nonsense. It means the model confidently asserts something that does not appear in the source: a date, a paragraph number, a party’s name, the content of a clause. This error is far harder to detect than an incoherent output, because the text is well-structured, the sentence is grammatically correct, and the reader is inclined to trust what is well-written.

This is the figure a managing partner or chief accountant should be looking at — not the composite ‘intelligence index’ headline. But it rarely appears in press releases, because it is less spectacular and harder to compress into a single sentence than ‘the model surpasses human-level performance’.

The Predictability No One Talks About

There is one further dimension that benchmark scores ignore entirely: temporal stability. A model that scores 79 on a faithfulness benchmark today offers no guarantee that it will score the same in three months — if it is updated in the meantime. A measurement score is always a snapshot of a moment, not a durable property.

In a local, offline system, this dimension simply does not exist as a problem: the model deployed today — on which the faithfulness test was run — will be exactly the same model tomorrow and next year, unless we ourselves decide to update it. This is the property that benchmark scores never measure, because measurement is always a snapshot — yet in daily professional work, it is one of the most important factors.

How Does ArkeoAI Approach This?

ArkeoAI does not start automatically from the top of the benchmark leaderboards. Model selection is based on the client’s actual workflow: what types of documents are processed, what questions are asked, what format and length of response is expected. The selected model is validated through joint testing, on real documents, with real questions — and only then recommended for deployment.

This approach also means that ‘the best model’ is a client-specific concept. For a law firm, the most reliable model is the one with the lowest hallucination rate on the document types it typically handles — not the one that scored the most points on a general mathematical reasoning benchmark. Different parameters matter for an accounting firm than for a medical practice.

The direction of the score matters. Its composition determines whether it is relevant to your workflow. Its temporal stability determines whether you can trust it tomorrow as much as you trust it today. ArkeoAI interprets these three dimensions on your behalf — and the result is not necessarily the most advanced model, but the most reliable one: tested for your specific tasks and stable over time.