Why Local and Cloud-Based RAG Are Fundamentally Different ?

A silent failure mode that almost nobody talks about

What Is RAG, and Why Does It Matter?

RAG — Retrieval-Augmented Generation — is one of the most widely used methods for enabling an AI system to answer not from general knowledge alone, but from specific, uploaded documents. In a law firm, this means the firm’s own contracts, statutory texts, and internal guidelines. For an accountant, it means regulations, tax codes, and client files. For a physician, protocols, treatment guidelines, and patient histories.

A RAG system consists of two interconnected components. The first is the retrieval unit: it stores documents in vector form — so-called embeddings — and, for any incoming question, finds the most relevant text passages. The second is the generative model: it takes the retrieved passages and composes a coherent, natural-language response. Both components are driven by AI models — and this is the point at which a rarely discussed but decisive difference arises between cloud-based and local deployments.

When someone hears ‘RAG-based system’, they typically picture the AI: the generative model that answers questions. The retrieval component — the embedding model and the vector database — almost never comes to the foreground. This oversight is precisely what creates a concrete, measurable, yet hard-to-detect risk in cloud-based systems.

The Hidden Risk of Cloud RAG: Model Version Drift

If a RAG system is built on a cloud-based API — any endpoint from OpenAI, Anthropic, Google, or any other major provider — the models running beneath the system can change at any time, without the developer’s knowledge or consent. This is not a theoretical concern: large providers regularly update their models in the background. Endpoints labelled ‘-latest’ or ‘newest’ do not guarantee the same model version from week to week.

This phenomenon is known in the literature as model version drift. It can cause problems at the level of the generative model — but it is particularly dangerous at the embedding model level, because its consequence is a second, far harder-to-detect failure mode: embedding drift.

A RAG system stores documents in vector form. These vectors — embeddings — are produced by an embedding model: it converts a sentence or paragraph into a numerical vector of several hundred dimensions. Texts that lie close to each other in the vector space’s geometry are considered semantically similar — and this is the basis on which the system finds the most relevant document passages for a given question.

If the cloud provider updates the embedding model — even with a minor shift in emphasis — it will produce a slightly different vector for the same text than it did before. This means that documents uploaded and embedded in January inhabit a different geometric ‘space’ than those uploaded in September, if the embedding model changed in between. The vector space becomes internally inconsistent: old documents sit at old coordinates, new ones at slightly shifted coordinates.

The system does not stop — it generates no error message whatsoever. Retrieval accuracy simply degrades silently. Questions that previously returned precisely the relevant passage now return less fitting text. The generative model receives this imprecise input and produces a correspondingly imprecise response. The user may not notice for months — until the consequence surfaces in a flawed summary, an incorrect citation, or a missed contractual clause.

How Do Cloud-Based Systems Try to Defend Against This?

Developers of serious, production-grade cloud RAG systems are aware of this problem and attempt to defend against it. The most common method: every document chunk’s metadata records which embedding model version was used to process it. At query time, the system checks whether the currently running model version matches the one used during storage. If they do not match, the affected documents must be re-embedded using the current model.

This defensive mechanism is technically sound — but it carries a hidden cost that is rarely communicated. Re-embedding takes time, computational capacity, and — in an API-based system — direct financial cost. For a large legal or medical corpus of several thousand documents, re-embedding is a matter of hours, not minutes. In the meantime, the system either goes offline or continues operating with the old, inconsistent vector space — in either case, with a compromise.

In a local system, this defensive mechanism is unnecessary — not because the defence is better, but because the problem does not structurally exist: the model version does not change on its own.

Local RAG: What Is Downloaded Once Stays Static

ArkeoAI systems run entirely local, offline models. The model weights — both the retrieval embedding model and the generative model — are downloaded once and remain on the machine. There is no API connection behind them, no background update, no version change without the knowledge of the system’s owner.

In practice, this means: if the system runs a specific embedding model and a specific generative model today, it will run exactly the same models tomorrow, next week, and next year — unless the owner explicitly decides to update. Document vectors remain consistent. The vector space into which contracts were uploaded in January represents exactly the same coordinate system in September as it did in January. Retrieval quality does not degrade silently.

This does not necessarily mean a local model surpasses a cloud-based one in every respect. It means it is more predictable. In a legal or medical workflow where the stability of output quality is not a wish but a professional requirement, this is a decisive difference — one that cloud providers almost never highlight in their marketing materials.

A Practical Analogy from the Office

Imagine a firm that has operated in the same building for years, and everyone knows exactly which shelf and cabinet holds each client’s file. One day — without anyone being informed — someone rearranges the cabinets. Not conspicuously, just slightly: the civil law shelf moves one floor up, the commercial law shelf one floor down. The file is still there — just somewhere slightly different.

The lawyer or assistant doesn’t notice at first — out of habit they reach for the old location and usually find something. But not always exactly what they were looking for. In some cases, the folder containing the most important precedent has been moved to a different cabinet and doesn’t come up in the search. The office keeps working — but retrieval accuracy has silently deteriorated.

This is precisely what can happen in a cloud-based RAG system when the embedding model, after an update, ‘arranges’ documents slightly differently in the vector space. In a local, offline system, the ‘cabinets’ never move on their own — and that is the guarantee provided by the architecture, not by a promise.

The Context Window: The Other Factor Nobody Asks About

RAG system performance is affected not only by the stability of the embedding model, but also by the size of the generative model’s context window — and this too is a factor that cloud providers rarely communicate upfront.

The context window determines how many text tokens the generative model can ‘see’ and process at one time. If the retrieval unit returns several relevant passages from a long document, but their combined length exceeds the context window, the model simply cuts off the excess — silently, without warning. The user assumes the model answered based on all the relevant material, while in reality it only ‘saw’ a portion of the documents.

In ArkeoAI systems, the context window size is set and fixed by us — according to the client’s workflow, not the provider’s default setting. This is another dimension of control: the question is not ‘what does the provider allow?’ but ‘what does the client actually need?’

Why Is This Never Discussed?

Embedding drift and model version drift are not secret phenomena — they are well documented in the RAG literature and known within serious developer communities. Yet they never appear in cloud providers’ client communications, sales presentations, or marketing materials.

The reason is simple: the problem is invisible. It generates no error message, causes no outage, sends no alert. It simply degrades the system’s retrieval quality slowly and gradually — until someone notices that the AI ‘behaves differently than it used to.’ By then, weeks or months have typically passed, and tracing the cause and precise onset of the decline is difficult.

ArkeoAI’s approach is based on consciously avoiding this problem — not with sophisticated monitoring tools, but through architecture. What is not connected to a network cannot overwrite a model. This is not a trade-off between convenience and security; it is a decision justified by the client’s professional requirements.