Retrieval-Augmented Generation is talked about everywhere, almost always in the same way: you take a language model, you “plug in” your company documents, and from that moment on it answers by citing its sources. Reality, as so often happens, is more interesting than the brochure.
We know this because we have actually built and put into production a RAG platform — entirely on-premise, on hardware within our own perimeter, with open models. Today it is the foundation of ACME ECMS IA & RAG Integration, and on the dedicated page you will find the infrastructure diagrams and a demo of how it works. Here, instead, we tell you what the diagrams do not: the five lessons that changed the way we design these systems.
1 · Quality is decided at retrieval, not in the model
The temptation for anyone starting out is to focus on the model: which LLM, how many parameters, which version. It is the wrong part of the problem. If the steps that retrieve the documents bring the model the wrong content, the most powerful model in the world will do exactly one thing: summarise the wrong thing well.
The real work happens upstream: how documents are split (the chunking), which representations they are indexed with (the embeddings), and above all how much structure you give the knowledge base. In our platform, a knowledge graph lives alongside the vector index: entities and relationships extracted from the documents, which make it possible to answer even questions that span several documents — where semantic similarity alone is not enough.
2 · Hybrid retrieval plus re-ranking: two passes, not one
A single search pass has to choose between two conflicting goals: missing nothing relevant and letting no noise through. The solution that worked for us was to separate them.
The first pass is wide and hybrid — vectors, keywords and graph together — and optimised for recall: better one candidate too many than one too few. The second pass is a re-ranker, a dedicated model that reorders the candidates by their genuine relevance to the question and cuts what looks similar but does not answer it. It is one of the components that did most to raise the perceived quality of the answers: the cited sources stop being “close to the topic” and start being the right ones.
3 · Not every question deserves RAG
A common mistake is to treat every request the same way: whatever the user asks, the whole pipeline is launched — search, re-ranking, generation with context. The result is slow, expensive answers even when none of that was needed.
In our architecture the first decision is made by a planner: a component that classifies the request and chooses the route. A greeting or a general-knowledge question takes the direct route, without touching the document base; a question about contracts, procedures or projects triggers the full pipeline. It sounds like a detail: in production it is the difference between a system that “answers when it can” and a service with predictable latency and consumption.
4 · Without metrics you are just guessing
You change the embedding model and the answers “seem” better. You rework the chunking and they “seem” worse. Seem to whom? On which questions? A developer's gut feeling is not a metric, and with a system that answers in natural language the eye gets used to things quickly.
That is why the platform includes systematic quality evaluation (we use the RAGAS framework, with a test set built in Italian around our use cases): with every change we measure how faithful to the sources the answers remain, how relevant to the question they are, and how good the retrieved context is. It is the same principle as our method — measure, analyse, improve, verify — applied to AI: every intervention starts from a number and comes back to a number.
The right question is not “does the AI answer well?”, but “compared with what, measured how, and better or worse than yesterday?”.
5 · On-premise is feasible — and sometimes it is the only way
Perhaps the most counterintuitive lesson: an enterprise RAG does not need a hyperscaler. Open models on GPUs sized to the use case, inside a governed infrastructure, cover the vast majority of real needs — and bring an advantage no remote API can offer: the data never leaves the perimeter.
For entire sectors this is not a nicety, it is a requirement. In Healthcare, health-related data enjoys the enhanced protection of Article 9 of the GDPR; in the Public Administration, data sovereignty and cloud guidance carry real weight. An on-premise architecture (or one on a private cloud in Italy) solves the problem at the root — and, with open models, it also eliminates vendor lock-in.
The other half of the lesson: a system like this must be run as a service, not as an experiment. Component monitoring, knowledge-base backups, controlled model updates: the “operations” side matters as much as the design side.
Where to start
If we had to condense the five lessons into a single piece of advice: start small, but start measuring. A concrete use case, a defined document corpus, quality metrics from day one — and an architecture that does not force you to choose between innovating and keeping your data where it belongs.
How our platform is built — infrastructure diagrams, logical flow and an animated demo of the life cycle — is described on the AI, Agents and RAG page. If instead you want to understand what it could do with your document estate, let's talk.
