How do I build my own RAG?

Hosting

Building

• Ingestion and parsing. Connect to the source systems (SharePoint, file shares, intranets, case management) and get clean text out of messy formats. Scanned PDFs need OCR, tables need structure-aware extraction, and a badly parsed document can mess up every answer that touches it. You need to budget real time here. It’s never the bit people expect, but it makes or breaks the quality of the outputs and therefore user confidence.
• Chunking and embedding. Split documents into retrievable pieces with sensible boundaries and overlap, attach metadata (source, date, access level), then convert each chunk to a vector via an embedding model. Note the embedding model choice now, because changing it later means re-embedding the entire index. For large indexes, that can take some time and effort.
• Vector store. Somewhere to hold and search those vectors. Our tool of choice is Azure AI Search.
• Retrieval and generation. Query comes in, gets embedded, you search (ideally hybrid, keyword plus semantic, with a reranking pass), then feed the top chunks plus the question to the model with a prompt that forces it to ground answers in the retrieved context and cite sources. Grounding and citation are super important for obvious reasons and guard against hallucinations and the associated nonsense.
• The security layer. This is probably the most important section for obvious reasons. It ensures people don’t see things they are not supposed to, and stops poison code or document injections. Your security people will understand all of this, but the headlines are:

• • Permission-aware retrieval. Bizarrely, this is the single most-missed requirement by people doing it themselves. A user must only ever retrieve from documents they’re already entitled to see. That means propagating Access Control Lists (ACLs) from the source system into the index and filtering at query time, not just bolting auth onto the UI. Get this wrong and you’ve built a very efficient data-leak engine.
  • Authentication and Role-Based Access Control (RBAC) via Single Sign-On (SSO), ideally tied to existing identity.
  • Encryption in transit and at rest, network isolation (private endpoints, no public ingress), and proper management.
  • Audit logging of who asked what and what was returned, both for security and for the inevitable “prove it” conversation with an auditor (or worse, law enforcement).
  • Input/output guarding – prompt-injection defences (a poisoned document can carry instructions), Personally Identifiable Information (PII) redaction where needed, and output filtering.
  • A Data Protection Impact Assessment (DPIA) and data-flow map done up front, not retrofitted.

Ongoing maintenance

• Up-to-date content. Source documents change, and crucially they get deleted. Stale or withdrawn content in your index is a genuine liability. You need incremental re-sync that adds, updates, and removes.
• Quality and evaluation. Maintain an evaluation set of representative questions with known-good answers and run it as a regression test whenever anything changes. Capture user feedback and actually review the failures. Retrieval quality drifts as the library grows.
• Model and embedding changes. A new LLM version means re-testing your prompts. This is super important because model behaviour shifts with every new release. This one catches out a lot of people who assume that output will be the same. It never is.
• Permission drift. People join, leave, change roles, documents get reclassified. Access controls have to stay in sync, or you have a problem.
• Security upkeep. Patch dependencies, rotate secrets, review access logs, and regularly pen-test. Prompt-injection techniques in particular keep evolving.
• Cost and capacity. Monitor token spend and infrastructure as usage climbs. RAG costs creep quietly and are likely to escalate as the labs try to monetise their investments.

In summary