Stumbling into AI: Part 3—RAG

First up: RAG stands for Retrieval-Augmented Generation.

Put another way, it’s about Generation (using LLMs, like we saw before and like you use every day), in which the prompt given to the LLM is Augmented by the Retrieval of additional context.

It is literally the difference between this prompt & response from an LLM:

and this one:

The second example’s generation has been augmented by the retrieval of additional information that’s then added to the prompt prior to submission to the LLM.

This may seem stupid—why would I tell the LLM the latest version, and then ask it the latest version? But it’s not me asking! In a typical interaction you’ll have the end-user entering a prompt, and the prompt given to the LLM will be the user’s question, plus context (information) added by some kind of lookup.

The LLM uses the additional context in order to provide an answer that is much more likely to be accurate, avoiding the issue of hallucinations (when LLMs make shit up) or simply out of date information (on which the LLM was trained; it’s not their fault really).

So far, so simple, really. RAG==a fancy way for giving LLMs more context with which to hopefully give you a better answer.

The Kafka version number example is somewhat trite and could be picked apart in multiple ways. Why do I need an LLM to tell me that? Why can’t I just go to the datasource that gave the additional context to the prompt myself?

The point with RAG is that you’re still wanting an LLM to do its thing—but to do so either with much greater accuracy (e.g. Kafka version), or in a way that it simply couldn’t do without the additional context.

RAGs really come into their own for enriching the behaviour of LLMs with data, often that’s held within our business. As a customer, we all expect companies to know "everything" about us, right? After all, we completely the forms, we checked the boxes—why are you asking me to type in a bunch of data that you already know?

Here’s a simple example of this concept. Imagine we want to know where our nearest airport is?

Yes yes, we could just open Google Maps, but hear me out.

If we ask an LLM this, it’s not got a clue—it has no context:

Whereas if we already hold information about our customer 'Rick', we can look that information up and pass it to the LLM:

So how do we get this additional context? Or rather, how do we do the Retrieval of data with which we can Augment our Generation?

Well, you could literally run a SQL query, as illustrated above. You could have your client make API calls to get data, you could even use an MCP server—the end result is the same: the LLM is getting additional context to help it do what you want it to do.

One common pattern for the implementation of RAG is the use of vector databases. In fact, so common I would say it’s become synonymous. The reason that it’s so widely used is that whilst a SQL lookup is excellent for working with structured data (e.g. "where does my customer live"), that’s often not the kind of additional context that we want to provide to the LLM.

What if we’re working with text? Whilst LLMs are trained on a metric crap-ton of publicly available text, there’s material out there that they don’t know about. That could be content that was created after the LLM was trained (for example, the release notes for the latest version of Apache Kafka). Often, in the context of RAG, it’s material internal to a company such as documentation, wikis, emails, Slack conversations…any manner of content.

To use this text data in RAG, a representation ("embedding") of the semantic meaning of the text is created, and stored as a vector. This is the RAG Ingestion part. Once the data is stored, it can be used in RAG Retrieval. What this does is take the user’s query and convert that to an embedding too. It then compares the query’s vector with those held in the vector store (populated by the ingest process) and finds the piece of textual data ("embedding") that’s most closely related to it. This piece of additional information is then included in the LLM request, in exactly the same manner we saw above—adding context to the existing prompt.

The Ollama blog has a nice example which I’m going to use here, demonstrating the Kafka version example I showed above.

In writing this article I made a rookie mistake. No surprise there; but one worth illustrating here.

I use LLMs a lot to help my writing—never to write but to help generate code, for example. The embeddings example on the Ollama blog had these documents:

Cute, but nothing to do with my domain. So I did as I often do—chuck it over to an LLM.

Feeling smug, I copied this into my Python code and added a couple more documents (chunks) to illustrate the point about release version freshness:

But when it came to comparing LLM responses using RAG and this data, it didn’t really make much difference in some cases:

Why’s the output basically the same? Because the LLM already knows what Apache Kafka is! If it didn’t, how would it have generated the documents array above?

(technically I used Claude 4 Sonnet to generate the array and Llama 3.2 in my Python script—but basically the same principle).

One lesson, but to express in two different ways from this:

There are probably a bunch of nuances to this. For example, could you use a more powerful LLM to distill down content for use in RAG by a smaller LLM (cheaper/faster to run)? Tell me in the comments below—I’m here to learn :)