The first chapter in this series made a simple point. ChatGPT is an application, not a mind. Behind every conversation, specialised systems handle specialised tasks, and the interface conceals the seams between them.
This chapter takes that point one step further. Whichever system is doing the reasoning, whether it is a language model inside ChatGPT or a model an organisation has connected to its own software, that system can only reason over the information it has actually been given. It does not know an organisation's business. It does not know where the evidence lives. Something has to find the right information and place it in front of the model before any reasoning can happen at all.
That something is retrieval. It is rarely discussed in board conversations about AI, and it is where a lot of the governance risk actually sits.
The problem enterprise systems already solved, mostly
Enterprise systems have spent decades solving one problem exceptionally well: locating information. A finance system knows how to find every unpaid invoice over a given amount. An HR system knows how to find every employee in a given department. This works because the data is structured: organised into tables, columns, and defined relationships that a query can traverse precisely.
What has changed is the question being asked. Instead of a person writing a query, a person now asks a question in ordinary business language, and the AI translates it into the query the system already knew how to answer.
Show me unpaid invoices over ₹1 million approved by Procurement.
Underneath that sentence, the model has to identify the right tables, the right joins, and the right filters, then execute a structured query exactly as a skilled analyst would have written one by hand.
It is tempting to treat this as a translation exercise: business language in, correct query out. It is not. The system must decide which entities are relevant, which relationships to traverse, and which subset of the organisation's data should be brought into the model's reasoning. Retrieval is a planning problem, not a language problem. An incorrect traversal does not produce an error message. It produces a confident answer built on the wrong evidence, and there is often nothing in the output to signal that anything went wrong.
Why text cannot simply be compared
Structured data solves part of the problem. A great deal of what matters inside an organisation is not structured at all. Some of it lives as whole documents: contracts, policy manuals, regulatory filings, PDFs and emails sitting in a document store, the kind of material a knowledge management or marketing team is used to indexing and searching. That is the traditional home of "unstructured data," and it is a real and important category on its own.
But there is a second, less obvious case, and it is the one this chapter is really about. Inside systems that are otherwise perfectly structured, individual fields are often free text. A support ticket has a rigid schema: an ID, a date, a priority level, all fixed and queryable. Sitting inside that same row is a Description field, written by a person, in their own words, with no fixed shape at all. It is not a document. It is one field inside one structured row. And a plain filter condition cannot search it, for the same reason a filter cannot search a PDF.
Text, whether it is a whole document or a single free text field, cannot be compared mathematically in its raw form. Before it can be embedded, it first has to be broken into chunks: manageable blocks of text, perhaps a paragraph or a few sentences long, small enough for a model to process and specific enough for a similarity comparison to mean something. A long contract does not become one embedding. It becomes dozens, one for each chunk it has been split into.
Embeddings solve the comparison problem by converting each chunk into a point within a high-dimensional mathematical space. A chunk is passed through a model and comes out the other side as a long list of numbers, for example a set of 512 coordinates:
The technical machinery that produces these coordinates does not need to be understood to use it well. What matters is the consequence: once text, whether a whole document or a single free text field, has been broken into chunks and converted into coordinates, a computer can ask a genuinely new kind of question.
A different kind of question
A structured query asks: which rows satisfy this condition. Vector search asks a different question entirely: which chunks of text are closest in meaning to this query.
Consider a support ticketing system. The table itself has a fixed schema. It is not a document store. But one of its fields is written in ordinary language, and that field often holds the actual answer to the question being asked.
AND CreatedDate < 2025‑01‑01
The table is structured. One field inside it is not. A retrieval pipeline built for this kind of question has to do both jobs in sequence: filter with precision on the attribute fields first, then decide whether the free text field even needs semantic search at all. If the SQL filter has already narrowed the population to a handful of tickets, the simplest and most reliable option is to hand their Description fields to the model directly. Vector search only earns its place when the narrowed population is still too large for that, and it is not run from scratch: the embeddings it searches were generated once, when the tickets were written, and stored in a vector index built for similarity search. What happens at query time is a search against that database, not the creation of it.
The Top-K problem
When semantic retrieval is required, the system must decide how many nearest neighbours to return. Most implementations choose a fixed number, commonly called K: the nearest five, ten, or twenty matches by similarity. The difficulty is that Top-K guarantees K neighbours, not K relevant neighbours.
Return to the authentication tickets. If only one ticket genuinely relates to authentication but Top-5 is requested, the remaining four are included simply because they are the next closest vectors. The model must now reason over one relevant record and four records that are merely similar, with no reliable signal distinguishing which is which. Increasing K improves the chance of finding the relevant one. It also increases the amount of irrelevant context the model has to reason through to find it.
Raising K does not resolve this. It increases recall and noise together, in roughly equal measure, because the underlying mechanism has no way to tell the two apart. There is no universally correct value of K, because K is an attempt to solve a relevance problem using a quantity control.
Put the two mechanisms side by side and a pattern appears. Structured retrieval reduces the search space deterministically: a row either satisfies the filter or it does not, with no ambiguity. Semantic retrieval reduces what remains probabilistically: it estimates relevance by distance, and distance is not the same thing as relevance. Each stage is a progressive reduction of uncertainty, not a menu of interchangeable techniques. The natural next question is whether that final stage of uncertainty, the estimation vector search performs, can be reduced further still.
Both of these patterns, text-to-SQL and vector search, are commonly described using a single term: Retrieval-Augmented Generation, or RAG. In practice the term is often used more narrowly than that, as shorthand for vector search specifically, embedding documents and retrieving the nearest matches. That narrower usage is inaccurate. RAG describes any architecture in which a system retrieves information before generating a response, whatever the retrieval mechanism actually is. A structured query against a database, run before the model reasons over the result, is RAG. A vector search over embedded chunks is also RAG. The two are frequently combined, as the ticket example above shows. Neither one is RAG on its own, and RAG is not a synonym for either.
Meaning, reconstructed from scratch, every time
Text-to-SQL and vector search look like different technologies solving different problems. Structurally, they are doing the same thing. Both are performing an act of interpretation at the moment a question is asked. Text-to-SQL is deciding, on the spot, what entities exist in the data and how they relate to one another. Vector search is deciding, on the spot, what counts as similar enough to matter and where the boundary of relevance should sit.
Every single query, from every single user, human or machine, reconstructs this understanding from nothing. The system has no memory of having answered a similar question before, no accumulated judgment about which interpretation was correct last time, and no way to improve except by being run through a newer model. The meaning of the organisation's own data is estimated fresh, every time, at runtime, under time pressure, with no human in a position to check the reasoning before the answer is delivered.
Governing meaning instead of estimating it
Both structured retrieval and vector search solve retrieval problems, but they do so by making decisions at runtime. Text-to-SQL decides which entities to traverse, how they relate, and which data belongs in the model’s context. Vector search decides what appears sufficiently similar to retrieve and where the boundary of relevance should sit. Every question reconstructs those decisions from scratch.
The next architectural step is different. Instead of estimating meaning every time a question is asked, organisations can establish meaning once, during ingestion, and allow every future retrieval mechanism to build upon it.
A governed semantic layer captures concepts rather than text. It records canonical definitions, synonyms, exclusions, relationships, similarity thresholds and human validation as part of the organisation’s own knowledge.
Retrieval therefore changes in kind. The question is no longer which records appear closest to this query. It becomes retrieve every record already governed as Third-Party Compromise. Runtime estimation becomes deterministic lookup. The judgement happened earlier, under governance, by someone accountable for making it correctly.
Where investment compounds
Language models will continue to improve. Embedding models will continue to improve. Text-to-SQL will become more accurate. Vector databases will become faster. None of those improvements belong to the organisation. They arrive as the industry advances.
The organisation’s investment should sit somewhere else.
For the past two decades, enterprises have invested heavily in semantic layers that describe the structure of their business. Those investments survived migrations from Oracle to Snowflake, from on-premise warehouses to cloud platforms, because they captured business meaning rather than database implementation. AI is pushing exactly the same idea one layer deeper.
The next durable investment is not another retrieval technique. It is a governed representation of the organisation’s own meaning. Every approved concept. Every validated relationship. Every recorded exclusion. Every resolved ambiguity. Every human decision about what the business means. These become assets that survive changes in retrieval techniques, embedding models and language models alike.
Today’s system may retrieve using SQL. Tomorrow’s may retrieve using vectors. Five years from now it may retrieve using something that has not yet been invented. The governed semantic state remains.
Most organisations are currently investing where technology depreciates fastest: prompts, retrieval pipelines and model selection. Those investments are necessary, but they are continually replaced by the next generation of tools. Investment in governed organisational meaning behaves differently. It compounds. Every concept captured today makes every future model more capable, because the model inherits a better understanding of the organisation rather than being asked to rediscover it from scratch.
The organisations that create lasting advantage will not necessarily own better models than their competitors. They will own a better representation of themselves.