The Third Umpire · Lesson 7 — The Wrong Footage ← Course

The wrong footage — when retrieval fails.

RAG is only as good as the search. Hand the umpire the wrong clip and it will rule, confidently, on the wrong clip.

RAG does not remove the hard problem — it moves it. The model is now only as reliable as the footage it is handed. So if the search brings the wrong clip, a missing one, or a stale one, the model grounds its answer in that, and states it with the same calm confidence as a correct one.

This is the sharp edge: a grounded-but-wrong answer can be more convincing than an obvious guess, because it arrives with a citation. The citation points to a real clip — just the wrong one.

The failure modes are simple. The record is missing (the search returns the nearest thing instead of nothing). The match is wrong (a clip that looks similar but isn't). Or the record is stale (the right clip, out of date). In every case the model reads faithfully — garbage in, grounded garbage out.

So most of the work in RAG lands on the search — building the archive well and pulling the right passages out of it. The model is the easy part.

Break the archive in different ways and watch the umpire rule, confidently, on the wrong footage — cited every time.

Same question every time. Break the search and watch the answer follow the footage.
the question
What was the verdict on the caught-behind appeal in the 12th over?
↓ the search returns ↓
↓ the umpire rules from it ↓
the umpire answers

The model only sees what the search returns.

The generate step is the same every time. Everything rides on what docs contains — get that wrong and the answer is wrong, and cited. 


      

    


    

      
Grounded is not the same as right.

      

        RAG shifts the burden onto the search. Retrieve the wrong, missing, or stale record and
        the model will ground a wrong answer in it — cited, fluent, and convincing. A citation
        proves where an answer came from, not that it is true.
      

      

        Most of the effort in a RAG system goes into retrieval: a complete, current, well-chunked
        archive and a search that returns the right passages — plus letting it say “no record”
        when nothing fits. Then verify what matters.
      

      

        On to Lesson 8 →
        
      

    


  


  

    

      ▸ Go deeper — making retrieval trustworthy optional
      

        
Missing should beat wrong — allow “not found”

        

          The most dangerous default is a search that always returns something. Set a
          relevance threshold: if the nearest match is not close enough, retrieve nothing and let
          the model say “no record”. An honest blank beats a confident answer grounded in an
          irrelevant clip.
        

        
Freshness

        

          An archive is a cache of the world, and caches go stale. Re-index when the source
          changes, and stamp each record with a date so the search can prefer the current version
          and you can spot an out-of-date answer.
        

        
Measure retrieval on its own

        

          Test whether the search returns the right passages, separately from whether the model
          writes a good answer. If retrieval is bad, no amount of prompting fixes it — the model
          is only ever reading what it was handed. This is where evals come in, later in the course.
        

        
Always show your sources

        

          Surface which chunks were used, in the answer. Citations are what let a human catch a
          wrong-footage answer — the one safeguard that turns a silent error into a visible one.