The Improviser · Lesson 7 — The Wrong Clip ← Course

The wrong clip — when retrieval fails.

RAG is only as good as its search. Hand the soloist the wrong record and it plays from it — confidently, and wrong.

RAG does not remove the hard problem — it moves it. The model is now only as reliable as the record it is handed. So if the search pulls the wrong take, a missing one, or a stale one, the model grounds its answer in that, and states it with the same calm confidence as a correct one.

This is the sharp edge: a grounded-but-wrong answer can be more convincing than an obvious guess, because it arrives with a citation. The citation points to a real record — just the wrong one.

The failure modes are simple. The record is missing (the search returns the nearest thing instead of nothing). The match is wrong (a take that looks similar but isn't). Or the record is stale (the right take, out of date). In every case the model reads faithfully — garbage in, grounded garbage out.

So most of the work in RAG is in the search: a complete, fresh, well-chunked library and a retrieval step that returns the right takes. The model is the easy part.

Break the library in different ways and watch the soloist answer, confidently, from the wrong clip — cited every time. Tap ▶ to hear what it was handed.

Same question every time. Break the search and watch the answer follow the record.
the question
What key is the closing ballad in?
↓ the search returns ↓
↓ the soloist answers from it ↓
the soloist answers

The model only sees what the search returns.

The generate step is the same every time. Everything rides on what records contains — get that wrong and the answer is wrong, and cited. 


      

    


    

      
Grounded is not the same as right.

      

        RAG shifts the burden onto the search. Retrieve the wrong, missing, or stale record and
        the model will ground a wrong answer in it — cited, fluent, and convincing. A citation
        proves where an answer came from, not that it is true.
      

      

        Most of the effort in a RAG system goes into retrieval: a complete, current, well-chunked
        library and a search that returns the right takes — plus letting it say “no record” when
        nothing fits. Then verify what matters.
      

      

        Next: the inner ear →
        
      

    


  


  

    

      ▸ Go deeper — making retrieval trustworthy optional
      

        
Missing should beat wrong — allow “not found”

        

          The most dangerous default is a search that always returns something. Set a
          relevance threshold: if the nearest match is not close enough, retrieve nothing and let
          the model say “no record”. An honest blank beats a confident answer grounded in an
          irrelevant take.
        

        
Freshness

        

          A library is a cache of the world, and caches go stale. Re-index when the source changes,
          and stamp each record with a date so the search can prefer the current version and you
          can spot an out-of-date answer.
        

        
Measure retrieval on its own

        

          Test whether the search returns the right takes, separately from whether the model writes
          a good answer. If retrieval is bad, no amount of prompting fixes it — the model is only
          ever reading what it was handed. This is where evals come in, later in the course.
        

        
Always show your sources

        

          Surface which takes were used, in the answer. Citations are what let a human catch a
          wrong-clip answer — the one safeguard that turns a silent error into a visible one.