DRAFT: We Really Need To Talk About Grounded AI

You probably heard about Google’s NotebookLM from a Verge article or podcast and really it is quite neat; a nice user interface on top of Google’s LLM; Gemini Pro. Talk with your notes, query your second brain[1], and produce automatic podcasts from text.

What’s interesting to me is the “grounded in your sources” pitch though:

This implies NotebookLM won’t hallucinate because it is using your notes and sources. This even as NotebookLM has “NotebookLM may still sometimes give inaccurate responses” in the UI when you use it:

“Grounded” is shorthand in the generative AI for RAG (Retrieval Augmented Generation) and all the glossing over of the LLM elephant in the room that comes with it; the quality-data you bring to the LLM is minuscule compared to the range of data already in the LLM. And in a system that uses weights and probability to predict the next word this is a rather large problem.

[1] Though the “second brain” movement typified by Roam hasn’t shown any real results and the science is shaky or even contrary to the hustle.