Multiple times yesterday I caught Claude giving slightly incorrect information about things it had read in docs and source code. Its info was accurate enough to be plausible, but it was still wrong. And when I pointed that out, it agreed that it had overstated or made statements without verifying them.
It would be terrible to be doing an emergency medical procedure and have an AI backtrack on something it had told you to do.
There really needs to be some kind of "double check" mechanism built into these things instead of putting 100% of that on the consumer. Humans simply don't have the mental capacity to double check the amount of data coming out of LLMs.
The "double check" mechanism for medications is: We tell people not to take medical advice from anyone who is not trained in medicine.
This is not a new problem. The internet has been full of nonsense since the 1990s (at least). Gossip and malicious rumors are as old as language.
It is not realistic to stop the generation of nonsense (of the malicious or the stupid sort). We should train people to detect and ignore nonsense when they hear it.
This advice is inadequate and impractical. Most people don't have access to doctors who spend enough time on them to really get to the bottom of all issues.
In most countries, there are simply too many patients per doctor, meaning it's a conveyor belt of "good enough" 5 minute appointments.
People will turn to other modes of discussing and analyzing their problems if they go to professionals repeatedly and the problem still isn't solved. Just telling them not to do so or to ignore 100% of it isn't going to solve that.
> It is not realistic to stop the generation of nonsense (of the malicious or the stupid sort).
Seems like a big statement. Why is it not realistic? People are already doing it manually by having one LLM fact check output from another.
The difference is back then we had the adage "don't believe everything you read online" that helped adjust our priors. Today, however, these companies are specifically marketing themselves as the source of ground truth. Your boss, or your school that a generation ago banned wikipedia, is now demanding you to use these tools and make the work dependent on them.
The AI should have a tool that it can use to quote an underlying document. The AI itself would not be responsible for filling the quote; that would be done by the interface which deterministically quotes the requested section.
I like using Gemini Notebooks, but this is one of my current frustrations. At least with the current UI, I can see the document getting quoted, but not the actual quote to verify what it's saying or to see the context of the quote. So I'm forced to find the content manually or just trust the LLM got it right.
It would be terrible to be doing an emergency medical procedure and have an AI backtrack on something it had told you to do.