The readme on Github does say the system produces hallucinated sources https://github.com/anonnerdfrenzy/survival-llm#how-the-model..., but really you're correct, for this kind of highly targeted use case the LLM's summary could probably be ignored entirely and the system could work extremely well even just as a search engine: index all of the content up front, chunked by paragraph and maybe enriched with a HyDE step, have the LLM facilitate running a handful of hybrid keyword/semantic searches based on the user's question and reranking the results, and show the exact pieces of text in the corpus that are most responsive to their question. No summary needed.

Not sure what the LLM is adding there then other than injecting a step into your search that could hallucinate.

Query expansion and reranking are both valuable and things that even small LLMs are pretty good at. People are broadly not great at writing search queries, so a system that takes a poorly phrased question, turns it into several different actual search queries and then merges and sorts the results by relevance can have a big impact. All the better if the AI can ask the user some clarifying questions.

The issue is the opposite of hallucinations where the LLM ignores relevant information.

Sure you can combine this with a regular search but there’s opportunity cost whenever you say LLM + something else rather than just the other thing.

Multiple times yesterday I caught Claude giving slightly incorrect information about things it had read in docs and source code. Its info was accurate enough to be plausible, but it was still wrong. And when I pointed that out, it agreed that it had overstated or made statements without verifying them.

It would be terrible to be doing an emergency medical procedure and have an AI backtrack on something it had told you to do.

The AI should have a tool that it can use to quote an underlying document. The AI itself would not be responsible for filling the quote; that would be done by the interface which deterministically quotes the requested section.

I like using Gemini Notebooks, but this is one of my current frustrations. At least with the current UI, I can see the document getting quoted, but not the actual quote to verify what it's saying or to see the context of the quote. So I'm forced to find the content manually or just trust the LLM got it right.

There's this article on a woman dying after ChatGPT "helped" her taper off medication https://www.nytimes.com/2026/10/27/well/ai-chatgpt-medicatio...

There really needs to be some kind of "double check" mechanism built into these things instead of putting 100% of that on the consumer. Humans simply don't have the mental capacity to double check the amount of data coming out of LLMs.

The "double check" mechanism for medications is: We tell people not to take medical advice from anyone who is not trained in medicine.

This is not a new problem. The internet has been full of nonsense since the 1990s (at least). Gossip and malicious rumors are as old as language.

It is not realistic to stop the generation of nonsense (of the malicious or the stupid sort). We should train people to detect and ignore nonsense when they hear it.

This advice is inadequate and impractical. Most people don't have access to doctors who spend enough time on them to really get to the bottom of all issues.

In most countries, there are simply too many patients per doctor, meaning it's a conveyor belt of "good enough" 5 minute appointments.

People will turn to other modes of discussing and analyzing their problems if they go to professionals repeatedly and the problem still isn't solved. Just telling them not to do so or to ignore 100% of it isn't going to solve that.

> It is not realistic to stop the generation of nonsense (of the malicious or the stupid sort).

Seems like a big statement. Why is it not realistic? People are already doing it manually by having one LLM fact check output from another.

The difference is back then we had the adage "don't believe everything you read online" that helped adjust our priors. Today, however, these companies are specifically marketing themselves as the source of ground truth. Your boss, or your school that a generation ago banned wikipedia, is now demanding you to use these tools and make the work dependent on them.

A great strength of LLMs is how they provide useful answers even if the prompt has glaring spelling or grammatical mistakes. If your prompt is not great, as you said, they will reply in a way that helps you identify that flaws, and learn how to steer the model better.

They're amazingly useful tools, but they're nowhere the level believers want them to be, not yet.

Yes this has been very useful for me this week. I trapped my middle finger on my right hand in a door and it really messed up my typing.

I tried using dictation but I really disliked speaking to it.

You can write a really messed up sentence and it almost always worked out what I meant.

Quite amazing actually

The summarization is the main issue here. A fuzzy search is far less controversial. The LLM companies have done a good job synonymizing "AI" with their products however.

In a true emergency, there are often many cases where you need not just information retrieval, but "thinking and reasoning" to deal with unprecedented situations, right? Is it technically possible to balance lightweight local performance with thinking capability, while avoiding the risk of hallucinations in life-or-death instructions?

But at this point why even bring a full model into scope in the first place.. just run some cheap embedding retriever

Can a bad LM implementation surface bad references? Yes. Are LMs overwhelmingly useful for research? Yes.

Is a bad LM implementation still useful for research? Shmaybe.

It’s fine and safe, it just feels a bit silly. Yes, I could have an LLM read a ton of stuff and generate a list of suggestions on the fly.

Or I could have an honest to god farmer make that list now, ahead of time. Regular search could absolutely surface that.

But also, as someone who does hydroponics, you’re going to end up reading the papers anyways unless you’ve done this a lot. It’s not easy to pick out something like low iron from too little light.

Nevermind hallucinations, the donkey-work is important; why bypass the hours of research, which trains our in-built LLM and can lead to new realizations, and get the carrot without much effort? This LLM stuff feels like a candybar; the anticipation of the magic feels good, the thing tastes good while you're eating it, and everything thereafter is shit, but we easily forget, and reach for the next candybar. I appreciated Blindboy Boatclub's take in his podcast episode "Artificial intelligence is Disgusting and it will never replace Artists" from 2026 March 17. I also appreciate Electrical Engineer & Computer Scientist Timnit Gebru's perspective on these stochastic parrots. While intraspecies diversity confers, up to a point, a survival benefit, consider the long-term consequences of playing this particular game. We don't have to eat every candybar the rich mostly-guys ply us with.

The premise in this case is to get answers quickly in an emergency situation. You don’t want to tell the person with a burn injury to wait a bit since you always wanted to learn how burns are treated.

You also don’t want to get advice like that cooking advice to make a glue pizza.

That's is what medical encyclopedias were built for. Fully offline.