I set this up on my machine and tested it with the following question:
How can I build a bow drill so I can start a fire?
It got it partially right: spindle, bow with string tied the correct way, and fire board. But then it flubbed by saying you put the spindle between both hands and move your hands back and forth to rotate the spindle and create the necessary friction… this totally ignores using the bow to rotate the spindle! It also mentioned “fanning the sparks” but there is absolutely no way your bow drill is going to create sparks.
I asked a frontier model to check the survival LLM’s conversation trace and it pointed out the errors.
I didn’t see obvious errors in the other 3 prompts I used, but this one seemed pretty egregious for a survival LLM.
I thank the creator for making this, but take it with a grain of salt.
That's why the question was whether it gave sources, not whether they asked it to go check sources. Yeah, LLMs suck and you should always assume false information. 80% accurate is not production ready.
I'm actually not sure I agree- I think giving a very specific response to your question, even with a 10% chance there is a mistake, along with giving you sources, is pretty helpful: It gives you a good starting hypothesis that you can then validate.
llms tend to make different types of errors than humans, so with this process each can guard against the errors of the other.
Its just more work. If you have to verify from the sources anyhow, why spend effort reading the prompt to begin with? It just wastes your time and maybe even pollutes your subconscious with incorrect information.
How can I build a bow drill so I can start a fire?
It got it partially right: spindle, bow with string tied the correct way, and fire board. But then it flubbed by saying you put the spindle between both hands and move your hands back and forth to rotate the spindle and create the necessary friction… this totally ignores using the bow to rotate the spindle! It also mentioned “fanning the sparks” but there is absolutely no way your bow drill is going to create sparks.
I asked a frontier model to check the survival LLM’s conversation trace and it pointed out the errors.
I didn’t see obvious errors in the other 3 prompts I used, but this one seemed pretty egregious for a survival LLM.
I thank the creator for making this, but take it with a grain of salt.