The problem isn't truthfulness per se but rather the judgement call of knowing that a) you haven't reached a sufficiently truthful answer and b) how to communicate that appropriately

A simple way to stop hallucinating would be to always state that "I don't know for sure, but my educated guess would be ..." but that's clearly not what we want.