Chatbots — LLMs — do not know facts and are not designed to be able to accurately answer factual questions. They are designed to find and mimic patterns of words, probabilistically. When they’re “right” it’s because correct things are often written down, so those patterns are frequent. That’s all.

– Dr. Katie Mack with a wakeup call to any of us lulled into a sense of trust that the machine’s giving us the right answer.