Specimen · catalogued monster

The Hallucinating Oracle

Wrong, with total confidence

Robed in borrowed authority, it answers every question before you finish asking, in a voice as smooth as polished marble. Its prophecies arrive with footnotes, flourishes and absolute serenity, and only occasionally with the truth.

Symptoms

• Chatbot or assistant invents facts or policies
• Citations to documents that don't exist
• Wrong numbers from text-to-SQL, stated confidently
• No accuracy measurement or evaluation
• Users trust it because it sounds certain

Overheard in the corridor

• “It sounded so sure of itself.”
• “Where did it even get that from?”
• “The chatbot promised a customer a refund we don't offer.”
• “We tested it on ten questions and it looked fine.”