confident wrongness

[ KON-fi-duhnt RAWNG-nis ]noun

An answer that reads well, looks tidy, sounds certain, and is wrong.

Explanation

The signature failure of this whole field. People ramble and hedge when they are unsure; machines do not. So sounding sure stopped being proof of anything.

Related words

• hallucination — When an AI confidently makes something up: a command, a quote, a setting, a fact that does not exist.
• verify ruthlessly — The closest thing this crowd has to a motto: use the tools hard, then check what comes back without mercy.

Prompts that deal with it

• Hand It An Error — Give it more than “it broke”
• Stop It Agreeing — Make it argue against you first
• Show Your Work — “Implemented” is not proof