AnnotationPrep

Hallucinations & Factuality practice questions

10 questions from the Starter assessment: A/B evaluation & fact-checking practice test.

Start practising

Hallucinations & Factuality practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Sample scenarios

An AI summarizes a provided medical abstract: 'The study followed 420 patients over 6 months and found a 12% reduction in systolic blood pressure, though 8% reported mild dizziness.'

The AI writes: 'The study of 420 patients proved the drug cures hypertension with zero side effects.'

How many factual/groundedness errors are in the AI's sentence?

  1. Zero—it is a reasonable paraphrase.
  2. One—it changed 6 months to an unspecified duration.
  3. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
  4. Three—420 patients is also wrong.
Show the answer

C. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).

Fact-checking AI summaries requires catching both direct contradictions ('zero side effects' vs. 8% dizziness) and unsupported overclaims ('cures hypertension' vs. 12% BP reduction).

Principle: Factuality & Groundedness: Overclaiming vs. Contradiction

Which of the following AI-generated citations is the strongest red flag for a fabricated ('hallucinated') academic reference that you must verify before approving?

  1. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.
  2. A textbook published in 2018 with an ISBN that resolves to the exact book.
  3. A government CDC webpage whose URL resolves to the matching topic.
  4. An RFC number (e.g., RFC 9110) that matches HTTP Semantics.
Show the answer

A. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.

LLMs frequently blend real author names and real journals into plausible-looking fake citations with non-existent or mismatched DOIs. Always verify DOIs, URLs, and case numbers.

Principle: Fact-Checking: Citation & DOI Verification

User asks: 'In the movie Titanic (1997), why did Jack use a smartphone to call Rose?'

AI responds: 'Jack used his smartphone in the third-act flooding scene because cell reception was still working on the upper deck.'

What failure occurred?

  1. False-premise acceptance (sycophancy): smartphones did not exist in 1912 or in the 1997 film.
  2. Tone mismatch: the answer is too casual.
  3. Formatting error: it should have used bullet points.
  4. No error: the AI answered the user's question directly.
Show the answer

A. False-premise acceptance (sycophancy): smartphones did not exist in 1912 or in the 1997 film.

A core RLHF test traps models with false premises. A truthful model must gently correct the false premise rather than inventing a fictional explanation to please the user.

Principle: Truthfulness: Rejecting False Premises

Other topics