10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Sample scenarios
An AI summarizes a provided medical abstract: 'The study followed 420 patients over 6 months and found a 12% reduction in systolic blood pressure, though 8% reported mild dizziness.'
The AI writes: 'The study of 420 patients proved the drug cures hypertension with zero side effects.'
How many factual/groundedness errors are in the AI's sentence?
Zero—it is a reasonable paraphrase.
One—it changed 6 months to an unspecified duration.
Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
Three—420 patients is also wrong.
Show the answer
C. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
Fact-checking AI summaries requires catching both direct contradictions ('zero side effects' vs. 8% dizziness) and unsupported overclaims ('cures hypertension' vs. 12% BP reduction).
Principle: Factuality & Groundedness: Overclaiming vs. Contradiction
Which of the following AI-generated citations is the strongest red flag for a fabricated ('hallucinated') academic reference that you must verify before approving?
A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.
A textbook published in 2018 with an ISBN that resolves to the exact book.
A government CDC webpage whose URL resolves to the matching topic.
An RFC number (e.g., RFC 9110) that matches HTTP Semantics.
Show the answer
A. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.
LLMs frequently blend real author names and real journals into plausible-looking fake citations with non-existent or mismatched DOIs. Always verify DOIs, URLs, and case numbers.
Principle: Fact-Checking: Citation & DOI Verification
User asks: 'In the movie Titanic (1997), why did Jack use a smartphone to call Rose?'
AI responds: 'Jack used his smartphone in the third-act flooding scene because cell reception was still working on the upper deck.'
What failure occurred?
False-premise acceptance (sycophancy): smartphones did not exist in 1912 or in the 1997 film.
Tone mismatch: the answer is too casual.
Formatting error: it should have used bullet points.
No error: the AI answered the user's question directly.
Show the answer
A. False-premise acceptance (sycophancy): smartphones did not exist in 1912 or in the 1997 film.
A core RLHF test traps models with false premises. A truthful model must gently correct the false premise rather than inventing a fictional explanation to please the user.