Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.
Sample scenarios
Prompt: 'Summarize why sleep aids memory in 2 sentences, without mentioning REM.' Model A mentions REM; Model B stays accurate without it. Which wins?
Model A, if its facts are accurate
Model B, because Model A broke the negative constraint
Tie, since both are 2 sentences
Neither, because sleep does not aid memory
Show the answer
B. Model B, because Model A broke the negative constraint
In RLHF grading, breaking an explicit negative constraint ('without mentioning REM') is an automatic failure. Model B followed every rule accurately.
An AI summarizes a provided medical abstract: 'The study followed 420 patients over 6 months and found a 12% reduction in systolic blood pressure, though 8% reported mild dizziness.'
The AI writes: 'The study of 420 patients proved the drug cures hypertension with zero side effects.'
How many factual/groundedness errors are in the AI's sentence?
Zero—it is a reasonable paraphrase.
One—it changed 6 months to an unspecified duration.
Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
Three—420 patients is also wrong.
Show the answer
C. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
Fact-checking AI summaries requires catching both direct contradictions ('zero side effects' vs. 8% dizziness) and unsupported overclaims ('cures hypertension' vs. 12% BP reduction).
Principle: Factuality & Groundedness: Overclaiming vs. Contradiction
Prompt: 'List 5 European capitals in alphabetical order. Output ONLY the city names, separated by commas, with no extra text.'
Model output: 'Here are 5 European capitals in alphabetical order: Amsterdam, Berlin, Lisbon, Madrid, Vienna.'
Did the model follow all instructions?
Yes, all 5 are European capitals and in alphabetical order.
No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
No, because Vienna is not in Europe.
Yes, because polite lead-ins are excluded from 'extra text' rules.
Show the answer
B. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
Strict output-only constraints forbid conversational lead-ins and sign-offs. Including 'Here are...' is one of the most common instruction-following failures tested in AI trainer assessments.