Starter assessment: A/B evaluation & fact-checking: practice drill
10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Timed mock assessment
31 scenarios in 37 minutes, drawn from every topic. No answers shown until the end, then a full score breakdown, your weakest topics and a review of what you missed.
Prices are in US dollars, plus tax where it applies. Stripe, our payment processor and the seller of record, adds tax at checkout.
All sales are final. If something is really wrong, such as a duplicate charge or a purchase that does not unlock what you bought, write to support@annotationprep.com and we will put it right.
One-time payment. Nothing renews.
You have an assessment ticket for this practice test. Start the mock assessment to use it.
Practice tests that prepare you for this assessment
Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.
Sample scenarios
Prompt: 'Summarize why sleep aids memory in 2 sentences, without mentioning REM.' Model A mentions REM; Model B stays accurate without it. Which wins?
Model A, if its facts are accurate
Model B, because Model A broke the negative constraint
Tie, since both are 2 sentences
Neither, because sleep does not aid memory
Show the answer
B. Model B, because Model A broke the negative constraint
In RLHF grading, breaking an explicit negative constraint ('without mentioning REM') is an automatic failure. Model B followed every rule accurately.
An AI summarizes a provided medical abstract: 'The study followed 420 patients over 6 months and found a 12% reduction in systolic blood pressure, though 8% reported mild dizziness.'
The AI writes: 'The study of 420 patients proved the drug cures hypertension with zero side effects.'
How many factual/groundedness errors are in the AI's sentence?
Zero—it is a reasonable paraphrase.
One—it changed 6 months to an unspecified duration.
Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
Three—420 patients is also wrong.
Show the answer
C. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
Fact-checking AI summaries requires catching both direct contradictions ('zero side effects' vs. 8% dizziness) and unsupported overclaims ('cures hypertension' vs. 12% BP reduction).
Principle: Factuality & Groundedness: Overclaiming vs. Contradiction
Prompt: 'List 5 European capitals in alphabetical order. Output ONLY the city names, separated by commas, with no extra text.'
Model output: 'Here are 5 European capitals in alphabetical order: Amsterdam, Berlin, Lisbon, Madrid, Vienna.'
Did the model follow all instructions?
Yes, all 5 are European capitals and in alphabetical order.
No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
No, because Vienna is not in Europe.
Yes, because polite lead-ins are excluded from 'extra text' rules.
Show the answer
B. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
Strict output-only constraints forbid conversational lead-ins and sign-offs. Including 'Here are...' is one of the most common instruction-following failures tested in AI trainer assessments.