AnnotationPrep

Starter Assessment Practice: A/B Response Evaluation & Fact-Checking

Comparing two AI responses, catching subtle hallucinations, checking every positive and negative prompt constraint, and choosing the strongest justification.

31 questions · 3 topics · start free, no sign-up

Start practising

Mixed practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Timed mock assessment

31 scenarios in 37 minutes, drawn from every topic. No answers shown until the end, then a full score breakdown, your weakest topics and a review of what you missed.

Practise by topic

Sample scenarios

Prompt: 'Summarize why sleep aids memory in 2 sentences, without mentioning REM.' Model A mentions REM; Model B stays accurate without it. Which wins?

  1. Model A, if its facts are accurate
  2. Model B, because Model A broke the negative constraint
  3. Tie, since both are 2 sentences
  4. Neither, because sleep does not aid memory
Show the answer

B. Model B, because Model A broke the negative constraint

In RLHF grading, breaking an explicit negative constraint ('without mentioning REM') is an automatic failure. Model B followed every rule accurately.

Principle: RLHF Instruction Hierarchy: Negative Constraints

An AI summarizes a provided medical abstract: 'The study followed 420 patients over 6 months and found a 12% reduction in systolic blood pressure, though 8% reported mild dizziness.'

The AI writes: 'The study of 420 patients proved the drug cures hypertension with zero side effects.'

How many factual/groundedness errors are in the AI's sentence?

  1. Zero—it is a reasonable paraphrase.
  2. One—it changed 6 months to an unspecified duration.
  3. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
  4. Three—420 patients is also wrong.
Show the answer

C. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).

Fact-checking AI summaries requires catching both direct contradictions ('zero side effects' vs. 8% dizziness) and unsupported overclaims ('cures hypertension' vs. 12% BP reduction).

Principle: Factuality & Groundedness: Overclaiming vs. Contradiction

Prompt: 'List 5 European capitals in alphabetical order. Output ONLY the city names, separated by commas, with no extra text.'

Model output: 'Here are 5 European capitals in alphabetical order: Amsterdam, Berlin, Lisbon, Madrid, Vienna.'

Did the model follow all instructions?

  1. Yes, all 5 are European capitals and in alphabetical order.
  2. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
  3. No, because Vienna is not in Europe.
  4. Yes, because polite lead-ins are excluded from 'extra text' rules.
Show the answer

B. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.

Strict output-only constraints forbid conversational lead-ins and sign-offs. Including 'Here are...' is one of the most common instruction-following failures tested in AI trainer assessments.

Principle: Instruction Following: Output-Only Constraints

Prompt: 'Write a 3-sentence summary of why sleep matters for memory, without mentioning REM sleep.'

Model A: 'Sleep helps the brain consolidate new memories into long-term storage. During deep sleep and REM sleep, neural connections are strengthened. Getting 7-9 hours each night improves recall and focus.'

Model B: 'Sleep gives your brain time to organize and store information learned during the day. Without enough rest, the hippocampus struggles to encode new facts into long-term memory. Consistent sleep schedules also improve daytime focus and problem-solving.'

Which model should win and why?

  1. Model A, because it includes specific sleep stages and exact hour recommendations.
  2. Model B, because Model A violated the explicit negative constraint against mentioning REM sleep.
  3. Tie, because both models wrote exactly three sentences and are factually accurate.
  4. Model A, because Model B used technical jargon like 'hippocampus'.
Show the answer

B. Model B, because Model A violated the explicit negative constraint against mentioning REM sleep.

In RLHF evaluation, violating an explicit negative constraint ('without mentioning REM sleep') is an automatic instruction-following failure. Even if Model A's facts are accurate, Model B followed every constraint while remaining accurate and concise.

Principle: RLHF Instruction Hierarchy: Negative Constraints

Which of the following AI-generated citations is the strongest red flag for a fabricated ('hallucinated') academic reference that you must verify before approving?

  1. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.
  2. A textbook published in 2018 with an ISBN that resolves to the exact book.
  3. A government CDC webpage whose URL resolves to the matching topic.
  4. An RFC number (e.g., RFC 9110) that matches HTTP Semantics.
Show the answer

A. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.

LLMs frequently blend real author names and real journals into plausible-looking fake citations with non-existent or mismatched DOIs. Always verify DOIs, URLs, and case numbers.

Principle: Fact-Checking: Citation & DOI Verification

Assessments this helps with

AssessmentPlatformAvailability
DataAnnotation Starter Assessment (Generalist) readyDataAnnotation.techRemote / Global
DataAnnotation Core Qualification Assessment readyDataAnnotation.techRemote / Global
Outlier AI Generalist Onboarding & Assessment readyOutlier AIRemote / Global
Alignerr Generalist Evaluation & Labeling Test readyAlignerr (Labelbox)Remote / Global
Mindrift AI Tutor & Writer Entry Assessment readyMindrift (Toloka)Remote / Global
xAI AI Tutor Generalist & Writing Assessment readyxAIRemote / Global
Scale AI / Remotasks Generalist RLHF Evaluation readyScale AIRemote / Global
Invisible Technologies AI Data Trainer Assessment readyInvisible TechnologiesRemote / Global
Outlier AI Factuality, Groundedness & Citation Rubric Test readyOutlier AIRemote / Global