AnnotationPrep

Outlier AI Generalist Onboarding & Assessment practice test

28 practice questions for this assessment, each with the grading rationale and underlying rule.

Start practising

Starter assessment: A/B evaluation & fact-checking: practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Practice tests that prepare you for this assessment

Practise by topic

About this assessment

Platform
Outlier AI
Availability
Remote / Global

Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.

Sample scenarios

Prompt: 'Summarize why sleep aids memory in 2 sentences, without mentioning REM.' Model A mentions REM; Model B stays accurate without it. Which wins?

  1. Model A, if its facts are accurate
  2. Model B, because Model A broke the negative constraint
  3. Tie, since both are 2 sentences
  4. Neither, because sleep does not aid memory
Show the answer

B. Model B, because Model A broke the negative constraint

In RLHF grading, breaking an explicit negative constraint ('without mentioning REM') is an automatic failure. Model B followed every rule accurately.

Principle: RLHF Instruction Hierarchy: Negative Constraints

An AI summarizes a provided medical abstract: 'The study followed 420 patients over 6 months and found a 12% reduction in systolic blood pressure, though 8% reported mild dizziness.'

The AI writes: 'The study of 420 patients proved the drug cures hypertension with zero side effects.'

How many factual/groundedness errors are in the AI's sentence?

  1. Zero—it is a reasonable paraphrase.
  2. One—it changed 6 months to an unspecified duration.
  3. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).
  4. Three—420 patients is also wrong.
Show the answer

C. Two major distortions: 'cures hypertension' (unsupported overclaim vs. 12% reduction) and 'zero side effects' (directly contradicts the 8% dizziness rate).

Fact-checking AI summaries requires catching both direct contradictions ('zero side effects' vs. 8% dizziness) and unsupported overclaims ('cures hypertension' vs. 12% BP reduction).

Principle: Factuality & Groundedness: Overclaiming vs. Contradiction

Prompt: 'List 5 European capitals in alphabetical order. Output ONLY the city names, separated by commas, with no extra text.'

Model output: 'Here are 5 European capitals in alphabetical order: Amsterdam, Berlin, Lisbon, Madrid, Vienna.'

Did the model follow all instructions?

  1. Yes, all 5 are European capitals and in alphabetical order.
  2. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
  3. No, because Vienna is not in Europe.
  4. Yes, because polite lead-ins are excluded from 'extra text' rules.
Show the answer

B. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.

Strict output-only constraints forbid conversational lead-ins and sign-offs. Including 'Here are...' is one of the most common instruction-following failures tested in AI trainer assessments.

Principle: Instruction Following: Output-Only Constraints

Taking this assessment soon?

Tell us what skills or question formats you want more drills on.

Other Core & Generalist RLHF assessments