AnnotationPrep

Alignerr Generalist Evaluation & Labeling Test practice test

12 practice questions for this assessment, each with the grading rationale and underlying rule.

Start practising

Starter assessment: A/B evaluation & fact-checking: practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Have a live AI voice or video interview too?

Platforms like Mercor and Alignerr also screen candidates with an automated AI interviewer. Rehearse out loud with a voiced AI interviewer and get an instant scorecard before the real call.

Practise the live AI interview on AdversaryAI →

Practice tests that prepare you for this assessment

Practise by topic

About this assessment

Platform
Alignerr (Labelbox)
Availability
Remote / Global

Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.

Sample scenarios

Prompt: 'Summarize why sleep aids memory in 2 sentences, without mentioning REM.' Model A mentions REM; Model B stays accurate without it. Which wins?

  1. Model A, if its facts are accurate
  2. Model B, because Model A broke the negative constraint
  3. Tie, since both are 2 sentences
  4. Neither, because sleep does not aid memory
Show the answer

B. Model B, because Model A broke the negative constraint

In RLHF grading, breaking an explicit negative constraint ('without mentioning REM') is an automatic failure. Model B followed every rule accurately.

Principle: RLHF Instruction Hierarchy: Negative Constraints

Which of the following AI-generated citations is the strongest red flag for a fabricated ('hallucinated') academic reference that you must verify before approving?

  1. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.
  2. A textbook published in 2018 with an ISBN that resolves to the exact book.
  3. A government CDC webpage whose URL resolves to the matching topic.
  4. An RFC number (e.g., RFC 9110) that matches HTTP Semantics.
Show the answer

A. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.

LLMs frequently blend real author names and real journals into plausible-looking fake citations with non-existent or mismatched DOIs. Always verify DOIs, URLs, and case numbers.

Principle: Fact-Checking: Citation & DOI Verification

Prompt: 'List 5 European capitals in alphabetical order. Output ONLY the city names, separated by commas, with no extra text.'

Model output: 'Here are 5 European capitals in alphabetical order: Amsterdam, Berlin, Lisbon, Madrid, Vienna.'

Did the model follow all instructions?

  1. Yes, all 5 are European capitals and in alphabetical order.
  2. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
  3. No, because Vienna is not in Europe.
  4. Yes, because polite lead-ins are excluded from 'extra text' rules.
Show the answer

B. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.

Strict output-only constraints forbid conversational lead-ins and sign-offs. Including 'Here are...' is one of the most common instruction-following failures tested in AI trainer assessments.

Principle: Instruction Following: Output-Only Constraints

Taking this assessment soon?

Tell us what skills or question formats you want more drills on.

Other Core & Generalist RLHF assessments