Starter assessment: A/B evaluation & fact-checking: practice drill
10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Have a live AI voice or video interview too?
Platforms like Mercor and Alignerr also screen candidates with an automated AI interviewer. Rehearse out loud with a voiced AI interviewer and get an instant scorecard before the real call.
Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.
Sample scenarios
Prompt: 'Summarize why sleep aids memory in 2 sentences, without mentioning REM.' Model A mentions REM; Model B stays accurate without it. Which wins?
Model A, if its facts are accurate
Model B, because Model A broke the negative constraint
Tie, since both are 2 sentences
Neither, because sleep does not aid memory
Show the answer
B. Model B, because Model A broke the negative constraint
In RLHF grading, breaking an explicit negative constraint ('without mentioning REM') is an automatic failure. Model B followed every rule accurately.
Which of the following AI-generated citations is the strongest red flag for a fabricated ('hallucinated') academic reference that you must verify before approving?
A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.
A textbook published in 2018 with an ISBN that resolves to the exact book.
A government CDC webpage whose URL resolves to the matching topic.
An RFC number (e.g., RFC 9110) that matches HTTP Semantics.
Show the answer
A. A paper with a real author's name, a plausible-sounding title, a real journal name, and a DOI that returns a 404 or points to an unrelated chemistry paper.
LLMs frequently blend real author names and real journals into plausible-looking fake citations with non-existent or mismatched DOIs. Always verify DOIs, URLs, and case numbers.
Principle: Fact-Checking: Citation & DOI Verification
Prompt: 'List 5 European capitals in alphabetical order. Output ONLY the city names, separated by commas, with no extra text.'
Model output: 'Here are 5 European capitals in alphabetical order: Amsterdam, Berlin, Lisbon, Madrid, Vienna.'
Did the model follow all instructions?
Yes, all 5 are European capitals and in alphabetical order.
No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
No, because Vienna is not in Europe.
Yes, because polite lead-ins are excluded from 'extra text' rules.
Show the answer
B. No, it violated the 'Output ONLY the city names... with no extra text' constraint by adding 'Here are 5 European capitals in alphabetical order:'.
Strict output-only constraints forbid conversational lead-ins and sign-offs. Including 'Here are...' is one of the most common instruction-following failures tested in AI trainer assessments.