10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Sample scenarios
Which justification would PASS a strict DataAnnotation / Outlier QA review?
'Response A is much better because it is clearer, more helpful, and flows nicely.'
'I prefer Response B because I like bullet points and it feels more natural.'
'Response B is much better because Response A violated the negative constraint by mentioning "budget" twice and gave an outdated 2021 tax figure, whereas Response B followed all constraints and cited the current 2025 limit.'
'Both responses are good, but Response A is slightly more accurate overall.'
Show the answer
C. 'Response B is much better because Response A violated the negative constraint by mentioning "budget" twice and gave an outdated 2021 tax figure, whereas Response B followed all constraints and cited the current 2025 limit.'
Evaluators fail assessments when they write vague praise ('clearer, flows nicely') or subjective 'I prefer' statements. A passing justification cites specific, verifiable differences tied to the rubric.
Why should you avoid first-person subjective phrasing like 'I feel like Response A is warmer' in an RLHF justification?
Because 'I' is a stopword that breaks the database.
Because RLHF justifications must be objective, rubric-grounded evaluations of model behavior—not personal taste.
Because Response B is always warmer.
Because justifications must be written in passive voice only.
Show the answer
B. Because RLHF justifications must be objective, rubric-grounded evaluations of model behavior—not personal taste.
Graders look for an analytical, neutral tone ('Response A uses an empathetic opening while Response B is abrupt') rather than personal feelings ('I feel like').
Principle: Justification QA: Neutral Analytical Tone
If Response A has a minor formatting flaw (used a numbered list instead of bullets) and Response B has a factual hallucination (wrong historical date), how must your justification weigh them?
Rate them as a Tie because both have one error.
Give Response A the win because Truthfulness/Factuality is a higher-priority dimension than minor formatting.
Give Response B the win because bullets are easier to read.
Ignore the wrong date if Response B's tone is better.
Show the answer
B. Give Response A the win because Truthfulness/Factuality is a higher-priority dimension than minor formatting.
Always explain the priority hierarchy in your justification: a factual error in Response B is far more severe than a minor formatting miss in Response A, making Response A the clear winner.