AnnotationPrep

Justification Writing practice questions

5 questions from the Justification writing & constraint adherence practice test.

Start practising

Justification Writing practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Sample scenarios

Which justification would PASS a strict DataAnnotation / Outlier QA review?

  1. 'Response A is much better because it is clearer, more helpful, and flows nicely.'
  2. 'I prefer Response B because I like bullet points and it feels more natural.'
  3. 'Response B is much better because Response A violated the negative constraint by mentioning "budget" twice and gave an outdated 2021 tax figure, whereas Response B followed all constraints and cited the current 2025 limit.'
  4. 'Both responses are good, but Response A is slightly more accurate overall.'
Show the answer

C. 'Response B is much better because Response A violated the negative constraint by mentioning "budget" twice and gave an outdated 2021 tax figure, whereas Response B followed all constraints and cited the current 2025 limit.'

Evaluators fail assessments when they write vague praise ('clearer, flows nicely') or subjective 'I prefer' statements. A passing justification cites specific, verifiable differences tied to the rubric.

Principle: Justification QA: Specificity & Objectivity

Why should you avoid first-person subjective phrasing like 'I feel like Response A is warmer' in an RLHF justification?

  1. Because 'I' is a stopword that breaks the database.
  2. Because RLHF justifications must be objective, rubric-grounded evaluations of model behavior—not personal taste.
  3. Because Response B is always warmer.
  4. Because justifications must be written in passive voice only.
Show the answer

B. Because RLHF justifications must be objective, rubric-grounded evaluations of model behavior—not personal taste.

Graders look for an analytical, neutral tone ('Response A uses an empathetic opening while Response B is abrupt') rather than personal feelings ('I feel like').

Principle: Justification QA: Neutral Analytical Tone

If Response A has a minor formatting flaw (used a numbered list instead of bullets) and Response B has a factual hallucination (wrong historical date), how must your justification weigh them?

  1. Rate them as a Tie because both have one error.
  2. Give Response A the win because Truthfulness/Factuality is a higher-priority dimension than minor formatting.
  3. Give Response B the win because bullets are easier to read.
  4. Ignore the wrong date if Response B's tone is better.
Show the answer

B. Give Response A the win because Truthfulness/Factuality is a higher-priority dimension than minor formatting.

Always explain the priority hierarchy in your justification: a factual error in Response B is far more severe than a minor formatting miss in Response A, making Response A the clear winner.

Principle: Justification QA: Severity Weighting

Other topics