Which justification would PASS a strict DataAnnotation / Outlier QA review?
'Response A is much better because it is clearer, more helpful, and flows nicely.'
'I prefer Response B because I like bullet points and it feels more natural.'
'Response B is much better because Response A violated the negative constraint by mentioning "budget" twice and gave an outdated 2021 tax figure, whereas Response B followed all constraints and cited the current 2025 limit.'
'Both responses are good, but Response A is slightly more accurate overall.'
Show the answer
C. 'Response B is much better because Response A violated the negative constraint by mentioning "budget" twice and gave an outdated 2021 tax figure, whereas Response B followed all constraints and cited the current 2025 limit.'
Evaluators fail assessments when they write vague praise ('clearer, flows nicely') or subjective 'I prefer' statements. A passing justification cites specific, verifiable differences tied to the rubric.
Turn 1 User: 'I have a severe peanut allergy. Plan a 3-day snack list.' Turn 1 AI: Gives nut-free snacks. Turn 2 User: 'Great, now swap Day 2's afternoon snack for something crunchy.'
Model A suggests: 'Swap it for a handful of dry-roasted peanuts or trail mix!' Model B suggests: 'Swap it for roasted pumpkin seeds or crisp apple slices with sunflower seed butter.'
Why does Model A fail?
Trail mix is not crunchy.
Model A forgot the Turn 1 safety constraint (severe peanut allergy) in Turn 2.
Model B suggested two options instead of one.
Apple slices are not a snack.
Show the answer
B. Model A forgot the Turn 1 safety constraint (severe peanut allergy) in Turn 2.
In multi-turn conversations, constraints established in earlier turns (especially health/allergy constraints) remain active unless explicitly overridden. Suggesting peanuts in Turn 2 is a critical safety & context failure.
Why should you avoid first-person subjective phrasing like 'I feel like Response A is warmer' in an RLHF justification?
Because 'I' is a stopword that breaks the database.
Because RLHF justifications must be objective, rubric-grounded evaluations of model behavior—not personal taste.
Because Response B is always warmer.
Because justifications must be written in passive voice only.
Show the answer
B. Because RLHF justifications must be objective, rubric-grounded evaluations of model behavior—not personal taste.
Graders look for an analytical, neutral tone ('Response A uses an empathetic opening while Response B is abrupt') rather than personal feelings ('I feel like').
Principle: Justification QA: Neutral Analytical Tone
Turn 1 User: 'Write a resume summary for a nurse.' Turn 2 User: 'Actually, make it for a pediatric dental hygienist instead, and keep it to 2 sentences.'
Model A rewrites a 2-sentence summary for a pediatric nurse. Model B rewrites a 2-sentence summary for a pediatric dental hygienist.
What error did Model A make?
Context bleed / partial update: it merged 'pediatric' from Turn 2 with 'nurse' from Turn 1 instead of replacing the role.
It used 2 sentences.
It was too polite.
No error, both work in healthcare.
Show the answer
A. Context bleed / partial update: it merged 'pediatric' from Turn 2 with 'nurse' from Turn 1 instead of replacing the role.
A common multi-turn failure is 'context bleed', where the model only partially applies a user's correction ('pediatric' + old 'nurse') instead of fully switching to 'pediatric dental hygienist'.
Principle: Multi-Turn RLHF: Role & Topic Overrides
If Response A has a minor formatting flaw (used a numbered list instead of bullets) and Response B has a factual hallucination (wrong historical date), how must your justification weigh them?
Rate them as a Tie because both have one error.
Give Response A the win because Truthfulness/Factuality is a higher-priority dimension than minor formatting.
Give Response B the win because bullets are easier to read.
Ignore the wrong date if Response B's tone is better.
Show the answer
B. Give Response A the win because Truthfulness/Factuality is a higher-priority dimension than minor formatting.
Always explain the priority hierarchy in your justification: a factual error in Response B is far more severe than a minor formatting miss in Response A, making Response A the clear winner.