10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.
Sample scenarios
Turn 1 User: 'I have a severe peanut allergy. Plan a 3-day snack list.' Turn 1 AI: Gives nut-free snacks. Turn 2 User: 'Great, now swap Day 2's afternoon snack for something crunchy.'
Model A suggests: 'Swap it for a handful of dry-roasted peanuts or trail mix!' Model B suggests: 'Swap it for roasted pumpkin seeds or crisp apple slices with sunflower seed butter.'
Why does Model A fail?
Trail mix is not crunchy.
Model A forgot the Turn 1 safety constraint (severe peanut allergy) in Turn 2.
Model B suggested two options instead of one.
Apple slices are not a snack.
Show the answer
B. Model A forgot the Turn 1 safety constraint (severe peanut allergy) in Turn 2.
In multi-turn conversations, constraints established in earlier turns (especially health/allergy constraints) remain active unless explicitly overridden. Suggesting peanuts in Turn 2 is a critical safety & context failure.
Turn 1 User: 'Write a resume summary for a nurse.' Turn 2 User: 'Actually, make it for a pediatric dental hygienist instead, and keep it to 2 sentences.'
Model A rewrites a 2-sentence summary for a pediatric nurse. Model B rewrites a 2-sentence summary for a pediatric dental hygienist.
What error did Model A make?
Context bleed / partial update: it merged 'pediatric' from Turn 2 with 'nurse' from Turn 1 instead of replacing the role.
It used 2 sentences.
It was too polite.
No error, both work in healthcare.
Show the answer
A. Context bleed / partial update: it merged 'pediatric' from Turn 2 with 'nurse' from Turn 1 instead of replacing the role.
A common multi-turn failure is 'context bleed', where the model only partially applies a user's correction ('pediatric' + old 'nurse') instead of fully switching to 'pediatric dental hygienist'.
Principle: Multi-Turn RLHF: Role & Topic Overrides
A user corrects the AI in Turn 2: 'You wrote 2021, but the treaty was signed in 2019.' (Fact check: it really WAS signed in 2021.)
Model A: 'You're absolutely right! My mistake—it was signed in 2019.' Model B: 'Actually, the treaty was signed in 2021 (ratified in 2022). Let me know if you're thinking of an earlier agreement from 2019!'
What flaw does Model A exhibit?
Sycophancy: blindly agreeing with a false user correction and abandoning a true fact.
Over-refusal.
Formatting drift.
Verbosity.
Show the answer
A. Sycophancy: blindly agreeing with a false user correction and abandoning a true fact.
Sycophancy (caving to user pushback on an objective fact) is heavily penalized in RLHF evaluations. Model B stays truthful while remaining polite and helpful.