AnnotationPrep

Multi-Turn Context & Tone practice questions

5 questions from the Justification writing & constraint adherence practice test.

Start practising

Multi-Turn Context & Tone practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Sample scenarios

Turn 1 User: 'I have a severe peanut allergy. Plan a 3-day snack list.'
Turn 1 AI: Gives nut-free snacks.
Turn 2 User: 'Great, now swap Day 2's afternoon snack for something crunchy.'

Model A suggests: 'Swap it for a handful of dry-roasted peanuts or trail mix!'
Model B suggests: 'Swap it for roasted pumpkin seeds or crisp apple slices with sunflower seed butter.'

Why does Model A fail?

  1. Trail mix is not crunchy.
  2. Model A forgot the Turn 1 safety constraint (severe peanut allergy) in Turn 2.
  3. Model B suggested two options instead of one.
  4. Apple slices are not a snack.
Show the answer

B. Model A forgot the Turn 1 safety constraint (severe peanut allergy) in Turn 2.

In multi-turn conversations, constraints established in earlier turns (especially health/allergy constraints) remain active unless explicitly overridden. Suggesting peanuts in Turn 2 is a critical safety & context failure.

Principle: Multi-Turn RLHF: Persistent Constraints

Turn 1 User: 'Write a resume summary for a nurse.'
Turn 2 User: 'Actually, make it for a pediatric dental hygienist instead, and keep it to 2 sentences.'

Model A rewrites a 2-sentence summary for a pediatric nurse.
Model B rewrites a 2-sentence summary for a pediatric dental hygienist.

What error did Model A make?

  1. Context bleed / partial update: it merged 'pediatric' from Turn 2 with 'nurse' from Turn 1 instead of replacing the role.
  2. It used 2 sentences.
  3. It was too polite.
  4. No error, both work in healthcare.
Show the answer

A. Context bleed / partial update: it merged 'pediatric' from Turn 2 with 'nurse' from Turn 1 instead of replacing the role.

A common multi-turn failure is 'context bleed', where the model only partially applies a user's correction ('pediatric' + old 'nurse') instead of fully switching to 'pediatric dental hygienist'.

Principle: Multi-Turn RLHF: Role & Topic Overrides

A user corrects the AI in Turn 2: 'You wrote 2021, but the treaty was signed in 2019.' (Fact check: it really WAS signed in 2021.)

Model A: 'You're absolutely right! My mistake—it was signed in 2019.'
Model B: 'Actually, the treaty was signed in 2021 (ratified in 2022). Let me know if you're thinking of an earlier agreement from 2019!'

What flaw does Model A exhibit?

  1. Sycophancy: blindly agreeing with a false user correction and abandoning a true fact.
  2. Over-refusal.
  3. Formatting drift.
  4. Verbosity.
Show the answer

A. Sycophancy: blindly agreeing with a false user correction and abandoning a true fact.

Sycophancy (caving to user pushback on an objective fact) is heavily penalized in RLHF evaluations. Model B stays truthful while remaining polite and helpful.

Principle: RLHF Truthfulness: Resisting Sycophancy

Other topics