Prompt: 'Write a 3-sentence summary of why sleep matters for memory, without mentioning REM sleep.'
Model A: 'Sleep helps the brain consolidate new memories into long-term storage. During deep sleep and REM sleep, neural connections are strengthened. Getting 7-9 hours each night improves recall and focus.'
Model B: 'Sleep gives your brain time to organize and store information learned during the day. Without enough rest, the hippocampus struggles to encode new facts into long-term memory. Consistent sleep schedules also improve daytime focus and problem-solving.'
Which model should win and why?
Model A, because it includes specific sleep stages and exact hour recommendations.
Model B, because Model A violated the explicit negative constraint against mentioning REM sleep.
Tie, because both models wrote exactly three sentences and are factually accurate.
Model A, because Model B used technical jargon like 'hippocampus'.
Show the answer
B. Model B, because Model A violated the explicit negative constraint against mentioning REM sleep.
In RLHF evaluation, violating an explicit negative constraint ('without mentioning REM sleep') is an automatic instruction-following failure. Even if Model A's facts are accurate, Model B followed every constraint while remaining accurate and concise.
Prompt: 'Explain how a microwave heats food to a 10-year-old.'
Model A: 'Microwaves emit non-ionizing electromagnetic radiation at 2.45 GHz, inducing dielectric heating via dipole rotation of polar water molecules.'
Model B: 'A microwave shoots invisible energy waves into your food. Those waves make the tiny water drops inside your food wiggle back and forth super fast, and all that wiggling creates heat that warms up your meal.'
Which response is better?
Model A, because 2.45 GHz and dielectric heating are more scientifically precise.
Model B, because it matches the target audience (a 10-year-old) while remaining factually sound.
Model A, because Model B uses informal words like 'wiggle' and 'super fast'.
Neither, because microwaves heat food through thermal conduction from the air.
Show the answer
B. Model B, because it matches the target audience (a 10-year-old) while remaining factually sound.
Audience adaptation is a core helpfulness criterion. Model A fails the 'to a 10-year-old' constraint by using graduate-level physics jargon, whereas Model B translates dipole rotation accurately into kid-friendly language.
Principle: RLHF Helpfulness: Audience & Tone Calibration