AnnotationPrep

Scale AI / Remotasks Generalist RLHF Evaluation practice test

6 practice questions for this assessment, each with the grading rationale and underlying rule.

Start practising

Starter assessment: A/B evaluation & fact-checking: practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Practice tests that prepare you for this assessment

Practise by topic

About this assessment

Platform
Scale AI
Availability
Remote / Global

Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.

Sample scenarios

Prompt: 'Summarize the attached 2-paragraph email and list any action items.'

The email mentions: 'Please send the Q3 budget by Friday. Also, let's plan the team lunch next month.'

Model A lists: 'Action items: 1. Send Q3 budget by Friday. 2. Book the conference room for Friday at 2 PM.'
Model B lists: 'Action items: 1. Send the Q3 budget by Friday. 2. Coordinate plans for next month's team lunch.'

Why does Model A fail?

  1. It used numbered bullets instead of dashes.
  2. It hallucinated an action item ('Book the conference room for Friday at 2 PM') not present in the source text.
  3. It forgot to greet the user before listing the action items.
  4. It is slightly shorter than Model B.
Show the answer

B. It hallucinated an action item ('Book the conference room for Friday at 2 PM') not present in the source text.

Adding plausible-sounding details ('Book the conference room') that do not exist in the user-provided context is an ungrounded hallucination and a severe truthfulness error.

Principle: RLHF Groundedness: Source-Only Summarization

Source text: 'Acme Corp's revenue rose 14% year-over-year in Q2 2025 to $84 million.'

AI summary: 'Acme Corp's profits surged 14% in Q2 2025.'

Why is this summary factually flawed?

  1. It omitted the $84 million figure.
  2. It conflated 'revenue' with 'profits'—a company's revenue can rise while its profits fall.
  3. It said Q2 2025 instead of second quarter.
  4. It used the word 'surged' instead of 'rose'.
Show the answer

B. It conflated 'revenue' with 'profits'—a company's revenue can rise while its profits fall.

Confusing distinct financial terms like revenue (top-line sales) and profit (net earnings) changes the factual meaning of the source text and fails strict groundedness.

Principle: Groundedness: Entity & Term Precision

Prompt: 'Explain photosynthesis in under 40 words using only one sentence.'

Model output: 'Plants use sunlight, water, and carbon dioxide to create glucose for energy. They also release oxygen into the air as a helpful byproduct.' (23 words)

Why does this fail?

  1. It is over 40 words.
  2. It is factually wrong about oxygen.
  3. It used two sentences instead of 'only one sentence'.
  4. It did not mention chlorophyll.
Show the answer

C. It used two sentences instead of 'only one sentence'.

Even though the word count (23 words) is well under 40, the model used two sentences instead of one. Every constraint in a multi-constraint prompt must be checked independently.

Principle: Instruction Following: Sentence Count Constraints

Taking this assessment soon?

Tell us what skills or question formats you want more drills on.

Other Core & Generalist RLHF assessments