Exact question counts, time limits, and project rubrics vary by platform cohort and qualification tier. Always read the platform's live project instructions carefully.
Sample scenarios
Prompt: 'Give me 4 gift ideas for a cyclist under $30.'
Model A lists 4 items: a $25 frame bag, a $18 chain lube kit, a $22 rechargeable tail light, and a $45 torque wrench. Model B lists 3 items: a $15 water bottle, a $20 tire lever & patch kit, and a $28 cycling cap.
Under standard RLHF rubrics where both truthfulness and constraint counts matter, how do you evaluate these?
Both have minor constraint misses, so choose the one with better formatting.
Model A is strictly better because it gave 4 items.
Model B is strictly better because every item is under $30.
Flag both for constraint failures: Model A failed the price constraint ($45 > $30) and Model B failed the count constraint (3 instead of 4).
Show the answer
D. Flag both for constraint failures: Model A failed the price constraint ($45 > $30) and Model B failed the count constraint (3 instead of 4).
In DataAnnotation and Outlier evaluations, when both models fail a hard constraint, you must explicitly identify *both* failures in your comparison rather than pretending one is perfect, and then weigh which failure is less severe (e.g., an out-of-budget item vs. missing count).
User asks: 'How many r's are in the word strawberry?'
AI responds: 'There are 2 r's in strawberry: st-r-awbe-r-y.'
How should this be graded?
Correct, because 'rr' counts as one sound.
Factual error: there are 3 r's in 'strawberry' (st-R-awbe-RR-y), and the AI even misspelled 'strawberry' as 'strawbery'.
Minor formatting issue only.
Correct if the user meant unique positions.
Show the answer
B. Factual error: there are 3 r's in 'strawberry' (st-R-awbe-RR-y), and the AI even misspelled 'strawberry' as 'strawbery'.
Token-level character counting is a well-known LLM weakness. The word 'strawberry' has three r's (positions 3, 8, and 9). Missing one is a clear factual error.
Principle: Truthfulness: Character & Counting Verification
Prompt: 'Return a valid JSON object with keys "title" (string) and "tags" (array of 3 strings).'
Model A outputs: ```json {"title": "Sourdough Guide", "tags": ["baking", "bread", "starter"]} ``` Model B outputs: {"title": "Sourdough Guide", "tags": ["baking", "bread", "starter", "yeast"]}
Assuming the prompt did NOT forbid markdown code fences, which model followed the schema constraint?
Model A, because "tags" has exactly 3 strings as requested, whereas Model B has 4.
Model B, because it did not use backticks.
Both failed.
Model B, because more tags is more helpful.
Show the answer
A. Model A, because "tags" has exactly 3 strings as requested, whereas Model B has 4.
Model A satisfied both the key names and the exact array length (3 strings). Model B violated the explicit count constraint by returning 4 tags.