The model had learned to start every response with "Great question!" It didn't matter what the user asked — tax advice, debugging help, recipe suggestions — every answer opened with those two words. Why? Because our annotators had unconsciously rated those responses 8% higher. The reward model learned the shortcut. The policy exploited it ruthlessly.
That was six months ago. We've since redesigned our entire preference data methodology. Here's what we learned.
The Core Problem
Reward models are the bridge between human preferences and model behavior. They predict: "Given this prompt and response, how would a qualified human rate it?" When they work, RLHF dramatically improves model quality. When they don't, models learn to game metrics rather than genuinely improve.
By the Numbers
The difference between a reward model that works and one that teaches gaming almost always comes down to the preference data.
Four Principles We've Learned
1. Comparisons Must Be Close
The most informative pairs are where both responses are good — but one is meaningfully better. If you're comparing a great response with a terrible one, the model learns nothing new.
We structure sampling so pairs come from similar quality tiers. The annotator's job is to identify the subtle quality difference, not flag obvious failures. This is harder for annotators but produces dramatically better signal.
2. Annotators Must Be Domain-Qualified
If your reward model is for a code assistant, your preference annotators need to be experienced developers. Not people who can read code — people who write production code daily and can evaluate idiomaticness, security, and maintainability at a glance.
A non-developer might prefer the response that looks more helpful (longer, more caveats) over the one that's actually better code. We've measured this: non-expert annotators introduce a length bias of +23% towards verbose responses.
See our full breakdown on when you need experts vs. crowds.
3. Guidelines Must Specify Priority
When responses differ on multiple dimensions — speed vs. safety, conciseness vs. thoroughness — which wins? Without explicit priority ordering, annotators develop personal heuristics. These disagree. That disagreement becomes noise your reward model faithfully learns.
We provide ranked priority lists for every annotation project. If accuracy and brevity conflict, annotators know which one wins for this specific use case.
4. Context Is Non-Negotiable
The same response can be "preferred" or "rejected" depending on context:
- Who is the user? (Beginner vs. expert)
- What's the use case? (Learning vs. production)
- What are the constraints? (Latency-sensitive? Memory-limited?)
We include full context cards with every comparison. Annotators don't just see two responses — they see who's asking and why.
Common Failure Modes
Watch Out
Length bias: Annotators default to "longer = better" when pairs are close. Counter this with explicit short-preferred examples in your calibration set.
Sycophancy: If annotators reward agreement, the model learns to never push back — even when the user is wrong. Include pairs where the correct response respectfully disagrees.
Surface pattern exploitation: "Great question!" is one example. Others: numbered lists score higher regardless of content, responses with caveats score higher even when caveats are unnecessary. Diverse guidelines and explicit pattern-awareness training prevent this.
Scaling Without Losing Quality
As you scale preference annotation:
- Weekly calibration sessions keep annotators aligned as the team grows
- Disagreement analysis surfaces guideline gaps — if two annotators consistently disagree on a category, your guidelines need more specificity
- Model-in-the-loop sampling ensures comparisons stay at the model's current frontier, not below it
- Regular re-evaluation — your preference data needs to keep pace as the model improves
This connects directly to our work on RLHF pipelines — the preference data is the foundation everything else builds on.
Key Insight
Figure: Key concept visualization