Why RLHF Still Needs Humans in the Loop

Last quarter, a Series B startup came to us after spending four months on a pure RLAIF pipeline. Their model was fluent, confident — and completely sycophantic. It agreed with everything, hedged on nothing, and users hated it within a week of beta launch. The automated preference signal had optimized for "sounds agreeable" because that's what the AI judge rewarded.

They needed humans. Specifically, they needed experts.

Automated alignment works for what's measurable: code that compiles, math that verifies, facts you can check against a database. Constitutional AI and RLAIF have pushed this boundary meaningfully.

But here's what they can't do: tell you which of two technically correct solutions is better in context. That requires taste, experience, and professional judgment.

We've run preference annotation across dozens of projects now. The pattern is consistent:

• Code generation: Two solutions pass all tests. One is idiomatic Go with proper error handling. The other works but would get rejected in any production code review. Automated metrics see them as identical.
• Sensitive topics: Both responses are "safe." One demonstrates genuine nuance. The other is so over-cautious it's useless. Safety classifiers can't distinguish these.
• Technical writing: Both are accurate. One builds the reader's intuition. The other is a wall of facts. No loss function captures "teaches well."

Not all human feedback is equal. We've measured this directly (and wrote about the full comparison in Crowdsourcing vs. Expert Networks):

• Crowd annotators are fast and cheap. They catch obvious errors. But on tasks requiring professional judgment — code quality, medical accuracy, legal nuance — their agreement rates drop below 60%.
• Domain experts cost more per label. But the signal-to-noise ratio is 3-5x higher. For frontier models already operating above crowd-level capability, experts are the only useful signal source.

The cost calculus flips when you account for re-annotation, noisy training signal, and additional compute burned on confused gradients.

After building preference pipelines for multiple frontier labs, here's what we've converged on:

• Automated filters handle the baseline: compilation checks, safety classifiers, factual verification
• Expert annotators provide the nuanced preference signal that actually moves the needle on model quality
• Continuous calibration between the model's current capability and annotator expectations — because as the model improves, easy comparisons become useless

The key insight we share with teams building reward models: your preference data needs to stay at the frontier of difficulty. Easy pairs teach nothing.

As models get better, the bar for useful feedback rises. You need people who are genuinely better than the model at the task. This is why we maintain networks of senior engineers, researchers, and domain specialists across 40+ specializations.

The future of RLHF isn't less human involvement. It's more targeted, higher-quality involvement from people who can teach models things they can't learn from themselves.