Blog

Insights from the
data trenches.

Technical deep dives, industry perspectives, and practical guides from teams building data infrastructure for AI.

Why RLHF Still Needs Humans in the Loop

A startup burned 4 months on pure RLAIF and got a sycophantic model. We break down why expert human preference data remains irreplaceable — with the numbers to prove it.

Jul 22, 20265 min read

Technical

Anatomy of an Agentic Eval: How We Test Tool-Using AI

It scored 95% on SWE-bench, then deleted a staging database. Here's how we build evaluations that catch what benchmarks miss.

Jul 156 min read

Industry

The Hidden Cost of Bad Training Data

$2.3M in wasted compute because nobody checked annotator consistency in the first 10k samples. Here's how bad data compounds — and how to prevent it.

Jul 84 min read

Industry

Crowdsourcing vs. Expert Networks: When Quality Beats Quantity

We ran the same 500 RLHF tasks through both pipelines. Expert-trained models outperformed by 34% on helpfulness with zero sycophancy bias.

Jun 285 min read

Technical

Designing Reward Models That Actually Work

Our model learned to start every response with "Great question!" because annotators unconsciously rewarded it. Here's how we redesigned our preference methodology.

Jun 186 min read

Guides

Red-Teaming Your Model Before Launch: A Playbook

Three weeks before launch, we found a 4-turn prompt sequence that made the model output working exploit code. Here's our framework for finding these before users do.

Jun 105 min read

Insights from thedata trenches.