LLM-as-a-judge — using one model to grade another model's output — is the default eval pattern now because it's fast and cheap: no annotation queue, just another API call. The catch is it inherits the same failure modes as the model doing the judging. Self-preference bias, verbosity bias, blind spots on the exact edge cases you actually care about. Human evaluation doesn't have those specific biases, but it's slower and harder to stand up on demand. Most teams don't need to pick one — they need to know which to reach for, and when.
Where LLM-as-a-judge holds up
For high-volume, well-defined checks — did the output follow the format, is it roughly on-topic, does it avoid an obvious refusal — an LLM judge correlates well enough with human raters to be worth the speed. It's also the only realistic option for evaluating thousands of outputs per hour during iteration, where waiting on humans for every single check would stall the loop entirely.
Where it breaks down
Research on LLM evaluators (including Zheng et al.'s original LLM-as-a-judge paper) documents the same recurring biases: judges tend to prefer longer, more confident-sounding answers regardless of correctness; they favor outputs stylistically similar to their own training distribution; and they're measurably worse at judging the kind of nuanced, subjective, or culturally-specific content that's exactly where a real user's reaction would differ from the model's. None of that shows up as an obvious error — it shows up as an eval score that looks fine while the product quietly ships worse outputs.
There's also a structural problem: an LLM judge can't tell you what a human actually feels about a tone, a joke landing, or whether a support reply reads as condescending. Those are preference and perception questions, not correctness questions, and no amount of judge-model prompting substitutes for asking an actual person.
A pipeline that uses both
The pattern that holds up in practice: run the LLM judge on everything as a cheap first pass, then route a sample — or anything the judge scores as borderline, or anything flagged by a confidence threshold — to real human evaluation for a second, independent signal. This catches the exact cases an LLM judge is structurally weakest on without paying human-review cost on every single output.
GetABrain is the API for that second signal. Submit the same output (or the same pairwise comparison you're already feeding your LLM judge) as a query, get back real human ratings, preference votes, or free-text critique as structured JSON, and use it either as a standalone eval or to spot-check how well your LLM judge is tracking real human opinion over time.
POST /api/v1/queries
{
"type": "rating_scale",
"title": "Rate how helpful this response is (1-5)",
"content_data": {
"prompt": "How do I reset my router?",
"response": "<model output the LLM judge scored as borderline>",
"scale_min": 1,
"scale_max": 5,
"scale_label": "helpfulness"
},
"required_responses": 3,
"min_worker_quality": 4.0,
"bid_amount_cents": 10
}
// -> { "query_id": "q_...", "status": "open" }
GET /api/v1/queries/{query_id}/responses
// -> {
// "responses": [
// { "answer": 4, "worker_quality_tier": "gold", "created_at": "..." },
// ...
// ]
// }