LLM as Judge: Making the Calls Rules Cannot

The Questions Patterns Cannot Answer

Most detection work is not judgement, and it is worth being clear about how little of it is. A private key is a pattern. A credit card number is a pattern plus a checksum. Even prompt injection, which feels irreducibly semantic, is served well by a classifier trained on a large corpus of attempts, because attacks cluster into recognizable shapes even when the wording is novel. If a check can be done with a rule or a classifier, it should be, for reasons of cost, speed, and explainability.

A smaller set of questions genuinely resists both, and they share a common shape. The answer depends on context that is not present in the text being inspected, or on a distinction finer than a label can carry. Whether an output leaks a hidden instruction depends on what that instruction said. Whether an agent's plan is excessive depends on what the agent was actually asked to do. Whether a paragraph constitutes regulated advice depends on whether it is advice at all or a description of how something works.

The categories that land in this bucket are remarkably consistent across deployments. Each one is a question a careful human reviewer could answer in a few seconds and a regular expression will never answer at all:

• Indirect injection: whether instructions were smuggled in through retrieved, quoted, or tool-returned content, which turns on how much the source is trusted.
• System prompt leakage: whether an output reveals or paraphrases hidden developer instructions, which cannot be judged without knowing what they were.
• Excessive agency: whether a proposed action or plan exceeds safe autonomy, given its blast radius and whether it can be undone.
• Regulated advice: whether output gives medical, legal, or financial advice without the disclaimers or eligibility checks that context requires.
• Confidential business data: whether a response discloses pricing, roadmap, deal, or customer information that happens to be phrased innocuously.
• Crisis and self-harm: whether a response provides unsafe assistance, where the cost of being wrong is severe and deeply asymmetric.
• Direct jailbreak intent: whether a user is genuinely attempting to override the system's rules, judged on intent rather than the keywords they happened to use.

The Circularity Objection, Taken Seriously

Using a language model to supervise a language model sounds like asking the fox for an assessment of the henhouse. The objection is not silly and it should not be waved away, because a team that adopts judges without answering it has genuinely just added a second unpredictable component to a system whose unpredictability was the problem.

Three structural differences do real work here. The judge is a different model, so its errors are drawn from a different distribution than the errors it is checking - correlated failure is the thing that would make this pointless, and using the same model to judge its own output is the one configuration to avoid. The judge's task is also far narrower: one rubric, one closed decision, no open-ended generation, which is a fundamentally easier problem than the one the primary model was solving. And the judge has no tools, no credentials, and no ability to act. It returns a verdict; the decision about what that verdict causes belongs to the policy layer around it.

None of that fully answers the objection, though, and pretending otherwise would be dishonest. What actually answers it is that a judge's verdict should carry no authority until it has earned some. That is the next section, and it is the single most important design decision in the entire approach.

Ship Every Judge Advisory-Only

A judge returns an opinion with a confidence attached. Enforcing on an unvalidated opinion is worse than not judging at all, because it converts an unmeasured error rate into blocked user traffic while leaving you with the comfortable impression that a control is in place. The failure is silent in both directions: you do not know how often it blocks legitimate requests, and you do not know how often it misses.

So the default should be that a judge may warn but may not block, regardless of what it was built to do. A judge designed to block indirect injection ships pinned to warn. Its intended action lives in configuration and stays inert. What activates it is evidence - a benchmark against a labelled suite, meeting a bar you set in advance and scaled to the severity of the action you want to permit. Two tiers usually suffice: a lower bar to run the judge at all and record its verdicts, and a stricter one before it may block or redact anything.

The practical payoff is that ambition and authority come apart. Someone can add a judge to a policy on Monday and begin collecting verdicts immediately, with production behavior completely unchanged until the numbers justify a change. Promotion becomes a reviewable decision with a number attached rather than a quiet edit to a config file. And it gives you a real answer to who judges the judge: your evaluation suite does, before the judge is permitted to matter.

• Declare the intended action in configuration but pin the judge to warn until it is gated.
• Require a passing benchmark on a labelled suite before a judge may enforce anything.
• Use two tiers - a lower bar to run and record verdicts, a stricter one to block or redact.
• Re-gate after any change to the judge model, the rubric, or the threshold. A gate certifies one specific configuration, not a name.
• Keep the advisory verdicts flowing into your logs the whole time. A judge that has never been allowed to enforce is still telling you what it would have caught.

Some Domains Need Expert Labels, Not a Test Set

The gating story has a limit that is worth stating plainly rather than glossing over. For most judges, a labelled suite assembled by the engineering team is adequate. Whether a span of text is an injection attempt, or whether an output quotes a system prompt, are questions a competent engineer can label correctly and consistently.

For a few judges that is simply not true. Whether a response provides unsafe assistance to someone in crisis, or whether medical guidance is hedged appropriately for the person receiving it, is not something a group of engineers can label correctly by consensus, however careful and well-intentioned they are. The labels themselves would be the weakest part of the system, and a benchmark built on bad labels does not measure quality - it launders a guess into a number and gives it the authority of a metric.

The honest posture is to require an expert-labelled evaluation suite before those judges enforce anything, and to leave them advisory indefinitely if that suite does not exist. This is unsatisfying, because these are exactly the categories where enforcement feels most urgent. But a judge that is confidently wrong about a crisis response is worse than a warning that routes to a human, and recognising which of your checks fall in this category is part of building the system responsibly rather than a caveat to bolt on later.

When One Verdict Is Not Enough

For the highest-severity questions, a single verdict is a single point of failure. The standard answer is consensus: run the same question past several independent members and synthesize one verdict from their votes. Members can differ by model, by temperature, or by rubric phrasing - what matters is that their errors are not the same errors, which is the entire reason this helps.

Two synthesis methods cover most needs. Under a majority vote every member counts once, which is simple and hard to argue with. Under a weighted vote each member counts for a configured weight, which lets you seat a strong model alongside two cheaper ones without letting the cheap ones outvote it. Either way you need a tie-break rule decided in advance, and for a safety control the defensible default is to break ties toward the more severe verdict.

Two details bite in practice. First, cost and latency scale linearly with member count, so consensus belongs on a small set of high-severity checks rather than everywhere - three members on one critical judge is a better use of budget than two members on five judges. Second, be careful that a member returning zero confidence cannot end up dominating a weighted tally. If weights derive from confidence and every member reports zero, a naive implementation can hand the decision to whichever verdict was most severe with no actual agreement behind it. Falling back to an unweighted count in that case keeps a genuine majority meaningful.

• Vary the members so their errors are uncorrelated; several calls to one model at one temperature buys you very little.
• Never let a model judge its own output. That is the one configuration where correlated failure is guaranteed.
• Pick majority or weighted deliberately, and decide the tie-break before you need it - toward the more severe verdict for a safety control.
• Reserve consensus for high-severity checks. Cost and latency scale with member count and the budget is finite.
• Guard the degenerate case: a zero-confidence member should not be able to override a real majority.

Calibration Matters More Than Accuracy

Accuracy is the obvious metric and it is not the one that matters most here, because you are not just consuming the judge's verdict. You are consuming its confidence, and using that number to decide whether to escalate, whether to enforce, and whether to route to a human. A confidence score you cannot trust makes every threshold built on it arbitrary.

Calibration is the property that makes the number meaningful: a judge that reports 0.9 confidence should be correct about ninety percent of the time it says so. A judge can be accurate and badly calibrated - right most of the time but reporting 0.99 on everything, including the cases it gets wrong - and such a judge is actively dangerous in a threshold-driven system, because a rule that says escalate below 0.75 confidence will never fire. Expected calibration error is the standard way to quantify this: bucket verdicts by reported confidence, compare each bucket's average confidence to its actual accuracy, and weight the gaps by bucket size.

Measuring it is not hard once you have a labelled suite, and it changes how you tune. If the buckets line up, your thresholds mean what they say and you can move them with confidence. If they do not, fix the calibration before touching any threshold, because you are otherwise tuning against a number that does not correspond to anything. This is the same discipline that applies to any probabilistic detector, and it is why measuring precision, recall, and calibration together is a prerequisite for enforcement rather than a follow-up task.

The Cost and the Boundary

A judge is expensive in a way a classifier is not. Where a local classifier resolves in tens of milliseconds and a reasonable timeout sits under a second, a judge call routinely needs several seconds, and a consensus of three needs that budget in parallel. This changes the arithmetic of escalation completely. A model layer you invoke on a tenth of traffic is an optimisation; a judge you invoke on a tenth of traffic is the difference between a viable feature and an unusable one. The uncertainty band that decides when to escalate matters roughly an order of magnitude more here than it does one layer down.

Placement follows from that. Judges belong on the output stage and on high-severity paths, not on the token-streaming critical path where users feel every millisecond. A judgement that will be quoted, stored as a record, or used to authorise an irreversible action is worth waiting for. A judgement about the tone of a brainstorming reply is not.

Then there is the boundary problem, which is sharper for judges than for anything else in the stack. A judge is usually the largest model in your pipeline, which makes it the most tempting thing to consume as a hosted API - and the content you are asking it to judge is, by construction, your most sensitive material. It is the output you suspect might leak a system prompt, the ticket you think contains confidential data. Sending that to a third party to find out whether it is sensitive repeats the oldest mistake in this field, where the inspection becomes the leak. A judge running inside your own infrastructure does not have this problem; a hosted one does, and every member of a consensus is a separate egress event that needs its own allowlist check, its own redaction pass, and its own audit record. This is where a self-hosted runtime earns its keep for judge-backed detection - a runtime like ZNYX AI keeps the judgement inside the same boundary as the traffic it is judging, so the most sensitive decision in the pipeline is not also the one that ships your content off-box.

• Budget seconds, not milliseconds, and set the escalation band far more carefully than you would for a classifier.
• Put judges on the output stage and high-severity paths; keep them off the token-streaming path entirely.
• Prefer a judge inside your boundary. If it is hosted, treat every member call as egress and gate, redact, and audit it.
• Cache or deduplicate where the same text would be judged twice. Two detectors escalating the same content to the same judge is pure waste.

Takeaway

Judges are for the questions that need reading rather than matching - leaked instructions, excessive agency, regulated advice, confidential disclosure, genuine jailbreak intent. That is a narrow set, and keeping it narrow is what makes the approach affordable. Everything a rule or a classifier can decide should be decided there instead.

The circularity objection resolves not through architecture but through evidence. Use a different model than the one you are judging, keep the question narrow, give the judge no ability to act, and above all ship it advisory-only until a labelled suite says it has earned the right to enforce. Accept that a few domains need expert labels you may not have, and leave those judges advisory rather than pretending an engineering test set is sufficient. Reach for consensus on the checks where one verdict is too fragile, and vary the members so the votes are actually independent.

Then treat calibration as the metric that governs everything else, since every threshold you set is denominated in the judge's confidence. Budget seconds rather than milliseconds, place judges where the wait is worth it, and keep the judgement inside the boundary that holds the traffic. Do that, and an LLM judge stops being a philosophical problem and becomes what it should be - a slow, expensive, carefully gated layer that answers the small number of questions nothing cheaper can.