This is a ready-to-use protocol for qualifying an LLM judge, the automated scorer used to rate groundedness, completeness, or quality at scale. An unvalidated judge produces numbers that look rigorous and are not, because the judge is itself a probabilistic system that can share the blind spots of the model it grades. This protocol measures the judge against human expert scores before you rely on it. Replace each <<FILL: ...>> placeholder with your own specifics. A filled specimen follows.
Approval page
| Role | Name | Signature | Date |
|---|---|---|---|
| Author (data science) | <<FILL>> | ||
| SME reviewer | <<FILL>> | ||
| QA approver | <<FILL>> |
1. Objective
To demonstrate that the LLM judge <<FILL: judge model and prompt version>>, used to score <<FILL: metric(s), e.g. groundedness and completeness>> for <<FILL: system under test>>, agrees with human expert scores closely enough to rely on its automated scores across the full golden dataset.
2. Scope
This protocol covers the qualification of the judge for the named metric(s) and use case only. A judge qualified for groundedness is not thereby qualified for completeness or for a different use case; each metric and context is qualified on its own evidence.
3. System description
| Field | Entry |
|---|---|
| Judge model and version | <<FILL: pinned model id and version>> |
| Judge prompt version | <<FILL: prompt id, under version control>> |
| Sampling parameters | <<FILL: temperature, etc., held fixed for the study>> |
| Metric(s) scored | <<FILL>> |
| Scoring scale | <<FILL: e.g. binary supported/unsupported per claim, or a defined rubric score>> |
| Human reference rubric | <<FILL: reference to the rubric and rater training>> |
4. Prerequisites
- PRQ-01 A human reference rubric exists, raters are trained on it, and rater qualifications are recorded.
- PRQ-02 A labeled agreement sample is drawn from the golden dataset, covering answerable, unanswerable, and adversarial categories, and is large enough to estimate agreement with a stated confidence interval.
- PRQ-03 The judge model version, prompt version, and sampling parameters are pinned and under version control for the duration of the study.
5. Roles
| Role | Responsibility |
|---|---|
| Data science | Builds the judge and the scoring harness; runs the study. |
| SME raters | Produce the human reference scores under the rubric. |
| QA | Owns the rubric and the agreement threshold; approves the conclusion. |
6. Acceptance criteria
- AC-01 The judge’s agreement with the human reference scores on the sample meets or exceeds the pre-defined threshold
<<FILL: e.g. 0.90 agreement, or a defined kappa>>, set before the study and justified against the consequence of a scoring error. - AC-02 Agreement holds across the categories, not only on easy answerable items; a judge that agrees on easy cases but fails on adversarial or unanswerable ones is not fit for those cases.
- AC-03 Any systematic bias (for example the judge consistently over-scoring groundedness) is characterized and either corrected or declared as a limitation with its effect on the metric bounded.
7. Test cases
| ID | Step | Expected | Actual | Pass/Fail | Tester | Date |
|---|---|---|---|---|---|---|
| TC-01 | Score the labeled sample with the judge; compute agreement with human scores | Agreement >= threshold | <<FILL>> | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-02 | Compare judge vs human on unanswerable items only | Agreement >= threshold on this category | <<FILL>> | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-03 | Compare judge vs human on adversarial / injection items only | Agreement >= threshold on this category | <<FILL>> | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-04 | Test judge stability: score a subset twice, same config | Judge scores are stable within <<FILL: tolerance>> | <<FILL>> | <<FILL>> | <<FILL>> | <<FILL>> |
| TC-05 | Characterize directional bias (judge vs human distribution) | Bias within <<FILL>> or declared and bounded | <<FILL>> | <<FILL>> | <<FILL>> | <<FILL>> |
8. Deviation handling
Any test case that fails, or any agreement below threshold, is recorded as a deviation with an impact assessment. The judge is not used for production scoring on the affected metric or category until the deviation is resolved (rubric clarified, judge prompt corrected and re-qualified, or the metric fallen back to human scoring). Reference <<FILL: SOP-ID for deviations>>.
9. Summary and conclusion
State whether the judge is qualified for the named metric(s) and use case, the measured agreement with its confidence interval, any category where it is not qualified, and any declared bias. A judge qualified only where agreement is high, with the rest scored by humans, is a valid and honest outcome.
10. Attachments
- The human reference rubric and rater qualifications.
- The labeled agreement sample and the judge and human scores.
- The agreement computation and confidence interval.
References
ICH Q9(R1) (Quality Risk Management) for sizing the qualification effort to the consequence of a scoring error. 21 CFR Part 11 and ALCOA+ for the integrity of the scoring records the judge produces. GAMP 5 (second edition) risk-based principles, applied to a model used as a measurement tool; described in original wording, not reproduced.
Confirm the current version of each reference before you rely on it.
Filled specimen (results)
Illustrative; replace with your own. Judge qualified for groundedness and completeness on a retrieval-QA assistant.
| ID | Metric / category | Human vs judge agreement | Threshold | Verdict |
|---|---|---|---|---|
| TC-01 | Overall (60-item sample) | 0.93 | 0.90 | Pass |
| TC-02 | Unanswerable items | 0.95 | 0.90 | Pass |
| TC-03 | Adversarial items | 0.88 | 0.90 | Fail |
| TC-04 | Judge stability (re-score) | within tolerance | tolerance | Pass |
| TC-05 | Directional bias | judge over-scored groundedness by ~2 points | declared | Pass (declared) |
Conclusion in the specimen: the judge is qualified for groundedness and completeness on answerable and unanswerable items, but not on adversarial items where agreement was 0.88, below the 0.90 threshold. For adversarial items the metric falls back to human scoring, and the ~2-point groundedness over-scoring bias is declared and its effect on the reported metric bounded. That honest split, qualified where the evidence supports it and human-scored where it does not, is far stronger than presenting a single “the judge agrees with humans” claim.
Common inspection findings this protocol prevents
- Using an LLM to grade an LLM with no measured agreement against human judgment, then presenting the percentages as objective.
- A judge validated on easy cases and silently trusted on the hard cases where it actually fails.
- A judge whose model or prompt version drifted after qualification, so the scoring tool is no longer the one that was qualified.
- An undeclared scoring bias that shifts the reported metric.
How to adapt this protocol
- Set your judge model, prompt version, and metric(s) in the system description.
- Set the agreement threshold before the study and justify it against the consequence of a scoring error.
- Size the labeled sample to estimate agreement with a confidence interval, and cover all item categories.
- Put the judge prompt and model version under the same change control as the rest of the system, and re-qualify the judge after any change to either.
- Confirm every regulation in the references against the current published version before you rely on it.