Test the evaluator before you trust the evaluation.
Teams increasingly grade their LLM outputs with another LLM. JudgeGuard tests that judge. It runs deterministic probes against it and blocks the release on the first one that breaks:
- Non-determinism: does the same input get the same verdict?
- Position bias: does answer order change the winner?
- Metadata leakage: is it reading things it shouldn't?
- Format sensitivity: does reformatting change the score?
- Missed defects: does it catch planted errors?
GitHub →