Skip to main content
A judge uses a model to grade output against a rubric, such as whether an answer follows an instruction or addresses a request. Hue hosts the model call and records the result. Use a verifier for rules that code can check directly.

​
Create and run a judge

  1. In Evaluators, choose Create evaluator → Verifier and enter a name.
  2. Open the evaluator and choose Publish version. Select LLM judge as the Evaluator type.
  3. Set the rubric, metrics, Gateway model ID, Provider ID, and Evidence bindings. Bind the input, output, expected values, or metadata the judge needs. Publish the version.
  4. Pin that version in your run. A one-shot CLI run executes pinned model judges by default; --no-judges or --no-output skips them:
Creating a judge does not run it. These steps publish an LLM judge for saved outputs. The Judge creation option configures a World judge for case outcomes. Your agent runs on your machine or CI. Hue grades the uploaded evidence. Only evidence bound to the judge is sent to its model. Missing required evidence produces a skipped result before a model call.

​
Inspect and regrade

Open the run in Runs to inspect the judge’s metrics and evidence. An unsuccessful quality verdict is a scored result, distinct from a model error or skipped check. To compare judge versions, create another scoring run over the saved outputs without running the agent again. Your coding agent can use Hue MCP’s scoring tools to create scoring runs and judge tools to queue and inspect hosted judge jobs.

​
Concurrency and funding

Hosted judges default to 50 concurrent checks per project, configurable up to 100 through Hue support. Existing pauses and custom limits still apply. Hue pays for hosted judge calls through your organization’s monthly AI budget, which defaults to $1,000 and is shared with other Hue-funded model calls. Your agent’s own model calls use your credentials. Hosted verifier compute has a separate allowance.