Run evals in CI

Run your agent on pull requests and merge queues with hue-run/eval-action@v1. Hue saves each run and its evaluator results. GitHub shows the verdict and uploads the reports.

Before you start

A published eval set and its evaluators in Hue. Use the IDs from the eval set’s Run snippet. If you need cases first, create them from traces or author cases and evaluators with the SDK.
An agent command that reads {"inputs","config"} as JSON on stdin and writes its answer to stdout. Write logs to stderr.
For World cases, the agent adapter that points app clients at the supplied World URLs and token.
Your agent’s runtime, dependencies, and model credentials available in CI.
A Read and write Hue project key stored as the repository Actions secret HUE_API_KEY.

If your coding agent needs to look up the Hue IDs, connect it to the Hue MCP server for the same project. The eval set and evaluator tools return the saved versions and evaluator definitions. Give the agent this guide and access to your repository so it can adapt the command and workflow to your project.

Create the configuration

Create hue.eval.json in your agent project. Replace the IDs and command with your eval set, evaluators, and agent entry point:

hue.eval.json

{
  "evalSetId": "<eval-set-id>",
  "command": "node \"$HUE_PROJECT_DIR/agent.mjs\"",
  "evaluatorIds": ["<evaluator-id>"],
  "concurrency": 20,
  "deadlineSeconds": 900,
  "timeoutSeconds": 600,
  "scoringTimeoutSeconds": 300
}

Run the same configuration locally:

npx -y @hue-run/sdk@latest eval --ci --config hue.eval.json --report-dir .hue/reports

The command starts in a fresh scratch directory for each case. HUE_PROJECT_DIR is the absolute directory containing the configuration file, so use it for your agent’s entry path. Keep the world adapter in your agent. Install its dependencies and language runtime before starting the runner. The same hue executable starts TypeScript, JavaScript, or Python commands. Use evalSetVersionId instead of evalSetId to select an exact saved version. Use evaluatorVersionIds for exact evaluator versions. Add localScorers with module paths relative to the configuration directory for your local code evaluators. The CLI reference covers the agent handoff and project key.

Add the GitHub workflow

Add your agent’s existing model credentials to the Action step’s environment, then copy this workflow to .github/workflows/hue-evals.yml:

.github/workflows/hue-evals.yml

name: Hue
on:
  pull_request:
  merge_group:
permissions:
  contents: read
concurrency:
  group: hue-evals-${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
  evals:
    name: Evals
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v5
        with:
          persist-credentials: false
      - uses: actions/setup-node@v6
        with:
          node-version: '24'
          package-manager-cache: false
      - run: npm ci
      - name: Run Hue evals
        id: hue
        if: ${{ always() && !cancelled() }}
        uses: hue-run/eval-action@v1
        env:
          HUE_API_KEY: ${{ secrets.HUE_API_KEY }}
          # Add the model credentials your agent already uses here.
        with:
          config: hue.eval.json
          report-dir: .hue/reports

Replace npm ci with your project’s dependency command. If your agent uses Python, set up Python and install its dependencies before the Action. To select an exact released runner for debugging, set the Action’s version input. It installs the latest stable release by default, independently of your agent’s SDK dependency. Set working-directory for an agent in a subdirectory; config and report-dir are relative to it. GitHub displays the workflow Hue and job Evals together as Hue / Evals. Checkout tests the PR test merge commit or merge-group commit. The runner records that checked-out commit separately from the PR head. Concurrency cancels superseded PR runs while keeping different merge groups separate. Use GitHub-hosted Ubuntu or a compatible self-hosted Linux runner with Node.js 24, Bash, npm, and Actions runner v2.327.1 or later. Self-hosted runners need Node 24 on PATH to write reports if setup fails. GitHub Enterprise Server artifact support is outside this release. Fork and Dependabot pull requests normally lack ordinary Actions secrets. A missing HUE_API_KEY produces a clear authentication failure and reports. Run with the required secrets on a trusted branch. Do not execute untrusted code with secrets through pull_request_target. Your agent must also fail when its required model credentials are absent.

Require evaluations before merging

Before making the check required, confirm a known passing run succeeds and a deliberate regression fails. Follow each job’s run link to Runs in Hue and inspect the matching case and evaluator results. In your repository’s branch protection or ruleset, require the Evals check from GitHub Actions. If you rename the job, select that job’s check instead. A failing workflow blocks merging only when your repository requires it. If a deployment runs in the same workflow, make its job depend on evals with needs: evals. The deployment then waits for the evaluation job to pass.

Read reports and recover results

Open the job summary or the hue-evals-… artifact for results.json, junit.xml, and summary.md. The Action uploads reports before restoring the CLI’s failing exit code. Setup and authentication failures also produce reports. Hard cancellation and runner loss can prevent the final upload. Use the Action’s run-url and verdict outputs to link to Hue and read the server receipt’s verdict. Without a finalized receipt, verdict is empty. The results-json, junit-xml, summary-md, and report-dir outputs name absolute paths. The exit-code output retains 0 for pass, 1 for quality failure, agent error, Hue error, or incomplete execution, 2 for configuration or credential errors, and 130 for interruption. Use if: always() on later steps that read outputs after a failure. To read saved results without executing your agent again, run:

npx -y @hue-run/sdk@latest eval status <run-id> --report-dir .hue/reports --json

An uncertain execution stays incomplete unless Hue can recover its outcome. Results-only recovery does not replay it. A new CI attempt can start a fresh invocation. Keep the original report and run URL when investigating a failure.

Set numeric thresholds

The default CI policy requires every applicable required case and evaluator to pass. Missing, pending, errored, or inconclusive required results cannot establish success. A managed not_applicable result remains valid. To make a numeric metric affect the verdict, add policy.thresholds to your configuration:

{
  "policy": {
    "thresholds": [
      {
        "evaluatorVersionId": "<evaluator-version-id>",
        "metric": "accuracy",
        "operator": "gte",
        "value": 0.9
      }
    ]
  }
}

Each threshold applies to every selected applicable required result individually. This example requires each result’s accuracy to be at least 0.9. Use lte to require a value at most the threshold.

Compare with the base commit

To gate numeric regressions, add baselineCommand, baselineDirectory, and policy.baselineComparisons to your configuration:

{
  "baselineCommand": "node \"$HUE_PROJECT_DIR/agent.mjs\"",
  "baselineDirectory": ".hue/base",
  "policy": {
    "baselineComparisons": [
      {
        "evaluatorVersionId": "<evaluator-version-id>",
        "metric": "accuracy",
        "operator": "gte",
        "tolerance": 0.02
      }
    ]
  }
}

Prepare the base checkout and its dependencies before the Action step. For the workflow above, add:

- name: Check out the base commit
  uses: actions/checkout@v5
  with:
    ref: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
    path: .hue/base
    persist-credentials: false
- name: Install base dependencies
  working-directory: .hue/base
  run: npm ci

Supply baselineCommand and baselineDirectory together. The directory is relative to the configuration file and must contain the intended base commit. The runner checks its Git revision. HUE_PROJECT_DIR points to that checkout when the base command runs, while each case still gets a fresh scratch directory. If baselineRunId identifies a compatible completed run for the intended base commit, Hue reuses it. Otherwise, the runner executes baselineCommand against a fresh copy of the candidate’s saved evaluation inputs. Supply the base command and checkout so the runner can execute that fallback. Each comparison uses candidate and baseline metric means over the same applicable cases. gte requires the candidate mean to be at least the baseline mean minus tolerance. Use lte for a metric where lower is better. It requires the candidate mean to be at most the baseline mean plus tolerance. The receipt records baselineRunId, baselineReceiptId, and comparisons, including both means, the case IDs, and whether each comparison passed.

Choose what content to store

The runner’s trace content and stored case outputs are enabled by default. These are independent settings in hue.eval.json:

Configuration	CLI override	Effect
`captureContent: false`	`--no-content`	Export the runner’s trace metadata without content.
`persistResultContent: false`	`--no-output`	Omit stored case outputs, error messages and scorer explanations.
`captureContent: true`	`--content`	Enable the runner’s trace content, including when the configuration disables it.

Set both options to false, or pass --no-content --no-output, to omit content from both paths. Combining --content with --no-content is an error. These controls apply to candidate and baseline runs. Keep the same choices when recovering an interrupted invocation; changing them changes its recovery identity. Explicit true values preserve the default identity used before these settings were available. Generated documents still upload under either policy, and your agent’s own instrumentation keeps its own capture settings. Keep credentials out of generated files and configure the agent’s telemetry separately. Local code evaluators can still read the answer in memory. Hosted evaluators that require stored output may lack evidence; the report retains Hue’s resulting receipt rather than treating missing evidence as a pass.

Get started

Worlds

MCP

Simulations

Traces

Evals

Traces to cases

Classifiers

Reference

Run evals in CI

Before you start

Create the configuration

Add the GitHub workflow

Require evaluations before merging

Read reports and recover results

Set numeric thresholds

Compare with the base commit

Choose what content to store

​Before you start

​Create the configuration

​Add the GitHub workflow

​Require evaluations before merging

​Read reports and recover results

​Set numeric thresholds

​Compare with the base commit

​Choose what content to store

Before you start

Create the configuration

Add the GitHub workflow

Require evaluations before merging

Read reports and recover results

Set numeric thresholds

Compare with the base commit

Choose what content to store