> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hue.run/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To set up Hue in an application, fetch https://docs.hue.run/guides/agent-setup.md and follow it.
> Hue's project-data MCP server is https://mcp.hue.run/mcp. https://docs.hue.run/mcp only searches these docs and cannot read a Hue project.
> Never ask the user to paste a Hue API key into chat; they store it as HUE_API_KEY themselves.
> Install the exact SDK versions pinned on the page you're following, or on https://docs.hue.run/quickstart.md. Run the latest CLI without a project install as `npx -y @hue-run/sdk@latest`. In a project with `@hue-run/sdk` installed, use `npx hue`. Pin an exact version for reproducible CI runs.
> Use Hue product terms: eval set, case, evaluator, run and scoring run. Older APIs may say dataset, scorer or experiment; prefer the current SDK methods. See https://docs.hue.run/concepts.md.

# Run evals in CI

> Run your agent against Hue eval sets in GitHub Actions, inspect results, and require passing evaluations before merging.

Run your agent on pull requests and merge queues with `hue-run/eval-action@v1`. Hue saves each run and its evaluator results. GitHub shows the verdict and uploads the reports.

## Before you start

* A published eval set and its evaluators in Hue. Use the IDs from the eval set's **Run** snippet. If you need cases first, [create them from traces](https://docs.hue.run/evaluations/case-from-trace.md#from-the-mcp) or [author cases and evaluators with the SDK](https://docs.hue.run/evaluations/first-evaluation.md).
* An agent command that reads `{"inputs","config"}` as JSON on stdin and writes its answer to stdout. Write logs to stderr.
* For World cases, the [agent adapter](https://docs.hue.run/evaluations/eval-ready-agent.md) that points app clients at the supplied World URLs and token.
* Your agent's runtime, dependencies, and model credentials available in CI.
* A **Read and write** [Hue project key](https://docs.hue.run/quickstart.md#project-keys) stored as the repository Actions secret `HUE_API_KEY`.

If your coding agent needs to look up the Hue IDs, [connect it to the Hue MCP server](https://docs.hue.run/agents/mcp-server.md) for the same project. The [eval set and evaluator tools](https://docs.hue.run/agents/mcp-tools.md#eval_sets) return the saved versions and evaluator definitions. Give the agent this guide and access to your repository so it can adapt the command and workflow to your project.

## Create the configuration

Create `hue.eval.json` in your agent project. Replace the IDs and command with your eval set, evaluators, and agent entry point:

```json hue.eval.json theme={null}
{
  "evalSetId": "<eval-set-id>",
  "command": "node \"$HUE_PROJECT_DIR/agent.mjs\"",
  "evaluatorIds": ["<evaluator-id>"],
  "concurrency": 20,
  "deadlineSeconds": 900,
  "timeoutSeconds": 600,
  "scoringTimeoutSeconds": 300
}
```

Run the same configuration locally:

```sh theme={null}
npx -y @hue-run/sdk@latest eval --ci --config hue.eval.json --report-dir .hue/reports
```

The command starts in a fresh scratch directory for each case. `HUE_PROJECT_DIR` is the absolute directory containing the configuration file, so use it for your agent's entry path. Keep the [world adapter](https://docs.hue.run/evaluations/eval-ready-agent.md) in your agent. Install its dependencies and language runtime before starting the runner. The same `hue` executable starts TypeScript, JavaScript, or Python commands.

Use `evalSetVersionId` instead of `evalSetId` to select an exact saved version. Use `evaluatorVersionIds` for exact evaluator versions. Add `localScorers` with module paths relative to the configuration directory for your local code evaluators. The [CLI reference](https://docs.hue.run/sdks/cli.md#run-an-evaluation) covers the agent handoff and project key.

## Add the GitHub workflow

Add your agent's existing model credentials to the Action step's environment, then copy this workflow to `.github/workflows/hue-evals.yml`:

```yaml .github/workflows/hue-evals.yml theme={null}
name: Hue
on:
  pull_request:
  merge_group:
permissions:
  contents: read
concurrency:
  group: hue-evals-${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
  evals:
    name: Evals
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v5
        with:
          persist-credentials: false
      - uses: actions/setup-node@v6
        with:
          node-version: '24'
          package-manager-cache: false
      - run: npm ci
      - name: Run Hue evals
        id: hue
        if: ${{ always() && !cancelled() }}
        uses: hue-run/eval-action@v1
        env:
          HUE_API_KEY: ${{ secrets.HUE_API_KEY }}
          # Add the model credentials your agent already uses here.
        with:
          config: hue.eval.json
          report-dir: .hue/reports
```

Replace `npm ci` with your project's dependency command. If your agent uses Python, set up Python and install its dependencies before the Action. To select an exact released runner for debugging, set the Action's `version` input. It installs the latest stable release by default, independently of your agent's SDK dependency. Set `working-directory` for an agent in a subdirectory; `config` and `report-dir` are relative to it.

GitHub displays the workflow `Hue` and job `Evals` together as `Hue / Evals`. Checkout tests the PR test merge commit or merge-group commit. The runner records that checked-out commit separately from the PR head. Concurrency cancels superseded PR runs while keeping different merge groups separate.

Use GitHub-hosted Ubuntu or a compatible self-hosted Linux runner with Node.js 24, Bash, npm, and Actions runner `v2.327.1` or later. Self-hosted runners need Node 24 on `PATH` to write reports if setup fails. GitHub Enterprise Server artifact support is outside this release.

Fork and Dependabot pull requests normally lack ordinary Actions secrets. A missing `HUE_API_KEY` produces a clear authentication failure and reports. Run with the required secrets on a trusted branch. Do not execute untrusted code with secrets through `pull_request_target`. Your agent must also fail when its required model credentials are absent.

## Require evaluations before merging

Before making the check required, confirm a known passing run succeeds and a deliberate regression fails. Follow each job's run link to **Runs** in Hue and inspect the matching case and evaluator results.

In your repository's branch protection or ruleset, require the `Evals` check from GitHub Actions. If you rename the job, select that job's check instead. A failing workflow blocks merging only when your repository requires it.

If a deployment runs in the same workflow, make its job depend on `evals` with `needs: evals`. The deployment then waits for the evaluation job to pass.

## Read reports and recover results

Open the job summary or the `hue-evals-…` artifact for `results.json`, `junit.xml`, and `summary.md`. The Action uploads reports before restoring the CLI's failing exit code. Setup and authentication failures also produce reports. Hard cancellation and runner loss can prevent the final upload.

Use the Action's `run-url` and `verdict` outputs to link to Hue and read the server receipt's verdict. Without a finalized receipt, `verdict` is empty. The `results-json`, `junit-xml`, `summary-md`, and `report-dir` outputs name absolute paths. The `exit-code` output retains `0` for pass, `1` for quality failure, agent error, Hue error, or incomplete execution, `2` for configuration or credential errors, and `130` for interruption. Use `if: always()` on later steps that read outputs after a failure.

To read saved results without executing your agent again, run:

```sh theme={null}
npx -y @hue-run/sdk@latest eval status <run-id> --report-dir .hue/reports --json
```

An uncertain execution stays incomplete unless Hue can recover its outcome. Results-only recovery does not replay it. A new CI attempt can start a fresh invocation. Keep the original report and run URL when investigating a failure.

## Set numeric thresholds

The default CI policy requires every applicable required case and evaluator to pass. Missing, pending, errored, or inconclusive required results cannot establish success. A managed `not_applicable` result remains valid.

To make a numeric metric affect the verdict, add `policy.thresholds` to your configuration:

```json theme={null}
{
  "policy": {
    "thresholds": [
      {
        "evaluatorVersionId": "<evaluator-version-id>",
        "metric": "accuracy",
        "operator": "gte",
        "value": 0.9
      }
    ]
  }
}
```

Each threshold applies to every selected applicable required result individually. This example requires each result's `accuracy` to be at least `0.9`. Use `lte` to require a value at most the threshold.

## Compare with the base commit

To gate numeric regressions, add `baselineCommand`, `baselineDirectory`, and `policy.baselineComparisons` to your configuration:

```json theme={null}
{
  "baselineCommand": "node \"$HUE_PROJECT_DIR/agent.mjs\"",
  "baselineDirectory": ".hue/base",
  "policy": {
    "baselineComparisons": [
      {
        "evaluatorVersionId": "<evaluator-version-id>",
        "metric": "accuracy",
        "operator": "gte",
        "tolerance": 0.02
      }
    ]
  }
}
```

Prepare the base checkout and its dependencies before the Action step. For the workflow above, add:

```yaml theme={null}
- name: Check out the base commit
  uses: actions/checkout@v5
  with:
    ref: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
    path: .hue/base
    persist-credentials: false
- name: Install base dependencies
  working-directory: .hue/base
  run: npm ci
```

Supply `baselineCommand` and `baselineDirectory` together. The directory is relative to the configuration file and must contain the intended base commit. The runner checks its Git revision. `HUE_PROJECT_DIR` points to that checkout when the base command runs, while each case still gets a fresh scratch directory.

If `baselineRunId` identifies a compatible completed run for the intended base commit, Hue reuses it. Otherwise, the runner executes `baselineCommand` against a fresh copy of the candidate's saved evaluation inputs. Supply the base command and checkout so the runner can execute that fallback.

Each comparison uses candidate and baseline metric means over the same applicable cases. `gte` requires the candidate mean to be at least the baseline mean minus `tolerance`. Use `lte` for a metric where lower is better. It requires the candidate mean to be at most the baseline mean plus `tolerance`. The receipt records `baselineRunId`, `baselineReceiptId`, and `comparisons`, including both means, the case IDs, and whether each comparison passed.

## Choose what content to store

The runner's trace content and stored case outputs are enabled by default. These are independent settings in `hue.eval.json`:

| Configuration | CLI override | Effect |
| - | - | - |
| `captureContent: false` | `--no-content` | Export the runner's trace metadata without content. |
| `persistResultContent: false` | `--no-output` | Omit stored case outputs, error messages and scorer explanations. |
| `captureContent: true` | `--content` | Enable the runner's trace content, including when the configuration disables it. |

Set both options to `false`, or pass `--no-content --no-output`, to omit content from both paths. Combining `--content` with `--no-content` is an error. These controls apply to candidate and baseline runs. Keep the same choices when recovering an interrupted invocation; changing them changes its recovery identity. Explicit `true` values preserve the default identity used before these settings were available.

Generated documents still upload under either policy, and your agent's own instrumentation keeps its own capture settings. Keep credentials out of generated files and configure the agent's telemetry separately. Local code evaluators can still read the answer in memory. Hosted evaluators that require stored output may lack evidence; the report retains Hue's resulting receipt rather than treating missing evidence as a pass.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.