Local evaluations

Use the SDK’s evaluation runner when you want to evaluate a function in your own process or CI, with evaluators you define in code. To run your agent against cases built from traces, use hue eval instead. This example creates one eval-set case, publishes an exact-match evaluator, and evaluates a deterministic uppercase function. It then scores the same saved output in a new scoring run. It makes no model call and consumes no hosted-judge allowance. Install the SDK for your language and set HUE_API_KEY to a project key created with the Read and write preset.

Content policy

Telemetry capture and evaluation storage are separate choices:

Choice	This example	Effect
`captureContent` / `capture_content`	`true`	Record the case input and target output in the trace, the recommended setting.
`persistResultContent` / `persist_result_content`	`true`	Store the target output so a later run can rescore it.

Setting the first switch to false sends metadata-only telemetry without changing what the evaluation stores. Eval-set inputs and reference answers are intentionally uploaded during setup. The second switch also controls target error messages, evaluator evidence, arbitrary explanations, and their local-checkpoint copies. With it disabled, local scores can still be uploaded, but historical evaluators skip unavailable output instead of invoking a callback on invented data. Declared metric values are always uploaded, including custom text metrics.

Create and run

Choose one language. The program creates setup once and saves IDs locally; later invocations reuse the same run and checkpoint directories. Add .hue-first-evaluation-* to .gitignore.

TypeScript
Python

Save this as first-evaluation.ts, then run node first-evaluation.ts with Node.js 22 or 24.

import { randomUUID } from "node:crypto";
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { createHue } from "@hue-run/sdk";
import { builtins, createEvaluationClient, runExperiment, rescore } from "@hue-run/sdk/evals";

const apiKey = process.env.HUE_API_KEY;
if (!apiKey) throw new Error("Set HUE_API_KEY in your server environment.");
const connection = {
  apiKey,
};
const client = createEvaluationClient(connection);
await client.checkConnection();
const directory = ".hue-first-evaluation-ts";
let fresh = false;
try {
  await mkdir(directory, { mode: 0o700 });
  fresh = true;
} catch (error) {
  if (!(error instanceof Error && "code" in error && error.code === "EEXIST")) throw error;
}

if (fresh) {
  const suffix = randomUUID().slice(0, 8);
  const dataset = await client.createDataset({
    name: "First uppercase evaluation",
    slug: `uppercase-${suffix}`,
  });
  const draft = dataset.versions[0];
  if (!draft) throw new Error("The dataset has no draft version.");
  const changed = await client.addCase(draft.id, {
    expectedRevision: draft.revision,
    externalKey: "greeting",
    inputs: "hello",
    expected: "HELLO",
  });
  // Freeze the cases and publish a scorer version for repeatable comparisons.
  const frozen = await client.freezeDatasetVersion(changed.version.id, changed.version.revision);
  const scorer = await client.createScorer({ name: "Exact match", slug: `exact-${suffix}` });
  const exact = await client.publishScorerVersion(scorer.id, builtins.exactMatch());
  const request = {
    idempotencyKey: randomUUID(),
    name: "Uppercase baseline",
    datasetVersionId: frozen.id,
    scorerVersionIds: [exact.id],
    config: { operation: "uppercase" },
  };
  await writeFile(`${directory}/creation-request.json`, JSON.stringify(request), {
    flag: "wx", mode: 0o600,
  });
  const experiment = await client.createExperiment(request);
  await writeFile(`${directory}/state.json`, JSON.stringify({
    experimentId: experiment.id,
    scorerVersionId: exact.id,
    rescoreKey: randomUUID(),
  }), { flag: "wx", mode: 0o600 });
}

const state = JSON.parse(await readFile(`${directory}/state.json`, "utf8")) as {
  experimentId: string; scorerVersionId: string; rescoreKey: string;
};
const hue = createHue({ ...connection, serviceName: "local-evaluation", captureContent: true });
try {
  // Run the target locally and save its output for later scoring.
  const report = await runExperiment({
    client, hue, experimentId: state.experimentId,
    checkpointDirectory: `${directory}/experiment`,
    persistResultContent: true,
    traceEvidence: { mode: "required" },
    target: async (inputs) => {
      if (typeof inputs !== "string") throw new TypeError("Expected a string dataset input.");
      return inputs.toUpperCase();
    },
  });
  // Score the saved output again without calling the target.
  const historical = await client.createEvaluationRun({
    idempotencyKey: state.rescoreKey,
    name: "Rescore saved uppercase output",
    subjectIds: report.subjectIds,
    scorerVersionIds: [state.scorerVersionId],
  });
  await rescore({
    client, runId: historical.id,
    checkpointDirectory: `${directory}/rescore`,
    persistResultContent: true,
  });
  console.log({ experimentId: state.experimentId, runId: report.runId, rescoreRunId: historical.id });
} finally {
  await hue.shutdown();
}

Save this as first_evaluation.py, then run .venv/bin/python first_evaluation.py.

import json
import os
from pathlib import Path
from uuid import uuid4
from hue_sdk import Hue
from hue_sdk.evals import EvaluationClient, TraceEvidence, builtin_scorers, run_experiment, rescore

api_key = os.environ["HUE_API_KEY"]
client = EvaluationClient(api_key=api_key)
client.check_connection()
directory = Path(".hue-first-evaluation-py")
fresh = False
try:
    directory.mkdir(mode=0o700)
    fresh = True
except FileExistsError:
    pass

def save_once(name, value):
    descriptor = os.open(directory / name, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
    with os.fdopen(descriptor, "w") as file:
        json.dump(value, file)

if fresh:
    suffix = uuid4().hex[:8]
    dataset = client.create_dataset(name="First uppercase evaluation", slug=f"uppercase-{suffix}")
    draft = dataset["versions"][0]
    changed = client.add_case(
        draft["id"], expected_revision=draft["revision"], external_key="greeting",
        inputs="hello", expected="HELLO",
    )
    version = changed["version"]
    # Freeze the cases and publish a scorer version for repeatable comparisons.
    frozen = client.freeze_dataset_version(version["id"], version["revision"])
    scorer = client.create_scorer(name="Exact match", slug=f"exact-{suffix}")
    exact = client.publish_scorer_version(scorer["id"], builtin_scorers.exact_match())
    request = dict(
        idempotency_key=str(uuid4()), name="Uppercase baseline",
        dataset_version_id=frozen["id"], scorer_version_ids=[exact["id"]],
        config={"operation": "uppercase"},
    )
    save_once("creation-request.json", request)
    experiment = client.create_experiment(**request)
    save_once("state.json", dict(
        experiment_id=experiment["id"], scorer_version_id=exact["id"], rescore_key=str(uuid4()),
    ))

state = json.loads((directory / "state.json").read_text())

def target(inputs, context):
    if not isinstance(inputs, str):
        raise TypeError("Expected a string dataset input.")
    return inputs.upper()

hue = Hue(api_key=api_key, service_name="local-evaluation", capture_content=True)
try:
    # Run the target locally and save its output for later scoring.
    report = run_experiment(
        client=client, hue=hue, experiment_id=state["experiment_id"], target=target,
        checkpoint_directory=str(directory / "experiment"), persist_result_content=True,
        trace_evidence=TraceEvidence("required"),
    )
    # Score the saved output again without calling the target.
    historical = client.create_evaluation_run(
        idempotency_key=state["rescore_key"], name="Rescore saved uppercase output",
        subject_ids=report.subject_ids, scorer_version_ids=[state["scorer_version_id"]],
    )
    rescore(
        client=client, run_id=historical["id"], checkpoint_directory=str(directory / "rescore"),
        persist_result_content=True,
    )
    print(dict(experiment_id=state["experiment_id"], run_id=report.run_id,
               rescore_run_id=historical["id"]))
finally:
    if not hue.shutdown():
        raise RuntimeError("Hue shutdown did not confirm successful delivery.")

The setup directory is created before registry writes. If setup stops before state.json, the next invocation stops rather than silently creating another run. Inspect Evals, Evaluators, and Runs before recovery. Registry writes have no automatic retry; the saved request can recover an uncertain acknowledgement with its original idempotency key and payload.

Read the result

Open Runs and select Uppercase baseline. The target should succeed and the exact-match metric should be true. The scoring run uses the same immutable subject ID and saved output. rescore has no target callback, so it cannot invoke the uppercase function again. To compare a different target configuration, create a new run with the same frozen eval-set and published evaluator versions. Use a new checkpoint directory; do not edit frozen pins or reuse a checkpoint.

Missing outputs and quality failures

Read presence flags such as hasOutput and hasExpected in API responses. Do not test value truthiness: JSON null, false, 0, and "" can all be present values. TypeScript target return undefined and Python’s exported MISSING sentinel represent unavailable output. Python None is a present null. A failed quality verdict is still a scored result, typically with passed: false. Target execution and evaluator errors have separate states. Missing evidence produces an explicit skipped result. For hue eval, each required evaluator must decide before a case can pass; an undecided required result leaves it inconclusive. Exact match preserves scalar types; false is not zero.

Resume an interrupted run

Keep the same run ID, content settings and checkpoint directory when retrying an interrupted upload. The runner saves permitted completion and scoring payloads before upload and reuses request keys. It will not rerun an uncertain target. If the process crashes after invoking a target but before saving its outcome, the runner raises UncertainExecutionError. Inspect the execution and its side effects. Removing the checkpoint or its lock is not permission to run the target again. Checkpoints are private local files, not encrypted storage. Confirm the old process has stopped before explicitly removing a leftover .lock. The example requires trace evidence. The runner ends the root span and waits for trace or log acknowledgement before completing the execution. An export failure does not silently become omitted evidence, and a fresh exporter cannot prove that an earlier export succeeded. Explicit omission is a separate policy: TypeScript { mode: "omit", reason: "This local run does not retain a trace snapshot." } or Python TraceEvidence("omit", "This local run does not retain a trace snapshot."); supply a meaningful reason if you choose it. Local built-ins also include string inclusion and JSON Schema draft 2020-12. Custom callbacks run on your machine without claimed side-effect cancellation. Manual and hosted-judge evaluator pins stay pending for their workflows; this example does not dispatch them or claim live hosted-model acceptance.

Get started

Worlds

MCP

Simulations

Traces

Evals

Traces to cases

Classifiers

Reference

Local evaluations

Content policy

Create and run

Read the result

Missing outputs and quality failures

Resume an interrupted run

​Content policy

​Create and run

​Read the result

​Missing outputs and quality failures

​Resume an interrupted run

Content policy

Create and run

Read the result

Missing outputs and quality failures

Resume an interrupted run