hue-run 0.1.2 on Python 3.10 or later, tested on 3.10 and 3.14. Start with the Python guide for installation and a runnable trace, and compatibility for supported runtimes and instrumentation. Use the installation guide for the available package release.
Imports
The SDK sends OpenTelemetry traces and correlated logs. Your application remains responsible for provider calls and the instrumentation that observes them.
Create a tracing client
Use the
api_key= keyword when omitting the origin. The key is required and validated even though the signature permits None to preserve existing Hue(base_url, api_key, ...) calls. The constructor does not load environment variables or an environment file automatically. Invalid options raise TypeError or ValueError.
Hue creates its own logger provider. With a borrowed tracer provider, your application owns that provider’s resources and other processors. Use one Hue instance for its provider lifecycle; OpenTelemetry has no public processor-removal API. See existing OpenTelemetry.
Spans and context
Usewith for the context managers below. They are ordinary synchronous context managers that may surround work inside an async function.
An exception escaping a Hue span records error type/status, then propagates unchanged. Python helpers always omit exception messages and stacks, including when capture is enabled. Initialize the client once, keep spans open until their work ends, and shut down when the process stops.
HueSpan
Choose logs or span attributes for the same content event to avoid duplicate storage.
log_inference still emits a metadata-only correlated log when content capture is disabled. hue.tracer, hue.tracer_provider and hue.logger_provider expose the configured OpenTelemetry objects for integrations that accept explicit instances.
Content policy
capture_content=False omits input/output content from Hue’s helpers before it enters an OpenTelemetry queue. True permits that helper content. The policy does not scrub arbitrary attributes, span names, session identifiers, external instrumentors or other exporters. Configure an external instrumentor’s own content policy before enabling it.
redactor(field, value) runs before helper JSON serialization and must handle your nested data format. A redaction or serialization failure raises a generic ValueError and omits that field. Captured fields are limited to 256 KiB. JSON None, False, 0 and "" remain distinct from omitted content. Received content is stored by Hue; the SDK does not set a retention timer.
Delivery and shutdown
validate_project() returns a frozen Project with id, name, slug and organization_id. Authentication, connection or invalid-response failures raise ProjectValidationError; the exception does not contain the key. Validation sends no trace.
force_flush() drains traces and logs. It returns False on timeout, a processor failure, a closed client, or any recorded failed export batch during this client’s lifetime. Inspect export_status.failed_trace_batches, failed_log_batches and ok; these are cumulative batch failure counters, not accepted-span counts.
A flush timeout bounds how long the caller waits; it does not cancel network work already running in the background. Timeout values must be positive integers. Queues are bounded and in memory; termination or overflow can lose telemetry. A successful flush does not prove that every operation was instrumented or that all parents have arrived.
shutdown() stops new helpers, drains and closes Hue-owned resources. Borrowed tracer providers and their other processors remain usable. A timeout returns False while shutdown continues; repeated calls wait for the same shutdown work. A with Hue(...) as hue: block calls shutdown on exit, but does not raise if that boolean is false. Use a checked flush when command success must require delivery:
Trace verification
TraceVerificationResult with verified and receipt: TraceReceipt | None. A receipt has trace_id, span_count, revision, fields, matched_span_ids, missing_span_ids, and trace_url. verified requires a stored receipt, every supplied expected span, and every required field. Timeout returns verified=False with the latest partial receipt, or None if none arrived. Other failures raise exported TraceVerificationError, with code and optional status_code, without response bodies or credentials.
expected_span_ids and required_fields accept sequences. Field names are input, output, model, usage, and session.
Trace IDs are 32 lowercase hexadecimal characters; expected span IDs are 16. All IDs must be nonzero. Supply at most 100 unique expected span IDs and unique required field names. The timeout defaults to 10,000 ms and must be positive, finite, and at most 60,000 ms. The receipt response is limited to 64 KiB.
Missing spans or required fields keep polling until the deadline. Only recognized trace-not-found responses, HTTP 429, and HTTP 503 are retried; connection failures, unsupported endpoints, redirects, and malformed responses raise a safe typed error. Verification reads stored normalized field presence across the trace, including supported correlated logs. It does not inspect values or prove that unlisted application operations were instrumented.
Flush the owning providers before verification; this method never flushes implicitly. An in-flight network read may finish in the background after the caller times out.
Evaluation client
EvaluationClient(base_url: str = "https://app.hue.run", api_key: str | None = None, *, timeout_seconds: float = 10) requires a valid project service key. Use api_key= for Hue Cloud or retain EvaluationClient(base_url, api_key) for an explicit origin; timeouts must be positive and finite. Registry writes require the key’s project_write capability.
Client methods are synchronous and return dictionaries. Arguments use snake_case; response fields and low-level completion payloads use HTTP camelCase. check_connection() returns the current project. List methods accept keyword-only after=None, limit=100; limits are 1–100. Responses contain items and nextCursor.
Evaluation responses are limited to 4 MiB. list_cases(version_id, ...) includes case content, so a valid row limit can still produce an oversized response when cases are large. Request a smaller page, such as limit=1, and pass each non-null nextCursor as after. The client does not automatically split or retry an oversized page.
Datasets and scorer definitions
Use
MISSING or omit expected to declare no reference. None is a present JSON-null reference. Freeze the dataset and publish scorer definitions before creating an experiment.
Scorer declarations and metrics
Scorerdefinition dictionaries use camelCase keys, even when supplied to a Python method.
Each metric definition has a
name and type: boolean, text, number (optional min/max), or category (required categories, a list of strings). Results must match the pinned definitions exactly.
Hosted config requires model, provider, rubric, bindings (a list of dictionaries with name, path, required), maxOutputTokens and timeoutMs; temperature is optional. The declaration states the requested policy; the server validates supported values and project execution settings when it is used.
Experiments, saved subjects and results
complete_execution accepts a camelCase payload: required idempotencyKey and terminal state (succeeded, error or cancelled), with optional output, error, expectedTraceRevision, traceEvidence (required or omit) and omissionReason. It returns executionId, subjectId, traceSnapshotId and evaluationItemId.
Each submitted result requires evaluationItemId, scorerVersionId and the Score fields described below; a local-code result also carries sourceDigest. A scored result has metrics entries with name, value and optional passed, plus a nonempty explanation or an evidence value. Error results contain error with type and optional message; skipped results contain an explanation.
Mutations do not retry implicitly. Preserve idempotency keys and returned IDs across retries after a lost response. HueApiError.status is the HTTP status when available; errors do not echo server bodies. The client rejects invalid JSON before writes, does not follow redirects, and limits request/response bodies to 1 MiB/4 MiB.
Runner target outputs and built-in JSON Schema declarations are limited to 200,000 serialized bytes. JSON validation also enforces nesting depth 32 and 20,000 nodes; for an HTTP write, those structural limits apply to the complete request body. Values must use finite JSON numbers, valid Unicode without NUL, string dictionary keys, lists and dictionaries without cycles. Python integers outside -(2**53 - 1) through 2**53 - 1 are rejected; use strings for larger exact integers. The server can apply additional field-specific limits.
Hosted judge jobs
get_judge_budget() reads configured allowance and availability. Call create_judge_jobs(run_id, *, idempotency_key, jobs) with jobs entries containing evaluationItemId and scorerVersionId; the result contains ids. Inspect list_judge_jobs(run_id, ...) or get_judge_job(job_id), and request cancellation with cancel_judge_job(job_id, *, reason).
Submitting hosted jobs is explicit and can consume the project’s allowance. Method availability does not establish that the project has a configured provider or budget. Local runners leave hosted and manual pins for their owning workflows.
Run locally and rescore
hue_sdk.evals and are synchronous entry points. They support sync or async target/scorer callbacks inside worker threads. From an async application, call the runner through asyncio.to_thread.
Telemetry and evaluation clients must use the same project and origin.
rescore requires an evaluation run created from saved subject IDs and has no target callback. RunnerReport exposes run_id, subject_ids, result_ids and deferred_scorer_version_ids.
Returning MISSING means unavailable output; returning None is JSON null. With persist_result_content=False, completions and local checkpoints omit raw output, target error messages, scorer evidence and arbitrary explanations. Typed metric values, including custom text values, remain stored. Existing dataset inputs/references are independent of this setting. Historical scoring skips unavailable output without invoking a scorer callback.
Local scorer exports
ScoreContext contains inputs, has_output, has_expected, optional output/expected, metadata and execution_state. A Score is scored with declared metrics and explanation/evidence, error with a typed error, or skipped with an explanation. A false quality verdict remains a scored result, with passed: False.
The source digest binds an explicitly supplied source declaration to the local scorer; it does not attest closures or installed dependencies. Custom callbacks have no general execution timeout or side-effect cancellation. JSON Schema uses an isolated subprocess with a deadline and no remote schema fetches. Hosted pins require judge-job dispatch; score_locally does not run a hosted judge.
Recovery errors
Use one checkpoint directory per experiment or rescore run. Checkpoint files contain allowed result content and are private local files, not encrypted storage. Resume an upload failure with the same run, directory and options so saved payloads and idempotency keys are reused. After a process crash, confirm its recorded owner has stopped before removing
.lock; removing the lock never authorizes another target invocation.
For uncertain work, investigate side effects before starting another attempt. The low-level start_execution method requires the latest previous_execution_id for a replacement attempt and allow_uncertain_retry=True to replace a still-started attempt. Restarting a runner does not grant either permission or automatically adopt an externally created attempt. Use a new experiment for an intentional fresh target run after investigating.
Required evidence waits for both signals after the root span ends. Omission is an explicit policy, never an automatic fallback after export failure. For a runnable example, see the public first evaluation guide.
Managed target adapter
ManagedTargetHandler from hue_sdk.managed requires machine_credential, a synchronous target callback and flush_telemetry. Optional arguments are tracer, base_url, max_execution_millis (default 90000, at most 90000) and finalization_millis (default 30000, at most 30000).
handle(payload, headers) accepts bounded raw JSON bytes or a dictionary and returns ManagedResponse with status_code, body and headers. The target receives ManagedTargetContext and returns ManagedTargetResult; generated files use ManagedOutputFile. Calls use the existing tracing providers. A false flush result leaves telemetry pending. See managed runs for framework integration, cancellation and recovery.