Skip to main content
Signals turn raw run transcripts into metrics. You define them once in a YAML config attached to an experiment; every run extracts its own values, and every iteration aggregates them into comparable numbers — no code required. Signals are distinct from goals: a goal is the pass/fail verdict for a single run, while a signal is a measurement aggregated across every run in an iteration. Goals answer “did this run succeed?”; signals answer “how do the environments differ?”

​
Signal types and scopes

A signal produces one value per run, typed as:
  • boolean — “did the agent fabricate an API?”
  • number — “how many tokens did the run use?”
  • category — “which install method did the agent choose?” (requires category_enums)
Signals are extracted at one of two scopes:
  • run — one observation per run (the default)
  • message — an observation per matching message, then folded into a single per-run value (sum, count, average, min, max, or histogram). Message-scoped signals require a target_role (assistant, user, or tool) and a fold.

​
Extraction methods

Run-scoped pattern and llm signals also declare a source — which part of the transcript to read: codeText, assistantText, userText, thinkingText, toolCalls, toolResults, finalAnswer, or any.

​
Aggregates

Aggregates fold per-run signal values into iteration-level metrics: count, count_where, rate, sum, avg, min, max, median, mode, count_by_category, avg_by_category, and distribution. Each aggregate references a signal by id and gets a display label. rate returns a fraction from 0 to 1 (truthy runs ÷ total runs), not a percentage.

​
Example config

Signal and aggregate ids must be kebab-case and unique. The config is validated on upload — mismatched scopes, missing folds, or aggregates referencing unknown signals are rejected with specific errors. For llm extraction, model must be one of the supported judge models: claude-haiku-4-5 (the default), claude-sonnet-4, claude-sonnet-4-5, claude-sonnet-4-6, gpt-4o-mini, gpt-4o, gpt-4.1-mini, gpt-4.1, or gpt-4.1-nano. Unsupported model ids pass validation but fail at extraction time.

​
Working with signal configs from the CLI

Validate locally before uploading — validation runs entirely client-side and needs no authentication:
The command exits 0 if the config is valid and 1 with specific errors if not, so it works as a CI check or pre-commit hook. Attach the config when creating the experiment, or update it later:
After an iteration completes, read the extracted values:

​
Versioning

The signal config is stored on the experiment with a content hash. Each iteration freezes the config it ran with, so changing the config later never rewrites historical aggregates — the next iteration simply uses the new version. Every extracted value also keeps evidence: the transcript event it came from and a short snippet, so you can audit why a signal fired.