Skip to main content
An experiment is a controlled comparison: a set of tasks, a set of environments, and an optional signal config. Running an experiment produces an iteration — one run for every task × environment pair, executed in parallel.

​
Setting up an experiment

  1. Create the experiment and give it a name
  2. Attach the tasks to evaluate
  3. Attach the environments to compare
  4. Optionally upload a signal config to extract custom metrics
  5. Trigger an iteration
Experiments live in your product’s Simulation section in the dashboard, and can also be created and run via the API or the CLI:
Triggering an iteration creates one run per task × environment pair. See Runs & iterations for what happens inside each run and what it records.

​
Iteration results

When every run in an iteration completes, results are generated at the iteration level:
  • Signal aggregates — each signal folded across runs into rates, averages, medians, and distributions, grouped by environment and task
  • Failure clusters — recurring friction patterns identified across failing runs, each with a root cause (what in your product, docs, or infra caused it) and a recommended fix
  • Summary — task-level scores and environment comparisons
Iterations are immutable. Re-running an experiment creates a new numbered iteration rather than overwriting the last one, so improvements stay measurable over time.

​
Running experiments from the CLI

Use --format json on any of these to feed results into scripts or CI. To drill into individual runs (tpc sim run get/logs/actions), see Runs & iterations.