Skip to main content

​
Agent Experience

Use tpc sim ax setup to configure an Agent Experience experiment for your product. Add --type benchmark to design a competitive benchmark instead.

​
Prerequisites

  • Install the CLI and sign in with tpc auth login. Setup requires OAuth login; API keys are not supported.
  • Supply your organization and product with --scope <org-slug>/<product-slug>.
  • For competitive benchmarks, your organization must have benchmarks enabled.

​
Guided setup

In an interactive terminal, benchmark setup asks for a brief and competitors. Enter competitors as NAME or NAME=DOMAIN, then submit a blank line to finish. Your own product is included automatically. Setup follows progress and prompts when the agent requests text, credentials, or files. Credential input is hidden. File requests accept a local file path and upload the file before submitting the answer. Uploads must be regular files no larger than 100 MiB. The backend does not accept archives for benchmark setup; provide individual files when requested. Optional questions can be declined with a blank answer. Required questions must be answered; press Ctrl+C to leave the run paused and reconnect later. On successful watched setup, the command prints the experiment ID and a tpc sim experiment run command. Setup does not start a benchmark iteration. Review the experiment and configure any task credentials before running it.

​
Supply benchmark inputs with flags

Omitting --type selects the existing product AX flow. Product AX setup refreshes the product’s singleton Agent Experience experiment. Each new benchmark setup run creates its own benchmark. Both types reattach to an existing in-flight run for the same product and type; resubmitting flags does not edit that in-flight run.

​
Resume onboarding

setup resume follows the existing run without starting another one. After it completes, use tpc sim experiment list --scope acme/my-product to inspect the generated experiment.

​
Automation and JSON output

Without an interactive terminal, or with --format json, benchmark setup requires --intent and at least one --competitor. It never reads answers from piped stdin.
Without --watch, JSON output contains the setup response, including runId and setupRunId. With --watch, the command emits one final JSON object:
  • Completion: runId, status, milestones, and experimentId for watched setup.
  • Failure: runId, status: "failed", and milestones, with a nonzero exit.
  • Required interaction: runId, status: "input_required", and pending containing the requested items, with a nonzero exit. Reconnect with setup resume in a terminal.
Progress-only status --watch and non-interactive setup resume also stop with a nonzero exit when input is needed. Error guidance is written separately from JSON output. Secret answers are never included in the CLI’s output.

​
Product AX credentials

These env commands apply to the product’s singleton Agent Experience experiment, not competitive benchmarks. They require that experiment to exist and do not answer an onboarding question. Use setup resume for pending onboarding requests.