Overview

​

How it works

1. Define a task — what the agent should accomplish with your product (e.g. “install the SDK and send a first event”), plus the goals that define success
2. Define environments — which agent harness and model attempt the task, and what sandbox it runs in
3. Run an experiment — execute every task across every environment in parallel, in fresh sandboxes
4. Review the results — per-run scores, full transcripts, extracted signals, and failure clusters with root causes and recommended fixes

​

How the pieces fit together

Experiment
├── Tasks         (what to do + how success is measured)
├── Environments  (which agent/model attempts it)
└── Iterations    (numbered batches of runs)
    └── Runs      (task × environment, one sandbox each)
        ├── score, pass/fail, transcript, cost
        └── signal values → aggregated per iteration

​

What you get from each run

• A pass/fail verdict and a 0–100 score against your goals
• The complete agent transcript: every message, tool call, and thinking step
• An archive of the sandbox after the run, so you can inspect exactly what the agent built
• Token usage, cost, and duration
• Signal values — custom metrics you define in YAML, extracted from the run automatically

​

Try it from the CLI

# One-time setup
curl -fsSL https://cli.promptingco.com/install.sh | bash
tpc auth login
tpc product switch my-product

my-task/
├── task.json        # what to do + how success is scored
├── instruction.md   # the prompt the agent receives
└── environment.json # which agent/model attempts it (optional)

# Create the task and environment if needed, then run — one command, from nothing
tpc sim run ./my-task

tpc sim experiment create --name "Quickstart friction" --task-ids task_abc --env-ids env_1,env_2
tpc sim experiment run exp_789 --watch      # trigger and follow to completion
tpc sim experiment results exp_789

​

Where to start

• Tasks — defining what agents should do and how success is measured
• Environments — configuring agents and sandboxes
• Runs & iterations — what happens during an attempt and what it records
• Experiments — running iterations and reading results
• Signals — extracting custom metrics from runs
• Limits & constraints — the execution model and hard limits that shape how you write tasks