Benchmark

Benchmark commands are product-scoped and read-only. Pass --scope <org-slug>/<product-slug>.

tpc benchmark summary --scope pro-org/my-product
tpc benchmark trend --scope pro-org/my-product
tpc benchmark suggestions --scope pro-org/my-product

These commands read your product’s results on the public TPC Benchmarks leaderboard: whether AI coding agents can go from your documentation to a first working integration. They do not create experiments. To set up your own competitive benchmark experiments, see Agent Experience and tpc sim ax setup --type benchmark. This is product data for signed-in TPC users. The public TPC Benchmarks plugin for ChatGPT is a separate product, described in TPC Benchmarks in ChatGPT. Every command accepts --format json and prints the raw response data instead of a table.

How your product matches a vendor

A product shows results when it matches a benchmarked vendor in one of two ways.

Match	How it happens	What you see
Linked	The product is linked to the vendor, for example after your team claims it on the Benchmarks page.	Results and suggestions.
Domain	The product’s website domain matches the vendor’s domain.	Results only. The CLI prints “Claim this product on its Benchmarks page to see suggestions”.

A product that matches no vendor prints Not benchmarked with a hint to request a benchmark from the product’s Benchmarks page in TPC.

Summary

tpc benchmark summary --scope pro-org/my-product

Prints the vendor, how it matched (linked or domain), the completion rate as passed over total, the experience score, the readiness score, the date it was last tested, the public results link, and a table with one row per task.

Field	Meaning
Completion rate	The share of benchmark runs where the agent finished the task.
Experience	A 0 to 100 score for how smooth the integration was.
Readiness	A 0 to 100 score from the vendor’s latest site readiness scan.

Tasks that TPC cannot score yet show the status unsupported and no completion rate.

Trend

tpc benchmark trend --scope pro-org/my-product

Prints the daily readiness score, then one table per harness and model setup with each day’s runs, completion rate, experience score, and the titles of the tasks tested that day. A day is flagged tasks changed when the set of tasks, or a task’s version, differs from the previous day of the same setup. Use the flag to tell a real change in results from a change in the benchmark itself. Run trends need runs on at least three different days. Until then the command prints Not enough runs yet.

Suggestions

tpc benchmark suggestions --scope pro-org/my-product

Suggestions appear only for linked products. Open suggestions are grouped by effort, quickest first: low, low-medium, and medium+.

Type	Where it comes from	What it shows
Readiness	The vendor’s latest site readiness scan.	The goal, the fix, links to the relevant specs, and `Still true as of` the scan date.
Experience	Friction that coding agents hit in benchmark runs.	The fix, the number of benchmark runs it is based on with a link to each run, `Still true as of` the last check that confirmed it or `Last evidence` when it was never rechecked, and your response if you accepted it.

Each experience suggestion is labeled Spec requirement when a cited specification requires it, with the specifications listed under Specs, or Optional · TPC readiness credit when it goes beyond the specification and earns TPC readiness credit. Experience suggestions are marked AI-generated. TPC drafts them from the failed or struggling benchmark runs and reviews them before publishing. A recheck confirms whether the problem still happens, so treat the evidence date as the freshness of the suggestion. Closed suggestions follow the open ones, newest first. A suggestion closes when the issue stops showing up in scans or runs, or when your team marks it fixed or declines it. Responding to a suggestion, claiming a vendor, and requesting a benchmark are done in TPC, not from the CLI.

Get started

Commands

Recipes

Reference

Benchmark

Benchmark

How your product matches a vendor

Summary

Trend

Suggestions

​Benchmark

​How your product matches a vendor

​Summary

​Trend

​Suggestions

Benchmark

How your product matches a vendor

Summary

Trend

Suggestions