Skip to main content
Ask your agent this, with <product-name> replaced by your product’s name:
The agent answers with your product’s results on the public TPC Benchmarks leaderboard, which measures whether AI coding agents can go from a product’s documentation to a first working integration. This is product data for signed-in TPC users. It is not the public TPC Benchmarks plugin for ChatGPT, which anyone can use without signing in. For that plugin, see TPC Benchmarks in ChatGPT.

​
Before you ask

  • Connect the server and sign in. See Connect the MCP server.
  • If you belong to more than one organization, put your organization slug in the same message. See Find your organization slug.
  • Use the product name as it appears in TPC. The agent searches for it.
  • The product needs a website domain that matches a benchmarked vendor, or a link to one. Suggestions need the link: a product that matches only by domain gets results but no suggestions until your team claims it on the product’s Benchmarks page in TPC.

​
What the agent does

The agent chooses its own calls. A typical answer uses these steps in this order.
1

Find the product

If you did not give your organization slug, list_organizations finds it. Then list_products with your organizationSlug and the product name as query returns the product’s slug and ID, which the next call needs.
2

Get the results and history

get_product_benchmarks takes a view. The summary view returns the matched vendor, whether it matched by link or by domain, completion rate, experience and readiness scores, the last tested date, and a per-task breakdown. The trend view returns daily readiness scores and, for each harness and model setup, daily runs with completion rate, experience score and the tested task titles.
3

Look at the suggestions

get_product_benchmarks with view: "suggestions" returns open and closed readiness and experience suggestions for a linked product. A product that is not benchmarked returns a null vendor.

​
What you get back

The agent writes a summary from those calls. Depending on which views it reads, the summary can include:
  • The vendor your product matches and how it matched
  • Completion rate, experience score, readiness score, and when the product was last tested
  • How each task did
  • How readiness and run results moved over time, and which days used a different set of tasks
  • Open suggestions grouped by effort, each with the fix and how recent its evidence is
  • Closed suggestions
Each experience suggestion is either a Spec requirement, meaning a cited specification requires it and its links are listed, or Optional, meaning it goes beyond the specification and earns TPC readiness credit. Experience suggestions are AI-generated from the vendor’s benchmark runs. Each one shows how many runs it is based on and your team’s response, if any. Readiness suggestions come from the vendor’s latest site readiness scan. The tool only reads. Responding to a suggestion, claiming a vendor, and requesting a benchmark are done in TPC. The same data is available from the terminal. See Benchmark in the CLI reference.