Skip to main content
TPC Benchmarks answers one question about a developer API, SDK, or tool: can a real AI coding agent go from the product’s documentation to a first successful integration? The Prompting Company (TPC) runs AI coding agents against each product’s public documentation and records what happens. Every attempt ends as passed, failed, or blocked. The plugin reports those results inside ChatGPT, so you can choose or compare products with measured data. The plugin is free and read-only. You do not sign in, it does not change anything in your accounts, and it does not ask for credentials.

​
Add the plugin

In ChatGPT, add the TPC Benchmarks plugin. There is nothing to configure. The plugin is a remote MCP server. To use it from ChatGPT developer mode or another MCP client that supports remote servers, add this URL:

​
Tools

All three tools are read-only.

​
Example prompts

  • Compare agent onboarding benchmark results for search APIs.
  • How have Exa and Firecrawl’s benchmark scores trended over recent weeks?
  • Show benchmark cards for email APIs that send transactional email.
You can ask about a product, a category, or a task in a category. If your question names no covered product or category, the plugin matches it to the closest category or task by meaning.

​
What the results include

For each product:
  • Completion rate: the share of attempts that reached a first successful integration.
  • Passed, failed, and blocked counts, and the total number of attempts.
  • Average successful run time.
  • An experience score from 0 to 100, with letter grades from S to D for five dimensions: Install, Auth, Execute, Docs, and Current.
  • The date of the last test.
  • A link to the product’s full benchmark page on promptingcompany.com.
A card shows the score, a passed, failed, and blocked bar, the five grades, the last test date, and a View benchmark button. A task that a product does not offer through its public API shows as not supported, with the documentation that shows it. It is left out of the product’s scores. Benchmark results are informational. They describe how AI coding agents performed on the days they were tested, and they are not a guarantee of how a product will work for you.

​
Coverage

The plugin covers every category on the live TPC benchmarks leaderboard. The categories and products grow as new benchmarks run. A product that is not on the leaderboard returns no results, and the plugin lists the categories it does cover.

​
Limits

Each user can make 60 lookups per 10 minutes. After that, the plugin asks you to try again shortly.

​
Data handling

  • The plugin receives the arguments ChatGPT sends, such as the search text, category, or product names.
  • Search text that names no covered product or category goes to a TPC-hosted embedding model, and to a language model judge if the match is uncertain. It is not used to train models.
  • The plugin stores a keyed hash of the ChatGPT user and session identifiers, card events such as shown and link opened, timestamps, and error and latency data, for 90 days.
  • It does not store your location, IP address, raw user agent, or raw ChatGPT identifiers.
  • Card links open through benchmarks.promptingcompany.com, which counts the visit and redirects to promptingcompany.com. It sets a visit cookie that lasts 30 minutes on that domain only.
See the privacy policy for details.

​
Support

Email support@promptingco.com with questions, problems, or feedback about TPC Benchmarks. The terms of use apply to the plugin.