Cloudflare Logpush

Cloudflare Logpush pushes batches of your zone’s HTTP request logs directly to The Prompting Company. Once the job is enabled, every AI crawler and agent visit (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and others) shows up in the AI traffic metric automatically — classification happens on our side, so you send everything and we keep what matters. Not on Cloudflare Enterprise? Use the raw events API to stream from any log source instead, or export a one-off CSV for history.

Before you start

You need a The Prompting Company API key (organization or service-account key) from Settings → API Keys.
The zone must be on a Cloudflare Enterprise plan — Logpush for the HTTP requests dataset is Enterprise-only.
Your DNS records must be proxied (orange cloud) — Logpush only captures traffic that flows through Cloudflare.

Configure the destination

https://logs.promptingco.com/v1/events/cf-logpush?header_Authorization=Bearer%20YOUR_API_KEY

Cloudflare converts header_* query parameters into request headers, so this sends your key as an Authorization: Bearer header with every push. Keep the value URL-encoded (%20 for the space).

Using the dashboard

In the Cloudflare dashboard, open your zone and go to Analytics & Logs → Logpush.
Click Create a Logpush job and select the HTTP requests dataset.
Choose HTTP destination and paste the destination URL above.
Select these fields: ClientIP, ClientRequestHost, ClientRequestMethod, ClientRequestReferer, ClientRequestSource, ClientRequestURI, ClientRequestUserAgent, WorkerSubrequest, EdgeStartTimestamp, EdgeResponseContentType, EdgeResponseStatus.
Save and enable the job. Cloudflare validates the destination with a test message — this succeeds automatically.

Using the API

curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
    "name": "tpc-crawler-logs",
    "dataset": "http_requests",
    "enabled": true,
    "destination_conf": "https://logs.promptingco.com/v1/events/cf-logpush?header_Authorization=Bearer%20YOUR_API_KEY",
    "max_upload_bytes": 5000000,
    "output_options": {
      "field_names": [
        "ClientIP", "ClientRequestHost", "ClientRequestMethod",
        "ClientRequestReferer", "ClientRequestSource", "ClientRequestURI",
        "ClientRequestUserAgent", "WorkerSubrequest", "EdgeStartTimestamp",
        "EdgeResponseContentType", "EdgeResponseStatus"
      ],
      "timestamp_format": "rfc3339"
    }
  }'

Any timestamp_format works (unixnano, unix, or rfc3339); rfc3339 is the easiest to debug.

What gets ingested

You don’t need to pre-filter traffic — send everything and The Prompting Company classifies AI crawlers and agents on ingestion. A few line types are skipped automatically so visits aren’t double-counted: Cloudflare’s destination-validation test message, Worker subrequests (WorkerSubrequest: true), non-eyeball requests (ClientRequestSource values like edgeWorkerFetch or purges), and lines without a ClientRequestHost. You can optionally add a Logpush filter to reduce volume, but it isn’t required for correctness.

Responses

Status	Meaning
`200`	Batch stored. Body reports `{"ok": true, "received": N, "ingested": N, "skipped": N}`
`401`	Missing or invalid API key — check the key in the destination URL
`429`	Key verification was rate limited — Cloudflare retries
`400`	Body could not be decoded
`413`	Decompressed batch too large — lower the job’s `max_upload_bytes`
`503`	Temporary ingestion outage — Cloudflare retries automatically

Delivery is at-least-once: failed batches are re-pushed in full, and occasional duplicates are expected and tolerated by the traffic metrics.

Verify it’s working

Real traffic starts appearing in the AI traffic dashboard within a few minutes of enabling the job. To exercise the pipeline immediately without waiting for a crawler visit, send one test event through the raw events API.

Get started

Dashboard overview

Content publishing

Analytics

Agent experience

Before you start

Configure the destination

Using the dashboard

Using the API

What gets ingested

Responses

Verify it’s working

​Before you start

​Configure the destination

​Using the dashboard

​Using the API

​What gets ingested

​Responses

​Verify it’s working

Before you start

Configure the destination

Using the dashboard

Using the API

What gets ingested

Responses

Verify it’s working