> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptingcompany.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiment repositories

> Author, validate, and preview local experiment definitions without running agents.

# Experiment repositories

Store experiment definitions and their referenced files in a directory that you can review and version in Git. The stable CLI provides three local commands:

| Command | Behavior |
| - | - |
| `tpc sim experiment init <dir> [--layout tasks]` | Create and validate a starter in a new directory. The default layout is v1. |
| `tpc sim experiment validate <dir>` | Detect the local layout and check its structure and paths. |
| `tpc sim experiment plan <dir>` | Validate and preview the run matrix for the detected layout. |

These commands require no authentication or scope. They make no network requests, send no telemetry, read no secret values, execute no scripts, and create no backend records. `init` writes local files; `validate` and `plan` read them. The existing experiment `create`, `update`, and `run` commands use experiment records in TPC.

## Task layout

Use `--layout tasks` to scaffold a repository in the [upstream Harbor task format](https://docs.harborframework.com/tasks/overview). The target directory must be new and its parent must exist.

```bash theme={null}
tpc sim experiment init ./my-experiment --layout tasks
tpc sim experiment validate ./my-experiment
tpc sim experiment plan ./my-experiment \
  --agent claude-code --agent gemini-cli --attempts 3
```

The starter contains one task. The example previews six runs.

```text theme={null}
my-experiment/
  tasks/
    hello-world/
      instruction.md
      task.toml
      environment/Dockerfile
      solution/solve.sh
      tests/test.sh
```

Put inputs for the agent in `environment/`, verifier fixtures and answer keys in `tests/`, and optional oracle code in `solution/`. The validator rejects symlinks, traversal paths, private-looking paths under the agent environment, and exact copies of recognizable private fixtures within its bounded scan. Review the contents of the instruction, environment assets and build definition before running a task. Filename and byte comparisons can only detect some placements.

`validate` and `plan` recognize a single task directory, a flat directory of tasks, or the default `tasks/` collection. If the root contains named dataset directories such as `set-a/` and `set-c/`, you can select them explicitly. The CLI discovers those directories when the root has no flat tasks.

| Task-layout option | Behavior |
| - | - |
| `--dataset <path>` | Repeat for each repository-relative directory or upstream dataset TOML manifest to include. Named manifest entries resolve by local task package name. The CLI checks digest syntax and uses the current local files for the preview. |
| `--agent <name>` | Repeat for each agent name in the plan. Defaults to `claude-code`. |
| `--task <task>` | Repeat to filter selected datasets by task key, package name, or repository-relative path. Omit to include all their tasks. |
| `--attempts <n>` | Attempts per agent and task in each dataset, from 1 to 1,000. Defaults to 1. |

```bash theme={null}
tpc --format json sim experiment plan ./pilot \
  --dataset set-a --dataset set-c \
  --agent claude-code --agent gemini-cli --attempts 3
```

The plan counts each selected task once per dataset, agent, and attempt. A task shared by two selected datasets contributes to both. JSON output has `layout: "tasks"`, `valid`, task summaries, and dataset membership. A plan also includes `agents`, `attempts`, and `totalRuns`.

Local validation accepts the structure of upstream schema 1.4 scaffolds, including blank instruction placeholders and separate verifier configurations. It does not start a container, build an image, run a test, fetch external tasks, resolve secrets, or check model availability. For execution through TPC Code Storage, put each task in its own repository-relative directory. Task directories cannot contain other task directories. Use single-step Linux tasks with nonempty instructions and an agent Dockerfile or compatible image.

TPC supports shared and separate verification. For separate verification, set `verifier.environment_mode = "separate"` and provide `tests/Dockerfile` that installs `/tests/test.sh` from the private test context. Declare the workspace paths the verifier needs in `artifacts`. The TPC verifier reads `/logs/verifier/reward.json` with lowercase letters, digits, hyphens, or underscores in check names and numeric rewards from zero to one. Existing upstream fixtures may require configuration or verifier changes before execution through TPC.

## Authored v1 repositories

The default `init` command creates an authored v1 repository. A directory that contains `tpc-experiment.json` uses v1 validation and plan output.

### Start an authored v1 repository

Run this in a directory where `my-experiment` does not exist. Its parent directory must already exist.

```bash theme={null}
tpc sim experiment init ./my-experiment
tpc sim experiment validate ./my-experiment
tpc sim experiment plan ./my-experiment
tpc --format json sim experiment plan ./my-experiment
```

`init` refuses an existing target, including an empty directory. Edit the generated instructions and definitions, then validate again. The starter has one task, one environment, and three repetitions: three planned runs. Agent and judge model IDs are placeholders. Replace them deliberately; validation checks their shape, but does not check model availability, provider credentials, pricing, or harness compatibility.

The commands support `--format text` (default) and `--format json`. JSON output from `init` and `validate` includes `valid`, `schemaVersion`, `fingerprint`, `experiments` (a count), and `totalRuns`. Invalid input produces an error and a nonzero exit status.

## Repository layout and schema

```text theme={null}
my-experiment/
  tpc-experiment.json
  schema.json
  tasks/
    answer.json
    instruction.md
  environments/
    default.json
  judges/
    evidence.json
  signals/
    cost.json
```

`tpc-experiment.json` is the entry point. Other filenames and directories are your choice: the manifest lists definition files explicitly. Unlisted files are not discovered automatically.

`schema.json` is a copy of the authoritative JSON Schema embedded in the CLI. It contains `$defs` for `manifest`, `experiment`, `task`, `goal`, `environment`, `judge`, and `signalConfig`. For editor assistance, the manifest can use `"$schema": "schema.json"`; a task can use `"$schema": "../schema.json#/$defs/task"`. These editor references are separate from repository file references, which are always relative to the repository root.

The CLI always validates against its embedded schema. Editing the copied schema or pointing `$schema` at a remote URL does not change validation and never fetches that URL. Unknown properties are rejected in the authored definition objects; an embedded evaluator `schema` is its own JSON object.

`schemaVersion: 1` identifies this limited authored repository format. Treat that version's meaning as immutable: incompatible shape or meaning changes require a new version. It is separate from onboarding `ExperimentSpec` v1 and portable task v1. It is not a lossless codec for those formats or for backend records, and it does not activate execution or migration. This MVP has no import, apply, upload, or repository-run command.

## Manifest and references

```json theme={null}
{
  "$schema": "schema.json",
  "schemaVersion": 1,
  "tasks": ["tasks/answer.json"],
  "environments": ["environments/default.json"],
  "judges": ["judges/evidence.json"],
  "experiments": [
    {
      "key": "geo-poc",
      "name": "GEO PoC example",
      "taskKeys": ["answer"],
      "environmentKeys": ["default"],
      "repetitions": 3,
      "signalConfigFile": "signals/cost.json"
    }
  ]
}
```

All five top-level fields other than `$schema` are required. `tasks`, `environments`, and `judges` contain unique paths to JSON definitions; each list must be nonempty. `experiments` is a nonempty array of inline definitions. An experiment requires `key`, `name`, `taskKeys`, `environmentKeys`, and `repetitions`; `description` and `signalConfigFile` are optional. Repetitions are integers from 1 to 1,000. Each experiment selects a nonempty, duplicate-free list of task and environment keys.

Keys are lowercase letters or digits separated by single hyphens, at most 100 characters. Task, environment, judge, and experiment keys must each be unique within their own kind. Goal keys must be unique within a task. References resolve to these local keys, never backend UUIDs. A goal's `judgeKey` must exist in the manifest's judges. Task and environment selections must exist even if their definition is otherwise valid. Every listed definition is loaded, including definitions unused by the experiment matrix.

## Tasks and goals

```json theme={null}
{
  "key": "answer",
  "name": "Answer with evidence",
  "category": "research",
  "taskType": "cli_execution",
  "timeLimitMs": 300000,
  "instructionFile": "tasks/instruction.md",
  "goals": [
    {
      "key": "grounded",
      "name": "Ground claims in evidence",
      "kind": "criterion",
      "passingThreshold": 70,
      "judgeKey": "evidence"
    }
  ]
}
```

A task requires every field shown. `description` is optional. Categories are `documentation`, `research`, `coding`, or `analysis`; `taskType` is `cli_execution`. `timeLimitMs` is an integer from 1 to 86,400,000. `instructionFile` points to nonempty UTF-8 text. A task requires at least one goal.

A goal requires `key`, `name`, `kind`, `passingThreshold`, and `judgeKey`. `description` is optional. A `criterion` describes a graded quality dimension with a threshold from 0 to 100; it may carry an optional stable usability `goalKey`. A `gate` describes a prerequisite: its threshold must be exactly 100, it must specify `gateRole` as `artifact`, `product_use`, or `no_fabrication`, and it cannot carry `goalKey`. Criteria cannot carry `gateRole`. These fields preserve the existing distinction between graded goals and prerequisites; the local commands do not score goals or evaluate gate outcomes.

## Environments and secrets

```json theme={null}
{
  "key": "default",
  "name": "Default agent",
  "agentConfig": {
    "harness": "claude",
    "model": "configure-your-agent-model"
  },
  "secretRefs": ["PROVIDER_API_KEY"],
  "initCommandsFile": "setup/init.sh",
  "initFiles": [{"source": "fixtures/input.txt", "destination": "input.txt"}]
}
```

Only `key`, `name`, and `agentConfig` are required. `description`, `secretRefs`, `initCommandsFile`, and `initFiles` are optional. The agent configuration accepts only nonempty `harness` and `model` strings. It does not support the full backend environment configuration or accept `sandboxProvider`.

`secretRefs` contains unique environment-style names such as `PROVIDER_API_KEY`, matching letters or underscore followed by letters, digits, or underscore. They are references only: the CLI does not look them up in the shell or a secret store. Keep secret values out of repository files. Plan JSON includes resolved instruction, command, and script text, so anything authored in those files appears in that output.

`initCommandsFile` points to nonempty UTF-8 command text. It is read as opaque text and never executed or interpreted as the backend's structured initialization commands. Each init file has a repository `source` and a relative `destination`; destinations must be unique per environment. Sources may contain arbitrary bytes. The destination is a proposed relative target, not an instruction to copy a file locally and not a backend absolute sandbox path.

## Reusable judges

A judge requires `key`, `name`, `evaluationType`, and `evaluationFunction`. `description` is optional. `evaluationFunction` contains `type` and `config`. A nonempty top-level `model` is required for an LLM judge and optional for other types. It is stored as authored metadata; the CLI does not select or invoke that model.

```json theme={null}
{
  "key": "evidence",
  "name": "Evidence quality",
  "evaluationType": "llm_judge",
  "model": "configure-your-judge-model",
  "evaluationFunction": {
    "type": "llm_judge",
    "config": {"description": "Score whether the answer supports its claims with evidence."}
  }
}
```

The supported evaluator shapes are:

| Function `type` | `evaluationType` | Required `config` | Optional `config` |
| - | - | - | - |
| `llm_judge` | `llm_judge` | `description` | `aspects` (string array), `examples` (`good` and/or `bad` string arrays) |
| `script_judge` | `script_judge` | `scriptFile` | `timeoutMs` (integer, 1–3,600,000) |
| `comparison` | `automatic` | `metric`, `operator`, `threshold` | — |
| `file_exists` | `automatic` | `path`, `required` (boolean) | — |
| `file_content_match` | `automatic` | `path`, `pattern`, `matchType` | — |
| `json_schema` | `automatic` | `path`, `schema` (object) | — |
| `python_script` | `automatic` | `scriptFile` | — |
| `bash_command` | `automatic` | `command` | `expectExitCode` (integer, 0–255) |

Comparison metrics are `duration`, `tokens`, or `cost`; operators are `<`, `>`, `<=`, `>=`, or `==`; thresholds are numbers. Content match types are `contains` or `regex`. `scriptFile` references nonempty UTF-8 text in the repository. Evaluator `path` values describe future output artifacts; the loader does not read those artifacts. The CLI checks definition shapes and resolves script files, but does not run checks, compile regex patterns, execute shell or Python, or evaluate an embedded JSON Schema against an artifact. Manual evaluation and the full backend configuration bags are outside this format.

## Signals and aggregates

An experiment may reference a signal configuration file:

```json theme={null}
{
  "version": 1,
  "signals": [
    {
      "id": "cost",
      "name": "Cost (USD)",
      "type": "number",
      "scope": "run",
      "extract": {"method": "stats", "stat": "cost"}
    }
  ],
  "aggregates": [
    {"id": "mean-cost", "signal_id": "cost", "fn": "avg", "label": "Mean cost (USD)"}
  ]
}
```

`version` must be 1. `signals` is required and nonempty; `aggregates` is optional. Each signal requires `id`, `name`, `type: "number"`, `scope: "run"`, and `extract` with `method: "stats"` and a supported numeric `stat`. Stats are `duration` (milliseconds), `duration_seconds`, `token_in`, `token_out`, `token_total`, `cost` (USD), `tool_calls`, `turns`, and `steps`.

Each aggregate requires `id`, `signal_id`, `fn`, and `label`. Functions are `count`, `sum`, `avg`, `min`, `max`, and `median`. Signal IDs must be unique; aggregate IDs must be unique within their separate list, and each `signal_id` must reference a signal in that file. IDs use the same lowercase hyphenated form as definition keys.

This is a numeric stats subset of the existing signal system. It does not include message extraction, collection, LLM rubric extraction, categorical signals, or the full aggregate library. Validation describes intended measurements; planning produces no signal values or aggregates.

## Plan output and fingerprint

Text output shows the repository fingerprint, experiment matrices, repetition counts, total planned runs, and secret reference names. JSON additionally contains:

| Field | Contents |
| - | - |
| `schemaVersion` | Repository format version. |
| `definitions` | All listed `tasks`, `environments`, and `judges`. Task instructions are resolved to `prompt`; judge script files to `evaluationFunction.config.script`; environment command files to `initCommands`. |
| `signalConfigs` | Loaded signal configs keyed by repository-relative file path. |
| `experiments` | Each experiment's key, name, repetitions, `totalRuns`, optional `signalConfigFile`, and `matrix` of `taskKey`/`environmentKey` pairs. |
| `secretRefs` | Sorted, unique reference names across all listed environments. |
| `files` | Referenced file inventory mapping paths to `sha256:` hashes of exact file bytes. |
| `fingerprint` | Deterministic SHA-256 digest of that inventory with a repository-v1 domain prefix. |
| `totalRuns` | Sum of matrix cell counts × repetitions across experiments. |

The fingerprint changes when referenced bytes or paths change, including JSON whitespace. It covers the manifest and its loaded reference closure, including init file sources, rather than all files in the directory. Unreferenced files and the copied `schema.json` are outside that inventory. It does not pin provider behavior, model availability, secrets, a runtime image, or execution results.

## Path and size limits

Repository file references and init file destinations use root-relative POSIX paths. Absolute paths, drive prefixes, backslashes, `.` or `..` components, and internal symlinks are rejected. Loaded files must be regular files. These checks reject traversal and static symlinks for a local checkout preview; the loader is not a security sandbox against concurrent hostile filesystem mutation. The repository root supplied on the command line may itself be a symlink; it is resolved before loading. JSON must be UTF-8, must not contain duplicate object keys, and is limited to 64 nesting levels. Number literals are limited to 256 characters, with any exponent between -1,000 and 1,000. Each load permits at most 100,000 decoded JSON values across authored documents; a signal configuration referenced by multiple experiments is decoded and validated once per load. Instructions, scripts, and command files must be nonempty UTF-8 text.

The loader permits at most 1 MiB per loaded file, 16 MiB across distinct loaded files, and 1,024 distinct loaded files. A separate 16 MiB budget limits resolved instruction, judge script, and initialization command text, counting every occurrence even when definitions reference the same source file. There may be at most 10,000 task × environment cells across all experiments, before repetitions. Manifest arrays, goals, and definition lists have additional bounds in the copied schema. A valid local preview establishes only that this authored subset is internally consistent within these limits.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.