Experiment repositories
Store experiment definitions and their referenced files in a directory that you can review and version in Git. The stable CLI provides three local commands:
These commands require no authentication or scope. They make no network requests, send no telemetry, read no secret values, execute no scripts, and create no backend records.
init writes local files; validate and plan read them. The existing experiment create, update, and run commands use experiment records in TPC.
Task layout
Use--layout tasks to scaffold a repository in the upstream Harbor task format. The target directory must be new and its parent must exist.
environment/, verifier fixtures and answer keys in tests/, and optional oracle code in solution/. The validator rejects symlinks, traversal paths, private-looking paths under the agent environment, and exact copies of recognizable private fixtures within its bounded scan. Review the contents of the instruction, environment assets and build definition before running a task. Filename and byte comparisons can only detect some placements.
validate and plan recognize a single task directory, a flat directory of tasks, or the default tasks/ collection. If the root contains named dataset directories such as set-a/ and set-c/, you can select them explicitly. The CLI discovers those directories when the root has no flat tasks.
layout: "tasks", valid, task summaries, and dataset membership. A plan also includes agents, attempts, and totalRuns.
Local validation accepts the structure of upstream schema 1.4 scaffolds, including blank instruction placeholders and separate verifier configurations. It does not start a container, build an image, run a test, fetch external tasks, resolve secrets, or check model availability. For execution through TPC Code Storage, put each task in its own repository-relative directory. Task directories cannot contain other task directories. Use single-step Linux tasks with nonempty instructions and an agent Dockerfile or compatible image.
TPC supports shared and separate verification. For separate verification, set verifier.environment_mode = "separate" and provide tests/Dockerfile that installs /tests/test.sh from the private test context. Declare the workspace paths the verifier needs in artifacts. The TPC verifier reads /logs/verifier/reward.json with lowercase letters, digits, hyphens, or underscores in check names and numeric rewards from zero to one. Existing upstream fixtures may require configuration or verifier changes before execution through TPC.
Authored v1 repositories
The defaultinit command creates an authored v1 repository. A directory that contains tpc-experiment.json uses v1 validation and plan output.
Start an authored v1 repository
Run this in a directory wheremy-experiment does not exist. Its parent directory must already exist.
init refuses an existing target, including an empty directory. Edit the generated instructions and definitions, then validate again. The starter has one task, one environment, and three repetitions: three planned runs. Agent and judge model IDs are placeholders. Replace them deliberately; validation checks their shape, but does not check model availability, provider credentials, pricing, or harness compatibility.
The commands support --format text (default) and --format json. JSON output from init and validate includes valid, schemaVersion, fingerprint, experiments (a count), and totalRuns. Invalid input produces an error and a nonzero exit status.
Repository layout and schema
tpc-experiment.json is the entry point. Other filenames and directories are your choice: the manifest lists definition files explicitly. Unlisted files are not discovered automatically.
schema.json is a copy of the authoritative JSON Schema embedded in the CLI. It contains $defs for manifest, experiment, task, goal, environment, judge, and signalConfig. For editor assistance, the manifest can use "$schema": "schema.json"; a task can use "$schema": "../schema.json#/$defs/task". These editor references are separate from repository file references, which are always relative to the repository root.
The CLI always validates against its embedded schema. Editing the copied schema or pointing $schema at a remote URL does not change validation and never fetches that URL. Unknown properties are rejected in the authored definition objects; an embedded evaluator schema is its own JSON object.
schemaVersion: 1 identifies this limited authored repository format. Treat that version’s meaning as immutable: incompatible shape or meaning changes require a new version. It is separate from onboarding ExperimentSpec v1 and portable task v1. It is not a lossless codec for those formats or for backend records, and it does not activate execution or migration. This MVP has no import, apply, upload, or repository-run command.
Manifest and references
$schema are required. tasks, environments, and judges contain unique paths to JSON definitions; each list must be nonempty. experiments is a nonempty array of inline definitions. An experiment requires key, name, taskKeys, environmentKeys, and repetitions; description and signalConfigFile are optional. Repetitions are integers from 1 to 1,000. Each experiment selects a nonempty, duplicate-free list of task and environment keys.
Keys are lowercase letters or digits separated by single hyphens, at most 100 characters. Task, environment, judge, and experiment keys must each be unique within their own kind. Goal keys must be unique within a task. References resolve to these local keys, never backend UUIDs. A goal’s judgeKey must exist in the manifest’s judges. Task and environment selections must exist even if their definition is otherwise valid. Every listed definition is loaded, including definitions unused by the experiment matrix.
Tasks and goals
description is optional. Categories are documentation, research, coding, or analysis; taskType is cli_execution. timeLimitMs is an integer from 1 to 86,400,000. instructionFile points to nonempty UTF-8 text. A task requires at least one goal.
A goal requires key, name, kind, passingThreshold, and judgeKey. description is optional. A criterion describes a graded quality dimension with a threshold from 0 to 100; it may carry an optional stable usability goalKey. A gate describes a prerequisite: its threshold must be exactly 100, it must specify gateRole as artifact, product_use, or no_fabrication, and it cannot carry goalKey. Criteria cannot carry gateRole. These fields preserve the existing distinction between graded goals and prerequisites; the local commands do not score goals or evaluate gate outcomes.
Environments and secrets
key, name, and agentConfig are required. description, secretRefs, initCommandsFile, and initFiles are optional. The agent configuration accepts only nonempty harness and model strings. It does not support the full backend environment configuration or accept sandboxProvider.
secretRefs contains unique environment-style names such as PROVIDER_API_KEY, matching letters or underscore followed by letters, digits, or underscore. They are references only: the CLI does not look them up in the shell or a secret store. Keep secret values out of repository files. Plan JSON includes resolved instruction, command, and script text, so anything authored in those files appears in that output.
initCommandsFile points to nonempty UTF-8 command text. It is read as opaque text and never executed or interpreted as the backend’s structured initialization commands. Each init file has a repository source and a relative destination; destinations must be unique per environment. Sources may contain arbitrary bytes. The destination is a proposed relative target, not an instruction to copy a file locally and not a backend absolute sandbox path.
Reusable judges
A judge requireskey, name, evaluationType, and evaluationFunction. description is optional. evaluationFunction contains type and config. A nonempty top-level model is required for an LLM judge and optional for other types. It is stored as authored metadata; the CLI does not select or invoke that model.
Comparison metrics are
duration, tokens, or cost; operators are <, >, <=, >=, or ==; thresholds are numbers. Content match types are contains or regex. scriptFile references nonempty UTF-8 text in the repository. Evaluator path values describe future output artifacts; the loader does not read those artifacts. The CLI checks definition shapes and resolves script files, but does not run checks, compile regex patterns, execute shell or Python, or evaluate an embedded JSON Schema against an artifact. Manual evaluation and the full backend configuration bags are outside this format.
Signals and aggregates
An experiment may reference a signal configuration file:version must be 1. signals is required and nonempty; aggregates is optional. Each signal requires id, name, type: "number", scope: "run", and extract with method: "stats" and a supported numeric stat. Stats are duration (milliseconds), duration_seconds, token_in, token_out, token_total, cost (USD), tool_calls, turns, and steps.
Each aggregate requires id, signal_id, fn, and label. Functions are count, sum, avg, min, max, and median. Signal IDs must be unique; aggregate IDs must be unique within their separate list, and each signal_id must reference a signal in that file. IDs use the same lowercase hyphenated form as definition keys.
This is a numeric stats subset of the existing signal system. It does not include message extraction, collection, LLM rubric extraction, categorical signals, or the full aggregate library. Validation describes intended measurements; planning produces no signal values or aggregates.
Plan output and fingerprint
Text output shows the repository fingerprint, experiment matrices, repetition counts, total planned runs, and secret reference names. JSON additionally contains:
The fingerprint changes when referenced bytes or paths change, including JSON whitespace. It covers the manifest and its loaded reference closure, including init file sources, rather than all files in the directory. Unreferenced files and the copied
schema.json are outside that inventory. It does not pin provider behavior, model availability, secrets, a runtime image, or execution results.
Path and size limits
Repository file references and init file destinations use root-relative POSIX paths. Absolute paths, drive prefixes, backslashes,. or .. components, and internal symlinks are rejected. Loaded files must be regular files. These checks reject traversal and static symlinks for a local checkout preview; the loader is not a security sandbox against concurrent hostile filesystem mutation. The repository root supplied on the command line may itself be a symlink; it is resolved before loading. JSON must be UTF-8, must not contain duplicate object keys, and is limited to 64 nesting levels. Number literals are limited to 256 characters, with any exponent between -1,000 and 1,000. Each load permits at most 100,000 decoded JSON values across authored documents; a signal configuration referenced by multiple experiments is decoded and validated once per load. Instructions, scripts, and command files must be nonempty UTF-8 text.
The loader permits at most 1 MiB per loaded file, 16 MiB across distinct loaded files, and 1,024 distinct loaded files. A separate 16 MiB budget limits resolved instruction, judge script, and initialization command text, counting every occurrence even when definitions reference the same source file. There may be at most 10,000 task × environment cells across all experiments, before repetitions. Manifest arrays, goals, and definition lists have additional bounds in the copied schema. A valid local preview establishes only that this authored subset is internally consistent within these limits.