Skip to main content
The onyx/ directory is the small repo-side setup surface the orchestrator uses to define a comparable research campaign, and hypothesis workers use to understand and measure experiments. You usually do not create it by hand. When you prompt an agent with /onyx ..., the orchestrator creates the initial files, commits them, runs a local setup validation, and then starts hypothesis workers. Setup files are frozen during Research unless the orchestrator stops the session, revises setup, and revalidates it. onyx setup init intentionally starts from a small explicit scaffold. It does not infer editable scope, source files, or evaluation behavior from the repository. When you pass --editable-scope or --eval-command, the CLI writes exactly those values, and the orchestrator should edit onyx/setup.json, onyx/onyx.md, and onyx/tools/* before validation and Research. Without an explicit eval command, the default evaluation tool fails until it is deliberately configured.

​
Files

For scoped projects, the directory lives under projectPath.

​
setup.json Defines the Setup

setup.json is the canonical local state for setup. The orchestrator writes it during Setup and onyx setup validate records the latest static checks in onyx/validation.json. Failed checks or stale validation block Research; warnings do not. Commit the onyx/ directory after validation and before starting Research. For GitHub-backed projects, push the commit so the server can verify the exact setup and evaluation inputs. Campaigns do not pin a mutable base. Each new session snapshots its own base commit, setup hash, evaluation revision, and one starting commit per selected hypothesis. If setup changes later, validate and commit it before creating another session. Use it to define:
  • what objective matters;
  • which metric to optimize;
  • the experiment policy, including the default single-candidate workflow mode;
  • what files agents may edit;
  • what APIs, files, or behaviors are off limits;
  • declared workflow tools and guardrail steps;
  • non-empty fingerprintPaths on the required metric tool;
  • shared resources or leases;
  • constraints and anti-gaming checks in narrative context plus command steps.
onyx.md is generated from the setup as a research spec for humans and agents. It captures durable project guidance such as metric interpretation, evaluation caveats, constraints, and useful starting points, but it is not the source of truth for enforceable setup policy. When the loop starts drifting because setup is wrong, stop the session and ask the orchestrator to revise setup and validate again.

​
Evaluation Fingerprints

fingerprintPaths are relative to projectPath and may name tracked regular files or directories. Directories expand recursively in sorted order. Onyx hashes committed git content—not dirty working-tree content—together with the metric contract, required metric step, and complete normalized metric-tool configuration. Missing, untracked, escaping, or symlinked inputs fail validation. The full setup hash controls whether a commit is valid for a session. The evaluation fingerprint controls comparability: a changed evaluator creates or reuses a separate evaluation revision, with its own charts and best-result projections.

​
Tools Connect to the Environment

The setup file declares the tools Onyx can run and the resources they need. Common tool IDs are:
  • evaluation.run;
  • reset.clean;
  • readiness.verify_idle;
  • reliability.check;
  • optional domain tools such as observe, collect, or calibrate.
onyx-worker exp run uses the workflow to acquire resources, run declared tools, parse the primary metric, run required guardrails, release resources, and record setup compliance. onyx-worker tools run <tool-name> runs a declared tool manually for diagnostics without creating workflow or experiment state.

​
Metric Output

The required metric step must print exactly one line for the configured primary metric:
If the metric tool exits nonzero, times out, emits no primary metric, or emits the wrong metric, the workflow attempt is recorded as failed.

​
Guardrails

Use workflow command steps with guardrail: true for tests, typechecks, lint, safety checks, simulator validity checks, or anti-gaming checks that should run after a valid metric. Required guardrail failures record the attempt as checks_failed. Optional command-step failures are recorded in workflow status but do not determine the experiment status.

​
Protected Setup Surface

These setup paths define the comparison and are frozen during Research:
  • onyx/onyx.md;
  • onyx/setup.json;
  • onyx/validation.json;
  • onyx/tools/*.
If you edit them manually, commit the change and ask the orchestrator to create a new validation report before starting more hypothesis workers.