Skip to main content
The onyx/ directory is the small repo-side setup surface the orchestrator uses to define a comparable research campaign, and hypothesis workers use to understand and measure experiments. You usually do not create it by hand. When you prompt an agent with /onyx ..., the orchestrator creates the initial files, commits them, runs a local setup validation, and then starts hypothesis workers. Setup files are frozen during Research unless the orchestrator stops the session, revises setup, and revalidates it. onyx setup init intentionally starts from a small explicit scaffold. It does not infer editable scope, source files, or evaluation behavior from the repository. When you pass --editable-scope or --eval-command, the CLI writes exactly those values, and the orchestrator should edit onyx/setup.json, onyx/onyx.md, and onyx/tools/* before validation and Research. Without an explicit eval command, the default evaluation tool fails until it is deliberately configured.

​
Files

For scoped projects, the directory lives under projectPath.

​
setup.json Defines the Setup

setup.json is the canonical local state for setup. The orchestrator writes it during Setup and onyx setup validate records the latest static checks in onyx/validation.json. Failed checks or stale validation block Research; warnings do not. Commit the onyx/ directory after validation and before onyx campaign setup. For GitHub-backed campaigns, push that setup base commit to the repository remote before remote campaign setup. Campaigns pin a base commit, and worker worktrees must be able to read the same frozen setup files from that base. If setup changes later, create or update the campaign base before restarting Research. Use it to define:
  • what objective matters;
  • which metric to optimize;
  • the experiment policy, including the default single-candidate workflow mode;
  • what files agents may edit;
  • what APIs, files, or behaviors are off limits;
  • declared workflow tools and guardrail steps;
  • shared resources or leases;
  • constraints and anti-gaming checks in narrative context plus command steps.
onyx.md is generated from the setup as a research spec for humans and agents. It captures durable project guidance such as metric interpretation, evaluation caveats, constraints, and useful starting points, but it is not the source of truth for enforceable setup policy. When the loop starts drifting because setup is wrong, stop the session and ask the orchestrator to revise setup and validate again.

​
Tools Connect to the Environment

The setup file declares the tools Onyx can run and the resources they need. Common tool IDs are:
  • evaluation.run;
  • reset.clean;
  • readiness.verify_idle;
  • reliability.check;
  • optional domain tools such as observe, collect, or calibrate.
onyx-worker exp run uses the workflow to acquire resources, run declared tools, parse the primary metric, run required guardrails, release resources, and record setup compliance. onyx-worker tools run <tool-id> runs a declared tool manually for diagnostics without creating workflow or experiment state.

​
Metric Output

The required metric step must print exactly one line for the configured primary metric:
If the metric tool exits nonzero, times out, emits no primary metric, or emits the wrong metric, the workflow attempt is recorded as failed.

​
Guardrails

Use workflow command steps with guardrail: true for tests, typechecks, lint, safety checks, simulator validity checks, or anti-gaming checks that should run after a valid metric. Required guardrail failures record the attempt as checks_failed. Optional command-step failures are recorded in workflow status but do not determine the experiment status.

​
Protected Setup Surface

These setup paths define the comparison and are frozen during Research:
  • onyx/onyx.md;
  • onyx/setup.json;
  • onyx/validation.json;
  • onyx/tools/*.
If you edit them manually, commit the change and ask the orchestrator to create a new validation report before starting more hypothesis workers.