Skip to main content
Onyx research is remote-first. Supabase and the /api/v1 API own projects, campaigns, sessions, hypotheses, worker leases, lifecycle state, experiment acceptance, knowledge, best projections, stop state, and git verification. Local workers still run in your checkout, but they report directly to the API instead of writing a local research database and replaying events later. .git/onyx/ remains important, but it is runtime storage, not the product source of truth. It keeps bounded logs, workflow files, transient pending-report outbox entries, resource locks, and disposable-worker diagnostics. The supervisor also maintains one purpose-built worker session-state brief. It combines complete immutable assignment and hypothesis guidance from the local worker context with bounded lifecycle, progress, peer, accepted-result, and selected complete knowledge facts from the API. There is no separate compact variant and selected strings or records are not clipped during serialization.

​
Local Runtime Files

Onyx stores local runtime artifacts under .git/onyx/. These files are safe to delete when you intentionally reset local runtime state, although doing so may remove logs and diagnostics for in-flight workers. They are not an offline campaign ledger or a recovery queue.

​
Online Requirement

Research product commands require API access. Campaign setup, session creation, worker lease acquisition, worker heartbeat, experiment reporting, knowledge, status, stop, and list commands read or write server state directly. If the API is unavailable, the command fails visibly instead of hiding the work behind a durable pending queue. Workers use detached disposable worktrees and never create worker branches. At teardown the harness may deliver exactly one already-terminal measured attempt; it never commits, evaluates, or reports dirty scratch work. Ref or API failures retain bounded terminal diagnostics, then the pending-report entry and worktree are discarded so a fresh worker can retry from its server-assigned starting commit.

​
Worker Leases and Acceptance

The server owns independently claimable worker slot rows and accepted experiment ordering. Slot claims use nonblocking row locks, so unrelated workers do not queue behind one session-wide lease lock.
  • Before creating a session, the supervisor verifies the provider executable, requested model, and absolute onyx-worker protocol handshake. A failed preflight creates no remote session.
  • For each requested worker, the supervisor generates a short-lived owx_worker_v1_... credential and submits it with the lease request; the server stores only its SHA-256 hash and never echoes the credential. It is scoped to one team, project, campaign, session, assignment, hypothesis, and worker, and becomes unusable when the lease expires or is released.
  • The supervisor owns routine remote control polling and normal batch heartbeat renewal, including coalesced terminal lifecycle updates. Supervised harnesses read the local control snapshot; stale or absent snapshots fall back to remote control. Direct worker brief, experiment, knowledge, and heartbeat calls use only the scoped /research/worker/* data plane; terminal heartbeats release slots. Team API keys stay in the supervisor.
  • Only transient provider rate-limit, overload, and unavailable failures enter launch backoff. Deterministic auth, model, protocol, network-policy, and sandbox failures stop the scheduler site immediately. Evaluation, git, and report-delivery failures remain distinct terminal outcomes without poisoning provider backoff.
  • Workers push HEAD to an immutable ref such as refs/onyx/experiments/<campaignId>/<runRef> before reporting an experiment.
  • Experiment reports never choose acceptedIndex. New reports normally return successfully with received disposition; workers do not wait for acceptance. Nonblocking server settlement later assigns accepted/discarded disposition and accepted indexes. Overflow past an exact target is retained only as discarded diagnostics. Operator history can request all, received, accepted, or discarded evidence, while rankings and routine worker briefs remain accepted-only.
  • Session progress exposes accepted and pending-received counts separately. Best results and campaign/revision counts are computed from indexed experiment facts when read rather than maintained as write-time projections.
  • Late reports are preserved as discarded diagnostics for lineage/debugging but do not count toward session targets, rankings, or best projections.

​
GitHub Repository Sync

The word “sync” still appears in one place: repository metadata refresh through the GitHub App. That process updates installation metadata, repository access, default branch heads, and code-read availability. It is separate from research state reporting and is not the old research event-replay layer.

​
Useful Commands

  • status shows the active profile, API target, and cached project/campaign pointers.
  • research status --summary provides bounded orchestrator progress and health; detailed research status and listen read live server state plus local supervisor runtime files when available.
  • workflow status shows local workflow runs, step summaries, metrics, block reasons, and log paths.
  • exp list and knowledge list read remote API state.
Use onyx research recover --runtime [--session <id>] [--dry-run] [--json] to reconcile interrupted local execution before bounded delivery. Dry-run does not create runtime files, refresh credentials or change remote state. JSON keeps the delivery summary and adds runtime.recovered, runtime.skippedActive, runtime.blocked, and bounded session actions/reasons. Blocked runtime recovery returns a nonzero status; pending report delivery is reported separately. Recovery verifies site/run ownership and boot/process identity, skips healthy supervisors, and retains artifacts when descendant termination is uncertain. It preserves terminal refs before removing disposable worktrees and credential homes. Explicit recovery never changes the remote session cutoff; stop an open session explicitly. Occupied resource slots require the existing idle reset, including onyx-runtime-reconciliation after an interrupted launch/recovery. Malformed evidence is retained. Repeated recovery never measures scratch work or resumes a session. Clean refuses unresolved runtime and removes worktrees through Git before deleting logs. Linux workers also retain private process-groups/*.json ownership records under their worker runtime directory while Onyx-managed command groups are active. These contain boot/PID/start identity, without command arguments or environment values. Recovery verifies those groups independently because an evaluator can outlive its provider group. Unknown identities and interrupted spawn records block cleanup; research clean also requires completed cleanup acknowledgement. The full CLI and worker CLI must both implement worker protocol 6.