Singularity benchmark

The benchmark is a versioned, bounded evaluation of Singularity-compatible tool-capable models through Jeden and Brama.

Inputs and isolation

The contract lives at dataset/benchmark.json. Every case runs in an isolated fixture workspace. Eligible models come from Jeden’s signed, caller-scoped Brama catalogue and execute through Jeden’s Skarbiec wrapper with bounded --jobs and optional repeated --model selection. Raw workspaces remain local.

Scoring

Deterministic JSON and file graders inspect each result, a workspace-boundary grader rejects escaped writes, and completion plus hard-failure accounting keeps refusals distinct from partial work. Verdicts are qualified, strong, partial, or refused, ranked by score, hard failures, completed cases, latency, and model id.

The Las catalogue case inspects the catalogue without invoking child tools. A Brama routing refusal is recorded as a refused evaluation rather than disguised as infrastructure success.

Outputs

The canonical run output is results/<timestamp>/report.json; results/latest.json is replaced atomically. Leaderboard Markdown is not a product artifact and the runner emits JSON only.