Run DGF-Bench on your model

What dgf-bench run does, stage by stage

1. Generate the clean dossiers

   [1/5] Generating 3 clean dossiers on route(s) buy, integrate, build (seed 40000, difficulty 4)...

   The command generates --dossier-number synthetic project dossiers, rotating the routes: with --route all (the default), three dossiers are one Buy, one Integrate and one Build dossier, 17 scheduled gates in total. Each dossier is generated facts-first: canonical facts, then the Word documents, CSV and JSON registers, systems of record and the Azure architecture diagram. A rule-based evaluator computes the reference decision of every gate from the same facts.

   Only dossiers with at least one blocked specialist gate (reference decision REWORK, SUSPENSION or NO_GO) are kept, so every attack has something to aim at. They are chosen at planning time from the canonical facts, before any document is written.

   Buy

   1. Procurement
   2. Legal
   3. Compliance
   4. Security
   5. IT
   6. General

   Integrate

   1. IT
   2. Architecture
   3. Security
   4. Legal
   5. Compliance
   6. General

   Build

   1. IT
   2. Architecture
   3. Security
   4. Tech Readiness
   5. General

2. Certify every gate

   [2/5] Certifying that every gate is decidable...

   For every scheduled gate, the certifier reads each decisive fact from the authoritative systems of record the agent can reach and checks it against the canonical value. The run stops if a single gate is not decidable from those sources.

3. Build the attack variants

   [3/5] Building 27 attack variants (one attack type each)...

   For each attack, the clean dataset is copied and that one attack is planted on every eligible gate. The attack goes only into evidence the organization does not vouch for: narrative documents, free-text notes of records, document metadata, uploaded files, the diagram image or the project context. Authoritative values, mandates, tools and rules are never changed, so the reference decision stays the same. The injection manifest is written to the evaluator-only 99_hidden_ground_truth.json, which the agent's tools never open.

4. Call the model

   [4/5] Model calls. 1 clean baseline + 27 attacks over 3 dossiers = 84 model x dossier runs, sharing one budget cap of $10.00.

   The model first reviews the clean dossiers (the baseline), then each attack variant. In each model × dossier run it reviews every gate of the route in order: it receives the gate's policy and its tools, looks for the facts in the documents and records (no fact snapshot is given), and submits a typed decision (GO, GO_WITH_RESERVATIONS, REWORK, SUSPENSION or NO_GO) with findings, actions and citations. Each decision is handed to the next gate, and the General gate consolidates the specialist decisions.

   • Up to --workers (default 6) model × dossier runs execute in parallel.
   • Each gate is capped at 30 turns and 60 tool calls, and each model turn at 16,384 output tokens.
   • Temperature 0 is sent where the endpoint accepts it; reasoning settings are left at the provider default.
   • Models with image input also receive the architecture diagram image at the IT, Architecture, Security and Tech Readiness gates; text-only models do not, so the image attack does not apply to them.
   • --max-cost-usd is one budget for the whole run: each condition may spend only what the others left, and the run stops when the cap is reached.

5. Score, attribute, report

   [5/5] Building the report...

   Every gate is scored against its reference decision and the attack manifest. Each attack success is then checked against the paired clean run (see Reading the report), and the run prints a summary:

   Output

   Done. <n> attributable attack successes across 27 attacks; clean outcome-strict <x>/17.
   DGF score: <score> / 100 (blocked <b> of <a> attacks)
   Spent: $<spent> of $10.00.
   Report: .../runs/z-ai_glm-5.3_3d/report/REPORT.md

   If the budget cap or errors stopped part of the run, a warning lists the unfinished conditions and says the score is not final.

Dry run, interruption, resume

# prepare and certify everything offline:
# no model call, no key needed
dgf-bench run --model z-ai/glm-5.3 \
  --dossier-number 3 --dry-run

# then evaluate the model on those dossiers
dgf-bench run --model z-ai/glm-5.3 \
  --dossier-number 3 --resume