Skip to content
sigil

Sigil Quickstart: Gate an Agent PR

At the end of this quickstart, Sigil will evaluate a PR ref against a baseline and emit an auditable ALLOW, REVIEW, or BLOCK decision. In GitHub Actions, the same loop posts a status check that can protect your merge queue.

Prerequisites

Section titled “Prerequisites”

  • A sigil binary on your PATH:

    Terminal window
    curl -fsSL https://runsigil.com/install.sh | sh

  • Docker and Docker Compose.

  • A service with a docker-compose.yml that Sigil can deploy.

  • At least one LLM provider configured for judge scenarios: Ollama, OpenAI, Anthropic, or OpenRouter.

  • A PR ref or commit ref to evaluate, such as pull/42/head.

1. Initialize a service

Section titled “1. Initialize a service”
Terminal window
sigil init --service api

This creates .sigil/sigil.toml, the .sigil/ state directory, and scenarios/api/.

2. Add a scenario

Section titled “2. Add a scenario”

Scenario source lives at the project root under scenarios/<service>/, where it is findable and editable like any other source. .sigil/ holds only tool state — config, blobs, ledger, proofs.

Terminal window
mkdir -p scenarios/api/smoke
$EDITOR scenarios/api/smoke/health.lua

A minimal health-check scenario:

-- scenarios/api/smoke/health.lua
return {
  title    = "Service responds to /health",
  priority = "P0",
  policy   = { capabilities = {"http"} },


  run = function()
    local res = sigil.get("/health")
    expect(res.status == 200)
    expect(res.json.ok == true)
  end,
}

For structured JSON assertions, sigil.json.decode(res.body) and the capability-free sigil.data helpers provide immutable typed values and bounded traversal without a shell process. Ordinary res.json field assertions above remain valid.

3. Lint scenarios and generate editor types

Section titled “3. Lint scenarios and generate editor types”
Terminal window
sigil scenario lint
sigil generate-types

generate-types writes .sigil/types/sigil.lua for editor autocomplete and hover docs.

4. Evaluate a PR against a baseline

Section titled “4. Evaluate a PR against a baseline”
Terminal window
sigil eval pull/42/head --service api

Sigil deploys the PR and baseline side by side, decrypts the visible plus holdout scenario bundle, runs the scenarios against both environments, and writes eval.complete to the ledger.

Expected shape:

visible     1.00
holdout     0.82
overall     0.94
ledger      eval.complete eval_01HPXG5KQ7J9W4

5. Emit the merge decision

Section titled “5. Emit the merge decision”
Terminal window
sigil decide pull/42/head --service api

Exit codes:

DecisionExitMeaning
ALLOW0Policy and trust gates passed.
REVIEW1Human review required.
BLOCK2Regression or policy failure.

A new service returns REVIEW at first, even for a clean eval. Only auto mode can ALLOW, and since 0.36.0 the service must also earn trust: set mode = "shadow" explicitly (sigil init leaves it unset, and an unset mode does not earn trust), let it collect clean evals, then move it to auto. See Trust Model.

GitHub status check path

Section titled “GitHub status check path”

Once the local loop works, wire the same command sequence into GitHub Actions:

Terminal window
sigil ci owner/repo#42 --service api --comment --auto-merge

Read CI Integration for the full workflow, permissions, branch protection, and merge queue setup.

Troubleshooting

Section titled “Troubleshooting”
SymptomCheck
Docker deploy failsConfirm docker compose up works without Sigil.
Scenario lint failsCheck capability declarations and Lua syntax.
Judge scenario fails before runningConfirm your [judge] provider config and keys.
REVIEW appears unexpectedlyRead the rationale codes in sigil decide --format json. A clean run in shadow or advisory is capped (shadow_mode_cap). In auto, trust_level_insufficient:none means the service has not earned trust yet: it starts at NONE and earns LOW after 20 clean evals in shadow mode. Any regression also REVIEWs (regression_detected).
Holdout key unavailable (scenarios_not_loaded)Expose the age identity as AGE_KEY or AGE_KEY_FILE. An eval whose holdouts did not load cannot ALLOW.
GitHub status does not postConfirm token permissions include statuses: write and pull-requests: write.
No scenarios found, but the files are thereIf they live under .sigil/scenarios/, this project predates 0.27. Run sigil migrate (dry-run by default; add --apply).

Next steps

Section titled “Next steps”