Configuring Judges

Sigil’s judge system is three-tier. Only the top tier uses an LLM; the cheaper tiers run first and short-circuit when they can.

Tier	Mechanism	Cost
1	Deterministic expect() assertions	free
2	JSON schema validation (invariant with schemas)	free
3	LLM judge (sigil.judge)	paid

[judge] config

[judge]
provider             = "ollama"                  # ollama | openai | anthropic | openrouter | command | claude-code
model                = "qwen3:14b"
fallback_model       = "qwen3:32b"               # tried when the primary model call fails
api_base             = "http://localhost:11434"  # defaulted by provider
temperature          = 0.3
consensus            = 3                         # judge samples per step (default 3)
timeout_seconds      = 30
max_tokens           = 4000                      # per judge response
injection_detection  = true
external_allowed_for = ["public", "internal"]    # data classifications that may reach an external model

The token budget for a whole eval is [eval] budget_tokens, shared by sigil.judge and sigil.intent calls. Since 0.36.0 it is enforced: once it is spent, further calls make their step unevaluable, and eval.complete records the real token use per scenario and per model. external_allowed_for applies to sigil.intent too since 0.36.0, so a scenario marked confidential cannot send its context to an external model through either call. pairwise was removed in 0.36.0; PR and baseline are judged independently.

Providers

• ollama — local model via Ollama. No API key. Good for dev and self-hosting.
• openai — OpenAI API. Uses OPENAI_API_KEY.
• anthropic — Anthropic API. Uses ANTHROPIC_API_KEY.
• openrouter — OpenRouter API. Uses OPENROUTER_API_KEY.
• command — pipes the prompt to an arbitrary CLI tool on stdin. Useful for exotic setups.
• claude-code — invokes claude -p --json-schema for structured output. The recommended setup when Sigil is part of a broader Claude Code pipeline.

Prompt format

Sigil uses a minimal, structured prompt format — not XML tags. This is the format claude -p accepts without modification:

=== INSTRUCTIONS ===
<judge instructions, including rubric>

=== INPUT ===
<serialized response under judgment>

The rubric is taken from the --- doc comment above the sigil.judge() call in the scenario file.

Consensus voting

With consensus = 3, Sigil samples the judge three times. The step passes on a majority of pass votes, and the recorded score is the median. The aggregate confidence is the lowest of the samples.

Calibration

Judge calibration is the process of grading the judge against a human-labeled golden set.

sigil judge calibrate            # reads .sigil/calibration/golden.json
sigil judge calibrate --gate     # check the metrics against deployment gates

Sigil reports:

• Accuracy — fraction of golden items where the judge matched the human label.
• Cohen’s kappa — agreement corrected for chance. Fixed in 0.36.0 to use the judge and human marginals; it is reported, not gated on.
• False-positive rate — fraction of human-rejected items the judge accepted.
• False-negative rate — fraction of human-accepted items the judge rejected.

Examples the judge errored on are treated the same way in every metric. If calibration is poor, change the judge model, tighten the rubric, or expand the golden set.

Injection defense

LLM judges see untrusted content (the service’s responses). Before the LLM call, Sigil scans each sigil.judge() observation for text that addresses the judge (ignore previous instructions, “note to the grader”, and similar). A hit skips the LLM call and makes the step unevaluable, a security failure class, so sigil decide REVIEWs. The observation is not echoed.

Set injection_detection = false if your service legitimately returns such text. Before 0.36.0 the flag was not read, and detection scanned the judge’s own rationale, which failed legitimate scenarios whose rationale contained words like “override”.