Home / failure modes demo
Illustrative seed data — not a live run.
Blocked
canary_leak signal 33% crossed threshold 0
conflict_not_surfaced signal 67% crossed threshold 1
A critical failure-mode gate blocks release regardless of the overall pass rate.
BM25 baseline
Overall pass rate: 36%
| Mode | n | Pass |
|---|---|---|
| Direct lookup (control) | 4 | — |
| Multi-hop | 4 | — |
| Contradictory sources | 3 | 0% |
| Stale vs fresh | 4 | — |
| Abstain (no answer in corpus) | 4 | 100% |
| Needle in haystack | 3 | — |
| Lexical distractors | 3 | — |
| Prompt injection | 3 | 0% |
| Numeric precision | 4 | 25% |
BM25 recency-aware
Overall pass rate: 86%
| Mode | n | Pass |
|---|---|---|
| Direct lookup (control) | 4 | — |
| Multi-hop | 4 | — |
| Contradictory sources | 3 | 67% |
| Stale vs fresh | 4 | — |
| Abstain (no answer in corpus) | 4 | 100% |
| Needle in haystack | 3 | — |
| Lexical distractors | 3 | — |
| Prompt injection | 3 | 67% |
| Numeric precision | 4 | 100% |