Alibi
Differential verification for AI-generated code
Built at the YC Fast Hackathon
The problem
AI coding agents now write code faster than anyone can review it line by line. The available options are a human reading the diff, which does not scale, or another AI reading the diff, which is another guess. Neither proves what the code actually does at runtime.
The approach
Differential testing, a decades-old technique from compiler and systems work, applied to a new problem: verifying AI-authored changes before they ship. Run the old version and the new version of a changed function against identical inputs, then compare the actual outputs.
Architecture
The key design decision
old_output == new_output
The comparison layer contains zero AI. It is a plain equality check: old_output == new_output. A model is never asked whether two values look different, because the language already answers that. AI re-enters only to judge one already-proven change against the ticket, which is a far smaller and more checkable question than reviewing an entire diff.
Results
"Add a 5 percentage-point discount for VIP orders over $100."
| Case | discount_rate | final_total | Classified |
|---|---|---|---|
| Regular $80 order | 0.0 -> 0.05 | $80.00 -> $76.00 | UNINTENDED |
| VIP $200 order | 0.15 -> 0.20 | $170.00 -> $160.00 | INTENDED |
FLAG, human review required. The ticket never mentioned regular customers.
Scope and limitations
Scoped to pure functions. Side effects, database calls, and non-determinism require mocking, which was out of scope for a four-hour build. This proves non-regression on the inputs tested, not universal correctness.