DynoInvoice · When green isn’t enough
A green pipeline says the tests that ran passed. It doesn’t say they ran, that they could fail, or that they test what was asked. Agents are good at producing green. Here is how DynoInvoice checks that green means something, and what happens when an agent is tempted to make a check easier.
The task was a convenience layer over the database library (INV-T-4224). Its done-when asked for negative tests on PostgreSQL 16, 17 and 18, with row-level security on and off: a stale version, another tenant’s row, no transaction, a protected table. The pipeline was green and listed six integration test classes. The first review round found one real gap and passed the rest.
Later, an agent working on something else opened the test report and noticed each class had run exactly one case, while their shared base class declared seventeen tests. Sixteen were written as fun name() = withService { ... }. Because withService returns the value of its block, those methods return something, and JUnit silently skips test methods that return a value. Every refusal test had passed through one review and two green pipelines without running.
Every test method returns nothing, and a new gate checks that each class’s reported case count equals the number of test methods in its file. The gate has its own tests: an expression-bodied test must make the count come up one short, and a missing class must count zero. Fixing only the sixteen methods was rejected, because the same mistake would come back unseen.
On re-review each of the six classes ran 28 of 28 cases, none failed or skipped; merged as daaaaabd. Reviewers now read the case count in the report, not just the colour. This was caught before merge, by an independent agent rather than the first review round, and that is exactly why the new gate exists.
The reviewer replaced a tenant check with if false. Its test stayed green, because a database policy refused the same insert with the same error. The check is now proved on its own (ef7e15ea).
The test built its database from the new schema, which already had the new grants, so an empty upgrade step would have passed. It now starts from the real previous version (954e9216).
A contract rule had never triggered, and four contract changes passed review without it before a fifth reviewer noticed. Now the first task that relies on a standard’s check must show it failing on a bad input (e0135024, b205d401).
A fake package registry accepted every credential, so a credential the real one would refuse went green, and failed at the first real publish. Stubs now must refuse a wrong credential, and the first live call is named as evidence.
Across about 490 reviewer threads on merged merge requests, we found no case of a coder deleting or disabling a test, or adding a baseline or suppression to turn red into green, that reached review unflagged. That only covers merged work, and reviewers are told in advance to look for exactly these shortcuts. What the record does show is the shortcut being offered as an option, and refused:
A contract gate was red on three invalid samples. The coder listed dropping them as an option, labelled it as weakening, and stopped. The ruling extended the validator to fail closed on keywords it didn’t support (1f9a1fe7, e1cd9dc3).
A new build release switched on stricter guards and produced 667 findings. A baseline file would have made them disappear. Instead every site was read and nine questions raised, each ruled “no exemption, fix it at source”; one needed a single exact exemption, and only that one went to the owner (70104366, 5760b015).
Building the convenience layer, the coder first tried allowing one more database setting. It wasn’t enough, and the module’s own documentation says batches are refused. The ruling: insert row by row, and leave the guard as it is.
Claims are checked, not trusted. Review compares what a merge request says with the pipeline and the code. “Verified locally across all 16 service directories” turned out to hide a failing code-generation job (6accc486). “One known flaky test” was an image-pull failure that left a feature with no passing proof (b3c0adf9). “Only two files changed” missed checker scripts and six new files (79bcde10). “The gate is clean” was read from the output instead of the exit status, and hid a breaking change (39302a0f). Each was corrected before merge.
Sometimes the check is the bug. A new schema was merged with a setting that breaks a published API standard: consumers must ignore unknown fields. It merged green because the gate compares each schema with its previous version, and a new file has none, so it was skipped. Once merged, it became the previous version for everyone else, and the next merge requests went red.
The ruling called it “our defect against a standard as written”. The schema was fixed through a new task, the downstream decoder task was revised to ignore unknown fields, and the gate was extended to check new files and then nested properties. Narrowing the gate or adding an exception was rejected as relaxing a check (d32d4532 → 9d02c9e7, b1f6d1b5, 73e676f5).
Green is evidence only if the tests ran, could fail and test the requirement. So reviewers plant faults to see tests go red, read case counts instead of colours, and check every claim against the pipeline. When a shortcut is possible, it appears as an option for a ruling, not as a quiet change. And when a gate turns out to be wrong, it is fixed and widened, never loosened.
None of this means agents never cut corners. It means corners can’t be cut quietly.
More: why does this line exist? · reviewing like an attacker · when the spec runs out · when a decision changes · one requirement and everything recorded about it · eight traces