You sized the test, then watched it like a hawk and called it the moment it crossed 95%. Every extra glance is another chance for pure noise to wander over the line β so the false-positive rate you actually ran at is far higher than the one you chose. This is how much higher.
| If you peek⦠| Real false-positive rate |
|---|---|
| once | 5.2% |
| 2 times | 8.1% |
| 3 times | 10.9% |
| 5 timesyou | 14% |
| 10 times | 19.6% |
| 20 times | 24.8% |
Check often enough β a live dashboard you refresh daily β and a test with no real effect is almost guaranteed to flash "significant" at some point. Monitoring continuously drives the true rate toward 100%.
The blunt, always-safe patch is BonferroniSplit your error budget evenly across the looks: test each at Ξ± Γ· (number of looks). Simple and never too lax β but conservative, since consecutive looks are highly correlated.: test each look at 99% confidence instead, so all 5 looks together still spend only your 5% budget.
Better still, use a method built for it β group-sequential boundaries (Pocock, O'BrienβFleming) or always-valid sequential tests β which let you watch continuously without inflating the rate, rather than just spending the budget more frugally.