Hacker Neue

Preferences
akoboldfrying parent
> the chance at least one is noise is 1 – 0.95⁵ ≈ 23 %

Yes, but that's not really the big deal that you're making it out to be, since it's (usually) not an all-or-nothing thing. Usually, the wins are additive. The chance of each winner being genuine is still 95% (assuming no p-hacking), and so the expected number of wins out of those 5 will be be 0.95 * 5 = 4.75 wins (by linearity of expectation), which is a solid win rate.

kgwgk
>> the chance at least one is noise is 1 – 0.95⁵ ≈ 23 %

> The chance of each winner being genuine is still 95%

Not really. It depends on what’s the unknown (but fixed in a frequentist analysis like this one) difference between the options - or absence thereof.

If there is no real difference it’s 100% noise and each winner is genuine with probability 0%. If the difference is huge the first number is close to 0% and the second number is close to 100%.

akoboldfrying OP
You're right, I made the classic frequentist statistics mistake. All we can talk about with certainty are the probabilities of things happening under a specific null model.
ec109685
Good point. The 23% in the example refers to the worst case where 5 tests are all null throughout the period.

This item has no comments currently.