The standard for science a much higher ie. publishing a effect when it arose by chance as an academic
When you A/B test generally mistakes are reversible and will not make your company bankrupt or lose your job. Something being a 1 in 20 fluke is acceptable risk, you'll get most decisions right. Compare this however to hairy decisions on entering a new market or creating a new product line, there are no A/B tests or scientific frameworks here, you gather all the evidence you can, estimate the risk and make a decision
When you A/B test generally mistakes are reversible and will not make your company bankrupt or lose your job. Something being a 1 in 20 fluke is acceptable risk, you'll get most decisions right. Compare this however to hairy decisions on entering a new market or creating a new product line, there are no A/B tests or scientific frameworks here, you gather all the evidence you can, estimate the risk and make a decision