Nice stuff for explaining what p-hacking is, but the big difference between medical "does this drug work" tests and software "does this layout improve signups" tests, is that in the latter, the downsides are much less. Most pharmaceuticals have undesirable side-effects, and they always cost money to produce as well as develop; a layout change (unless it has some huge code complexity, which it typically won't) has very little. The cost of developing it has already been paid. If it is more likely to help than not, the right decision is to ship it. This makes the importance of not p-hacking far less for software, than in the pharmaceutical case.
You may have some kind of ongoing cost (code complexity or server load or whatever), but in the example given, I would not expect that, and it wasn't addressed in the article. The cost of a false positive is an important factor. If you get ten probably-helpful, or even ten 50%-likely-to-be-helpful changes out, instead of 2 or 3 definitely-helpful changes, this is a net benefit.
You may have some kind of ongoing cost (code complexity or server load or whatever), but in the example given, I would not expect that, and it wasn't addressed in the article. The cost of a false positive is an important factor. If you get ten probably-helpful, or even ten 50%-likely-to-be-helpful changes out, instead of 2 or 3 definitely-helpful changes, this is a net benefit.