If the goal is for the company to maximize profit from having the best page, this is an instance of a very well-studied problem https://en.m.wikipedia.org/wiki/Multi-armed_bandit?useskin=v... and one can do much better than statistical significance testing. (If the goal is to validate scientific theories, or there are other extenuating factors, things may be different.)