do you use any tool for this, or simply crunk up slightly the dial each day
There are multi armed bandit algorithms for this. I don’t know the names of the public tools.
This is especially useful for something where the value of the choice is front loaded, like headlines.
We've used this Python package to do this: https://github.com/bayesianbandits/bayesianbandits
But continue a percentage of A/B/n testing as well.
This allows for a balancing of speed vs. certainty