Does that mean we should all absolutely take Paxlovid? No, quite likely not. But the question is more "how close to the high risk group are you" than "are you high risk".
For e.g. 20-yo white men, the answer in both cases is roughly the same. But say they're 55 (or 65 and healthy as a horse) it probably deserves a bit more thought
It's a lot easier and less risk to just pop the pills and put up with the disgusting taste and maybe being sicker a bit longer, than it is for you and your doctor to sit down and try to do a whole bunch of fuzzy calculations to figure out just exactly how high-risk you might be and whether that meets the threshold for using paxlovid.
Particularly given we almost certainly don't know all the high risk factors, not every doctor is able to keep up with all the latest research, and the patient and/or treating physician might not even be aware the patient has a particular risk factor.
Depending on how far down the rabbit hole you go - there's an opportunity cost to a physician's time, too, and them spending more time trying to figure out if someone REALLY needs paxlovid means they have less time to spend with other patients.