Jokes aside, it's because it is a way to signal that you are part of the in-group by sharing in a particular activity. And what better way than to make that the kind of activity that implicitly shows off your privilege? Golf involves a large amount of grass fields set aside and maintained just for you and your friends that can't be used for anything else, and spending time on it means you can afford to not use your time on something more productive. It's gatekeeping by its very nature. It's showing off how rich you are by being wasteful with two of the most precious resources available (I think there's even a name for this in the context of evolutionary biology).
Having said that, I won't deny that it can be enjoyable in the same sense that going for a walk can be enjoyable. Going on (the equivalent of) a walk, together or alone, is a healthy thing to do. But as a social activity those other factors also play a role.
Although maybe croquet qualifies as posh enough... Now you're having me wonder if we can categorize lawn games by the classes that play them?
I'm not trying to tell anyone that golf is inherently bad and that they should feel bad for enjoying it (although I do stand by my earlier comment on it being a display of having excess land to waste, at least if we're talking about golf courses in areas where it does not fit into the natural landscape and where available land is scarce).
Similarly, whether or not the class comments apply to you specifically is not up to me to decide. I don't know you or your context.
Now, to answer your question: I would say that the entire category of Lawn Games would be available here. I already linked the wiki page before but I'll do so again in case that comment gets lost:
https://en.wikipedia.org/wiki/Lawn_game
There's also Tai Chi, or yoga, or just going for a walk, and we can keep going. Also, looking at my own parents, badminton and tennis are perfectly fine sports to play for retired people if your opponents are of the same age and you both take it easy. My mom plays badminton and tennis with her friends, and my father still likes to shoot hoops in his backyard. So basically, there are plenty of ways to get mild exercise out there, that is not an argument favoring golf over anything else.
I think you're thinking of the handicap principle.
* You can talk with your rich buddies while doing it
* You don't get sweaty or tired from doing it
* However you can still claim that you exercised
* You don't meet filthy poor people
* At the end you can replenish the 250 kilocalories you burned playing 9 holes by eating a big 400 kcal tenderloin steak and washing it away with 5 glasses of wine (also 400 kcal)
It's not really my cup of tea, though.
What are your hobbies? Whatever they are, they probably seem deathly dull to someone out there.
Why are wealthy people so much into golf? It seems like the dullest of activities. Is it just because other rich people say they like golf, and they're all such conformists?