And you can't easily switch between several virtualized machines like that. Windows will bluescreen if you remove a graphics card while in use, and I'm not sure OSX/Linux guests will fare better.
Passthrough only really works reliably if you only have one guest active at a time and never try to use the graphics card from the host.
Suprisingly OS X actually has decent support for a wide array of video cards out of the box, all of the AMD GCN 1.0 (7000 series and the R7/R9 2XXX rebrands of them) work out of the box as does the R9 290X/390X, along with a bunch of the older HD4000-HD6000 cards. NVidia provides native drivers for all their cards online, and if you're using an Intel GPU that was ever in a Mac it has support as well.
Of course, it's in violation of the macOS EULA to run it on non-Apple hardware in the first place, but you'd be surprised just how much hardware it supports.
VGA passthrough is also possible with KVM. I wonder if Oracle has any plans to add it to VirtualBox? It seems they have made room for KVM support, so in theory they could.
I have seen all OSX versions from Leopard to El Capitan successfully running as a VB guest, but the graphics support has always been bad, since there are no Guest Additions for OSX (BTW this also means no bidirectional clipboard, drag & drop of files, etc..)
There's some support:
VMware only has passthrough support for ESXi/vSphere. Not for VMware workstation or VMware Player. So unfortunately not with Linux as a host.
For the record, VMware Fusion also does not support PCIe passthrough.
Not sure if it's possible with OSX.
What we really need are video drivers for OSX, for virtualbox and/or other virtualisation solutons, so OSX as a guest can work really well.