The issues I know of with GPU passthrough are:
Non-Quadro NVIDIA cards don't want to work in a VM so you need to disguise KVM to get them to work. NVIDIA says this is a bug but I think they just want you to buy quadro cards. [1]
AMD Bonaire and Hawaii architecture cards have issues with resetting in a VM, so you can't let the VM go to sleep or shutdown. Newer (R9 480) or older AMD cards don't have this issue. [2]
[1]: http://vfio.blogspot.com/2014/08/vfiovga-faq.html?m=1 [2]: http://vfio.blogspot.com/2015/04/progress-on-amd-front.html?...
(not my blog)
They don't want to work, but they do. I have no issue with my GTX1070 after a few minutes after playing with some KVM settings.
The last time I looked, there were significant challenges with getting a Nvidia or iGPU to work.