octopus parent
To be fair, VirtualBox runs flawlessly on OSX and Windows. I've also used VirtualBox without problems on Ubuntu. What Linux distribution you use ?
I've had Ubuntu 12.04 guests crash hard (like the commenter mentioned) on OSX and Linux hosts with VirtualBox.
Probably has something to do with the 3D acceleration Unity needs.
In my case (MacBook Air 2013 with 8GB of RAM I run Ubuntu 12.04.4 without a problem).
My installs are headless, no 3D accel or Unity needed.
As originally mentioned, the VBox kernel modules are of poor quality. You may or may not trigger an edge case in your daily use, with or without a desktop environment.
I personally had a terrible time using VirtualBox on a Lenovo T420 running Windows 7 x64 about 2 years ago. I was having frequent blue screens, so Lenovo sent out a tech to swap the motherboard. The issue persisted, and after some time, I noticed that it seemed to always occur when I had multiple VMs running. The problem went away when I migrated to VMware Workstation.
Anecdotally, I've had issues on OSX with VirtualBox, so much so that I switched to VMWare Fusion on Mac hosts...