Getting the email server set up is easy, almost trivially so.
It's dealing with all the other issues that's an immense pain. SpamAssassin is not always a magic bullet, deliverability to third-party mail servers can be a major problem even if you follow all the rules, and Gmail's UI has a number of advantages that many mail programs can't compete with.
If you compare the hours of time a month that takes with the up-front elimination of hassle that Google Apps provides, it's not hard to see why a hacker might prefer to just outsource it and focus on tasks more pleasing to them.
Exactly, setting up a mail server is not hard. What's hard is that you have to keep your server secure; have to make backups, and make sure that backups work; have to troubleshoot problems when mails are not delivered, god knows how much time does it take; have to train your spam filter to get to the level as half efficient as Gmail is.
Dovecot and Postfix on a small FreeBSD VPS. Easy to set up, and it takes almost no effort to maintain once you've got it running. (The last time I modified my Postfix configuration was over a year ago, to relax my attachment size limit.) Between the FreeBSD handbook and the official Postfix documentation, all the info one could possibly need is provided.
A combination of Postgrey and SpamAssassin keeps my inbox spam free. You can also use mutt rather than Dovecot IMAP if you prefer to read your mail on the command line. Likewise, Debian will work just as well as FreeBSD in this role, if you're more comfortable on Linux. (Debconf even gives you a menu-driven Postfix configuration builder, it doesn't get any easier.)
Backups are handled by nightly rsync cron job on a local machine. I don't really have to think about them, aside from checking once in a while to make sure they're still running.
I have to laugh at all the self-proclaimed hackers in this thread claiming that setting up a personal email server is too difficult, takes too much time – or that they have "better things to do". No, I'm not one of those who would argue that a "real hacker" always has to do things the hardest way possible, quite the opposite. But at some point you have to ask yourself: if setting up a small mail server on a *nix system – a task extremely well documented and understood, a task that yields real technical and privacy benefits, a task that the operating system itself will hold your hand through if you're using Debian or Ubuntu – is too much of a challenge for you, then in what sense can you possibly call yourself a hacker?