Unless I misunderstood you, the government would know who I specifically voted for in your scheme? That would be a huge no-go!
Yes, you misunderstood me. Imagine there are many election authorities, maybe perhaps every candidate has their own election authority. Each election authority creates their own public/private key and the public key of the election is created by joining the public key of all authorities.
Using a mixnet (note: Tor uses mixnets too), you can anonymize and decrypt the ballots while preserving the secrecy of the ballot. Not even the servers know at any point that you voted to Hillary/Trump, but the tally is done. And the whole process is end-to-end verifiable by any external party.
It's the magic of math.
You create a plaintext ballot, then you encrypt the ballot with the election authorities' keys, then you sign your encrypted ballot with your own private key (then you send the ballot to the server).
You are signing a ballot that is already encrypted.