The mugger, through no coercion of Bentham, chooses to go down a finger. It is obvious that the mugger has an insane utility function, but it isn't obvious that Bentham letting him act it out is causing a drop in overall utility.
If the mugger doesn't want his own finger, it is Bentham can choose to trust him that 9 fingers are better than 10. Maybe the mugger is even behaving rationally, maybe the 10th finger has cancer, who knows. As the story illustrates, giving him $10 didn't stop him from losing his finger. There are many factors here that make the situation unclear.
Not really, my utility function weighs some mugger being hurt at 0
I imagine you could define utility that way, but presumably the mugger could increase the cost (two fingers? an arm?) until the argument works. Also, if you do definite a utility function like that (say, "there is more utility in this £10 being mine rather than yours than the utility of your arm") then that's a pretty questionable morality.