I couldn't care less about callouses, but I always wear gloves for two reasons:
1) I need to focuse 100% on the weight I'm lifting, and not on my hands hurting. I also tend to pinch the skin on my hands during some exercises (eg. lat pulldowns)
2) I really couldn't care less about being a 'real man' in the gym, I'm there to lift big and go home. I will put myself through a lot of pain if there is a purpose, to endure pain for no reason at all is crazy to me
If someone could give me a valid (eg. they reduce performance) reason to not wear gloves, I will throw mine out today, but I have not seen a reason yet to make me get rid of the gloves