Tnation emailed me an article a few weeks ago that echoed my own observations on this to an extent...
A few years ago, along with drugs I also quit drinking. I bring this up not because I want a high five but because of the interesting range of responses declining a drink illicits from others. It seems (as a generalization of course) that people who are happy or comfortable with themselves, and more specifically their attitude/relationship with alcohol have pretty much zero issues with a non-drinker. Conversely, people who I would privately suspect drink too much, or often or whatever are the ones who are deeply uncomfortable, or even offended by someone who is able to turn down alcohol.
This seems to also apply, at least in my (generalized) experience with exercise/fitness and particularly nutrition. People who are either fit or at least comfortable with their level of fitness have next to no reaction to either a casual mention of "no thanks I'm planning to hit the gym still today" or even "I'll pass on seconds, great meal though!" (or even 'no cake for me today thanks, watching what I eat this weekend') etc. Out of shape, or overweight people seem to be some of the only ones who can't fathom this kind of willpower and feel the need to comment, usually snidely about something to do with working out or watching macros etc...
Anyways enough of my opinion/observations, this is the article from T-nation: