I think your ability to gain naturally tapers off as your natural test levels go down, but another factor is that it is easier to gain when you are new to training and farther from your genetic potential.
I can't pinpoint when things went "downhill", I would say late twenties. I could get away with more then.
But I am in better shape now at 34, just because I train like clockwork now, no extended vacations since 1997. Plus my diet is a lot better and I don't drink much at all any more. I was never a problem drinker, but even moderate drinking is empty calories and really puts the kabosh on protein synthesis.