Not sure if the OP gave his age, but urologists will tell you that as men age, penis size decreases. This is due to decreasing natural testosterone production, which drops about 10% a decade, or 1% per year.
The penis retains its vitality and size, not through sexual intercourse or regular wanking, but by regular "workouts" during the sleep cycle. Throughout the night, the penii of men with normal T cycle back and forth between tumescence and flaccidity. This consistent exercise keeps the penis healthy and strong, even if your woman is imaginary and you haven't been laid in six months.
Around the age of 50 when natural T levels are really dropping, most men will have fewer and fewer nighttime erections and slowly lose firmness and size (about an inch a decade). So, if you're waiting on testosterone replacement therapy (TRT), think about that...
This may be relevant to AAS users who fail to PCT properly and are essentially left with the T of a man in his 50s or 60s.
Everybody's different, but the takeaway here, is pay attention to your nighttime erections. If you aren't having any, you need to fix that.