I think that the problem lies with a tight anterior deltoid (front of the shoulder) Lack of flexibility is often a problem with muscle soreness during exercise.
To increase the range of movement and flexibility, try holding a press up (push up) at the very bottom of the movement for 20-25 seconds, and do 3 sets resting for 1 minute inbetween.