Yes, listen to the doctor - if you do too much stuff you will end up swelling. Trust me, I know from experience. You will end up with four nut sized items in your sack and only two of them being your actual nuts. If that happens, you need to take Ibuprofen like you invented the stuff. I was given 1600mg to be taken twice daily...by the second dose I was feeling much better.
Take the week off, you will thank yourself for it later.