I'm on 1.6mg per day and 18.5mg of Zopiclone. Here's my story:
Had severe anxiety attacks (Probably my fault. Not use to the hot humid Japanese summers. Got dehydrated but scared the crap out of me.) Anyways, I was (Am) on a cocktail of meds. One of those is Lorazepam (Ativan in Western countries).
Years later and several ups and downs, I'm now on 1.6mg daily along with Zopiclone still at 18.5mg. Zopiclone is a tranquilizer. Because of the half-life of the drugs, you can go through withdrawal before your next dose. People looking to taper off the meds usually break up their doses so that you get a steady level throughout the day. If you're on 4mg daily (Quite high) I would try taking 2mg midday and 2mg before bed. Like TRT everyone is different and reacts differently. Once you've got that under control for more than a few weeks you can try and remove about 10 percent of the total dose (You may have to titrate with water or milk.) Tapering more than that can lead to more anxiety and may not happen THAT day but weeks later. Which is why after you taper down, you stabilize for a few weeks before your next taper.
Once you've gotten to a point where your Ativan is low enough, I would consider crossing over to Valium. You can get valium in smaller dosages and it's easier to taper (No water titration etc). Valium also has a longer half-life and you'll feel less anxiety during and after the crossover. From there, you still taper at the same amounts. That's how you get yourself off the meds (If you that's what you want to do).
If you need the meds, then I would look into crossing over into something longer lasting and keeping anxiety at bay with more regular dosing. However, that's something you'll have to talk to your doctor with. I can provide more experience concerning this is you like, just PM me.
That all said, I've taken 0.4mg Ativan and 3.5mg Zopiclone just an hour before getting into the gym and I'm fine. I don't feel it. It just stabilizes me. Sometimes I forget and take it after the gym. While not great for my health, it certainly does kick in harder after and I feel more relaxed going home. Sometimes I forget altogether and I end up with headaches until I get my dose in.
So...you can workout. Depending on how your body reacts to these drugs you can get in and get a good workout without any need to feel like you're 'too relaxed'. I do HIIT 3 times a week and it's brutal. If you go in on leg day and you crawl out, you know you're hitting it hard.
(Please, take everything I just wrote with a grain of salt. This is based on my own personal experience. With a doctor prescribing 4mg Ativan, I'd look at maybe getting a different doc. Sounds like a pill pusher.)