Worst case scenario, you can try self-medicating. Sometimes you need to take things into your own hands. It's up to you. If you feel fine, then don't mess with it. You could experiment though, the thyroid tends to bounce back much more easily than the reproductive system. Speaking of balls, this one study looking at hypo/hyperthyroid and it's effect on sex hormones in rats found that fixing the hypo increased their testes size back to normal. Interesting stuff, endocrinology is fascinating.
OP is in Netherlands
I'm seeing a lot of advice stating to find a new doctor. My family doctor and my urologist doesn't seem to be able to do anything because of the insurance companies. I have been to a couple of urologists over the past couple of years and neither one seem to be able to help with the low T issue. So you can understand somebody's "desperation" to find resolution. I"m in the U.S. by the way.
Get my latest blood results
TSH = 3.16 mU/l and range is 0.400-4.00
Free T4 = 12.6 pmol/l and range is 10.0-24.0
Albumin = 45 g/l and range is 35-50
Prolactin = 142 mU/l and range is 53-360
Based on these number the Dr. sees no reason to look further into hypothyroidism and I agree with him.
All that is left is to see what the Belgium endocrinologist has to say regarding my testosterone being around 10.4 nmol/l (300 ng/dl). I told my current Dr that I am going to Belgium and he said that he would be surprised if a low testosterone is causing my problems as it is within the range of 8 to 28 nmol/l (230 to 800 ng/dl). I will know more in a few weeks.
TSH above 3.00 is considered hypothyroidism as we discussed previously. Did you do your own research on this?