I guess doctors are not that different from the general population in that they sometimes voice baseless opinions and preconceptions.
I don't think there is any evidence, or even plausible mechanism for testosterone causing birth defects. Probably not even in a pregnant female, though I suspect it would terminate the pregnancy at sufficient doses.