Yes, Anagarika, and yet it’s interesting that the tradition seems to offer different Buddhas for different kinds of believers.
One thing we can keep in mind is that there were people in the Buddha’s circle - including his cousin Ananda, and aunt/stepmother Mahapajapati - who would have had plenty of intimate knowledge about what the Buddha looked like as a boy and young man. So, for example, the fact that the Buddha was a “shaveling” monk doesn’t mean we can’t trust the claim hat he had very curly black hair as a young man.
And if he did have unusual, intersex primary sexual features, Pajapati would certainly have known, even if the Buddha himself were very discreet about the fact. So that’s one possible way in which the report might have passed into the tradition.
Also, if there were something physically unusual about the Buddha’s sexual organ, that might have contributed to the feeling of “not fitting in” among his fellow young khattiya braves, and add an additional poignant touch to the ultimate decision to go forth from household life.
But to me, the account of the marks has the feeling of something that came from a legendary package of traits. Perhaps there had been some great leader or figure in the past who had possessed such an unusual set of physical features. (for example, the Egyptian Pharaoh Akhenaten is frequently hypothesized to have have possessed some genetic abnormalities - including elongated arms and fingers.)