This is mistaken. The religion of the brahmins was not Vedism in the time of the Buddha and this is extremely clear from non-Buddhist sources and the suttas themselves. The Brāhmanas are memeorized and contained within each Veda: So the RgVeda as a whole contains the Aitareya Brāhmana for instance depending on the specific school. The Atharvaveda also has some material even older than the RgVeda despite being slightly later as a whole collection, but we’re talking hundreds of years before the Buddha either way. Much of the cosmology in the EBTs is Atharvavedic as well as we would expect.
The Samhitās (the earliest portion you are referring to) are much more debatable, yes. The oldest parts of the RgVeda don’t even talk of the afterlife; only the later sections of it start bringing it up and there seems to be developing ideas. But the RgVedic religion of the Samhitās was of semi-nomadic and recently settled people, some of it probably going back to maybe around a thousand years before the Buddha; not the religion of Brahmanism in the Buddha’s time that developed substantially from earlier Vedism (despite also having incredible continuity in many regards).
Also, the reason the ‘three Vedas’ are mentioned so much is because these have the powerful mantras recited during ritual and so forth. The Brahmanas explain it and provide instructions/stories/meaning/secret connections/etc. So on top of being contained within the triple Veda, they are not the same type of text.
Things like rebirth and the meaning of ritual can be debated, but one thing that is completely agreed upon is the prominence and primacy of the Brāhmanas even pre-Buddha.