The belief certainly wasn’t completely ubiquitous in ancient India, since even in the suttas there are materialists who deny rebirth. The materialists are also argued with in quite some detail in the Hindu and Jain texts, so appear to have been quite prominent at the time. Search the internet for “Ancient Indian materialism” and you’ll find several scholarly books talking about this.
Also of interest is that most scholars agree the Vedas do not contain the idea of rebirth. It only appears in the Upanishads, which implies that the belief may have been adopted from the native Indians.
I asked a similar question here: