Typically, we imagine that the Buddha, after leaving his first two teachers and going off to practice his period of extreme austerities, was doing something identical to or similar to the Jains. We tend to think that there’s a strong likelihood he was initiated first in some sort of Brahminical tradition, and then practiced in some Jain-like tradition, and then went off on his own. This provides a pretty balanced picture when we think about the traditions from the time period that survive to this day. Hinduism developed out of Brahminism in interaction with other philosophies and religions, and Jainism developed over time to where it is today.
I don’t have access to the book, but Patrick Olivelle has one titled “Ascetics and Brahmins: Studies in Ideologies and Institutions” (2011). Having skimmed through some of the section summaries and previews, it seems that he talks about how there are plenty of attestations of Brahminical ascetics who would live off of left-over food, wander for alms, go naked, wear bark or animal hides, and do all kinds of austerities.
Now, if we go off of tradition, the Buddha’s first three sermons were on (1) the middle way between asceticism / indulgence, and the 4 noble truths, (2) non-self, and (3) the fire sermon. The last two are clearly addressed to more Brahminical audiences, in that they are responding to Brahminical concepts. The attā in the 2nd sermon is one that is an inner controller, as in the Upaniṣads. The third sermon plays on the symbolism and imagery of fire in response to the ritual fire maintained by brahmins. If I recall correctly, the second sermon is traditionally considered to be taught to the same group as the first sermon as well.
Knowing that extreme asceticism was potentially practiced in Brahminical circles, is it possible that the Bodhisatta was still inspired from a more Brahminical school of thought in starting these austerities? Maybe he thought—after things didn’t work out with Āḷāra Kālāma/Uddaka Rāmaputta—that he should ramp up his practice in-line with some of the other Brahminically-initiated ascetics who opted for an extreme lifestyle of austerities, rather than switching to a completely new school of thought. He never mentions being a student of Nigaṇṭha Nātaputta, nor any of his disciples. Moreover, in MN 26 (one of the more reliable accounts of the Bodhisatta’s backstory), he simply mentions leaving Uddaka Rāmaputta to find the supreme state, and then going to an area in Magadha and eventually realizing it there. There is no mention of an extreme ideological shift. As Bhante @sujato pointed out in his post on the Padhāna Sutta (Snp 3.2), there seems to be less of a clean break than we are sometimes led to believe in a more neat-and-tidy version of the Buddha’s backstory. That is not to say that there wasn’t a major shift in his mindset and practice, but when we combine this with MN 26 and the knowledge of Brahmin asceticism striving for liberation already being established, it would almost make more sense to me that this was the rough trajectory.
Indeed, in MN 85 where he expands on what he did once he arrived at Magadha, he simply realized that he needed to withdraw further from sensuality and did some extreme practices to try and force his mind and body into seclusion from pleasures (he even says that this was an insight of his, giving the simile of wet sticks). There is nothing inherently Jain about this, and the idea of tapas certainly cropped up in Brahminical circles as well. What he describes is a clean transition from (seemingly) Brahminical circles—perhaps focused more on theory and meditation rather than heavy austerity—to experimenting with different forms of fasting, asceticism, and austerities on his own, and then eventually turning to a more balanced approach with jhānic pleasure and less mortifying striving. No mention of Jainism or any other school of thought.
Certainly it would make sense that he may have mixed, mingled, discussed, and borrowed ideas from other ascetics and śramaṇas. We get a very diverse picture of people doing all kinds of practices in the Buddha’s day with some recurring stock formulas. Perhaps the transition was not so abrupt, but he simply decided to take on the austerities (as I said above), and he tried out some different austerities that other people were doing and that he thought could be helpful in this time period. Although he may have left behind a good amount of the ideology of his former teachers, he certainly had the same goal, the conceptual frameworks, the meditation practices, and the spirit to strive for the far-shore and cross over saṁsāra instilled in him, all of which was shared with certain groups of Brahmins (especially the more ascetic ones). Maybe the five ascetics who came with him and attended on him were former ascetic/spiritual-seeking Brahminical students as well who accompanied him on a journey to live a more extreme and austere life like some of their [Vedic] forefathers—or, more simply, their ascetic Brahminical peers who seem to appear constantly throughout the suttas (see note on Bāhiya, or any of the stock formulae for ascetics and brahmins doing austerities). He was quite accomplished according to his former teachers, so perhaps these five ascetics followed him for his skill, convinced that they could figure out the way to liberation (where their teachers got close but failed) living in heavier seclusion and austerity.
We do often hear the Buddha and some of his disciples talk about how good brahmins of the past lived austere, meditative lives, and this seems to be a narrative that Yājñavalkya / muni-minded Vedic groups tended to partake of as well. Perhaps they even had some legendary sages in their lineage under the former Kosalan teachers who had lived hermit or bhikkhu lifestyles.
A lot of the similarities with Jainism could easily be a case of just shared ascetic (samanabrāhmana) ideas in the East. Bausch has already offered what, to me, is a strong case for the āsavās in the Buddha’s teaching being more of Brahminical descent ideologically (perhaps from sramanic cross pollination; the BAU teachings on them in relation to colors and the self certainly have parallels in Jain or even Ājīvika doctrine). All the far-shore language and imagery is easily the same, and it being found across these ascetic traditions is no surprise; they probably pulled from the same ideological substratam at different points in time. I feel like perhaps the Buddha was a casteless sramana who had been initiated in internal/ascetic Brahminical circles, went off on his own as an accomplished meditator and teacher, and experimented with other forms of asceticism without ever making a hard switch over or adopting major doctrinal ideas.
Note: Bāhiya of the Bark Cloth is a case study for this. Not only was he likely involved in an Upanisadic tradition (at least in the narrative, be it later or not) based off of the teaching presented to him in terms of the 4-fold Upanisadic epistemology that is also related to the ātman, but also his bark attire and extreme restraint/austerity (tapas, mentioned by Yājñavalkya in the BAU) led him to believe he was enlightened—which he was not far off from; all he needed was to let go of his metaphysical assumptions and accept the ‘bare’ reality of conditioned experience, leaving behind the epistemological leaps characteristic of most/all Upanisadic systems.
EDIT: In MN 56, the Buddha discusses with a disciple of Nigaṇṭha Nātaputta (Mahāvīra), and is corrected on a basic doctrinal point about how they refer to actions/deeds. This, to me, is more indicative of the fact that the Buddha was not familiar with Jain doctrine personally, and was only informed by word-of-mouth and in encounters with disciples. This is such a fundamental and basic aspect of Jainism (action/karma, here called daṇḍa) that it does not really make sense for the Buddha to be ignorant of it had he been a practitioner. Not a definite, but another clue perhaps.
EDIT2: Both Alexander Wynne, in “The Origin of Early Buddhist Meditation,” and Bronkhorst, in “The Two Traditions of Meditation in Ancient India,” despite disagreeing on some grounds, show that extreme asceticism was a part of early Brahminical meditation, even if somewhat marginal. This includes an extreme form of breath-restraint meditation—sometimes characterized by pressing the tongue to the mouth or clenching the teeth—nearly identical to what the Bodhisatta describes doing in his period of austerities and searching. Mortification and self-denial were in the repertoire in order to free oneself from karmic retribution (which Bausch demonstrated was a concept already present in the Kānva Satapatha Brāhmana) and liberate the ātman, essential goals of early Brahminical contemplative practice. This is, again, more evidence that the Bodhisatta was simply opting for a more extreme and austere form of Brahminical-adjacent practice after departing from his former teachers.