I’m not an expert, and I think even experts would say this is all speculative, but I was taught that around this time, in the so called axial age, Indian society was experiencing a period of "class conflict” between the varnas (which likely had looser boundaries than today) due to the change to an urbanized agriculturalist society.
It is roughly hypothesized that all Indo-European peoples had a common idea of a three class society from the original social context of a non-urbanized pastoralist society. This probably divided people roughly into those with family traditions of craftsmanship, family traditions of lore, and family traditions of martial leadership. Or maybe the categories were different, it’s all hypothetical. But what’s clear is that in the new context of urban civilization these roles, and their relationships changed.
The secular view is that the 16 Great States of the Buddha’s time were probably only ~200 years old at most, and nothing like them had ever before existed in their culture (India’s first urbanization having been done by dravidians).
So, Indian society at this time is still struggling to come to terms with what it means that there are now people like Anathapindika and Bimbisara, people with wealth and power far beyond what was ever possible in prehistoric, pre-urban times. Meanwhile, there had been no proportionate change in spheres that were culturally coded as “brahminical”.
Realistically, there probably wasn’t anything like a universal set of caste laws everyone followed consistently. Rather, brahmins had an oral tradition that placed them at the top (we can be fairly confident it was mostly the same as what we have in the three main Vedas today) and then people made their own decisions based on particular social context.
So what the Buddha does in that sutta, is essentially match legend for legend, and then point out a lot of social realities. “Ok, you think your varna is the best, but what would happen in the following situations?"
It’s a little loose, but imagine as analogy the following hypothetical dialog between a wealthy industrialist and a landless hereditary aristocrat in the west during the industrial revolution
Aristocrat: God blessed my ancestors and decreed they rule
Industrialist: God predestines all things and my commercial success is indicative of his divine favor
Aristocrat: traditionally, my social group is above yours
Industrialist: But wealthy men are able to petition and be granted peerages, while peers cannot simply petition to be granted wealth. Wealth is also respected in all nations, while your noble rank is only respected within a few nations. Wealth can buy things, while nobility cannot. Practically, wealth is superior to nobility.
Aristocrat: I suppose you are right.
Prior to the conversation, there was cognitive dissonance, where one party simultaneously held the view that “my group is the highest” but also knew all of these practical social truths. It’s only when they’re explicitly drawn out and contrasted that the cognitive dissonance is resolved and the old prideful view has to be abandoned.
Then historically what happened in India is aristocratic power was consolidated more and more in through the Mauryan Empire. The Mauryans went so far as to outlaw vedic animal sacrifice, showing that in their time, khattiya’s were clearly supreme over brahmins. But in turn they were overthrown by a brahmin general (an example of why khattiya =/= warrior) to form the Shunga empire, which patronized a lot of critical Hindu cultural and religious works that helped develop and enshrine a solid position for that varna moving forward, even after their empire fell.
However, we can’t be confident exactly how this went, because of issues common to Indian historiography. For example, the puranas (important Hindu holy texts) make the claim that under the Shunga empire four million Buddhists were killed. C.f. The Holocaust / Shoah, where, with the advantage of higher populations, advanced technology, and sophisticated organizational structures, Hitler was able to systematically murder ~ six million Jewish people. It is generally considered implausible that this Shunga genocide of Buddhists actually happened as described. But when events that are that enormous are totally unclear, it’s impossible to accurately and precisely understand what happened. Imagine trying to describe how European society has evolved if you have no idea whether or not the Holocaust happened.
Somehow, eventually, we end up in a situation where, contrary to the Buddha’s examples, all inter-varna marriages are forbidden (but still happen), there’s a stricter sense of jatti, and probably a somewhat novel idea of gotra, with an associated ‘incest’ prohibition that forbids marriage between two persons with the same ‘gotra’ (patrilineal family line) which would have made his Sakyan origin story disgraceful. But how and when that happened is totally unclear. There’s no “anti-varna mixing law of 400AD” to point to.
So, tl;dr the history of the caste system is mysterious and likely involved a lot of changing and redefining over time, including at the time of the Buddha. So it’s not that weird that the arrogant brahmin had some cognitive dissonance going on.