It doesn’t seem to be the case that all householders wore white, for the Piṅgiyānīsutta says:
Now on that occasion five hundred Licchavis were visiting the Blessed One. Some Licchavis were blue, with a blue complexion, clothed in blue, wearing blue ornaments. Some Licchavis were yellow, with a yellow complexion, clothed in yellow, wearing yellow ornaments. Some Licchavis were red, with a red complexion, clothed in red, wearing red ornaments. Some Licchavis were white, with a white complexion, clothed in white, wearing white ornaments. Yet the Blessed One outshone them all with his beauty and glory.
(AN5.195)
It may have been the case the majority wore white, but given the abundance of sutta references to cloth-dyeing I wouldn’t count on it:
“Bhikkhus, suppose a cloth were pure and bright, and a dyer dipped it in some dye or other, whether blue or yellow or red or carmine; it would look well dyed and pure in colour. Why is that? Because of the purity of the cloth. So too, when the mind is undefiled, a happy destination may be expected.”
(MN7)
What is likely, however, is that all or most adult male brahmins wore white, for this is something that the Manusmṛti requires them to do after they’ve completed their apprenticeship.
If so, then I believe this would be the likely provenance of the odātavasana gīhi expression, for it happens not infrequently that the style or colour of dress that’s perceived as most characteristic or emblematic of a particular group is not necessarily that worn by the majority, but rather, that worn by some social or cultural elite. Think, for example, of the top-hatted Englishmen in Nazi propaganda posters (though a top hat would actually have been way beyond the means of most Englishmen at that time) or the practice of 19th century illustrators of depicting all Chinese with pigtails (when it was in fact only the northeastern Manchus who wore their hair in a queue).