Yes, there does seem to be some confusion in all this.
If we just look at the use of nikāya in the sense of “school, sect, group” rather than “body of texts”, then in the old days it was used for what we call the “18” schools of Buddhism. However, the meaning of these “18” schools is not as clear as one might think. There seem to be overlapping usages:
- Doctrinal: a more loose grouping corresponding to certain philosophical characteristics.
- Vinaya: apparently more precise, corresponding to the allegiance to one particular set of Vinaya texts and one monastic lineage found in one monastery or clearly defined group of monasteries.
Thus one “doctrinal” nikāya might encompass a number of Vinaya nikāyas. The smaller Vinaya nikāyas might nominally be said to have minor doctrinal differences, but I doubt how reliable this is: there has never been a monastery where even two monks agreed about every point of interpretation.
So much for the old days. What about now? I’ll leave aside the Chinese and Tibetan traditions, as I don’t know much about them.
In the modern Theravada, which might be called the Theravāda-nikāya, there is ostensibly only one canon in Pali and only one Vinaya tradition. It’s not quite that simple, because some canonical books are not universally accepted, and there are differences between various canons as passed down in Thailand, Mynmar, and Sri Lanka. But still, this is a reasonable approximation, and importantly, all the monastics in the “Theravada” regions perceive themselves as belonging to the “Theravada”. They might agree or disagree as to whether the Milinda should be included in the canon, but they don’t see this as a schismatic issue. This is in notable contrast to the old days, where in the Dipavamsa for example, such differences definitely were grounds for schism.
Now, within the Theravada, there are a number of formally constituted nikāyas in each country. Some of the main ones are already mentioned above. These are not distinguished by doctrine, textual canon, or Vinaya, but by administrative group, to some extent based on ordination lineage.
It is true, the different nikāyas might have different Vinaya practices, but this is not what defines the nikāya. Historically, for example, the Dhammayuttika Nikāya is often said to have a stricter policy than the Mahā Nikāya when it comes to handling money. But I am Mahā Nikāya, and I don’t use money at all, while many Dhammayuttika monks, probably the majority, do. In fact, even prominent members of the Dhammayuttika have in recent years claimed that there is no longer any real difference in Vinaya practice between these two groups. As another example, in Sri Lanka, the Siam Nikāya does ordination based on caste, which the other Nikayas reject. Yet not all Siam Nikaya monks base ordination on caste.
So any Vinaya difference is at the most a broad generalization.
The origins of these nikāyas depend nominally on ordination lineage. Thus the Siam Nikaya is the lineage re-introduced to Sri lanka from Thailand. The Rāmañña Nikaya hails from Burma, as does the Dhammayuttika.
But once again, these things are not clear cut. In Thailand, for example, the Dhammayuttika arose gradually in the 19th century as a result of the reforms introduced by King Mongkut. When he was a monk, he doubted the validity of his ordination, believing that so many of the monks had committed expulsion offences that there was no way to have a valid ordination. He sought a trustworthy source of ordination from the so-called kalyāṇī sīmā tradition in Burma, and this became the origin of the Dhammayuttika. But there are obvious problems with this procedure, not least of which that he himself apparently served as preceptor at ordinations while still not having enough vassas to qualify. Nevertheless, to this day the Dhammayuttika for the most part refuses to perform acts of the Sangha with Maha Nikaya monks (though this itself is against the Vinaya), and they treat us as sāmaṇera (novices). But this depends on the individual, and not everyone follows this.
Anyway, so the Dhammayuttika gradually came to perceive itself as a distinct nikāya, and at a certain point this became legally recognized. The rest of the monks were called Mahā Nikāya.
But what is Mahā Nikāya? So far as I can tell, this idea did not exist at all before the Dhammayuttika, and is nothing more than a negative definition: monks who are not Dhammayuttika. In his book on Thai ordination lineages published in the early 70s, Somdet Nyanasamvara—soon to become chief patriarch—investigated the history of Thai ordination lineages and concluded that there must have been so many comings and goings and different traditions that it was impossible to say anything with certainty. For hundreds of years, monks and nuns must have been coming to Thailand from different parts of India, Bangladesh, Burma; from parts east, Indonesia, Vietnam, and so on; and from the then-diverse schools in Sri Lanka. There must have been all sorts of ordinations with all sorts of combinations of monks in different parts. In the north, for example, the cult of Upagupta is strong, suggesting a close connection with north India schools such as the Mulasarvastivada, whose monks may well have traveled overland to northern Thailand—then a separate nation—quite distinct from what was happening in the south.
So the Mahā Nikāya is not really an “ordination lineage” in any recognizable sense. It is just a generic term for the multiple different, largely unknowable, ordination lineages that existed in Thailand before the 19th century.
Really these groups are primarily administrative categories. They are loosely organized “fraternities”. And don’t neglect the word “loosely” here. In all my time in Thailand, I had hardly any contact with a representative of any of the administration. For the most part, monasteries just get on with what they are doing.
What is the purpose of these administrative groupings? From what I can see, the primary aim is not Buddhist education or monastic training, but the acquisition and control of real estate.
In addition to these “formal” groups, there are of course multiple overlapping informal hierarchies. The “Forest Tradition” is one such. At some points in history, the Forest Tradition was a recognized nikāya, and there have been calls to reintroduce this in modern Thailand. Certainly the forest monks of whatever nikāya have more in common with each other than they do with the city monks. Yet “Forest Tradition” is another problematic term: many monks under this umbrella have never, in fact, lived in the forest. Many “Forest Monasteries” have no forest, and are little more than tourist traps.
Again, within these are more local, teacher-based lineages. Occasionally these become very large, like the Ajahn Chah tradition, but mostly they make up a small circle of monasteries, maybe one main one, a couple of branches, and a retreat spot or three. Relations between these groups may be as fluid or as circumscribed as the monks wish.
And finally, moving within and through all these traditions are monks and nuns who think all of this is kind of silly and stay away from it.