Statistical analysis of Early Buddhist Texts

[Vimala]

@josephzizys

Just having a clean file for each of the Nikayas would be a good thing for machine stuff, with no punctuation marks, no capitals, only single spaces, etc etc

You can very easily filter these things out with regex inside your code.

@josephzizys

Down the track it should be possible to do things like determine the number of case endings in a given sutta/vagga/nikaya - at least in non compound words… it should probably be feasible to identify where words are compound in many cases where they occur as repetitions with non compounded occurances elsewhere in the canon by wildcard searching for string matches with and without spaces.

Indeed, we are currently also working on a Pali stemmer. We have a Sanskrit stemmer here: Buddhanexus and SuttaCentral has a Pali stemmer somewhere in the depths of the code … I think the stemmer is used within Elasticseach so you would need to look into the SC backend code in python. It also uses a stemmer at the frontend (I suspect) for breaking down compounds. In any pali text, turn on the Pali->English lookup tool and click on a long word to see it work.
BuddhaNexus takes the compounds into account when comparing texts to find parallels.

This file might be of help. It has both the stemmer and replacement of all typography: suttacentral/client/elements/lookups/sc-lookup-pli.js at master · suttacentral/suttacentral · GitHub