These are two Sanskrit plurals for “dharma”. Dharmāḥ, afaik, is very well attested in Buddhavacana, dictionaries etc. But dharmāṇa appears in Buddhavacana as well, but not in any dictionaries I can find.
What is the origin of these two different means of forming plurals?
Yes. It might be an eccentricity of so-called “Buddhist Hybrid Sanskrit” versus regular Sanskrit? BHS has both dharmāḥ & dharmāṇa accounted for in it.
AFAIK, dhammānu, if it is a plural & a genitive, in Sanskrit is dharmāṇām. Very close. But a single “m” & a lengthened vowel (ā) can make all the difference, especially in Sanskrit & Sanskrit-derived languages.
I think so. I’ve seen a couple of opinions that proper Sanskrit writers would never have ended up with the constructions seen in BHS, perhaps an innuendo that BHS is simply pidgin Sanskrit, at best.