What is matter after all ? It is a set of conditions, of possibilities, etc. If one were to create a virtual world that is purely mathematical, we would have an object having the property of “12” or of “010101”, we would have an algorithm that says object X with property Y would react like this when in contact with this, etc.
Matter, consciousness, perception etc. all are a set of conditions and a set of algorhitms that governs how the conditionality between them works. For example there is a certain conditionality going on between matter and perception. If one gets hit by a brick, a certain perception will arise.
If we are to find one thing that unifies them all it is conditionality. Matter, consciousness, etc. - everything is linked through conditionality that is going on between them all. That’s why we have “samsara” the conditioned world. Every single thing is this world is linked to the rest through conditionality. Every single thing has a special relationship with other things through this conditionality and the algorhitms that govern it.
What we need to do is try to find how exactly does this conditionality work, what kind of algorhitms governs this conditioned world. And here we have materialism, solipsism, buddhism with it’s 5 aggregate model, etc. trying to explain how this conditionality works. And our job is to find out which one is correct.
There is no particular distinction made between matter and non-material aggregates in the suttas. As B.Sujato said, “the knife is sticked between consciousness and the other 4 aggregates”. As far as we are concerned, matter is just the same as consciousness or perception - it is a set of posibilities, or things that have characteristics, governed by specific algorhitms just like perception or consciousnss or etc. There is no fundamental difference between the 2. Only thing that we need to find out is the exact algorhitms that govern the relationship between them.