The kind of inferring you seem to be referring to is known today as induction. While inference in general can be misapplied, induction seem particularly prone to misuse. I think it’s certainly difficult, in a non-sarcastic way.
It’s clear to me these pitfalls were well known by the Buddha. A general teaching on problems of inference is found in the Cūlahatthipadopama Sutta (MN 27).
But we also find the particular issue of induction in DN 1:
Here, monks, a certain ascetic or Brahmin has by means of effort, extortion, application, earnestness and right attention attained to such a state of mental concentration that he thereby recalls past existences – one birth, two births, … ten births, a hundred births, a thousand births […] Thus he remembers various past lives, their conditions and details. And he says “the self and the world are eternal, barren like a mountain-peal, set firmly as a post. There beings rush round, circulate, pass away and re-arise, but this remains eternally. Why so? I have by means of effort, extortion, attained to such a state of mental concentration that I have thereby recalled various past existences…That is how I know the self and the world are eternal”
…while the Buddha appears talking about the “origin of the world” in rather differently manner (SN 22.100):
Bhikkhus, this saṃsara is without discoverable beginning. A first point is not discerned of beings roaming and wandering on hindered by ignorance and fettered by craving….
That is, while the induction is reproved in the former quote (particularly, in the case of the world), it is carefully absent in the latter.
I think the same care can be seen in MN 22:
Bhikkhus, you may well cling to that doctrine of self that would not arouse sorrow, lamentation, pain, grief, and despair in one who clings to it. But do you see any such doctrine of self, bhikkhus?”—“No, venerable sir.”—“Good, bhikkhus. I too do not see any doctrine of self that would not arouse sorrow, lamentation, pain, grief, and despair in one who clings to it.
So, in the absence of an explicit assertion, It seems prudent to take to heart the advice the Buddha gave to Mālunkyāputta (MN 63):
Therefore, Mālunkyāputta, remember what I have left undeclared as undeclared, and remember what I have declared as declared.
Not only prudent, but probably the safest route to preserve his words as they were handed down.
Finally, we read the Buddha saying the following of the Noble Disciple (SN 12.20):
When, bhikkhus, a noble disciple has clearly seen with correct wisdom as it really is this dependent origination and these dependently arisen phenomena, […] he will now be inwardly confused about the present thus: ‘Do I exist? Do I not exist? What am I? How am I? This being—where has it come from, and where will it go?’
At the same time, he praised investigating the teachings:
‘How is this? What is the meaning of this?’ They make open what isn’t open, make plain what isn’t plain, dispel doubt on its various doubtful points.
So, I think there’s value in discussing how the suttas should be interpreted and how the dhamma should be practiced. But, if following the example of a Noble Disciple is of value, discussing whether there’s a self or not, apart from the texts…perhaps it’s not something valuable – maybe, the opposite.