I have to disagree with you here that “there’s no other way around it” because there are many ways around to attain to non-returner.
(1) Be clear that just Jhannas alone are not enough for one to attain to non-returner and one is still considered as “uneducated ordinary person / assutavatā puthujja.”
" …a mendicant, quite secluded from sensual pleasures, secluded from unskillful qualities, enters and remains in the first absorption, which has the rapture and bliss born of seclusion, while placing the mind and keeping it connected. They enjoy it and like it and find it satisfying. If they abide in that, are committed to it, and meditate on it often without losing it, when they die they’re reborn in the company of the gods of Brahmā’s Host. The lifespan of the gods of Brahma’s Host is one eon. An ordinary person stays there until the lifespan of those gods is spent, then they go to hell or the animal realm or the ghost realm. But a disciple of the Buddha stays there until the lifespan of those gods is spent, then they’re extinguished in that very life. This is the difference between an educated noble disciple and an uneducated ordinary person, that is, when there is a place of rebirth."
AN4.123 SuttaCentral
One will need Jhannas + see impermanent to end of defilement or at least to attain to non-returner.
“…a person, quite secluded from sensual pleasures, secluded from unskillful qualities, enters and remains in the first absorption … They contemplate the phenomena there—included in form, feeling, perception, choices, and consciousness—as impermanent, as suffering, as diseased, as an abscess, as a dart, as misery, as an affliction, as alien, as falling apart, as empty, as not-self. When their body breaks up, after death, they’re reborn in the company of the gods of the pure abodes. This rebirth is not shared with ordinary people.”
AN4.124 SuttaCentral
‘The first absorption is a basis for ending the defilements.’ That’s what I said, but why did I say it? Take a mendicant who, quite secluded from sensual pleasures, secluded from unskillful qualities, enters and remains in the first absorption. They contemplate the phenomena there—included in form, feeling, perception, choices, and consciousness—as impermanent, as suffering, as diseased, as an abscess, as a dart, as misery, as an affliction, as alien, as falling apart, as empty, as not-self. They turn their mind away from those things, and apply it to the deathless: ‘This is peaceful; this is sublime—that is, the stilling of all activities, the letting go of all attachments, the ending of craving, fading away, cessation, extinguishment.’ Abiding in that they attain the ending of defilements. If they don’t attain the ending of defilements, with the ending of the five lower fetters they’re reborn spontaneously, because of their passion and love for that meditation. They are extinguished there, and are not liable to return from that world. "
AN9.36 SuttaCentral
(2) One who’s practicing a heart full of love, compassion, rejoicing, and equanimity + seeing impermanence can also attain to non-returner.
“a person meditates spreading a heart full of love (compassion, rejoicing, and equanimity) to one direction, and to the second, and to the third, and to the fourth. In the same way above, below, across, everywhere, all around, they spread a heart full of love to the whole world—abundant, expansive, limitless, free of enmity and ill will. They contemplate the phenomena there—included in form, feeling, perception, choices, and consciousness—as impermanent, as suffering, as diseased, as an abscess, as a dart, as misery, as an affliction, as alien, as falling apart, as empty, as not-self. When their body breaks up, after death, they’re reborn in the company of the gods of the pure abodes. This rebirth is not shared with ordinary people”.
AN4.126 SuttaCentral
(3) One who’s practicing the other progressive meditations such as the dimension of infinite space, the dimension of infinite consciousness, the dimension of nothingness + seeing impermanence can also attain to non-returner. AN9.36 SuttaCentral
(4) One who has “faith, conscience, prudence, energy, and wisdom regarding skillful qualities” + ending of the five lower fetters can also attain to non-returner.
And what kind of person rises up then finds a footing? It’s the kind of person who, rising up, thinks: ‘It’s good to have faith, conscience, prudence, energy, and wisdom regarding skillful qualities.’ With the ending of the five lower fetters they’re reborn spontaneously. They are extinguished there, and are not liable to return from that world. This kind of person rises up then finds a footing.
AN7.15 SuttaCentral
(5) Once how’s observing impermanence + ending five lower fetters can also attain to non-returner
Next, take a person who meditates observing impermanence in all conditions. With the ending of the five lower fetters they’re extinguished between one life and the next. …
With the ending of the five lower fetters they’re extinguished upon landing. …
With the ending of the five lower fetters they’re extinguished without extra effort. …
With the ending of the five lower fetters they’re extinguished with extra effort. …
With the ending of the five lower fetters they head upstream, going to the Akaniṭṭha realm. This is the seventh person.