Very good point.
Lapsing in the precepts, sense restraint, and the overall development of a wholesome lifestyle (leaning towards renunciation) should be far more disconcerting than lack of time spent on the cushion. Always good to remember that sīlabbata-parāmāsa (holding to virtue and duty) applies to anything done out of mere routine, as a duty in itself, with the assumption that mere resolve in that sense is contributing towards their development:
“Ānanda, are all precepts and observances, lifestyles, and spiritual paths fruitful when taken as the essence?”
“This is no simple matter, sir.”
“Well then, Ānanda, break it down.”
“Take the case of someone who cultivates precepts and observances, a lifestyle, and a spiritual path, taking this as the essence. If unskillful qualities grow while skillful qualities decline, that’s not fruitful. However, if unskillful qualities decline while skillful qualities grow, that is fruitful.”
That’s what Ānanda said, and the teacher approved.
-AN 3.78
Keeping in mind AN 10.53, it is quite plausible that merely sitting everyday can induce either a standstill or a deterioration if the basis of the practice has not been purified and fortified. Couple that with:
Well then, Bahiya, purify the very starting point of wholesome states. And what is the starting point of wholesome states? Virtue that is well purified and view that is straight. Then, Bahiya, when your virtue is well purified and your view is straight, based upon virtue, established upon virtue, you should develop the four establishments of mindfulness.
(…)
When, Bahiya, based upon virtue, established upon virtue, you develop these four establishments of mindfulness in such a way, then whether night or day comes, you may expect only growth in wholesome states, not decline.” -SN 47.15
…and it seems clear that lifestyle is the real key to making use of time spent sitting, as opposed to sitting being the key to a wholesome lifestyle: