Yes, there’s a bit of an ambiguity in how annihilationsim is viewed. Sometimes it’s seen in very negative terms, otherwise as being philosophically close to Buddhism.
I haven’t studied the issue in detail, but I think the distinction is between the philosophy and the ethical implications that might be drawn from that. If there is a simple view that there is just one life, and if ethical and spiritual practice are included in that, it is probably not such an issue, and may indeed be close to letting go.
On the other hand, if annihilationism is paired with ethical nihilism—as it is sometimes in the suttas—then it is deeply pernicious.
It is common in religious circles to assume that these two views are closely related. But the evidence seems rather the opposite: that religious people are less ethical than non-religious people. This is true even when testing for issues regarded as moral issues by religious people—teen sex, abortion, drug taking, and so on. Such things are regularly higher in regions with strong religious belief. That is to say, the more religious people are, the more likely they are to engage in underage sex, have abortions, get addicted to drugs, and so on.
Such studies as I have seen have been carried out in Christian and partly Muslim areas, so I couldn’t say if it applies to Buddhists as well.
Obviously this doesn’t support the idea that you have to believe in God or heaven or whatever in order to be moral. In my view, when people make these arguments, it’s usually just performative moralism: they want to look like moral people.
On the other hand, it’s probably wise to avoid drawing any causal relations from this information. More highly religious areas in southern US states or southern Europe (where such studies have been done) tend to be poorer and less well educated, with more corrupt governments, less welfare support and so on.