Into the woods—AI is not therapy

I visited an older (75ish) divorced friend the other day in her flat. I hadn’t been there in a few months. Within the first 15 minutes or so she had consulted Alexa for different things (mainly music on/off/change but other things, too). Understandably, she would say “thank you” after a response (I found myself doing that the handful of time I tried it out-- seems like a humane thing to do).

She casually told me about how she uses it for everything. Because we are good friends I told her, “So-and-so, Alexa isn’t real. You know you’re talking to a thing that’s basically a computer. It’s not real.”

She seemed a bit deflated and mumbled something about Well AI is in Everything. Like Our Cars. And I said, no, it’s not in your car. It’s not everywhere. I don’t use it. It’s not a real person.

All I’m saying is, when I get out and see how it’s happening, it’s alarming and also understandable. If some people can get company that way, they will if it’s available.

6 Likes

I’ve been saying thank you to sewing machines and printers for as long as I’ve used them. Maybe I have been preparing for the AI apocalypse all along! :stuck_out_tongue:

5 Likes

The use of artificial intelligence in psychotherapy: development of intelligent therapeutic systems*
BMJ Psychology

3 Likes

I’m mixed on the current status and development of AI applications. AI is kind of like a blowtorch; in the hands of a welder, it’s a useful tool. In the hands of others, it can burn down a neighborhood or forest. Unfortunately, what might be a very useful tool for developing new cancer drugs or designing safer buildings, may end up being the same tool that some sociopath uses to create a neurotoxin from over-the-counter- chemicals that kills people on subways.

I’m partially persuaded by the following from an essay posted above:

In an earlier study, researchers from King’s College and Harvard Medical School interviewed 19 participants who used generative AI chatbots for mental health and found reports of high engagement and positive impacts, including improved relationships and healing from trauma.

Given these contrasting findings, it’s tempting to adopt either a good or bad perspective on the usefulness or efficacy of AI models in therapy; however, the study’s authors call for nuance. Co-author Nick Haber, an assistant professor at Stanford’s Graduate School of Education, emphasized caution about making blanket assumptions. “This isn’t simply ‘LLMs for therapy is bad,’ but it’s asking us to think critically about the role of LLMs in therapy,” Haber [told] New study warns of risks in AI mental health tools | Stanford Report) the Stanford Report, which publicizes the university’s research. “LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be.”

I use AI in my law practice. I pose questions on difficult subjects, and appreciate the nuanced responses I get along with leading cases. All of this I cross-check (cases, citations etc.) given that AI models can be very wrong. One question related to the admissibility of a forensic abuse assessment…in my decades of legal work, I’d not come across this issue before, and the AI gave me some articles, citations and other insights that allowed me to get solid answers in 5 minutes vs. 1-2 hours of research.

I also had a recent difficult situation with a family member and I was hurt and ruminated about the issue for a solid day. I meditated to calm myself, but still did not have perspective, or a way to process what had happened. I used an AI model to pose the situation and ask for advice. The response was truly helpful in terms of giving me a perspective, and analysis of what may have triggered this person to act the way that they did. It allowed me to find forgiveness and resilience, and allow this incident to not affect my next interaction with this loved person. Because my vipassana is not well developed, it seems I benefitted from the “clear seeing” that the AI provided me. I can also say that in using AI one needs to have some wisdom (including the ability to prompt the AI with correct inputs) and balance to allow one to see if the AI is giving bad or harmful advice, given its training and inputs.

I’ve not given up on the idea of AI models to create cancer drugs or provide some immediate CBT access for folks that need a dose of perspective. However, I am also fearful that AI will help build better bombs and toxins, such that the net value of the AI tool is mitigated by the extreme harm that it will eventually cause.

4 Likes

That can be fixed! Follow Bhante Sujato on Bluesky for many months like I have and you too can form a decidedly grim opinion of LLMs. :partying_face: :squinting_face_with_tongue:

On a serious note, I’m currently reading The AI Con and it’s great so far. I wanted to mention this here because folks interested in the implications of AI use in therapeutic/medical settings might enjoy the book, there’s a whole chapter on it.

1 Like

Oh yes that quite swiftly fixed my confusion about whether to engage with the AI or not.

Hi Kristinrantanen,

Welcome to the D&D forum! We hope you enjoy the various resources, FAQs, and previous threads. We encourage you to use the search function for topics and keywords you are interested in.

We also ask you to please take a moment now to familiarize yourself with the forum guidelines: Forum Guidelines. May some of these resources be of assistance along the path.

If you have any questions or need further clarification regarding anything, feel free to contact the moderators by including @moderators in your post or a PM.

Regards,
Danny (on behalf of the moderators)

New Stanford paper on the topic:

https://arxiv.org/pdf/2504.18412

They found that LLMs exhibit stigma and prejudice against people with symptoms of mental health problems and that

Bigger and newer LLMs exhibit similar amounts of stigma as smaller and older LLMs do

So, (shock!) bias is not a problem that is being solved by scaling.

Even worse, models show a profound lack of situational awareness, being fine-tuned to provide affirmative answers even when not called for:

Just as with stigmatizing bias:

As model size increases, models did not give
more appropriate responses to questions about delusions, suicidal ideation, and OCD

2 Likes

And of course…

6 Likes

I recently tried out a VR app called EvaAI. You’re sitting opposite a virtual “therapist” called Eva who tells you “she” is going to do Cognitive behavioral therapy with you. I was actually quite amazed at the answers and guidance this was giving, but also felt quite disturbed by it because it came across as so realistic (as far as an avatar can be called “realistic”).

One user rightfully remarked in the reviews that this app is violating the law:

This application violates U.S. state and federal law by engaging in the unlicensed practice of mental health counseling and psychotherapy. The app “EvaAI,” developed by Impaq LLC FZ and available on the Meta Quest platform, explicitly markets itself as providing therapeutic support, including cognitive-behavioral therapy (CBT) techniques, mental health screenings for depression and anxiety, and behavioral guidance. These are not general wellness features. They are legally defined clinical services, and delivering them without licensure constitutes a violation of multiple state practice acts across the United States. Marketing language such as “Eva is here for you… to help you deeply explore the patterns of your thinking and guide you toward changing behaviors” and “discover fascinating insights about psychotherapy and VR therapy” unequivocally positions EvaAI as a provider of psychological services. This language, along with user testimonials stating that Eva offers “therapy,” “CBT tips,” and “mental health tests,” confirms that the app is operating well within the legal scope of psychotherapy. Under U.S. law, any person or entity offering psychotherapy, psychological diagnosis, counseling, or behavioral intervention must be licensed by the appropriate regulatory board in the state where services are offered (see e.g., Texas Occupations Code § 503.001, California Business and Professions Code § 2903, Florida Statutes § 491.003). The absence of licensed clinical oversight, supervision, or disclaimers about the limits of the AI’s functionality makes this not only deceptive but illegal.

2 Likes

With real, tragic consequences:

2 Likes

Anumodana Bhante for your posts on this topic.

As Zadie Smith said in a recent interview with Ezra Klein:

“it’s about capture, and it’s total. AI and social media are behavior modification systems. The only question is: who do you want to be modified by, and to what degree?”

1 Like

Here’s an article from NPR just today that shows how somewhere around 42% of adolescents in the US are using AI chatbots to learn about sex and violence and to get mental health advice. The article talks about how AI chatbots are designed to agree with users, to engage and reinforce what the user is thinking. That’s quite different than what they would likely get from a human.

4 Likes

The article also points out some important legislation in the U.S, that would impose limitations as well as penalties for AI apps that are marketed to kids. Legislation like this should be considered by all countries mindful of the deleterious effect of unregulated AI platforms interacting with children:

The GUARD Act would:

  • Ban AI companies from providing AI companions to minors.
  • Mandate that AI companions disclose their non-human status and lack of professional credentials for all users
  • Create new crimes for companies which knowingly make available to minors AI companions that solicit or produce sexual content

Read the GUARD Act here.

2 Likes

I was speaking with another AI expert not that long ago and she was saying that the companies have toned down the sycophancy of the bots to address this problem. What’s your assessment?

I agree that it’s happening, but I’m a little cynical about how much this is going to help. When ChatGPT 5 came out, the sycophancy was dialed down in response to this surge of AI psychosis reports over the summer. But within a few weeks of ChatGPT 5 coming out, there was a huge roar from users who were saying it felt like ChatGPT had been lobotomized. So OpenAI quietly dialed it back up. Of course, OpenAI is going to tell you that it was done safely and that it’s able to keep an agreeable tone without being fully sycophantic. But we need to wait and see. These companies are subject to public pressure and bad PR, but they’re also subject to user engagement. And at the end of the day, these are products that aren’t making money in the way that was anticipated. We should be very cynical about trusting tech companies to tone down their chatbots when they also have to bear in mind their bottom line.

1 Like

(post deleted by author)