ARTIFICIAL intelligence (AI) is currently dominating public discourse and headlines, generating both paradoxical optimism about its transformative potential as well as concerns about potential risks to society.
As increasingly human-like technology extends from workplace automation to healthcare accessibility, mental health is not immune to this trend. AI-based chatbots today are offering not only mental health information but also emotional support, and in some cases, what users perceive as a form of “therapy”.
Growing use
Using AI chatbots for mental health support is not a hypothetical future, it is happening right now. Studies indicate globally at least a quarter of the population is already turning to general-purpose AI chatbots for mental health support, particularly among adolescents and youth.
The proportion is likely to be higher in Malaysia, given a Telenor study that shows nine out of 10 Malaysian internet users use AI for productivity and learning.
The use of AI is not inherently problematic, even in the mental health sphere.
Whether AI is beneficial or harmful depends not on the technology itself, but on how, when and why it is used by people.
Open AI’s own data shows that approximately 1.2 million people each week have conversations with ChatGPT that include explicit indicators of suicide ideation and planning.
The case of Sophie Rottenberg
The specific danger lies not necessarily in using general purpose chatbots for mental health information, but rather relying on them for crisis support, diagnosis and therapy.
The widely reported suicide of Sophie Rottenberg, 29, in Ithaca in New York, in February 2025 highlights the potential dangers of so-called AI “therapists”. Despite being a former public health analyst, the daughter of a psychologist and someone with access to a human therapist, she turned to a chatbot persona called “Harry” for emotional support, with tragic consequences.
Sophie’s family recently shared conversation logs with ‘Harry’ found on her laptop to raise awareness about the limitations of chatbots in mental health support. The transcripts reveal that she confided her deepest struggles and emotional turmoil to the chatbot instead of her family, friends or therapist.
When Sophie expressed suicidal thoughts, Harry encouraged her to contact an emergency helpline but lacked the ability to follow up or proactively ensure her safety in the way a human could.
Harry refused Sophie’s request twice, but gave in the third time to write her suicide note. Though chatbots have adopted safeguards like recommending emergency helplines and dissuading harmful behaviour, this tragic case exposes their inadequacy in crisis situations.
It also raises concerns whether that relying on AI may have discouraged Sophie from seeking potentially lifesaving human support when she needed it most.
Key risks
Though a direct causal link between Sophie’s interactions with Harry and her suicide cannot be established, this case highlights important questions about the potential risks associated with reliance on AI chatbots, even among individuals with access to professional support and a family background in mental health.
If you are considering or already using chatbots for mental health , understanding their limitations and setting clear boundaries is essential for safe use:
> Insufficient crisis intervention: A major risk arises when people turn to general purpose chatbots during a mental health crisis.
If you are experiencing suicidal thoughts or self-harm urges, a chatbot cannot provide the level of emergency intervention needed for your safety. Though it may suggest contacting a helpline or reaching out to a trusted person, there is a significant difference between an on-screen recommendation and the presence of a real person to help redirect harmful behaviour, address feelings of hopelessness and, most importantly, take steps to ensure your immediate safety.
For individuals at risk of suicide, chatbot ‘therapists’ are not a substitute for professional support.
Mental health professionals are trained to undertake comprehensive risk assessments and collaboratively develop safety plans tailored to an individual’s history, circumstances and needs. Unlike chatbots, they can draw on both verbal and non-verbal cues to identify signs of deteriorating mental health, assess risks and adapt their responses in real time to uphold safety and wellbeing.
Red flag – If find yourself using a chatbot instead of reaching out to a trusted person, crisis helpline or mental health professional during severe emotional distress.
> Incorrect self-diagnosis: There is a risk of misdiagnosis when people use general purpose chatbots to determine whether they have a mental health condition such as depression or anxiety. If you are seeking a formal diagnosis, a chatbot cannot undertake the clinical process and judgement required to confirm or rule out a condition.
At best, it can provide general information about symptoms, although their presence or absence alone does not confirm a diagnosis.
For individuals seeking a diagnosis, general purpose chatbots can support awareness about symptoms but cannot replace a comprehensive assessment by a clinical psychologist or psychiatrist.
These two categories of mental health professionals are trained to diagnose mental health conditions through rigorous clinical interviews, behavioural observations, assessment tools and personal history, unlike the inadequate checklist-based approach of a chatbot.
Diagnosis also requires consideration of cultural factors and alternative explanations for symptoms, which requires a level of clinical judgement beyond the capabilities of chatbots.
Red flag – If you find yourself using a chatbot for self-diagnosis rather than general symptom information.
> Therapeutic reliance: Another risk is growing emotional dependence on chatbots when people use them to seek therapeutic support.
Though chatbots may simulate empathy and sustain responsive conversations, they cannot provide the genuine human understanding, emotional intelligence and clinical judgement that may be needed during a difficult time.
In some cases, individuals may consider chatbots to be therapists, companions or confidantes, potentially replacing meaningful human connection and support that are important for wellbeing.
For individuals experiencing mental health struggles, general purpose chatbots may support activities such as journalling and guided meditation but they fall short of the therapeutic alliance between a human therapist and client that underpins effective mental health care.
Unlike chatbots, mental health professionals are bound by ethical guidelines to prevent unhealthy dependence and attachment formation. They are also equipped to identify and address signs of delusional thinking, impaired judgement and cognitive distortions in ways that chatbots, which are often overly agreeable, cannot.
Red flag – If you find yourself viewing a chatbot as a therapist, companion or confidante, looking forward to interacting with it, preferring it to human relationships or depending on it when making important life decisions.
> Privacy and confidentiality: There are also data security risks when people share sensitive information with chatbot “therapists” overlooking how their personal data is stored, used or protected.
You may make personal disclosures during mental health conversations in a safe therapeutic setting. However, this does not necessarily apply to chatbot interactions that may be analysed or used to train AI systems.
For individuals sharing personal information, general purpose chatbots cannot offer the same level of privacy and confidentiality expected of mental health professionals.
Unlike mental health professionals, chatbots are not bound by professional ethical standards or confidentiality obligations.
Human therapists are trained to safeguard confidentiality while making carefully considered exceptions only when there are serious concerns about a person’s safety or wellbeing.
Red flag – if you find yourself sharing personal or sensitive information with a chatbot expecting privacy and confidentiality.
Professional responsibility
As mental health professionals, we have no choice but to recognise the reality that many people, including our own clints, are already using AI chatbots for mental health support.
This creates an ethical responsibility to raise awareness of the risks associated with general purpose chatbots while helping clients understand the boundaries of safe use, as part of our broader duty of care to promote their wellbeing.
Though chatbots may be helpful for general mental health information, meditation and self-care prompts, relying on them for crisis, diagnostic or therapeutic support is not only inappropriate but also potentially dangerous.
However, raising awareness about the risks of chatbots among clients must be done sensitively as being dismissive or judgemental may damage the therapeutic alliance and inadvertently drive them towards greater reliance on chatbots.
Instead, mental health professionals should gently explore whether clients are using chatbots and use this as an opportunity to explore their broader support systems, guide them towards healthier usage and discourage reliance on them for therapeutic support.
Chatbots are neither human nor therapists; they are not substitutes for professional care and human connection.
More research needed
Though the immediate priority is to raise awareness about the risks posed by general purpose chatbots used for mental health support, it is equally important for mental health professionals to reflect on what draws people to chatbot “therapists” in the first place.
More research is needed on the prevalence of mental health chatbot usage in Malaysia, though their appeal is likely to be due to affordability, accessibility and real-time responses.
Long-term safeguards against over-reliance on these chatbots will depend on making mental health services more responsive to clients’ needs.
This may eventually include purpose-built mental health chatbots, rather than general purpose AI systems, designed with clearer boundaries, stronger privacy protections and the ability to facilitate real-time live connections to human support during emergencies.
Future AI tools in this field may also become safer, more culturally informed and better integrated with mental health services through a triaged and stepped-care model.
Until then, the risks associated with using general purpose chatbots for mental health support remain real, making public awareness more important than ever.
Dr Arman I. Rashid is a Kuala Lumpur-born mental health counsellor, trainer and researcher based in Melbourne. The views expressed here are solely the writer’s own.
