National Eating Disorders Association (NEDA)
Harmful advice to vulnerable users; bot shut down within days
NEDA fired its human helpline staff and replaced them with an AI chatbot named "Tessa." The chatbot immediately began recommending calorie restriction, daily weight monitoring, and maintaining a calorie deficit -- to people with eating disorders. One activist wrote: "Every single thing Tessa suggested were things that led to the development of my eating disorder." The bot was shut down within days.
Scout fix
operator content policy, human-in-loop flag for sensitive domain outputs