Clinical safety vs. User engagement: a problem for AI companies to solve

Hello everyone!

I'm a trained psychologist working in a mental health AI startup. My role is twofold: making the service useful and engaging, as well as making it clinically safe.

The problem? User engagement and clinical safety aren't easy to combine. Many features which seems to boost engagement (overvalidation/sycophancy, making AI more humanlike, adding quick fun games, use streaks, etc.) may also endanger clinical safety at the same time. It seems to be quite difficult to maximize both at the same time. Most likely to service will either be more engaging, or more safe.

This feels hard from not only design perspective, but also from the professional standpoint. As a psychologist I find it hard to participate in anything, which will possibly make people unsafe. At the same time, a service/product which is not engaging to users and does not appease the masses, will very possible lose the race against more aggressive and unetheical competitors.

This raises big questions about the direction of development. Should the engagement be raised even if it raises risk of bad mental health outcomes? Or should the safety be put first, even if it raises risk of going completely out of business, also forfeiting the future chance of building something actually safe? The morally right answer is obvious of course, but it seems difficult to actually implement in real life.

Just some thoughts I'm struggling with daily. I'm sure many people in similar positions are struggling with the same theme.

Any thoughts or feelings on this?