From Transparency to Delegation: A Structured Scoping Review and Risk-Tiered Framework for AI Decision Support in Mental-Health Care
DOI:
https://doi.org/10.66635/ejnvnf86Keywords:
artificial intelligence, mental health, clinical decision support, digital mental health, AI governance, human oversight, explainability, risk escalation, scoping review, algorithmic delegationAbstract
Background: Artificial intelligence is increasingly used in mental-health contexts, including public-facing chatbots, large language models, screening tools, triage systems, risk-prediction models, and clinical decision-support applications. Existing governance frameworks emphasize transparency, safety, validation, accountability, and human oversight, but they often leave unresolved the operational question of delegation: when may AI merely inform, suggest, rank, pre-select, escalate, or act in relation to mental-health care?
Objective: This review aimed to map safeguards proposed, discussed, or evaluated for AI systems that influence mental-health care decisions, with particular attention to delegation, human oversight, explanation, validation, auditability, accountability, and crisis escalation.
Methods: A structured scoping review with framework synthesis was conducted. Academic sources, official policy documents, clinical AI reporting standards, regulatory guidance, and governance literature were identified through academic searching, targeted publisher/platform searches, official-document searches, and citation chaining. After duplicate and clearly overlapping records were removed, 121 records were screened, 85 full texts or official documents were assessed, and 72 sources were included in the extraction matrix and synthesis.
Results: Six recurring AI roles were identified: informing, suggesting, ranking, pre-selecting, escalating, and acting. Seven safeguard categories were mapped across the corpus: disclosure and transparency, explanation, validation, human oversight, auditability and documentation, consent and privacy, and accountability with escalation. The literature contained strong commitments to broad principles but was less consistent in specifying operational thresholds. Human oversight, crisis escalation, explanation, validation, and audit logging were often endorsed without clear guidance on who should review, when review is mandatory, what must be documented, or how safeguards should vary by clinical risk.
Conclusions: Responsible AI governance in mental-health care requires more than general commitments to transparency or human oversight. Safeguards should vary according to both the degree of AI influence and the level of clinical risk. The proposed risk-tiered delegation framework links AI roles and risk levels to minimum safeguards, offering a practical structure for deciding when AI may responsibly guide mental-health care and when human review, documentation, or escalation must be mandatory.
References
1.Abd-Alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019). An overview of the features of chatbots in mental health: A scoping review. International Journal of Medical Informatics, 132, 103978. https://doi.org/10.1016/j.ijmedinf.2019.103978
2.Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020). Effectiveness and safety of using chatbots to improve mental health: Systematic review and meta-analysis. Journal of Medical Internet Research, 22(7), e16021. https://doi.org/10.2196/16021
3.Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Medical Informatics and Decision Making, 20, 310. https://doi.org/10.1186/s12911-020-01332-6
4.Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19-32. https://doi.org/10.1080/1364557032000119616
5.Aromataris, E., & Munn, Z. (Eds.). (2020). JBI manual for evidence synthesis. JBI. https://doi.org/10.46658/JBIMES-20-01
6.Auf, H., Svedberg, P., Nygren, J., Nair, M., & Lundgren, L. E. (2025). The use of artificial intelligence in mental health services to support decision-making: Scoping review. Journal of Medical Internet Research, 27, e63548. https://doi.org/10.2196/63548
7.Carroll, C., Booth, A., Leaviss, J., & Rick, J. (2013). Best fit framework synthesis: Refining the method. BMC Medical Research Methodology, 13, 37. https://doi.org/10.1186/1471-2288-13-37
8.Challen, R., Denny, J., Pitt, M., Gompels, L., Edwards, T., & Tsaneva-Atanasova, K. (2019). Artificial intelligence, bias and clinical safety. BMJ Quality & Safety, 28(3), 231-237. https://doi.org/10.1136/bmjqs-2018-008370
9.Chancellor, S., & De Choudhury, M. (2020). Methods in predictive techniques for mental health status on social media: A critical review. npj Digital Medicine, 3, 43. https://doi.org/10.1038/s41746-020-0233-7
10.Char, D. S., Shah, N. H., & Magnus, D. (2018). Implementing machine learning in health care: Addressing ethical challenges. New England Journal of Medicine, 378(11), 981-983. https://doi.org/10.1056/NEJMp1714229
11.Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., Boulesteix, A.-L., Camaradou, J., Celi, L. A., Denaxas, S., Denniston, A. K., Glocker, B., Golub, R., Harvey, H., Heinze, G., ... Andaur Navarro, C. L. (2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, e078378. https://doi.org/10.1136/bmj-2023-078378
12.Cruz-Gonzalez, P., He, A. W.-J., Lam, E. P., Ng, I. M. C., Li, M. W., Hou, R., Chan, J. N.-M., Sahni, Y., Vinas-Guasch, N., Miller, T., Lau, B. W.-M., & Sánchez Vidaña, D. I. (2025). Artificial intelligence in mental health care: A systematic review of diagnosis, monitoring, and intervention applications. Psychological Medicine, 55, e18. https://doi.org/10.1017/S0033291724003295
13.Cruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., & SPIRIT-AI and CONSORT-AI Working Group. (2020). Guidelines for clinical trial protocols for interventions involving artificial intelligence: The SPIRIT-AI extension. Nature Medicine, 26(9), 1351-1363. https://doi.org/10.1038/s41591-020-1037-7
14.Cui, X., Gu, Y., Fang, H., & Zhu, T. (2025). Development and evaluation of LLM-based suicide intervention chatbot. Frontiers in Psychiatry, 16, 1634714. https://doi.org/10.3389/fpsyt.2025.1634714
15.D'Alfonso, S. (2020). AI in mental health. Current Opinion in Psychology, 36, 112-117. https://doi.org/10.1016/j.copsyc.2020.04.005
16.De Freitas, J., Uğuralp, A. K., Oğuz-Uğuralp, Z., & Puntoni, S. (2024). Chatbots and mental health: Insights into the safety of generative AI. Journal of Consumer Psychology, 34(3), 481-491. https://doi.org/10.1002/jcpy.1393
17.Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126. https://doi.org/10.1037/xge0000033
18.Elyoseph, Z., Gur, T., Haber, Y., Simon, T., Angert, T., Navon, Y., Tal, A., & Asman, O. (2024). An ethical perspective on the democratization of mental health with generative AI. JMIR Mental Health, 11, e58011. https://doi.org/10.2196/58011
19.European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 of 13 June 2024 laying down harmonised rules on artificial intelligence and amending certain Union legislative acts. Official Journal of the European Union, L, 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
20.Fiske, A., Henningsen, P., & Buyx, A. (2019). Your robot therapist will see you now: Ethical implications of embodied artificial intelligence in psychiatry, psychology, and psychotherapy. Journal of Medical Internet Research, 21(5), e13216. https://doi.org/10.2196/13216
21.Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017). Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): A randomized controlled trial. JMIR Mental Health, 4(2), e19. https://doi.org/10.2196/mental.7785
22.Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018). Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: Randomized controlled trial. JMIR Mental Health, 5(4), e64. https://doi.org/10.2196/mental.9782
23.Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121-127. https://doi.org/10.1136/amiajnl-2011-000089
24.Graham, S., Depp, C., Lee, E. E., Nebeker, C., Tu, X., Kim, H.-C., & Jeste, D. V. (2019). Artificial intelligence for mental health and mental illnesses: An overview. Current Psychiatry Reports, 21, 116. https://doi.org/10.1007/s11920-019-1094-0
25.Grote, T., & Berens, P. (2020). On the ethics of algorithmic decision-making in healthcare. Journal of Medical Ethics, 46(3), 205-211. https://doi.org/10.1136/medethics-2019-105586
26.Guo, Z., Lai, A., Thygesen, J. H., Farrington, J., Keen, T., & Li, K. (2024). Large language models for mental health applications: Systematic review. JMIR Mental Health, 11, e57400. https://doi.org/10.2196/57400
27.Habli, I., Lawton, T., & Porter, Z. (2020). Artificial intelligence in health care: Accountability and safety. Bulletin of the World Health Organization, 98(4), 251-256. https://doi.org/10.2471/BLT.19.237487
28.Inkster, B., Sarda, S., & Subramanian, V. (2018). An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: Real-world data evaluation mixed-methods study. JMIR mHealth and uHealth, 6(11), e12106. https://doi.org/10.2196/12106
29.International Medical Device Regulators Forum. (2014). Software as a Medical Device: Possible framework for risk categorization and corresponding considerations (IMDRF/SaMD WG/N12FINAL:2014). https://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-140918-samd-framework-risk-categorization-141013.pdf
30.International Medical Device Regulators Forum. (2017). Software as a Medical Device: Clinical evaluation (IMDRF/SaMD WG/N41FINAL:2017). https://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-clinical-evaluation_1.pdf
31.Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1, 389-399. https://doi.org/10.1038/s42256-019-0088-2
32.Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17, 195. https://doi.org/10.1186/s12916-019-1426-2
33.Khera, R., Simon, M. A., & Ross, J. S. (2023). Automation bias and assistive AI: Risk of harm from AI-driven clinical decision support. JAMA, 330(23), 2255-2257. https://doi.org/10.1001/jama.2023.22557
34.Kretzschmar, K., Tyroll, H., Pavarini, G., Manzini, A., Singh, I., & NeurOx Young People's Advisory Group. (2019). Can your phone be your therapist? Young people's ethical perspectives on the use of fully automated conversational agents in mental health support. Biomedical Informatics Insights, 11, 1178222619829083. https://doi.org/10.1177/1178222619829083
35.Lawrence, H. R., Schneider, R. A., Rubin, S. B., Matarić, M. J., McDuff, D. J., & Jones Bell, M. (2024). The opportunities and risks of large language models in mental health. JMIR Mental Health, 11, e59479. https://doi.org/10.2196/59479
36.Le Glaz, A., Haralambous, Y., Kim-Dufor, D.-H., Lenca, P., Billot, R., Ryan, T. C., Marsh, J., DeVylder, J., Walter, M., Berrouiguet, S., & Lemey, C. (2021). Machine learning and natural language processing in mental health: Systematic review. Journal of Medical Internet Research, 23(5), e15708. https://doi.org/10.2196/15708
37.Lee, E. E., Torous, J., De Choudhury, M., Depp, C. A., Graham, S. A., Kim, H.-C., Paulus, M. P., Krystal, J. H., & Jeste, D. V. (2021). Artificial intelligence for mental healthcare: Clinical applications, barriers, facilitators, and artificial wisdom. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 6(9), 856-864. https://doi.org/10.1016/j.bpsc.2021.02.001
38.Levac, D., Colquhoun, H., & O'Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5, 69. https://doi.org/10.1186/1748-5908-5-69
39.Li, H., Zhang, R., Lee, Y.-C., Kraut, R. E., & Mohr, D. C. (2023). Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being. npj Digital Medicine, 6, 236. https://doi.org/10.1038/s41746-023-00979-5
40.Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., & SPIRIT-AI and CONSORT-AI Working Group. (2020). Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: The CONSORT-AI extension. Nature Medicine, 26(9), 1364-1374. https://doi.org/10.1038/s41591-020-1034-x
41.Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90-103. https://doi.org/10.1016/j.obhdp.2018.12.005
42.London, A. J. (2019). Artificial intelligence and black-box medical decisions: Accuracy versus explainability. Hastings Center Report, 49(1), 15-21. https://doi.org/10.1002/hast.973
43.Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019). Resistance to medical artificial intelligence. Journal of Consumer Research, 46(4), 629-650. https://doi.org/10.1093/jcr/ucz013
44.McBain, R. K., Cantor, J. H., Zhang, L. A., Baker, O., Zhang, F., Burnett, A., Kofner, A., Breslau, J., Stein, B. D., Mehrotra, A., & Yu, H. (2025). Evaluation of alignment between large language models and expert clinicians in suicide risk assessment. Psychiatric Services, 76(11), 944-950. https://doi.org/10.1176/appi.ps.20250086
45.Miner, A. S., Milstein, A., Schueller, S., Hegde, R., Mangurian, C., & Linos, E. (2016). Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA Internal Medicine, 176(5), 619-625. https://doi.org/10.1001/jamainternmed.2016.0400
46.Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1, 501-507. https://doi.org/10.1038/s42256-019-0114-4
47.Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2020a). From what to how: An initial review of publicly available AI ethics tools, methods and research to translate principles into practices. Science and Engineering Ethics, 26, 2141-2168. https://doi.org/10.1007/s11948-019-00165-5
48.Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020b). The ethics of AI in health care: A mapping review. Social Science & Medicine, 260, 113172. https://doi.org/10.1016/j.socscimed.2020.113172
49.Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18, 143. https://doi.org/10.1186/s12874-018-0611-x
50.National Institute for Health and Care Excellence. (2022). Evidence standards framework for digital health technologies. NICE. https://www.nice.org.uk/what-nice-does/digital-health/evidence-standards-framework-esf-for-digital-health-technologies
51.National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework: AI RMF 1.0 (NIST AI 100-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1
52.National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.600-1
53.Nilsen, P., Svedberg, P., Nygren, J., Frideros, M., Johansson, J., & Schueller, S. M. (2022). Accelerating the impact of artificial intelligence in mental healthcare through implementation science. Implementation Research and Practice, 3, 26334895221112033. https://doi.org/10.1177/26334895221112033
54.Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., Arnaout, R., Kohane, I. S., Saria, S., Topol, E., Obermeyer, Z., Yu, B., Butte, A. J., & the Minimum Information About Clinical Artificial Intelligence Modeling Investigators. (2020). Minimum information about clinical artificial intelligence modeling: The MI-CLAIM checklist. Nature Medicine, 26(9), 1320-1324. https://doi.org/10.1038/s41591-020-1041-y
55.Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342
56.Omar, M., Soffer, S., Charney, A. W., Landi, I., Nadkarni, G. N., & Klang, E. (2024). Applications of large language models in psychiatry: A systematic review. Frontiers in Psychiatry, 15, 1422807. https://doi.org/10.3389/fpsyt.2024.1422807
57.Organisation for Economic Co-operation and Development. (2024). Recommendation of the Council on Artificial Intelligence (OECD/LEGAL/0449). OECD. https://oecd.ai/en/ai-principles
58.Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015). Guidance for conducting systematic scoping reviews. International Journal of Evidence-Based Healthcare, 13(3), 141-146. https://doi.org/10.1097/XEB.0000000000000050
59.Pichowicz, W., Kotas, M., & Piotrowski, P. (2025). Performance of mental health chatbot agents in detecting and managing suicidal ideation. Scientific Reports, 15, 31652. https://doi.org/10.1038/s41598-025-17242-4
60.Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022). AI in health and medicine. Nature Medicine, 28(1), 31-38. https://doi.org/10.1038/s41591-021-01614-0
61.Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020). A governance model for the application of AI in health care. Journal of the American Medical Informatics Association, 27(3), 491-497. https://doi.org/10.1093/jamia/ocz192
62.Ryan, K., Yang, H.-J., Kim, B., & Kim, J. P. (2025). Assessing the impact of AI on physician decision-making for mental health treatment in primary care. npj Mental Health Research, 4, 16. https://doi.org/10.1038/s44184-025-00124-y
63.Sedlakova, J., & Trachsel, M. (2023). Conversational artificial intelligence in psychotherapy: A new therapeutic tool or agent? The American Journal of Bioethics, 23(5), 4-13. https://doi.org/10.1080/15265161.2022.2048739
64.Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019). Machine learning in mental health: A scoping review of methods and applications. Psychological Medicine, 49(9), 1426-1448. https://doi.org/10.1017/S0033291719000151
65.Su, C., Xu, Z., Pathak, J., & Wang, F. (2020). Deep learning in mental health outcome research: A scoping review. Translational Psychiatry, 10, 116. https://doi.org/10.1038/s41398-020-0780-3
66.Tavory, T. (2024). Regulating AI in mental health: Ethics of care perspective. JMIR Mental Health, 11, e58493. https://doi.org/10.2196/58493
67.Timmons, A. C., Duong, J. B., Simo Fiallo, N., Lee, T., Vo, H. P. Q., Ahle, M. W., Comer, J. S., Brewer, L. C., Frazier, S. L., & Chaspari, T. (2023). A call to action on assessing and mitigating bias in artificial intelligence applications for mental health. Perspectives on Psychological Science, 18(5), 1062-1096. https://doi.org/10.1177/17456916221134490
68.Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25(1), 44-56. https://doi.org/10.1038/s41591-018-0300-7
69.Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., ... Straus, S. E. (2018). PRISMA Extension for Scoping Reviews: Checklist and explanation. Annals of Internal Medicine, 169(7), 467-473. https://doi.org/10.7326/M18-0850
70.UNESCO. (2021). Recommendation on the ethics of artificial intelligence. UNESCO. https://www.unesco.org/en/artificial-intelligence/recommendation-ethics
71.U.S. Food and Drug Administration, Health Canada, & Medicines and Healthcare products Regulatory Agency. (2021). Good machine learning practice for medical device development: Guiding principles. https://www.gov.uk/government/publications/good-machine-learning-practice-for-medical-device-development-guiding-principles
72.Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019). Chatbots and conversational agents in mental health: A review of the psychiatric landscape. The Canadian Journal of Psychiatry, 64(7), 456-464. https://doi.org/10.1177/0706743719828977
73.Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B. F., Ibrahim, M., Liu, X., Mateen, B. A., Mathur, P., McCulloch, P., Morgan, L., Rogers, C., Saria, S., Ting, D. S. W., Watkinson, P., Weber, W., ... DECIDE-AI Expert Group. (2022). Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nature Medicine, 28, 924-933. https://doi.org/10.1038/s41591-022-01772-9
74.Vayena, E., Blasimme, A., & Cohen, I. G. (2018). Machine learning in medicine: Addressing ethical challenges. PLOS Medicine, 15(11), e1002689. https://doi.org/10.1371/journal.pmed.1002689
75.Vo, V., Woode, M. E., Carter, S. M., Degeling, C., & Chen, G. (2025). Unravelling public preferences for the use of artificial intelligence mobile health applications in Australia. Value in Health, 28(11), 1696-1704. https://doi.org/10.1016/j.jval.2025.06.013
76.Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., Ossorio, P. N., Thadaney-Israni, S., & Goldenberg, A. (2019). Do no harm: A roadmap for responsible machine learning for health care. Nature Medicine, 25, 1337-1340. https://doi.org/10.1038/s41591-019-0548-6
77.Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & PROBAST Group. (2019). PROBAST: A tool to assess the risk of bias and applicability of prediction model studies. Annals of Internal Medicine, 170(1), 51-58. https://doi.org/10.7326/M18-1376
78.World Health Organization. (2021). Ethics and governance of artificial intelligence for health: WHO guidance. World Health Organization. https://www.who.int/publications/i/item/9789240029200
79.World Health Organization. (2025). Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models. World Health Organization. https://www.who.int/publications/i/item/9789240084759
