Bridging Technology and Therapy: Assessing the Quality and Analyzing the Impact of Human Editing on AI-Generated SOAP Notes in Pediatric Rehabilitation

Abstract

This thesis explores the integration of artificial intelligence (AI) into clinical documentation, focusing on evaluating AI-generated SOAP (Subjective, Objective, Assessment, and Plan) notes in pediatric rehabilitation settings. AI-powered tools, such as Microsoft's Copilot and the University of Waterloo-developed KAUWbot, offer potential efficiencies by automating aspects of clinical documentation. However, their quality, reliability, and applicability to clinical practice have remained largely unexamined.

The research aims to assess and compare the quality of human-authored SOAP notes and AI-generated notes across five key criteria: clarity, completeness, conciseness, relevance, and organization. A dataset of 432 SOAP notes, divided into four pools, was evaluated using a custom rubric. The pools included human-authored notes, Copilot-generated notes edited by occupational therapists, unedited KAUWbot-generated notes, and KAUWbot-generated notes edited by occupational therapists. A rigorous anonymization process ensured evaluator impartiality.

Findings indicate that AI-generated notes, particularly when edited by clinicians, achieve comparable or superior quality to human-authored notes. After editing, notes generated by KAUWbot, a model fine-tuned on pediatric occupational therapy data, exhibited notable improvements in relevance and organization. Statistical analyses demonstrated some differences among note pools, with edited AI-generated notes consistently receiving the highest ratings. This highlights the importance of human oversight in enhancing AI output and tailoring it to specific clinical needs.

The research shows the potential of AI to augment clinical documentation processes, reduce clinician workload, and improve documentation quality. However, it also emphasizes the necessity of human-AI collaboration and robust training to mitigate limitations such as contextual inaccuracies and misclassifications. These findings provide a foundation for future research and practical recommendations for integrating AI into healthcare documentation workflows.