Hsu, Tien-Wei, Tseng, Ping-Tao, Tsai, Shih-Jen, Ko, Chih-Hung, Thompson, Trevor ORCID: 0000-0001-9880-782X , Hsu, Chih-Wei, Yang, Fu-Chi, Tsai, Chia-Kuang, Tu, Yu-Kang, Yang, Szu-Nian, Liang, Chih-Sung ORCID: 0000-0003-1138-5586 and Su, Kuan-Pin (2024) Plagiarism, quality, and correctness of AI-generated vs. human-written abstracts for the same psychiatric research paper. Psychiatry Research. p. 116145. ISSN 0165-1781 (doi:https://doi.org/10.1016/j.psychres.2024.116145)

PDF (Author's Accepted Manuscript) 47807 THOMPSON_AI-generated_Vs_Human-written_Abstracts_For_The_Same_Psychiatric_Research_Paper_(AAM)_2024.pdf - Accepted Version Restricted to Repository staff only until 17 August 2025. Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (510kB) | Request a copy

Abstract

This study aims to assess the ability of artificial intelligence (AI)-based chatbot to generate abstracts from academic psychiatric articles. We provided 30 full-text psychiatric papers to ChatPDF (based on ChatGPT) and prompted generating a similar style structured or unstructured abstract. We further used papers from Psychiatry Research as active comparators (unstructured format). We compared the quality of the ChatPDF-generated abstracts with the original human-written abstracts and the similarity, plagiarism, AI content, and correctness of the AI-generated abstracts. Five experts evaluated the quality of the abstracts using a blinded approach. They also identified the abstracts written by the original authors and validated the conclusions produced by ChatPDF. The study findings showed that the similarity and plagiarism were relative low with only 14.07% and 8.34%. The detected AI-content was 31.48% for generated structure-abstracts, 75.58% for unstructured-abstracts, and 66.48% for active comparators abstracts. For quality, generated structured-abstracts rated similarly to originals, but unstructured ones had significantly lower scores. Experts had 40% accuracy with structured abstracts, 73% with unstructured ones, and 77% for active comparators. However, 30% of AI-generated abstract conclusions were wrong. In conclusion, the data organization capabilities of AI language models hold significant potential for applications in clinical psychiatry. However, the use of ChatPDF to summarize psychiatric papers requires caution of correctness.

Item Type:	Article
Uncontrolled Keywords:	ChatGPT, ChatPDF, academic writing, artificial intelligence
Subjects:	H Social Sciences > H Social Sciences (General)
Faculty / School / Research Centre / Research Group:	Faculty of Education, Health & Human Sciences Faculty of Education, Health & Human Sciences > Institute for Lifecourse Development Faculty of Education, Health & Human Sciences > Institute for Lifecourse Development > Centre for Chronic Illness and Ageing Faculty of Education, Health & Human Sciences > School of Human Sciences (HUM)
Last Modified:	21 Aug 2024 10:46
URI:	http://gala.gre.ac.uk/id/eprint/47807

Actions (login required)

View Item

Downloads