Subjects: Library Science，Information Science >> Information Retrieval submitted time 2023-08-28
Abstract： Purpose/Significance Explore the similarities and differences between ChatGPT generation and Chinese paper abstracts written by scholars, and analyze the differences in content characteristics between the two, providing reference for AI generated academic paper detection and related research. Method/Process Firstly, taking the field of information resource management as an example, we extracted 500 highly cited papers from library science, information science, and archival science in the past three years. Based on the obtained paper titles, we used the Prompt method to apply the ChatGPT tool to generate corresponding abstract texts and construct a dataset; Secondly, 9 machine learning and deep learning algorithms were used to classify and detect abstract texts generated by ChatGPT and written by scholars; Finally, analyze the similarities and differences between the two from multiple perspectives, including text features, topic models, and ROUGE evaluation, in order to reveal the similarities and differences between the two. Result/Conclusion Mainstream machine learning and deep learning algorithms trained on datasets can effectively distinguish whether abstracts are generated by AI or written by scholars, with BERT and ERNIE performing the best, while RF and Xgboost perform the best among machine learning algorithms. The number of abstract characters and sentences generated by ChatGPT is higher than that written by scholars, and the keywords are mostly template based transitional words; The themes of the two texts are mostly the same, but there are differences in themes such as "disciplinary system" and "digital humanities"; The quantitative analysis of ROUGE and cosine similarity indicates that the abstracts generated by ChatGPT have a significant "resemblance" rather than a "resemblance" to the abstract texts written by scholars.
Peer Review Status:
Awaiting Review
Hits
Hits
Downloads
Comment