Turing Auditing – Detecting AI Generated Text
A University of Pittsburgh researcher has developed a method, Turing Auditing, to accurately detect the presence of text produced by generative artificial intelligence (GenAI) in a work product. Through the combination of proprietary forensic and investigative reading techniques alongside commercially available AI tools, Turing Auditing can be used to assess any written work product (e.g., academic work, cover letters, transcripts, legal filings). The potential applications of this method include detecting academic integrity violations, plagiarism detection, prevention of copyright infringements, and identifying malicious activity online.
Description
With the rise in commercially available GenAI software there is a growing concern, particularly in academia, of GenAI being used to generate work product that is fraudulently presented as the original work of a human author. This is a violation of most universities’ academic integrity standards and can promote plagiarism, inhibit creativity, and reduce real understanding of a topic. GenAI can also produce incorrect results and lead to legal liability if an AI-driven decision is incorrect or dangerous. There is a growing need to accurately identify the use of GenAI in writing. Turing Auditing is a multistep method to assess writing work and calculate the probability of GenAI being used in the writing process. Turing Auditing can be used to produce a standard report format for clear communication with students during any disciplinary process.Applications
• Examining academic work for AI generated text• Assessing any written work for use of AI generated text
Advantages
Automated AI detectors are commonly used by universities to check students work; however, research has demonstrated these lack accuracy. In addition, sophisticated prompts and AI editors can produce work that evades detection.Turing Auditing combines close reading and distant reading skills with automated examination of text to score writing for a probability of GenAI usage. Document metadata and cited source tracking are also used to assess written work. Turing Auditing uses proprietary metadata-defined agent persona (MDAP) prompt engineering and commercially available large language models (LLMs) to grade writing. As LLMs favor LLM writing styles, an elevated grade or glowing testimonial from an LLM strongly suggests GenAI was used to produce work. Finally, Turing Auditing also include a qualitative formulaic metric and assessment criteria in addition to a report template to ensure full compliance with any disciplinary process.