Project ROBOT-TALK

Recognizing the Origin of roBOtic Texts. Task Automatization and Linguistic Knowledge (ROBOT-TALK)

Project PID2022-140897OB-I00 funded by MICIU/AEI /10.13039/501100011033/ and FEDER Una manera de hacer Europa

september 2023-2026

Go to the Spanish version

ROBOT-TALK is a multidisciplinary applied research project addressing the digital transition to determine semi-automatically whether digital texts “have been produced by automated processes without human intervention” (Carta Derechos Digitales 2021, Chapter XV. 2).  The main objective of the project was to define a methodology based on linguistic, forensic and computational knowledge to identify texts completely generated by AI systems (metaphorically referred to as robots) in Spanish. This methodology is designed to be transferable to other types of forensic authorship analysis. The project’s goal to identify texts generated by Large Language Models (LLMs) is crucial for cybersecurity, truthfulness and information integrity in an increasingly digitalised society. The main hypothesis behind the project was that an author's idiolect can be detected through the linguistic analysis regardless of whether it was written by a human or generated by AI.

The project's results (within the scope of the research conducted so far) demonstrate the following:

(1) There is a set of linguistic features that distinguishes human-written texts from AI-generated texts in Spanish.

(2) These linguistic features are distinctive enough to identify human-written texts from AI-generated texts. Furthermore, these features could even identify which LLM generated a text, provided that a reference corpus is available containing texts by identified authors (both human and AI) and covering different text genres. In addition, these reference texts should be sufficiently long to allow the linguistic features to be identified, and their genre should be comparable to that of the text being analysed.

(3) For authorship attribution, the ROBOT-TALK corpus has been created. ROBOT-TALK is a comparable corpus consisting of a collection of texts with identified authors, providing a baseline for contrastive linguistic analysis, i.e. human-written texts versus texts generated by five different LLMs across three genres, ordered from the most to the least structured: academic writing, news articles and reviews. Regarding the authorship attribution methodology, the project has demonstrated that discriminative linguistic features can be identified manually, through qualitative linguistic analysis, and automatically, through quantitative statistical analysis. Both approaches are possible, provided a large reference corpus such as ROBOT-TALK.

(4) Combining both qualitative and quantitative approaches allows for accurate linguistic analysis of anonymous texts while maintaining reasonable coverage.

(5) Finally, text analysis and linguistic profiling can be further optimised through an analytical dashboard that allows users to visualise, analyse, compare and interpret measurable features extracted from both reference texts and texts under analysis. The dashboard offers different illustrations and visualizations through charts, tables, statistical indicators and interactive tools.

The project's findings pave the way for new research opportunities aiming at the development of linguistic tools that focus on analysing human-written texts, AI-generated texts and texts produced through a combination of human and AI authorship. Although this latter category was not explored within ROBOT-TALK, it is currently being investigated in the new AUTHOR_IA project.