Detecting AI-generated text by measuring the asserted author's understanding of selected words and/or phrases in the text
This invention is a method for detecting AI-written material which evaluates authorship more directly than purely machine-based methods of AI detection. This method automatically selects relatively-rare key words and/or phrases used in a submitted body of text which is asserted to have been written by a specific person and creates a test to measure understanding of those key words and/or phrases which is administered to that person.
1. A method of using a computer to detect AI-generated content in a written assignment submitted by a student, the method comprising:
creating a plurality of field-specific databases of words and/or phrases based on analysis of one or more bodies of text, wherein a field is a field of study, discipline, field of research, type of profession, and/or news topic, wherein a database in the plurality of databases has a record for each word and/or phrase in the database, wherein each record includes at least one synonym for the word or phrase, and wherein each record also includes a rarity score and/or vocabulary level for the word or phrase;
receiving a written assignment submitted by a student;
scanning the assignment to identify words and/or phrases in the assignment;
using machine-learning to (a) identify the field of the assignment, (b) select a database specific to the field of the assignment, and (c) adjust the rarity scores of words and/or phrases in the database based on the frequency of words and/or phrases identified in the assignment;
assigning rarity scores and/or vocabulary levels from the database to the words and/or phrases in the assignment;
selecting key words and/or phrases from among the identified words and/or phrases, wherein the key words and/or phrases have high rarity scores and/or vocabulary levels in the selected database; and
creating a test to be taken by the student which measures the student's understanding of the key words and/or phrases, wherein the test asks the student to match each key word and/or phrase with a synonym for the key word and/or phrase, wherein a lower percentage of correct answers on the test indicates a higher probability that the student used AI to generate the assignment.