Document compliance evaluation using a large language model

1 . A method of determining a compliance of a first document and ensuring the first document does not include a hallucination using a first large language model, the method comprising:

separating, by a compliance evaluation system having at least one computer processor, the first document into multiple chunks of text;

evaluating a first chunk of the multiple chunks of text to determine whether the first chunk is a standard chunk type or a nonstandard chunk type, wherein:

in response to the first chunk being a standard chunk type:

determining a purpose of the first chunk;

retrieving a first template chunk having a similar purpose; and

determining a first similarity score representative of a similarity between the first chunk and the first template chunk;

in response to the first chunk being a nonstandard chunk type;

providing the first chunk to the first large language model;

prompting the first large language model to evaluate the first chunk for a hallucination; and

determining, by the first large language model, a first hallucination score representative of at least an amount of the first chunk that is a hallucination created when the first chunk was generated by another large language model;

evaluating, concurrently with the evaluation of the first chunk, a second chunk of the multiple chunks of text to determine whether the second chunk is a standard chunk type or a nonstandard chunk type, wherein:

in response to the second chunk being a standard chunk type:

determining a purpose of the second chunk;

retrieving a second template chunk having a similar purpose; and

determining a second similarity score representative of a similarity between the second chunk and the second template chunk;

in response to the second chunk being a nonstandard chunk type;

providing the second chunk to the first large language model;

prompting the first large language model to evaluate the second chunk for a hallucination; and

determining, by the first large language model, a second hallucination score representative of an amount of the second chunk that is a hallucination; and

formulating a compliance score dependent on the first similarity score, the first hallucination score, the second similarity score, and the second hallucination score with the compliance score being representative of a level of compliance of the first document.

2 . The method of claim 1 , further comprising:

evaluating each chunk of the multiple chunks similarly to the evaluation of the first chunk and the evaluation of the second chunk, wherein the compliance score is representative of all similarity scores and hallucination scores from each chunk of the multiple chunks.

3 . The method of claim 1 , further comprising:

evaluating the first document as a whole to determine whether the first document includes all required chunks of the multiple chunks of text by comparing the first document to a first template document, wherein the first document and the first template document are the same type of document; and

formulating a minimum required chunks score dependent upon the comparison of the first document to the second document.

4 . The method of claim 3 , wherein the compliance score is also dependent upon the required chunks score.

5 . The method of claim 3 , wherein the first template document is saved in a template repository.

6 . The method of claim 3 , wherein the first document is a contract that has been edited by a first party and the first template document is an unedited version of the first document.

7 . The method of claim 3 , wherein each of the first similarity score, the first hallucination score, and the minimum required chunks score are equally represented in the compliance score.

8 . The method of claim 3 , further comprising:

in response to the evaluation of the first document revealing that the first document does not include all required chunks, communicating a notification to a user.

9 . The method of claim 1 , wherein the first hallucination score is additionally representative of at least one of: a number of hallucinations present in the first chunk and a content of the first chunk that is a hallucination.

10 . The method of claim 1 , wherein the first document is a contract that is a statement of work.

11 . The method of claim 1 , further comprising:

normalizing the compliance so as to be a value between 0 and 1.

12 . The method of claim 1 , further comprising:

evaluating each chunk of the multiple chunks, wherein the compliance score is representative of all similarity scores and hallucination scores.

13 . The method of claim 1 , further comprising:

communicating the compliance score to a user.

14 . The method of claim 1 , wherein the step of determining a first similarity score representative of the similarity between the first chunk and the first template chunk further comprises:

converting the first chunk to a first vector embedding;

retrieving a second vector embedding corresponding to the first template chunk; and

comparing the first vector embedding to the second vector embedding to determine the similarity between the first chunk and the first template chunk, wherein the first similarity score is representative of the similarity between the first vector embedding and the second vector embedding.

15 . The method of claim 1 , wherein the determination of whether the first chunk is a standard chunk type or a nonstandard chunk type is performed by a classifier module that at least partially includes a computer processor.

16 . The method of claim 1 , wherein the step of separating the first document into multiple chunks of text is performed automatically by a chunk separator module that at least partially includes a computer processor.

17 . The method of claim 1 , further comprising:

receiving the first document; and

storing the first document in storage media.

18 . The method of claim 1 , wherein the nonstandard chunk type is a chunk that was generated via a second large language model.