Retraining document-tagging machine-learned model based on anonymized data
A document management system trains a machine-learned model using a first training set of tagged documents to, when applied to a document, tag one or more portions of the document. The document management system applies the machine-learned model to a target document. One or more portions of the target document incorrectly tagged by the machine-learned model are identified. A feature vector representative of the target document is generated. Each entry of the feature vector is representative of a characteristic of the target document without including private information from the target document. The document management system queries a corpus of documents using the feature vector to identify a set of documents that correspond to the feature vector. A second training set of tagged documents is generated using the identified set of documents. The document management system retrains the machine-learned model using the second training set of tagged documents.
1 . A method comprising:
training, by a document management system, a machine-learned model using a first training set of training documents, each training document in the first set of training documents labeled with first corresponding tags of a plurality of tags;
assigning, by the document management system and based on the machine-learned model processing a target document, a first tag of the plurality of tags to one or more portions of the target document;
identifying, by the document management system, whether a portion of the one or more portions of the target document is incorrectly assigned the first tag based on the machine-learned model processing the target document;
based on identifying that the portion of the one or more portions of the target document is incorrectly assigned the first tag based on the machine-learned model processing the target document, generating, by the document management system, a feature vector for the target document, each entry of the feature vector indicative of a characteristic of the target document, wherein entries of the feature vector include anonymized data created based on the document management system removing sensitive data included in the target document;
querying, by the document management system and based on the anonymized data of the feature vector, a corpus of documents using the feature vector to identify a set of documents corresponding to the feature vector;
generating, by the document management system and based on the identified set of documents, a second training set of training documents, each training document in the second training set of training document labeled with second corresponding tags of the plurality of tags;
retraining, by the document management system, the machine-learned model using the second training set of training documents to be a retrained machine-learned model; and
assigning, by the document management system and based on the retrained machine-learned model processing the target document, a second tag of the plurality of tags to the portion of the one or more portions of the target document.
2 . The method of claim 1 , wherein at least one entry of the feature vector includes at least one of: a Boolean representation, a decimal representation, a count representation, or a string representation.
3 . The method of claim 1 , wherein characteristics of the target document include at least one of: a word type, a word count, a clause type, a clause count, a spacing, a header, a document type, a renewal period, a renewal notice period, a termination date, a start date, a party type, a jurisdiction, a font, or a font size.
4 . The method of claim 1 , wherein querying the corpus of documents using the feature vector to identify the set of documents corresponding to the feature vector comprises:
generating, by the document management system, additional feature vectors, each additional feature vector associated with a document in the corpus of documents;
comparing the feature vector with the additional feature vectors; and
selecting additional feature vectors with a threshold similarity to the feature vector.
5 . The method of claim 1 , wherein the machine-learned model is a first machine-learned model, and wherein the second training set of training documents is generated using a second machine-learned model.
6 . The method of claim 1 , wherein the second training set of training documents includes the first training set of training documents.
7 . The method of claim 1 , wherein retraining the machine-learned model comprises:
applying the machine-learned model to the second training set of training documents to generate predictions of tags for one or more portions of documents in the second set of training documents;
updating weights of the machine-learned model based on the predictions and the second corresponding tags; and
generating, based on the updating the weights of the machine-learned model, the retrained machine-learned model.
8 . The method of claim 1 , wherein generating the feature vector comprises:
identifying the sensitive data included in the target document; and
removing, based on performing anonymity operators, the sensitive data included in the target document.
9 . A system comprising:
processing circuitry; and
computer readable media comprising instructions that, when executed, cause the processing circuitry to:
train a machine-learned model using a first training set of training documents, each training document in the first set of training documents labeled with first corresponding tags of a plurality of tags;
assign, based on the machine-learned model processing a target document, a first tag of the plurality of tags to one or more portions of the document;
identify whether a portion of the one or more portions of the target document is incorrectly assigned the first tag based on the machine-learned model processing the target document;
based on identifying that the portion of the one or more portions of the target document is incorrectly assigned the first tag based on the machine-learned model processing the target document, generate a feature vector of the target document, each entry of the feature vector indicative of a characteristic of the target document, wherein entries of the feature vector include anonymized data created based on removing sensitive data included in the target document;
query, based on the anonymized data of the feature vector, a corpus of documents using the feature vector to identify a set of documents corresponding to the feature vector;
generate, based on the identified set of documents, a second training set of training documents, each training document in the second training set of training documents labeled with second corresponding tags of the plurality of tags;
retrain the machine-learned model using the second training set of training documents to be a retrained machine-learned model; and
assign, based on the retrained machine-learned model processing the target document, a second tag of the plurality of tags to the portion of the one or more portions of the target document.
10 . The system of claim 9 , wherein at least one entry of the feature vector includes at least one of: a Boolean representation, a decimal representation, a count representation, or a string representation.
11 . The system of claim 9 , wherein characteristics of the target document include at least one of: a word type, a word count, a clause type, a clause count, a spacing, a header, a document type, a renewal period, a renewal notice period, a termination date, a start date, a party type, a jurisdiction, a font, or a font size.
12 . The system of claim 9 , wherein to query the corpus of documents using the feature vector to identify the set of documents corresponding to the feature vector, the instructions cause the processing circuitry to:
generate additional feature vectors, each additional feature vector associated with a document in the corpus of documents;
compare the feature vector with the additional feature vectors; and
select additional feature vectors with a threshold similarity to the feature vector.
13 . The system of claim 9 , wherein the machine-learned model is a first machine-learned model, and wherein the second training set of training documents is generated using a second machine-learned model.
14 . The system of claim 9 , wherein the second training set of training documents includes the first training set of training documents.
15 . The system of claim 9 , wherein to retrain the machine-learned model, the instructions cause the processing circuitry to:
apply the machine-learned model to the second training set of training documents to generate predictions of tags for one or more portions of documents in the second set of training documents; and
update weights of the machine-learned model based on the predictions and the second corresponding tags; and
generate, based on updated weights of the machine-learned model, the retrained machine-learned model.
16 . The system of claim 9 , wherein to generate the feature vector, the instructions cause the processing circuitry to:
identify the sensitive data included in the target document; and
remove, based on performing anonymity operators, the sensitive data included in the target document.
17 . Computer-readable media encoded with instructions that, when executed, cause processing circuitry to:
train a machine-learned model using a first training set of training documents, each training document in the first training set of training documents labeled with first corresponding tags of a plurality of tags;
assign, based on the machine-learned model processing a target document, a first tag of the plurality of tags to one or more portions of the document;
identify whether a portion of the one or more portions of the target document is incorrectly assigned the first tag based on the machine-learned model processing the target document;
based on identifying the portion of the one or more portions of the target document is incorrectly assigned the first tag based on the machine-learned model processing the target document, generate a feature vector for the target document, each entry of the feature vector indicative of a characteristic of the target document, wherein entries of the feature vector include anonymized data created based on removing sensitive data included in the target document;
query, based on the anonymized data of the feature vector, a corpus of documents using the feature vector to identify a set of documents corresponding to the feature vector;
generate, based on the identified set of documents, a second training set of training documents, each training document in the second training set of training documents labeled with second corresponding tags of the plurality of tags;
retrain the machine-learned model using the second training set of training documents to be a retrained machine-learned model; and
assign, based on the retrained machine-learned model processing the target document, a second tag of the plurality of tags to the portion of the one or more portions of the target document.
18 . The computer-readable media of claim 17 , wherein at least one entry of the feature vector includes at least one of: a Boolean representation, a decimal representation, a count representation, or a string representation.
19 . The computer-readable media of claim 17 , wherein characteristics of the target document include at least one of: a word type, a word count, a clause type, a clause count, a spacing, a header, a document type, a renewal period, a renewal notice period, a termination date, a start date, a party type, a jurisdiction, a font, or a font size.
20 . The computer-readable media of claim 17 , wherein to generate the feature vector, the instructions cause the processing circuitry to:
identify the sensitive data included in the target document; and
remove, based on performing anonymity operators, the sensitive data included in the target document.