Method and system for implementing machine learning analysis of documents
Disclosed is an approach for performing auto-classification of documents. A machine learning framework is provided to analyze the document, where labels associated with certain documents can be propagated to other documents.
1 . A computer implemented method for determining a confidentiality status for documents using label propagation, comprising:
storing a plurality of documents in a document repository on at least one storage device of a cloud-based storage system, wherein a first subset of documents of the plurality of documents are labeled non-confidential, a second subset of documents of the plurality of documents are labeled confidential, and a third subset of documents of the plurality of documents are to be labeled; and
processing, using the cloud-based storage system, individual documents of the third subset of documents by:
accessing a document in the document repository on the at least one storage device of the cloud-based storage system of the third subset of documents to generate a feature vector representing the document of the third subset of documents and to store the feature vector in a database of feature vectors at the cloud-based storage system;
accessing the database of feature vectors at the cloud-based storage system to identify one or more documents of the first or second subsets of documents as one or more neighbor documents to the document based on at least a corresponding feature vector stored in the database of feature vectors; and
storing, at the cloud-based storage system, a recommended label for the document based on one or more labels for the one or more neighbor documents, and wherein the recommended label corresponds to at least one of confidential or non-confidential and the recommended label is generated using one or more rules for label propagation comprising a first rule that specifies propagation of a label when no conflicting labels are assigned to the one or more neighbor documents, a second rule that specifies propagation of a label when a majority of the one or more neighbor documents with labels have the same label, or a third rule that specifies propagation of a label only for a nearest neighbor document of the one or more neighbor documents.
2 . The computer implemented method of claim 1 , wherein the feature vector comprises a term-frequency vector.
3 . The computer implemented method of claim 2 , wherein the feature vector is generated by tokenizing the respective document, and counting a number of occurrences of respective tokens.
4 . The computer implemented method of claim 3 , wherein the feature vector is generated based on at least a local dictionary, and the local dictionary corresponds to a respective customer of a plurality of customers that used the cloud-based storage system.
5 . The computer implemented method of claim 4 , wherein one or more newly identified tokens are added to the local dictionary.
6 . The computer implemented method of claim 1 , wherein documents of the plurality of documents are clustered into a plurality of groups having a centroid feature vector and are represented in a graph maintaining connections between centroids representing relationships between the centroids.
7 . The computer implemented method of claim 1 , wherein the recommended label is presented to a user for acceptance of the label, and only accepted labels are used for label propagation.
8 . The computer implemented method of claim 1 , wherein accessing the database of feature vectors at the cloud-based storage system to identify documents of the plurality of documents as one or more neighbor documents to the respective document is further based on at least metadata comprising a folder name, a file name, or a document owner.
9 . The computer implemented method of claim 8 , wherein a weight is applied to the feature vector or the metadata that increases or decreases a significance of the feature vector or the metadata.
10 . A system, comprising:
a memory storing a set of instructions;
a processor that executes the set of instructions to cause a set of acts for determining a confidentiality status for documents using label propagation, the set of acts comprising:
storing a plurality of documents in a document repository on at least one storage device of a cloud-based storage system, wherein a first subset of documents of the plurality of documents are labeled non-confidential, a second subset of documents of the plurality of documents are labeled confidential, and a third subset of documents of the plurality of documents are to be labeled; and
processing, using the cloud-based storage system, individual documents of the third subset of documents by:
accessing a document in the document repository on the at least one storage device of the cloud-based storage system of the third subset of documents to generate a feature vector representing the document of the third subset of documents and to store the feature vector in a database of feature vectors at the cloud-based storage system;
accessing the database of feature vectors at the cloud-based storage system to identify one or more documents of the first or second subsets of documents as one or more neighbor documents to the document based on at least a corresponding feature vector stored in the database of feature vectors; and
storing, at the cloud-based storage system, a recommended label for the document based on one or more labels for the one or more neighbor documents, and wherein the recommended label corresponds to at least one of confidential or non-confidential and the recommended label is generated using one or more rules for label propagation comprising a first rule that specifies propagation of a label when no conflicting labels are assigned to the one or more neighbor documents, a second rule that specifies propagation of a label when a majority of the one or more neighbor documents with labels have the same label, or a third rule that specifies propagation of a label only for a nearest neighbor document of the one or more neighbor documents.
11 . The system of claim 10 , wherein the feature vector comprises a term-frequency vector.
12 . The system of claim 11 , wherein the feature vector is generated by tokenizing the respective document, and counting a number of occurrences of respective tokens.
13 . The system of claim 12 , wherein the feature vector is generated based on at least a local dictionary, and the local dictionary corresponds to a respective customer of a plurality of customers that used the cloud-based storage system.
14 . The system of claim 13 , wherein one or more newly identified tokens are added to the local dictionary.
15 . The system of claim 10 , wherein documents of the plurality of documents are clustered into a plurality of groups having a centroid feature vector and are represented in a graph maintaining connections between centroids representing relationships between the centroids.
16 . The system of claim 10 , wherein the recommended label is presented to a user for acceptance of the label, and only accepted labels are used for label propagation.
17 . The system of claim 10 , wherein accessing the database of feature vectors at the cloud-based storage system to identify documents of the plurality of documents as one or more neighbor documents to the respective document is further based on at least metadata comprising a folder name, a file name, or a document owner.
18 . The system of claim 17 , wherein a weight is applied to the feature vector or the metadata that increases or decreases a significance of the feature vector or the metadata.
19 . A computer program product embodied on a non-transitory computer usable medium having stored thereon a sequence of instructions which, when executed by a processor causes a set of acts for determining a confidentiality status for documents using label propagation, the set of acts comprising:
storing a plurality of documents in a document repository on at least one storage device of a cloud-based storage system, wherein a first subset of documents of the plurality of documents are labeled non-confidential, a second subset of documents of the plurality of documents are labeled confidential, and a third subset of documents of the plurality of documents are to be labeled; and
processing, using the cloud-based storage system, individual documents of the third subset of documents by:
accessing a document in the document repository on the at least one storage device of the cloud-based storage system of the third subset of documents to generate a feature vector representing the document of the third subset of documents and to store the feature vector in a database of feature vectors at the cloud-based storage system;
accessing the database of feature vectors at the cloud-based storage system to identify one or more documents of the first or second subsets of documents as one or more neighbor documents to the document based on at least a corresponding feature vector stored in the database of feature vectors; and
storing, at the cloud-based storage system, a recommended label for the document based on one or more labels for the one or more neighbor documents, and wherein the recommended label corresponds to at least one of confidential or non-confidential and the recommended label is generated using one or more rules for label propagation comprising a first rule that specifies propagation of a label when no conflicting labels are assigned to the one or more neighbor documents, a second rule that specifies propagation of a label when a majority of the one or more neighbor documents with labels have the same label, or a third rule that specifies propagation of a label only for a nearest neighbor document of the one or more neighbor documents.
20 . The computer program product of claim 19 , wherein the feature vector comprises a term-frequency vector.
21 . The computer program product of claim 20 , wherein the feature vector is generated by tokenizing the respective document, and counting a number of occurrences of respective tokens.
22 . The computer program product of claim 21 , wherein the feature vector is generated based on at least a local dictionary, and the local dictionary corresponds to a respective customer of a plurality of customers that used the cloud-based storage system.
23 . The computer program product of claim 22 , wherein one or more newly identified tokens are added to the local dictionary.
24 . The computer program product of claim 19 , wherein documents of the plurality of documents are clustered into a plurality of groups having a centroid feature vector and are represented in a graph maintaining connections between centroids representing relationships between the centroids.
25 . The computer program product of claim 19 , wherein the recommended label is presented to a user for acceptance of the label, and only accepted labels are used for label propagation.
26 . The computer program product of claim 19 , wherein accessing the database of feature vectors at the cloud-based storage system to identify documents of the plurality of documents as one or more neighbor documents to the respective document is further based on at least metadata comprising a folder name, a file name, or a document owner.
27 . The computer program product of claim 26 , wherein a weight is applied to the feature vector or the metadata that increases or decreases a significance of the feature vector or the metadata.