Data processing method, electronic device and computer program product
A method, an electronic device, and a computer program product for processing data is disclosed. The method includes training a classification model based on a plurality of reference documents describing different objects, the trained classification model respectively associating the plurality of reference documents with the described objects. The method further includes determining from the individual words identification information that can identify the objects based on contributions of individual words in the reference documents to the association. Identification information that can identify objects in documents describing the objects may be determined, so that an identification information data set is automatically generated for training a machine learning model that is used to determine the identification information.
1. A method for processing data, comprising:
training a classification model based on a plurality of reference documents describing different objects and a plurality of object identifiers associated with the different objects to obtain a trained classification model, wherein the object identifiers comprise at least one selected from a group consisting of an organization name, an organization code, and personal identifiers, wherein the trained classification model respectively associating the plurality of reference documents with the described objects, wherein the plurality of reference documents comprises at least one of the following:
biographies, medical records, criminal records, historical information of a company, resumes, and bills; and
based on contributions of individual words in the plurality of reference documents to the association, determining, from the individual words, identification information that can identify the objects, wherein the identification information is used to determine personally identifiable information associated with at least one of the described objects, wherein the identification information is further used to automatically generate an identification information data set without manually labeling the identification information, wherein the identification information data set comprises at least a portion of the plurality of reference documents.
2. The method according to claim 1 , wherein determining from the individual words the identification information that can identify the objects comprises:
determining weights assigned to the individual words by the trained classification model, the weights indicating the contributions; and
determining words having the weights exceeding a threshold in the individual words as the identification information.
3. The method according to claim 1 , further comprising:
composing the plurality of reference documents that comprise the identification information into the identification information data set,
using the identification information data set to train a named entity recognition (NER) model, and
using the trained NER model to determine identification information in a target document based on the respective features of a plurality of extents and the association.
4. The method according to claim 3 , wherein the identification information data set is combined with a general data set to train the NER model, and wherein a document in the general data set comprises at least one word that is manually determined as identification information.
5. The method according to claim 3 , wherein the NER model is a long-short term memory model using a conditional random field.
6. The method according to claim 1 , wherein the classification model is a hierarchical attention network.
7. An electronic device, comprising:
a processor; and
a memory coupled to the processor, the memory having instructions stored therein, and the instructions, when executed by the processor, causing the device to execute actions comprising:
training a classification model based on a plurality of reference documents describing different objects and a plurality of object identifiers associated with the different objects to obtain a trained classification model, wherein the object identifiers comprise at least one selected from a group consisting of an organization name, an organization code, and personal identifiers, wherein the trained classification model respectively associating the plurality of reference documents with the described objects, wherein the plurality of reference documents comprises at least one of the following:
biographies, medical records, criminal records, historical information of a company, resumes, and bills; and
based on contributions of individual words in the plurality of reference documents to the association, determining from the individual words identification information that can identify the objects, wherein the identification information is used to determine personally identifiable information associated with at least one of the described objects, wherein the identification information is further used to automatically generate an identification information data set without manually labeling the identification information, wherein the identification information data set comprises at least a portion of the plurality of reference documents.
8. The device according to claim 7 , wherein determining from the individual words the identification information that can identify the objects comprises:
determining weights assigned to the individual words by the trained classification model, the weights indicating the contributions; and
determining words having the weights exceeding a threshold in the individual words as the identification information.
9. The device according to claim 7 , wherein the actions further comprise:
composing the plurality of reference documents that comprise the identification information into the identification information data set,
using the identification information data set to train a named entity recognition (NER) model, and
using the trained NER model to determine identification information in a target document.
10. The device according to claim 9 , wherein the identification information data set is combined with a general data set to train the NER model, and wherein a document in the general data set comprises at least one word that is manually determined as identification information.
11. The device according to claim 9 , wherein the NER model is a long-short term memory model using a conditional random field.
12. The device according to claim 7 , wherein the classification model is a hierarchical attention network.
13. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for processing data, the method comprising:
training a classification model based on a plurality of reference documents describing different objects and a plurality of object identifiers associated with the different objects to obtain a trained classification model, wherein the object identifiers comprise at least one selected from a group consisting of an organization name, an organization code, and personal identifiers, wherein the trained classification model respectively associating the plurality of reference documents with the described objects, wherein the plurality of reference documents comprises at least one of the following:
biographies, medical records, criminal records, historical information of a company, resumes, and bills; and
based on contributions of individual words in the plurality of reference documents to the association, determining, from the individual words, identification information that can identify the objects, wherein the identification information is used to determine personally identifiable information associated with at least one of the described objects, wherein the identification information is further used to automatically generate an identification information data set without manually labeling the identification information, wherein the identification information data set comprises at least a portion of the plurality of reference documents.
14. The non-transitory computer readable medium of claim 13 , wherein determining from the individual words the identification information that can identify the objects comprises:
determining weights assigned to the individual words by the trained classification model, the weights indicating the contributions; and
determining words having the weights exceeding a threshold in the individual words as the identification information.
15. The non-transitory computer readable medium of claim 13 , further comprising:
composing the plurality of reference documents that comprise the identification information into the identification information data set,
using the identification information data set to train a named entity recognition (NER) model, and
using the trained NER model to determine identification information in a target document based on the respective features of a plurality of extents and the association.
16. The non-transitory computer readable medium of claim 15 , wherein the identification information data set is combined with a general data set to train the NER model, and wherein a document in the general data set comprises at least one word that is manually determined as identification information.
17. The non-transitory computer readable medium of claim 16 , wherein the NER model is a long-short term memory model using a conditional random field.
18. The non-transitory computer readable medium of claim 15 , wherein the NER model is a long-short term memory model using a conditional random field.