Auto-correcting framework for open information extraction systems
A method for auto-correcting information extracted by an open information extraction system includes inputting an input triple extracted from an input sentence to a grammaticality model trained to output whether the input triple is grammatical or ungrammatical. The input triple is corrected based on the grammaticality model outputting that the input triple is ungrammatical. Embodiments of the present invention can be used in a variety of machine learning and artificial intelligence applications, including but not limited to several anticipated use cases in material informatics, data security, data extraction, and medical/healthcare, for example, for supporting decision making or optimizing and calibrating of extracted chemical compounds from libraries of textbooks and publications.
1 . A method for auto-correcting information extracted by an open information extraction system, the method comprising:
inputting an input triple extracted from an input sentence to a grammaticality model trained to output whether the input triple is grammatical or ungrammatical; and
correcting the input triple based on the grammaticality model outputting that the input triple is ungrammatical.
2 . The method according to claim 1 , further comprising identifying which slot or slots of the input triple are ungrammatical.
3 . The method according to claim 2 , wherein the identifying which slot or slots of the input triple are ungrammatical is performed by computing a grammaticality score for each slot of the input triple and comparing the grammaticality scores of the slots to an overall grammaticality score computed for the input triple.
4 . The method according to claim 3 , further comprising generating a set of new triples by sampling from the input sentence and replacing the slot or slots of the input triple identified to be ungrammatical with candidate words or phrases given a fixed relation with slot or slots of the input triple that were not identified to be ungrammatical.
5 . The method according to claim 4 , wherein the input triple and the set of new triples are input to a similarity score model trained to determine similarity scores between the input triple and each of the new triples.
6 . The method according to claim 5 , wherein the input triple is corrected using one of the new triples determined by the similarity score model to be most similar to the input triple.
7 . The method according to claim 5 , wherein the similarity score model is trained by:
classifying triples extracted by the open information extraction system as being positive or negative; and
determining a shortest string similarity between each of the negative triples and the positive triples to determine one of the positive triples as being a correct triple for the respective negative triple, wherein the correct triples are used in training data for the similarity score model, the training data being in a form of an anchor, a positive and a random positive, wherein the anchor is the respective negative triple, the positive is the correct triple determined for the respective negative triple, and the random positive is a different one of the positive triples.
8 . The method according to claim 7 , wherein the positive triples are determined from annotated sentences used as input for the training, and wherein the negative triples are determined using an evaluation protocol and comparing the triples extracted by the open information extraction system to the positive triples.
9 . The method according to claim 8 , wherein the evaluation protocol indicates which slot or slots is/are erroneous for each of the negative triples, and wherein the shortest string similarity is determined between each of the erroneous slots and a same slot of the positive triples that have other slots that correspond to the respective negative triple.
10 . The method according to claim 9 , wherein the shortest string similarity is determined using a Levenshtein distance.
11 . The method according to claim 1 , wherein the grammaticality model is trained by:
computing abstract representations of unannotated sentences;
computing n-gram grammaticality scores of the abstract representations;
classifying triples extracted by the open information extraction system as being positive or negative;
computing abstract phrases for each of the positive and negative triples; and
using the n-gram grammaticality scores and the abstract phrases for each of the positive and negative triples as training data to a machine learning model.
12 . The method according to claim 11 , wherein the n-gram grammaticality scores are determined using perplexity measurements.
13 . The method according to claim 11 , wherein the machine learning model is a neural network classifier based on a long short-term memory or transformer-based neural model.
14 . A system for auto-correcting information extracted by an open information extraction system comprising one or more hardware processors which, alone or in combination, are configured to provide for execution of the following steps:
inputting an input triple extracted from an input sentence to a grammaticality model trained to output whether the input triple is grammatical or ungrammatical; and
correcting the input triple based on the grammaticality model outputting that the input triple is ungrammatical.
15 . A tangible, non-transitory computer-readable medium containing instructions which, upon being executed by one or more processors, provide for execution of the following steps:
inputting an input triple extracted from an input sentence to a grammaticality model trained to output whether the input triple is grammatical or ungrammatical; and
correcting the input triple based on the grammaticality model outputting that the input triple is ungrammatical.
16 . The method according to claim 1 , wherein the grammaticality model was trained based on n-gram grammaticality scores computed for unannotated sentences from medical reports, and wherein the input sentence is taken from one of the medical reports or a different medical report that is being processed by the open information extraction system.
17 . A method for training a grammaticality model for an open information extraction system, the method comprising:
computing abstract representations of unannotated sentences;
computing n-gram grammaticality scores of the abstract representations;
classifying triples extracted by the open information extraction system as being positive or negative;
computing abstract phrases for each of the positive and negative triples; and
using the n-gram grammaticality scores and the abstract phrases for each of the positive and negative triples as training data to a machine learning model.