Metric for assessing a quality of one or more paraphrases
A computing system includes a memory; and processing circuitry in communication with the memory. The processing circuitry is configured to: receive a paraphrase comprising a paraphrase text sample corresponding to an original text sample; and calculate a paraphrase metric value corresponding to the paraphrase, wherein the paraphrase metric value is calculated based on an adequacy score, a novelty score, and a fluency score of the paraphrase, the adequacy score indicating an extent to which the paraphrase text sample preserves a meaning of the original text sample, the novelty score indicating a level of difference between words and characters of the paraphrase text sample and words and characters of the original text sample, and the fluency score indicating an extent to which the paraphrase text sample is devoid of repetition, spelling, and grammatical mistakes.
1 . A computing system comprising:
a memory; and
processing circuitry in communication with the memory, wherein the processing circuitry is configured to:
receive a paraphrase comprising a paraphrase text sample corresponding to an original text sample;
calculate a paraphrase metric value corresponding to the paraphrase, wherein the paraphrase metric value is calculated based on an adequacy score, a novelty score, and a fluency score of the paraphrase, the adequacy score indicating an extent to which the paraphrase text sample preserves a meaning of the original text sample, the novelty score indicating a level of difference between words and characters of the paraphrase text sample and words and characters of the original text sample, and the fluency score indicating an extent to which the paraphrase text sample is devoid of repetition, spelling, and grammatical mistakes, and wherein the novelty score or the fluency score are configured to penalize the paraphrase metric value based on a longest common subsequence score of the paraphrase compared to a benchmark longest common subsequence score for a previously generated paraphrase;
save the paraphrase metric value corresponding to the paraphrase to a paraphrase metric database;
save the paraphrase and the paraphrase metric value to a set of training data for a paraphrase generation model;
update the paraphrase generation model using the set of training data including the paraphrase and the paraphrase metric value;
re-train, using first paraphrases associated with relatively high paraphrase metrics, the paraphrase generation model to generate paraphrases similar to the first paraphrases; and
re-train, using second paraphrases associated with relatively low paraphrase metrics, the paraphrase generation model to avoid generating paraphrases similar to the second paraphrases.
2 . The computing system of claim 1 , wherein the original text sample comprises a first set of characters and the paraphrase text sample comprises a second set of characters, wherein the second set of characters include one or more differences from the first set of characters.
3 . The computing system of claim 1 , wherein to calculate the paraphrase metric value, the processing circuitry is configured to:
determine the adequacy score;
calculate the novelty score based on the longest common subsequence score and the benchmark longest common subsequence score, wherein the benchmark longest common subsequence score corresponds to a group of paraphrase text samples each corresponding to the original text sample;
calculate the fluency score based on the longest common subsequence score of the paraphrase and the benchmark longest common subsequence score; and
calculate the paraphrase metric value based on the adequacy score, the novelty score and the fluency score.
4 . The computing system of claim 1 , wherein to calculate the paraphrase metric value, the processing circuitry is configured to calculate the paraphrase metric value based on the adequacy score, the novelty score, the fluency score and a paraphrase length score of the paraphrase, the paraphrase length score indicating an extent to which a length of the paraphrase text sample differs from a length of the original text sample.
5 . The computing system of claim 4 , wherein the processing circuitry is further configured to calculate the paraphrase length score based on the length of the paraphrase text sample and the length of the original text sample.
6 . The computing system of claim 4 , wherein to calculate the paraphrase metric value, the processing circuitry is configured to:
multiply the adequacy score, the novelty score, the fluency score, and the paraphrase length score such that:
when the novelty score is less than one, the novelty score weighs towards decreasing the paraphrase metric value so that the paraphrase metric value is lower than the adequacy score;
when the fluency score is less than one, the fluency score weighs towards decreasing the paraphrase metric value so that the paraphrase metric value is lower than the adequacy score; and
when the paraphrase length score is less than one, the paraphrase length score weighs towards decreasing the paraphrase metric value so that the paraphrase metric value is lower than the adequacy score.
7 . The computing system of claim 1 ,
wherein to calculate the paraphrase metric value, the processing circuitry is configured to multiply the adequacy score, the novelty score, and the fluency score such that:
when the novelty score is less than one, the novelty score weighs towards decreasing the paraphrase metric value, and
when the fluency score is less than one, the fluency score weighs towards decreasing the paraphrase metric value.
8 . The computing system of claim 1 , wherein the processing circuitry is further configured to:
execute the paraphrase generation model in order to generate the paraphrase text sample based on the original text sample,
wherein the paraphrase generation model uses an artificial neural network (ANN), a deep neural network (DNN), or another kind of neural network.
9 . The computing system of claim 8 , wherein the processing circuitry is configured to:
save the paraphrase and the paraphrase metric value to a set of testing data, wherein the set of testing data includes a set of paraphrases generated by the paraphrase generation model, and wherein each paraphrase of the set of paraphrases corresponds to a paraphrase metric value of a set of paraphrase metric values; and
test, based on the set of paraphrases generated by the paraphrase generation model and the corresponding set of paraphrase metric values, an ability of the paraphrase generation model to generate quality paraphrases, wherein the paraphrase metric value corresponding to each paraphrase of the set of paraphrases indicates a level of quality of the respective paraphrase.
10 . The computing system of claim 1 , wherein the processing circuitry is further configured to evaluate, based on the paraphrase metric value, a quality of the paraphrase text sample.
11 . The computing system of claim 1 , wherein the paraphrase is a first paraphrase, wherein the paraphrase text sample is a first paraphrase text sample, wherein the original text sample is a first original text sample, wherein the paraphrase metric value is a first paraphrase metric value, wherein the adequacy score is a first adequacy score, the novelty score is a first novelty score, and the fluency score is a first fluency score, and wherein the processing circuitry is configured to:
receive a second paraphrase comprising a second paraphrase text sample corresponding to a second original text sample;
calculate a second paraphrase metric value corresponding to the second paraphrase, wherein the second paraphrase metric value is calculated based on a second adequacy score, a second novelty score, and a second fluency score of the second paraphrase;
save the second paraphrase metric value corresponding to the second paraphrase to the paraphrase metric database; and
save the second paraphrase and the second paraphrase metric value to a set of training data for the paraphrase generation model.
12 . A method comprising:
receiving, by processing circuitry in communication with a memory, a paraphrase comprising a paraphrase text sample corresponding to an original text sample;
calculating, by the processing circuitry, a paraphrase metric value corresponding to the paraphrase, wherein the paraphrase metric value is calculated based on an adequacy score, a novelty score, and a fluency score of the paraphrase, the adequacy score indicating an extent to which the paraphrase text sample preserves a meaning of the original text sample, the novelty score indicating a level of difference between words and characters of the paraphrase text sample and words and characters of the original text sample, and the fluency score indicating an extent to which the paraphrase text sample is devoid of repetition, spelling, and grammatical mistakes, and wherein the novelty score or the fluency score are configured to penalize the paraphrase metric value based on a longest common subsequence score of the paraphrase compared to a benchmark longest common subsequence score for a previously generated paraphrase;
saving, by the processing circuitry, the paraphrase metric value corresponding to the paraphrase to a paraphrase metric database;
saving, by the processing circuitry, the paraphrase and the paraphrase metric value to a set of training data for a paraphrase generation model;
updating, by the processing circuitry, the paraphrase generation model using the set of training data including the paraphrase and the paraphrase metric value;
retraining, by the processing circuitry and using first paraphrases associated with relatively high paraphrase metrics, the paraphrase generation model to generate paraphrases similar to the first paraphrases; and
re-training, by the processing circuitry and using second paraphrases associated with relatively low paraphrase metrics, the paraphrase generation model to avoid generating paraphrases similar to the second paraphrases.
13 . The method of claim 12 , wherein the original text sample comprises a first set of characters and the paraphrase text sample comprises a second set of characters, wherein the second set of characters include one or more differences from the first set of characters.
14 . The method of claim 12 , wherein calculating the paraphrase metric value comprises:
determining, by the processing circuitry, the adequacy score;
calculating, by the processing circuitry, the novelty score based on the longest common subsequence score and the benchmark longest common subsequence score, wherein the benchmark longest common subsequence score corresponds to group of paraphrase text samples each corresponding to the original text sample;
calculating, by the processing circuitry, the fluency score based on the longest common subsequence score of the paraphrase and the benchmark longest common subsequence score; and
calculating, by the processing circuitry, the paraphrase metric value based on the adequacy score, the novelty score and the fluency score.
15 . The method of claim 12 , wherein calculating the paraphrase metric value comprises calculating the paraphrase metric value based on the adequacy score, the novelty score, the fluency score and a paraphrase length score of the paraphrase, the paraphrase length score indicating an extent to which a length of the paraphrase text sample differs from a length of the original text sample.
16 . The method of claim 15 , further comprising calculating, by the processing circuitry, the paraphrase length score based on the length of the paraphrase text sample and the length of the original text sample.
17 . The method of claim 15 , wherein calculating the paraphrase metric value comprises:
multiplying the adequacy score, the novelty score, the fluency score, and the paraphrase length score such that:
when the novelty score is less than one, the novelty score weighs towards decreasing the paraphrase metric value so that the paraphrase metric value is lower than the adequacy score;
when the fluency score is less than one, the fluency score weighs towards decreasing the paraphrase metric value so that the paraphrase metric value is lower than the adequacy score; and
when the paraphrase length score is less than one, the paraphrase length score weighs towards decreasing the paraphrase metric value so that the paraphrase metric value is lower than the adequacy score.
18 . The method of claim 12 , wherein calculating the paraphrase metric value comprises:
multiplying the adequacy score, the novelty score, and the fluency score such that:
when the novelty score is less than one, the novelty score weighs towards decreasing the paraphrase metric value, and
when the fluency score is less than one, the fluency score weighs towards decreasing the paraphrase metric value.
19 . A non-transitory computer readable medium comprising instructions that when executed cause one or more processors to:
receive a paraphrase comprising a paraphrase text sample corresponding to an original text sample;
calculate a paraphrase metric value corresponding to the paraphrase, wherein the paraphrase metric value is calculated based on an adequacy score, a novelty score, and a fluency score of the paraphrase, the adequacy score indicating an extent to which the paraphrase text sample preserves a meaning of the original text sample, the novelty score indicating a level of difference between words and characters of the paraphrase text sample and words and characters of the original text sample, and the fluency score indicating an extent to which the paraphrase text sample is devoid of repetition, spelling, and grammatical mistakes, and wherein the novelty score or the fluency score are configured to penalize the paraphrase metric value based on a longest common subsequence score of the paraphrase compared to a benchmark longest common subsequence score for a previously generated paraphrase;
save the paraphrase metric value corresponding to the paraphrase to a paraphrase metric database; and
save the paraphrase and the paraphrase metric value to one or more of a set of testing data or a set of training data for a paraphrase generation model;
update the paraphrase generation model using the set of training data including the paraphrase and the paraphrase metric value
re-train, using first paraphrases associated with relatively high paraphrase metrics, the paraphrase generation model to generate paraphrases similar to the first paraphrases; and
re-train, using second paraphrases associated with relatively low paraphrase metrics, the paraphrase generation model to avoid generating paraphrases similar to the second paraphrases.