Method and system for generating a conversational agent by automatic paraphrase generation based on machine translation
The present teaching relates to generating a conversational agent. In one example, a plurality of input utterances may be received from a developer. A paraphrase model is obtained. The paraphrase model is generated based on machine translation. For each of the plurality of input utterances, one or more paraphrases of the input utterance are generated based on the paraphrase model. For each of the plurality of input utterances, at least one of the one or more paraphrases is selected based on an instruction from the developer to generate selected paraphrases. The conversational agent is generated based on the plurality of input utterances and the selected paraphrases.
1. A method, implemented on a machine having at least one processor, storage, and a communication platform connected to a network for generating a conversational agent, the method comprising:
obtaining, based on a paraphrase model, one or more paraphrases for each of a plurality of input utterances, wherein the paraphrase model is trained to obtain the one or more paraphrases via machine translation of the plurality of input utterances;
selecting at least one of the one or more paraphrases for each input utterance to generate selected paraphrases, wherein the selecting is based on:
a user instruction of selecting the at least the one of the one or more paraphrases, and
an intent of the input utterance; and
generating a conversational agent based on the plurality of input utterances and the selected paraphrases.
2. The method of claim 1 , further comprising:
obtaining aggregated training data based on stored user-agent dialogs, the at least one of the one or more paraphrases, or the stored user-agent dialogs and the at least one of the one or more paraphrases, wherein the paraphrase model is trained using the aggregated training data.
3. The method of claim 2 , further comprising:
selecting, based on the aggregated training data, the paraphrase model from a plurality of paraphrase models.
4. The method of claim 3 , wherein a plurality of N-grams features are generated from the aggregated training data, selecting the paraphrase model comprises:
determining, based on the plurality of N-grams features, that the aggregated training data comprises new training data related to the paraphrase model, wherein the paraphrase model is updated based on the aggregated training data.
5. The method of claim 2 , further comprising:
generating a plurality of N-grams from the aggregated training data;
determining, based on the plurality of N-grams, that the aggregated training data is unrelated to any previously generated paraphrase model; and
generating the paraphrase model based on the aggregated training data.
6. The method of claim 1 , wherein the at least one of the one or more paraphrases selected is semantically equivalent to a corresponding input utterance.
7. The method of claim 1 , further comprising:
training, prior to the one or more paraphrases being obtained, the paraphrase model using training data, wherein the training data is obtained by:
generating texts in a first language,
translating the texts into one or more second languages based on machine translation to obtain translated texts,
translating each of the translated texts from the one or more second languages to the first language to obtain re-translated texts, and
generating the training data based on the texts in the first language and the re-translated texts.
8. A non-transitory machine-readable medium having information recorded thereon for generating a conversational agent, wherein the information, when read by the machine, effectuate operations comprising:
obtaining, based on a paraphrase model, one or more paraphrases for each of a plurality of input utterances, wherein the paraphrase model is trained to obtain the one or more paraphrases via machine translation of the plurality of input utterances;
selecting at least one of the one or more paraphrases for each input utterance to generate selected paraphrases, wherein the selecting is based on:
a user instruction of selecting the at least the one of the one or more paraphrases, and
an intent of the input utterance; and
generating a conversational agent based on the plurality of input utterances and the selected paraphrases.
9. The medium of claim 8 , wherein the operations further comprise:
obtaining aggregated training data based on stored user-agent dialogs, the at least one of the one or more paraphrases, or the stored user-agent dialogs and the at least one of the one or more paraphrases, wherein the paraphrase model is trained using the aggregated training data.
10. The medium of claim 9 , wherein the operations further comprise:
selecting, based on the aggregated training data, the paraphrase model from a plurality of paraphrase models.
11. The medium of claim 10 , wherein a plurality of N-grams features are generated from the aggregated training data, selecting the paraphrase model comprises:
determining, based on the plurality of N-grams features, that the aggregated training data comprises new training data related to the paraphrase model, wherein the paraphrase model is updated based on the aggregated training data.
12. The medium of claim 9 , wherein the operations further comprise:
generating a plurality of N-grams from the aggregated training data;
determining, based on the plurality of N-grams, that the aggregated training data is unrelated to any previously generated paraphrase model; and
generating the paraphrase model based on the aggregated training data.
13. The medium of claim 8 , wherein the at least one of the one or more paraphrases selected is semantically equivalent to a corresponding input utterance.
14. The medium of claim 8 , wherein the operations further comprise:
training, prior to the one or more paraphrases being obtained, the paraphrase model using training data, wherein the training data is obtained by:
generating texts in a first language,
translating the texts into one or more second languages based on machine translation to obtain translated texts,
translating each of the translated texts from the one or more second languages to the first language to obtain re-translated texts, and
generating the training data based on the texts in the first language and the re-translated texts.
15. A system, having at least one processor, storage, and a communication platform connected to a network for generating a conversational agent, comprising:
a paraphrase generator configured to obtain, based on a paraphrase model, one or more paraphrases for each of a plurality of input utterances, wherein the paraphrase model is trained to obtain the one or more paraphrases via machine translation of the plurality of input utterances;
a paraphrase data selector configured to select at least one of the one or more paraphrases for each input utterance to generate selected paraphrases, wherein the selecting is based on:
a user instruction of selecting the at least the one of the one or more paraphrases, and
an intent of the input utterance; and
a conversational agent generator configured to generate a conversational agent based on the plurality of input utterances and the selected paraphrases.
16. The system of claim 15 , further comprising:
a training data aggregator configured to obtain aggregated training data based on stored user-agent dialogs, the at least one of the one or more paraphrases, or the stored user-agent dialogs and the at least one of the one or more paraphrases, wherein the paraphrase model is trained using the aggregated training data.
17. The system of claim 16 , further comprising:
a paraphrase model selector configured to select, based on the aggregated training data, the paraphrase model from a plurality of paraphrase models.
18. The system of claim 17 , wherein a plurality of N-grams features are generated from the aggregated training data, the paraphrase model being selected comprises the paraphrase model selector being configured to:
determine, based on the plurality of N-grams features, that the aggregated training data comprises new training data related to the paraphrase model, wherein the paraphrase model is updated based on the aggregated training data.
19. The system of claim 16 , further comprising:
a paraphrase model selector configured to:
generate a plurality of N-grams from the aggregated training data, and
determine, based on the plurality of N-grams, that the aggregated training data is unrelated to any previously generated paraphrase model; and
a statistical paraphrase model training engine configured to generate the paraphrase model based on the aggregated training data.
20. The system of claim 15 , further comprising:
a machine translation based training data collector configured to train, prior to the one or more paraphrases being obtained, the paraphrase model using training data, wherein the training data is obtained by:
generating texts in a first language,
translating the texts into one or more second languages based on machine translation to obtain translated texts,
translating each of the translated texts from the one or more second languages to the first language to obtain re-translated texts, and
generating the training data based on the texts in the first language and the re-translated texts.