Natural language processing machine learning frameworks trained using multi-task training routines
Various embodiments of the present invention provide methods, apparatus, systems, computing devices, computing entities, and/or the like for performing natural language processing operations using an attention-based text encoder machine learning model that is trained using a multi-task training routine that is associated with two or more training tasks (e.g., a multi-task training routine that is associated with two or more sequential training tasks, a multi-training routine that is associated with two or more concurrent training tasks, and/or the like).
1. A computer-implemented method comprising:
generating, by one or more processors and using an attention-based text encoder machine learning model and based at least in part on an unlabeled document data object, an unlabeled document word-wise embedded representation set that comprises a plurality of unlabeled document word-wise embedded representations, each of the plurality of unlabeled document word-wise embedded representations associated with a respective unlabeled document word of the unlabeled document data object, wherein the attention-based text encoder machine learning model comprises a tunable parameter value that is generated based at least in part on a concurrent learning loss by:
identifying one or more training input document data objects comprising one or more training unlabeled document data objects and a plurality of labeled document data objects,
identifying a training document pair, wherein the training document pair comprises (i) a training unlabeled document data object of the one or more training unlabeled document data objects and (ii) a labeled document data object of the plurality of labeled document data objects that is related to the training unlabeled document data object based at least in part on ground-truth cross-document relationships,
generating a cross-document distance measure for the training document pair,
generating the concurrent learning loss based at least in part on: (i) a language modeling loss model that is based at least in part on the one or more training input document data objects, and (ii) a similarity determination loss model that is determined based at least in part on the cross-document distance measure, and
generating, using the concurrent learning loss, the tunable parameter value;
generating, by the one or more processors and using the attention-based text encoder machine learning model and based at least in part on a labeled document data object of the plurality of the labeled document data objects, a labeled document word-wise embedded representation set that comprises a plurality of labeled document word-wise embedded representations, each of the plurality of labeled document word-wise embedded representations associated with a respective labeled document word of the labeled document data object;
generating, by the one or more processors and based at least in part on the plurality of unlabeled document word-wise embedded representations and the plurality of labeled document word-wise embedded representations, a cross-document similarity measure comparing the unlabeled document data object and the labeled document data object; and
generating, by the one or more processors and for the unlabeled document data object, a document classification based at least in part on the cross-document similarity measure.
2. The computer-implemented method of claim 1 , wherein the language modeling loss model is defined in accordance with a language modeling training task and the language modeling training task comprises a text reconstruction sub-task.
3. The computer-implemented method of claim 1 , wherein the attention-based text encoder machine learning model comprises an attentional autoencoder machine learning model.
4. The computer-implemented method of claim 1 , wherein the cross-document distance measure comprises one or more cosine distance measures.
5. The computer-implemented method of claim 1 , wherein generating the cross-document similarity measure comprises:
for a given word pair of a plurality of word pairs that comprises a given unlabeled document word of the unlabeled document data object and a given labeled document word of the labeled document data object, generating a pairwise word similarity measure,
for the given word pair, generating a pairwise flow indicator based at least in part on the pairwise word similarity measure of the given word pair relative to other pairwise word similarity measures in a subset of the plurality of word pairs that is associated with the given unlabeled document word in the given word pair; and
generating the cross-document similarity measure based at least in part on the pairwise word similarity measure and the pairwise flow indicator.
6. The computer-implemented method of claim 5 , wherein the pairwise word similarity measure for the given word pair comprises a cosine similarity measure between a given unlabeled document word-wise embedded representation for the given unlabeled document word and a given labeled document word-wise embedded representation for the given labeled document word.
7. The computer-implemented method of claim 5 further comprising determining the pairwise word similarity measure in a manner that is configured to maximize a word mover's similarity measure for the unlabeled document data object and the labeled document data object.
8. A system comprising one or more processors and at least one memory storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
generating, using an attention-based text encoder machine learning model and based at least in part on an unlabeled document data object, an unlabeled document word-wise embedded representation set that comprises a plurality of unlabeled document word-wise embedded representations, each of the plurality of unlabeled document word-wise embedded representations associated with a respective unlabeled document word of the unlabeled document data object, wherein the attention-based text encoder machine learning model comprises a tunable parameter value that is generated based at least in part on a concurrent learning loss by:
identifying one or more training input document data objects comprising one or more training unlabeled document data objects and a plurality of labeled document data objects,
identifying a training document pair, wherein the training document pair comprises (i) a training unlabeled document data object of the one or more training unlabeled document data objects and (ii) a labeled document data object of the plurality of labeled document data objects that is related to the training unlabeled document data object based at least in part on ground-truth cross-document relationships,
generating a cross-document distance measure for the training document pair,
generating the concurrent learning loss based at least in part on: (i) a language modeling loss model that is based at least in part on the one or more training input document data objects, and (ii) a similarity determination loss model that is determined based at least in part on the cross-document distance measure, and
generating, using the concurrent learning loss, the tunable parameter value;
generating, using the attention-based text encoder machine learning model and based at least in part on a labeled document data object of the plurality of labeled document data objects, a labeled document word-wise embedded representation set that comprises a plurality of labeled document word-wise embedded representations, each of the plurality of labeled document word-wise embedded representations associated with a respective labeled document word of the labeled document data object;
generating, based at least in part on the plurality of unlabeled document word-wise embedded representations and the plurality of labeled document word-wise embedded representations, a cross-document similarity measure comparing the unlabeled document data object and the labeled document data object; and
generating, for the unlabeled document data object, a document classification based at least in part on the cross-document similarity measure.
9. The system of claim 8 , wherein the language modeling loss model is defined in accordance with a language modeling training task and the language modeling training task comprises a text reconstruction sub-task.
10. The system of claim 8 , wherein the attention-based text encoder machine learning model comprises an attentional autoencoder machine learning model.
11. The system of claim 8 , wherein the cross-document distance measure comprises one or more cosine distance measures.
12. The system of claim 8 , wherein generating the cross-document similarity measure, comprises:
for a given word pair of a plurality of word pairs that comprises a given unlabeled document word of the unlabeled document data object and a given labeled document word of the labeled document data object, generating a pairwise word similarity measure,
for the given word pair, generating a pairwise flow indicator based at least in part on the pairwise word similarity measure of the given word pair relative to other pairwise word similarity measures in a subset of the plurality of word pairs that is associated with the given unlabeled document word in the given word pair; and
generating the cross-document similarity measure based at least in part on the pairwise word similarity measure and the pairwise flow indicator.
13. The system of claim 12 , wherein the pairwise word similarity measure for the given word pair comprises a cosine similarity measure between a given unlabeled document word-wise embedded representation for the given unlabeled document word and a given labeled document word-wise embedded representation for the given labeled document word.
14. The system of claim 12 , wherein the operations further comprise determining the pairwise word similarity measure in a manner that is configured to maximize a word mover's similarity measure for the unlabeled document data object and the labeled document data object.
15. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
generating, using an attention-based text encoder machine learning model and based at least in part on an unlabeled document data object, an unlabeled document word-wise embedded representation set that comprises a plurality of unlabeled document word-wise embedded representations, each of the plurality of unlabeled document word-wise embedded representations associated with a respective unlabeled document word of the unlabeled document data object, wherein the attention-based text encoder machine learning model comprises a tunable parameter value that is generated based at least in part on a concurrent learning loss by:
identifying one or more training input document data objects comprising one or more training unlabeled document data objects and a plurality of labeled document data objects,
identifying a training document pair, wherein the training document pair comprises (i) a training unlabeled document data object of the one or more training unlabeled document data objects and (ii) a labeled document data object of the plurality of labeled document data objects that is related to the training unlabeled document data object based at least in part on ground-truth cross-document relationships,
generating a cross-document distance measure for the training document pair,
generating the concurrent learning loss based at least in part on: (i) a language modeling loss model that is based at least in part on the one or more training input document data objects, and (ii) a similarity determination loss model that is determined based at least in part on the cross-document distance measure, and
generating, using the concurrent learning loss, the tunable parameter values;
generating, using the attention-based text encoder machine learning model and based at least in part on a labeled document data object of the plurality of labeled document data objects, a labeled document word-wise embedded representation set that comprises a plurality of labeled document word-wise embedded representations, each of the plurality of labeled document word-wise embedded representations associated with a respective labeled document word of the labeled document data object;
generating, based at least in part on the plurality of unlabeled document word-wise embedded representations and the plurality of labeled document word-wise embedded representations, a cross-document similarity measure comparing the unlabeled document data object and the labeled document data object; and
generating, for the unlabeled document data object, a document classification based at least in part on the cross-document similarity measure.
16. The one or more non-transitory computer-readable storage media of claim 15 , wherein the language modeling loss model is defined in accordance with a language modeling training task and the language modeling training task comprises a text reconstruction sub-task.
17. The one or more non-transitory computer-readable storage media of claim 15 , wherein the attention-based text encoder machine learning model comprises an attentional autoencoder machine learning model.
18. The one or more non-transitory computer-readable storage media of claim 15 , wherein the cross-document distance measure comprises one or more cosine distance measures.
19. The one or more non-transitory computer-readable storage media of claim 15 , wherein generating the cross-document similarity measure comprises:
for a given word pair of a plurality of word pairs that comprises a given unlabeled document word of the unlabeled document data object and a given labeled document word of the labeled document data object, generating a pairwise word similarity measure,
for the given word pair, generating a pairwise flow indicator based at least in part on the pairwise word similarity measure of the given word pair relative to other pairwise word similarity measures in a subset of the plurality of word pairs that is associated with the given unlabeled document word in the given word pair; and
generating the cross-document similarity measure based at least in part on the pairwise word similarity measure and the pairwise flow indicator.
20. The one or more non-transitory computer-readable storage media of claim 19 , wherein the pairwise word similarity measure for the given word pair comprises a cosine similarity measure between a given unlabeled document word-wise embedded representation for the given unlabeled document word and a given labeled document word-wise embedded representation for the given labeled document word.