Automating large-scale data collection
Obtaining collections of sentences in different languages that are usable for training models in various applications of artificial intelligence is provided. A method is provided that obtains, from text corpus, webpages in a plurality of languages, each of the webpages corresponding to an URL; obtains annotations for each of the webpages based on its URL, to obtain annotated data entries corresponding to the webpages, each of the annotated data entries including a classification label corresponding to a sub-topic of one of a plurality of topics, where each of the plurality of topics includes a corresponding plurality of sub-topics; filters the annotated data entries to obtain topic-specific content in a target language based on the classification labels, the topic-specific content corresponding to one or more sub-topics; performs post-processing on the topic-specific content to obtain result data; and outputs the result data for the topic.
1 . A computer-implemented method comprising:
(a) obtaining, by a dataset generation system from a text corpus, a plurality of webpages in a plurality of languages, each of the plurality of webpages corresponding to a respective universal resource locator (URL);
(b) annotating, by the dataset generation system, each of the plurality of webpages based on the respective URL, to obtain annotated data entries, each of the annotated data entries corresponding to one of the plurality of webpages and comprising a classification label corresponding to a sub-topic of one of a plurality of topics, wherein each of the plurality of topics comprises a corresponding plurality of sub-topics;
(c) performing, by the dataset generation system, a first filtering for the annotated data entries to obtain topic-specific content in a target language among the plurality of languages by:
comparing one or more sub-topics among the corresponding plurality of sub-topics with classification labels correspondingly included in the annotated data entries, wherein the one or more sub-topics are associated with a topic among the plurality of topics, and
obtaining, from the annotated data entries, the topic-specific content corresponding to the one or more sub-topics based on the comparing; and
(d) performing, by the dataset generation system, post-processing on the topic-specific content to obtain result data, by performing at least one from among a second filtering and a normalizing on sentences included in the topic-specific content,
wherein (c) and (d) are performed for a plurality of different target languages among the plurality of languages, the target language being a first target language of the plurality of different target languages;
outputting, by the dataset generation system, the result data for each of the plurality of different target languages as a plurality of model training datasets for the topic for which (c) and (d) were performed;
transmitting, by the dataset generation system to a model training system, at least a first model training dataset among the plurality of model training datasets, the first model training dataset corresponding to the first target language;
training, by the model training system, a first machine learning model using the first model training dataset in the first target language, the training comprising inputting, into the first machine learning model, training examples from the first model training dataset, to obtain a set of values for model parameters for a first speech recognition model; and
generating, by the model training system, the first speech recognition model in the first target language, for the topic for which (c) and (d) were performed, wherein the first speech recognition model is configured with the set of values for a set of predetermined sentiments or a set of predetermined intents,
wherein the first speech recognition model is configured to, based on one or more sentences provided as an input by a user in the first target language, identify a certain sentiment from the set of predetermined sentiments or a certain intent from the set of predetermined intents, and output, for the one or more sentences, a prediction of the certain sentiment or a prediction of the certain intent.
2 . The computer-implemented method of claim 1 , wherein the performing the first filtering further comprises:
comparing keywords of a predetermined lexicon for the topic with words in sentences included in one from among the topic-specific content and the annotated data entries; and
obtaining the sentences containing the keywords, to generate topic-specific, lexicon-specific content,
wherein the performing the post-processing on the topic-specific content comprises performing the post-processing on the topic-specific, lexicon-specific content.
3 . The computer-implemented method of claim 1 , wherein the performing the first filtering further comprises:
obtaining, from the topic-specific content, variation-specific content corresponding to a variation of the target language, by searching URLs of webpages corresponding to the topic-specific content for an indication of a geographic locality corresponding to the variation,
wherein the performing the post-processing on the topic-specific content comprises performing the post-processing on the variation-specific content.
4 . The computer-implemented method of claim 1 , wherein the obtaining the plurality of webpages further comprises:
detecting a language of the plurality of webpages, respectively;
computing a language confidence score for the detected language; and
discarding, from the plurality of webpages, webpages whose language confidence score is less than a predetermined threshold, prior to the annotating.
5 . The computer-implemented method of claim 1 , wherein:
the obtaining the plurality of webpages further comprises tokenizing raw content included in each of the plurality of webpages into sentences, and
the sentences included in the topic-specific content are tokenized sentences.
6 . The computer-implemented method of claim 1 , wherein the performing the post-processing further comprises:
performing the second filtering on the sentences included in the topic-specific content to remove, from the topic-specific content, at least one from among an incomplete sentence and a grammatically incorrect sentence, to obtain a set of high-quality sentences; and
normalizing the set of high-quality sentences by removing, from at least one sentence of the set of high-quality sentences, at least one from among a bullet point and an emoji.
7 . The computer-implemented method of claim 1 , further comprising:
prior to the performing the post-processing, verifying the topic-specific content by extracting, from the topic-specific content, webpages corresponding to at least one from among a verified source domain, a verified publication, and a verified public source, to obtain the topic-specific content that is verified.
8 . The computer-implemented method of claim 1 , wherein:
each of the annotated data entries comprises a raw content section comprising raw content and a sentences section comprising sentences obtained by tokenizing the raw content, and
the annotating further comprises:
inserting tags in each of the annotated data entries, wherein the tags identify, for a webpage among the plurality of webpages, at least two from among the respective URL, the classification label for the URL, the target language, a target language confidence score, a webpage title, the raw content section, and the sentences section.
9 . The computer-implemented method of claim 1 , wherein:
(c) and (d) are performed for the plurality of topics in the plurality of different target languages, and
the outputting the result data further comprises providing sets of result data that are obtained for each of the plurality of topics in the plurality of languages as a plurality of training datasets for building speech recognition models in the plurality of different target languages for the plurality of topics.
10 . The computer-implemented method of claim 1 , wherein the performing the post-processing further comprises:
performing the second filtering on the sentences of the topic-specific content to identify sentences for a task-specific requirement,
wherein the task-specific requirement is a sentiment analysis.
11 . The computer-implemented method of claim 1 , further comprising:
prior to the annotating, receiving a first user selection input providing an identification of the target language; and
prior to the performing the first filtering, receiving a second user selection input providing an identification of the topic for which (c) and (d) are to be performed.
12 . The computer-implemented method of claim 11 , wherein the first user selection input and the second user selection input are provided by a customer of a cloud service provider.
13 . A system comprising:
a model training system; and
a dataset generation system comprising one or more processors, and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations including:
(a) obtaining, from a text corpus, a plurality of webpages in a plurality of languages, each of the plurality of webpages corresponding to a respective universal resource locator (URL),
(b) annotating each of the plurality of webpages based on the respective URL, to obtain annotated data entries, each of the annotated data entries corresponding to one of the plurality of webpages and comprising a classification label corresponding to a sub-topic of one of a plurality of topics, wherein each of the plurality of topics comprises a corresponding plurality of sub-topics,
(c) performing a first filtering for the annotated data entries to obtain topic-specific content in a target language among the plurality of languages by:
comparing one or more sub-topics among the corresponding plurality of sub-topics with classification labels correspondingly included in the annotated data entries, wherein the one or more sub-topics are associated with a topic among the plurality of topics, and obtaining, from the annotated data entries, the topic-specific content corresponding to the one or more sub-topics based on the comparing, and
(d) performing post-processing on the topic-specific content to obtain result data, by performing at least one from among a second filtering and a normalizing on sentences included in the topic-specific content,
wherein (c) and (d) are performed for a plurality of different target languages among the plurality of languages, the target language being a first target language of the plurality of different target languages,
outputting the result data for each of the plurality of different target languages as a plurality of model training datasets for the topic for which (c) and (d) were performed, and
transmitting, to the model training system, at least a first model training dataset among the plurality of model training datasets, the first model training dataset corresponding to the first target language, wherein the model training system is configured to:
training a first machine learning model using the first model training dataset in the first target language, the training including inputting, into the first machine learning model, training examples from the first model training dataset, to obtain a set of values for model parameters for a first speech recognition model, and
generating the first speech recognition model in the first target language, for the topic for which (c) and (d) were performed, wherein the first speech recognition model is configured with the set of values for a set of predetermined sentiments or a set of predetermined intents,
wherein the first speech recognition model is configured to, based on one or more sentences provided as an input by a user in the first target language, identify a certain sentiment from the set of predetermined sentiments or a certain intent from the set of predetermined intents, and output, for the one or more sentences, a prediction of the certain sentiment or a prediction of the certain intent.
14 . The system of claim 13 , wherein:
(c) and (d) are performed for the plurality of topics in the plurality of different target languages, and
the outputting the result data further includes providing sets of result data that are obtained for each of the plurality of topics in the plurality of languages as a plurality of training datasets for building speech recognition models in the plurality of different target languages for the plurality of topics.
15 . One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a computer system comprising a model training system and a dataset generation system to perform operations including:
(a) obtaining, using the dataset generation system from a text corpus, a plurality of webpages in a plurality of languages, each of the plurality of webpages corresponding to a respective universal resource locator (URL),
(b) annotating, using the dataset generation system, each of the plurality of webpages based on the respective URL, to obtain annotated data entries, each of the annotated data entries corresponding to one of the plurality of webpages and comprising a classification label corresponding to a sub-topic of one of a plurality of topics, wherein each of the plurality of topics comprises a corresponding plurality of sub-topics,
(c) performing, using the dataset generation system, a first filtering for the annotated data entries to obtain topic-specific content in a target language among the plurality of languages by:
comparing one or more sub-topics among the corresponding plurality of sub-topics with classification labels correspondingly included in the annotated data entries, wherein the one or more sub-topics are associated with a topic among the plurality of topics, and
obtaining, from the annotated data entries, the topic-specific content corresponding to the one or more sub-topics based on the comparing, and
(d) performing, using the dataset generation system, post-processing on the topic-specific content to obtain result data, by performing at least one from among a second filtering and a normalizing on sentences included in the topic-specific content,
wherein (c) and (d) are performed for a plurality of different target languages among the plurality of languages, the target language being a first target language of the plurality of different target languages,
outputting, using the dataset generation system, the result data for each of the plurality of different target languages as a plurality of model training datasets for the topic for which (c) and (d) were performed,
transmitting, from the dataset generation system to the model training system, at least a first model training dataset among the plurality of model training datasets, the first model training dataset corresponding to the first target language,
training, using the model training system, a first machine learning model using the first model training dataset in the first target language, the training including inputting, into the first machine learning model, training examples from the first model training dataset, to obtain a set of values for model parameters for a first speech recognition model, and
generating, using the model training system, the first speech recognition model in the first target language, for the topic for which (c) and (d) were performed, wherein the first speech recognition model is configured with the set of values for a set of predetermined sentiments or a set of predetermined intents,
wherein the first speech recognition model is configured to, based on one or more sentences provided as an input by a user in the first target language, identify a certain sentiment from the set of predetermined sentiments or a certain intent from the set of predetermined intents, and output, for the one or more sentences, a prediction of the certain sentiment or a prediction of the certain intent.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein:
(c) and (d) are performed for the plurality of topics in the plurality of different target languages, and
the outputting the result data further includes providing sets of result data that are obtained for each of the plurality of topics in the plurality of languages as a plurality of training datasets for building speech recognition models in the plurality of different target languages for the plurality of topics.