Systems and methods for intelligent, scalable, and cost-effective data categorization
Described herein are methods, systems, and computer-readable media for classification. Techniques may retrieve content datasets, gather first sets of input data from the content datasets, and preprocess the first sets of input data. Techniques may next generate second sets of input data by embedding associated first metadata and second metadata, determine a plurality of contextual similarities based on contextual information, and generate third sets of input data by grouping one or more sets of input data based on the determined plurality of contextual similarities. Techniques may further determine, for each content dataset of the plurality of content datasets using one or more machine learning models, one or more second categories associated with the content dataset.
1. A method for classifying content, the method comprising:
retrieving, from one or more databases, a plurality of content datasets, wherein each content dataset of the plurality of content datasets is associated with a person of a plurality of persons and includes:
content associated with the person,
first metadata associated with the person,
second metadata associated with the content, and
one or more first categories associated with the content dataset;
gathering first sets of input data from the plurality of content datasets, wherein each set of input data of the first sets of input data is associated with a person of the plurality of persons;
preprocessing the first sets of input data by:
cleaning the first sets of input data,
aggregating the cleaned first sets of input data,
dividing the aggregated input data into a plurality of first subsets of input data, and
normalizing the plurality of first subsets of input data;
generating second sets of input data by embedding associated first metadata and second metadata into each subset of the normalized plurality of first subsets of input data;
determining a plurality of contextual similarities based on contextual information associated with the second sets of input data, wherein each contextual similarity comprises a measure of similarity in context between two persons of the plurality of persons;
generating third sets of input data by grouping one or more sets of input data of the second sets of input data based on the determined plurality of contextual similarities; and
determining, for each content dataset of the plurality of content datasets using one or more machine learning models, one or more second categories associated with the content dataset.
2. The method of claim 1 , wherein the content comprises at least one of text data, audio data, video data, image data, presentation data, metadata, tag data, header data, author data, or identification information in at least one of human, computer, and machine languages.
3. The method of claim 1 , wherein the person of the plurality of persons is an author of the associated content.
4. The method of claim 1 , wherein gathering first sets of input data from the plurality of content datasets comprises gathering one or more keywords from the plurality of content datasets.
5. The method of claim 1 , wherein cleaning the first sets of input data comprises resolving grammatical or typographical errors in the gathered first sets of input data.
6. The method of claim 1 , wherein the plurality of first subsets of input data comprises no more than a predetermined number of subsets.
7. The method of claim 1 , wherein normalizing the plurality of first subsets of input data comprises at least one of:
removing one or more stop words from a corpus of keywords,
removing one or more common words from the corpus of keywords, or
normalizing text across different languages.
8. The method of claim 1 , wherein preprocessing the first sets of input data comprises:
cleaning the first sets of input data;
normalizing the cleaned first sets of input data to a specific range; and
performing one or more data augmentation techniques to create variations of the first sets of input data.
9. The method of claim 8 , wherein cleaning the first sets of input data comprises:
identifying one or more removeable sets of the first sets of input data to remove, wherein the one or more removeable sets are identified as corrupted or irrelevant; and
removing the one or more removeable sets of the first sets of input data.
10. The method of claim 8 , wherein cleaning the first sets of input data comprises modifying at least one of a resolution or format of one or more sets of the first sets of input data.
11. The method of claim 1 , further comprising:
generating tokenized data by tokenizing the second sets of input data based on the contextual information, wherein generating third sets of input data is based on the tokenized data.
12. The method of claim 11 , wherein generating tokenized data comprises tokenizing the second sets of input data into one or more single words or short sentences.
13. The method of claim 1 , wherein generating training data further comprises:
identifying a plurality of keywords from the plurality of content datasets;
ranking the plurality of keywords based on a frequency of occurrence, wherein a first keyword associated with a high frequency of occurrence is ranked higher than a second keyword associated with a low frequency of occurrence; and
generating a first set of categories based on the ranked plurality of keywords.
14. The method of claim 13 , wherein generating training data further comprises:
retrieving fourth sets of input data from the database; and
generating the training data based on the third sets of input data, fourth sets of input data, and first set of categories.
15. The method of claim 1 , wherein the one or more machine learning models are part of a monolithic system or a distributed system.
16. The method of claim 1 , further comprising:
generating training data based at least on the third sets of input data;
training the one or more machine learning models based on the training data; and
determining, for each content dataset of the plurality of content datasets using the one or more trained machine learning models, one or more second categories associated with the content dataset.
17. A system comprising:
one or more memory devices storing processor-executable instructions; and
one or more processors configured to execute instructions to cause the system to perform operations comprising:
retrieving, from a database, a plurality of content datasets, wherein each content dataset of the plurality of content datasets is associated with a person of a plurality of persons and includes:
content associated with the person,
first metadata associated with the person,
second metadata associated with the content, and
one or more first categories associated with the content dataset;
gathering first sets of input data from the plurality of content datasets, wherein each set of input data of the first sets of input data is associated with a person of the plurality of persons;
preprocessing the first sets of input data by:
cleaning the first sets of input data,
aggregating the cleaned first sets of input data,
dividing the aggregated input data into a plurality of first subsets of input data, and
normalizing the plurality of first subsets of input data;
generating second sets of input data by embedding associated first metadata and second metadata into each subset of the normalized plurality of first subsets of input data;
determining a plurality of contextual similarities based on contextual information associated with the second sets of input data, wherein each contextual similarity comprises a measure of similarity in context between two persons of the plurality of persons;
generating third sets of input data by grouping one or more sets of input data of the second sets of input data based on the determined plurality of contextual similarities; and
determining, for each content dataset of the plurality of content datasets using one or more machine learning models, one or more second categories associated with the content dataset.
18. A non-transitory computer readable medium including instructions that are executable by one or more processors to cause a system to perform operations for classification of content, the operations comprising:
retrieving, from a database, a plurality of content datasets, wherein each content dataset of the plurality of content datasets is associated with a person of the plurality of persons and includes:
content associated with the person,
first metadata associated with the person,
second metadata associated with the content, and
one or more first categories associated with the content dataset;
gathering first sets of input data from the plurality of content datasets, wherein each set of input data of the first sets of input data is associated with a person of the plurality of persons;
preprocessing the first sets of input data by:
cleaning the first sets of input data,
aggregating the cleaned first sets of input data,
dividing the aggregated input data into a plurality of first subsets of input data, and
normalizing the plurality of first subsets of input data;
generating second sets of input data by embedding associated first metadata and second metadata into each subset of the normalized plurality of first subsets of input data;
determining a plurality of contextual similarities based on contextual information associated with the second sets of input data, wherein each contextual similarity comprises a measure of similarity in context between two persons of the plurality of persons;
generating third sets of input data by grouping one or more sets of input data of the second sets of input data based on the determined plurality of contextual similarities; and
determining, for each content dataset of the plurality of content datasets using one or more machine learning models, one or more second categories associated with the content dataset.