Dynamic and precise website categorization using a large language model
A concept tagging model is trained based at least in part on a result of a large language model used to analyze training data. Text content of a website is received. The concept tagging model is applied to the text content to identify one or more concept terms in the text content. An embedding model is applied to the identified one or more concept terms to determine one or more embeddings. Using the one or more embeddings, a closest matching granular category in a taxonomy of categories is identified for the website.
1 . A method, comprising:
training a concept tagging model based at least in part on a result of a large language model used to analyze training data;
receiving text content of a website;
applying the concept tagging model to the text content to identify one or more concept terms in the text content;
applying an embedding model to the identified one or more concept terms to determine one or more embeddings;
using the one or more embeddings to identify, for the website, a closest matching granular category in a taxonomy of categories; and
in response to a determination that the closest matching granular category does not meet a match threshold, applying a fallback document classification model to the website to determine a closest matching general category in the taxonomy of categories and categorizing the website as belonging to the identified closest matching general category.
2 . The method of claim 1 , wherein the result of the large language model is generated in response to a prompt to identify in the training data one or more concept terms belonging to one or more specified concepts.
3 . The method of claim 1 , wherein the concept tagging model is a transformer language model.
4 . The method of claim 1 , wherein the website is one of a plurality of crawled websites to be categorized.
5 . The method of claim 1 , wherein the text content includes portions identified from the website as semantic text artifacts.
6 . The method of claim 1 , further comprising based on clustering the identified one or more concept terms in the text content, determining a potential new granular category to be included in the taxonomy of categories.
7 . The method of claim 1 , wherein the closest matching granular category is identified based on one or more comparisons of the one or more embeddings with a reference embedding for the closest matching granular category.
8 . The method of claim 1 , further comprising in response to a determination that the closest matching granular category meets a match threshold, categorizing the website as belonging to the identified closest matching granular category.
9 . The method of claim 8 , further comprising storing the categorization of the website in a Uniform Resource Locator (URL) filtering database.
10 . The method of claim 8 , further comprising determining whether to allow access to the website based on the categorization of the website.
11 . A system, comprising:
a processor configured to:
train a concept tagging model based at least in part on a result of a large language model used to analyze training data;
receive text content of a website;
apply the concept tagging model to the text content to identify one or more concept terms in the text content;
apply an embedding model to the identified one or more concept terms to determine one or more embeddings;
use the one or more embeddings to identify, for the website, a closest matching granular category in a taxonomy of categories; and
in response to a determination that the closest matching granular category does not meet a match threshold, apply a fallback document classification model to the website to determine a closest matching general category in the taxonomy of categories and categorize the website as belonging to the identified closest matching general category; and
a memory coupled to the processor and configured to provide the processor with instructions.
12 . The system of claim 11 , wherein the result of the large language model is generated in response to a prompt to identify in the training data one or more concept terms belonging to one or more specified concepts.
13 . The system of claim 11 , wherein the processor is further configured to: based on clustering the identified one or more concept terms in the text content, determine a potential new granular category to be included in the taxonomy of categories.
14 . The system of claim 11 , wherein the closest matching granular category is identified based on one or more comparisons of the one or more embeddings with a reference embedding for the closest matching granular category.
15 . The system of claim 11 , wherein the processor is further configured to: in response to a determination that the closest matching granular category meets a match threshold, categorize the website as belonging to the identified closest matching granular category.
16 . The system of claim 15 , wherein the processor is further configured to: store the categorization of the website in a Uniform Resource Locator (URL) filtering database.
17 . The system of claim 15 , wherein the processor is further configured to: determine whether to allow access to the website based on the categorization of the website.
18 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
training a concept tagging model based at least in part on a result of a large language model used to analyze training data;
receiving text content of a website;
applying the concept tagging model to the text content to identify one or more concept terms in the text content;
applying an embedding model to the identified one or more concept terms to determine one or more embeddings;
using the one or more embeddings to identify, for the website, a closest matching granular category in a taxonomy of categories; and
in response to a determination that the closest matching granular category does not meet a match threshold, applying a fallback document classification model to the website to determine a closest matching general category in the taxonomy of categories and categorizing the website as belonging to the identified closest matching general category.
19 . The computer program product recited claim 18 , further comprising computer instructions for in response to a determination that the closest matching granular category meets a match threshold, categorizing the website as belonging to the identified closest matching granular category.
20 . The computer program product recited claim 19 , further comprising computer instructions for storing the categorization of the website in a Uniform Resource Locator (URL) filtering database.
21 . The computer program product recited claim 19 , further comprising computer instructions for determining whether to allow access to the website based on the categorization of the website.