Mapping webpages to page groups
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for mapping webpages to page groups. The program and method provide for receiving plural Uniform Resource Locators (URLs), each URL of the plural URLs corresponding to a respective webpage of a website; generating a distance matrix with pairwise distances between the plural URLs; performing hierarchical clustering based on the distance matrix, to generate a dendrogram in which the plural URLs are arranged in hierarchical clusters; and determining, based on the dendrogram, a predicted page group for each of the plural URLs.
1. A method, comprising:
receiving plural Uniform Resource Locators (URLs), each URL of the plural URLs corresponding to a respective webpage of a website;
accessing, from a database, a set of terms, the set of terms having been predetermined as prioritized;
extracting distinct terms corresponding to a path level, a query key and a cvar key for the plural URLs;
computing a similarity score of the distinct terms with the set of terms;
identifying, based on the computing, URLs of the plural URLs having at least one term appearing within the set of terms;
applying weights to the identified URLs, to prioritize the identified URLs relative to other URLs of the plural URLs;
performing, based on applying the weights, hierarchical clustering with respect to the plural URLs, to generate a dendrogram in which the plural URLs are arranged in hierarchical clusters;
storing a representation of the dendrogram;
automatically determining, based on the stored representation of the dendrogram, a predicted page group for each of the plural URLs; and
causing, based on determining the predicted page group for each of the plural URLs, display of metrics corresponding to the website.
2. The method of claim 1 , further comprising:
computing URL patterns and conditions for each cluster within the hierarchical clusters.
3. The method of claim 1 , wherein the hierarchical clusters comprise agglomerative hierarchical clusters.
4. The method of claim 1 , wherein computing the similarity score corresponds to calculating a respective levenshtein ratio for the path level, the query key and the cvar key for the plural URLs, and
wherein the identifying comprises selecting URLs of the plural URLs with levenshtein ratios exceeding a predefined value with respect to the path level, the query key and the cvar key.
5. The method of claim 1 , wherein arranging the plural URLs into hierarchical clusters comprises:
performing grid searching with respect to a preset range of clusters;
computing a silhouette score for each cluster within the preset range of clusters; and
determining the hierarchical clusters based on the computed silhouette scores.
6. The method of claim 1 , further comprising:
generating, based on applying the weights, a distance matrix with pairwise distances between the plural URLs,
wherein the arranging comprises performing hierarchical clustering based on the distance matrix, to generate a dendrogram in which the plural URLs are arranged in hierarchical clusters, and
wherein the determining is based on a stored representation of the dendrogram.
7. The method of claim 6 , further comprising:
accessing, from a database, a set of terms, the set of terms having been predetermined as prioritized,
wherein determining the hierarchical clusters is further based on identifying nodes within the dendrogram having conditions that include at least one term within the set of terms.
8. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, configure the at least one processor to perform operations comprising:
receiving plural Uniform Resource Locators (URLs), each URL of the plural URLs corresponding to a respective webpage of a website;
accessing, from a database, a set of terms, the set of terms having been predetermined as prioritized;
extracting distinct terms corresponding to a path level, a query key and a cvar key for the plural URLs;
computing a similarity score of the distinct terms with the set of terms;
identifying, based on the computing, URLs of the plural URLs having at least one term appearing within the set of terms;
applying weights to the identified URLs, to prioritize the identified URLs relative to other URLs of the plural URLs;
performing, based on applying the weights, hierarchical clustering with respect to the plural URLs, to generate a dendrogram in which the plural URLs are arranged in hierarchical clusters;
storing a representation of the dendrogram;
automatically determining, based on a stored representation of the dendrogram, a predicted page group for each of the plural URLs; and
causing, based on determining the predicted page group for each of the plural URLs, display of metrics corresponding to the website.
9. The system of claim 8 , the operations further comprising:
computing URL patterns and conditions for each cluster within the hierarchical clusters.
10. The system of claim 8 , wherein the hierarchical clusters comprise agglomerative hierarchical clusters.
11. The system of claim 8 , wherein computing the similarity score corresponds to calculating a respective levenshtein ratio for the path level, the query key and the cvar key for the plural URLs, and
wherein the identifying comprises selecting URLs of the plural URLs with levenshtein ratios exceeding a predefined value with respect to the path level, the query key and the cvar key.
12. The system of claim 8 , wherein arranging the plural URLs into hierarchical clusters comprises:
performing grid searching with respect to a preset range of clusters;
computing a silhouette score for each cluster within the preset range of clusters; and
determining the hierarchical clusters based on the computed silhouette scores.
13. The system of claim 8 , the operations further comprising:
generating, based on applying the weights, a distance matrix with pairwise distances between the plural URLs,
wherein the arranging comprises performing hierarchical clustering based on the distance matrix, to generate a dendrogram in which the plural URLs are arranged in hierarchical clusters, and
wherein the determining is based on a stored representation of the dendrogram.
14. The system of claim 13 , the operations further comprising:
accessing, from a database, a set of terms, the set of terms having been predetermined as prioritized,
wherein determining the hierarchical clusters is further based on identifying nodes within the dendrogram having conditions that include at least one term within the set of terms.
15. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
receiving plural Uniform Resource Locators (URLs), each URL of the plural URLs corresponding to a respective webpage of a website;
accessing, from a database, a set of terms, the set of terms having been predetermined as prioritized;
extracting distinct terms corresponding to a path level, a query key and a cvar key for the plural URLs;
computing a similarity score of the distinct terms with the set of terms;
identifying, based on the computing, URLs of the plural URLs having at least one term appearing within the set of terms;
applying weights to the identified URLs, to prioritize the identified URLs relative to other URLs of the plural URLs;
performing, based on applying the weights, hierarchical clustering with respect to the plural URLs, to generate a dendrogram in which the plural URLs are arranged in hierarchical clusters;
storing a representation of the dendrogram;
automatically determining, based on a stored representation of the dendrogram, a predicted page group for each of the plural URLs; and
causing, based on determining the predicted page group for each of the plural URLs, display of metrics corresponding to the website.
16. The non-transitory computer-readable storage medium of claim 15 , the operations further comprising:
computing URL patterns and conditions for each cluster within the hierarchical clusters.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the hierarchical clusters comprise agglomerative hierarchical clusters.
18. The non-transitory computer-readable storage medium of claim 15 , wherein computing the similarity score corresponds to calculating a respective levenshtein ratio for the path level, the query key and the cvar key for the plural URLs, and
wherein the identifying comprises selecting URLs of the plural URLs with levenshtein ratios exceeding a predefined value with respect to the path level, the query key and the cvar key.
19. The non-transitory computer-readable storage medium of claim 15 , wherein arranging the plural URLs into hierarchical clusters comprises:
performing grid searching with respect to a preset range of clusters;
computing a silhouette score for each cluster within the preset range of clusters; and
determining the hierarchical clusters based on the computed silhouette scores.
20. The non-transitory computer-readable storage medium of claim 15 , the operations further comprising:
generating, based on applying the weights, a distance matrix with pairwise distances between the plural URLs,
wherein the arranging comprises performing hierarchical clustering based on the distance matrix, to generate a dendrogram in which the plural URLs are arranged in hierarchical clusters, and
wherein the determining is based on a stored representation of the dendrogram.