Systems and methods for identifying and measuring trends in consumer content demand within vertically associated websites and related content
View Patent ↗Embodiments of the invention can provide systems and methods for identifying and measuring trends in consumer content demand within a vertical network of websites and related content. In one embodiment, a method can include receiving content from webpages in at least one vertical; receiving data associated with a plurality of selected keywords associated with the at least one vertical, wherein one or more associations between at least a portion of the plurality of selected keywords can be generated; receiving session data associated with a plurality of consumers accessing content in webpages in the at least one vertical; determining third party data associated with the plurality of consumers accessing at least a portion of webpages comprising at least one of the selected keywords; and aggregating, based at least in part on the third party data, session data associated with visits to the webpages comprising at least one of the selected keywords.
1. A method comprising:
receiving, from a crawler application program, a first data stream comprising crawled content pertaining to a plurality of webpages associated with a vertical, the crawled content comprising keywords and locations for the keywords, wherein the crawler application program comprises instructions to extract the crawled content from a set of uniform resource locator (URL) fragments;
receiving, by a processor of a server, a second data stream comprising expert data associated with one or more predetermined keywords pertaining to the vertical, the expert data comprising one or more inferences associated with relationships between the one or more predetermined keywords, the one or more inferences drawn prior to identifying one or more of the predetermined keywords in the crawled content;
identifying at least one predetermined keyword in the crawled content to determine respective locations and a number of occurrences for the at least one predetermined keyword within the crawled content;
associating crawled content from the plurality of webpages with a third data stream comprising user activity data pertaining to a plurality of user computing devices accessing the plurality of webpages;
filtering, from the user activity data, a first set of activity data indicating a user is no longer viewing a set of webpages comprising the at least one predetermined keyword, wherein the filtering generates a second set of activity data;
receiving a fourth data stream comprising at least one of third party geolocation or demographic data associated with the plurality of consumers accessing at least a portion of the plurality of webpages comprising the at least one of the predetermined keywords;
determining one or more associations between the at least one of the third party geolocation or the demographic data and the at least one of the predetermined keywords in the crawled content;
transforming the first data stream, the second data stream, the third data stream and the fourth data stream to be indexed in a first database, wherein the first data stream, the second data stream, the third data stream and the fourth data stream are collected from different sources;
associating the second set of user activity data and the at least one of the third party geolocation or the demographic data based at least in part on the respective locations for the at least one predetermined keyword in the first database for identifying and measuring a trend in consumer content demand;
determining, using the first database, one or more vertical metrics of at least one of a product or a brand associated with the vertical;
collecting, from a plurality of input devices associated with the plurality of user computing devices, data corresponding to movement of the plurality of input devices;
determining an amount of interaction time of the plurality of user computing devices with one or more webpages corresponding to the one or more vertical metrics;
filtering, by the processor of the server, the one or more vertical metrics in view of the amount of interaction time to generate a filtered list of vertical metrics; and
displaying a report comprising the filtered list of vertical metrics and the amount of interaction time corresponding to each of the vertical metrics.
2. The method of claim 1 , further comprising:
matching one or more first URLs associated with the at least one of the third party geolocation or the demographic data with one or more second URLs associated with the one or more predetermined keywords.
3. The method of claim 1 , further comprising:
determining a classification of at least a portion of one of the plurality of webpages to generate the report comprising a statistic associated with the vertical.
4. The method of claim 1 , wherein the vertical is associated with at least one of an automobile industry, a type of vehicle, a consumer product industry, a type of consumer product, an entertainment industry, a type of entertainment, a music industry, a type of music, a motion picture industry, a type of motion picture, a pharmaceuticals industry, a type of pharmaceutical, an apparel industry, a type of apparel, a financial products industry, a financial services industry, a type of financial product, or a type of financial service.
5. The method of claim 1 , wherein the user activity data comprises at least one of a first URL, an Internet protocol (IP) address, alphanumeric text, a browser type, or a time associated with a consumer activity.
6. A system comprising:
a memory to store instructions; and
a processor operatively coupled to the memory, the processor to execute the instructions to:
receive, from a crawler application program, a first data stream comprising crawled content pertaining to a plurality of webpages associated with a vertical, the crawled content comprising keywords and locations for the keywords, wherein the crawler application program comprises instructions to extract the crawled content from a set of uniform resource locator (URL) fragments;
receive a second data stream comprising expert data associated with one or more predetermined keywords pertaining to the vertical, the expert data comprising one or more inferences associated with relationships between the one or more predetermined keywords, the one or more inferences drawn prior to identifying one or more of the predetermined keywords in the crawled content;
identify at least one predetermined keyword in the crawled content to determine respective locations and a number of occurrences for the at least one predetermined keyword within the crawled content;
associate crawled content from the plurality of webpages with a third data stream comprising user activity data pertaining to a plurality of user computing devices accessing the plurality of webpages;
filter, from the user activity data, a first set of activity data indicating a user is no longer viewing a set of webpages comprising the at least one predetermined keyword, wherein the filtering generates a second set of activity data;
receive a fourth data stream comprising third party geolocation or demographic data associated with the plurality of consumers accessing at least a portion of the plurality of webpages comprising the at least one of the predetermined keywords;
determine one or more associations between the third party geolocation or demographic data and the at least one of the predetermined keywords in the crawled content;
transform the first data stream, the second data stream, the third data stream and the fourth data stream to be indexed in a first database, wherein the first data stream, the second data stream, the third data stream and the fourth data stream are collected from different sources;
associate the second set of user activity data and the third party geolocation or demographic data based at least in part on the respective locations for the at least one predetermined keyword in the first database for identifying and measuring a trend in consumer content demand;
determine, using the database, one or more vertical metrics of at least one of a product or a brand associated with the one or more vertical metrics;
collect, from a plurality of input devices associated with the plurality of user computing devices, data corresponding to movement of the plurality of input devices;
determine an amount of interaction time of the plurality of user computing devices with one or more webpages corresponding to the one or more vertical metrics;
filter the one or more vertical metrics in view of the amount of interaction time to generate a filtered list of vertical metrics, wherein the filter improves processing based on analysis of the interaction time as a function of the one or more vertical metrics; and
display a report comprising the amount of interaction time corresponding to each of the one or more vertical metrics.
7. The system of claim 6 , the processor to execute the instructions to: match one or more first URLs associated with the at least one of the third party geolocation or the demographic data with one or more second URLs associated with the one or more predetermined keywords.
8. The system of claim 6 , the processor to execute the instructions to determine a classification of at least a portion of one of the plurality of webpages to generate the report comprising a statistic associated with the vertical.
9. The system of claim 6 , wherein the vertical relates to at least one of an automobile industry, a type of vehicle, a consumer product industry, a type of consumer product, an entertainment industry, a type of entertainment, a music industry, a type of music, a motion picture industry, a type of motion picture, a pharmaceuticals industry, a type of pharmaceutical, an apparel industry, a type of apparel, a financial products industry, a financial services industry, a type of financial product, or a type of financial service.
10. The system of claim 6 , wherein the user activity data comprises at least one of a first URL, an Internet protocol (IP) address, alphanumeric text, a browser type, or a time associated with a consumer activity.
11. The system of claim 6 , wherein the third party demographic data comprises at least one external sales data, consumer demographic information, consumer age data, IP (Internet Protocol) address, zip code, area code, location coordinate, or geocode.
12. The system of claim 6 , the processor to execute the instructions to receive data from a tracking tag associated with one of the plurality of webpages.
13. The system of claim 6 , the processor to execute the instructions to normalize the user activity data.
14. A non-transitory computer readable storage medium comprising instructions that, when executed by a processor, cause the processor to:
receive, from a crawler application program, a first data stream comprising crawled content pertaining to a plurality of webpages associated with a vertical, the crawled content comprising keywords and locations for the keywords, wherein the crawler application program comprises instructions to extract the crawled content from a set of uniform resource locator (URL) fragments;
receive a second data stream comprising expert data associated with one or more predetermined keywords pertaining to the vertical, the expert data comprising one or more inferences associated with relationships between the one or more predetermined keywords, the one or more inferences drawn prior to identifying one or more of the predetermined keywords in the crawled content;
identify at least one predetermined keyword in the crawled content to determine respective locations and a number of occurrences for the at least one predetermined keyword within the crawled content;
associate crawled content from the plurality of webpages with a third data stream comprising user activity data pertaining to a plurality of user computing devices accessing the plurality of webpages;
filter, from the user activity data, a first set of activity data indicating a user is no longer viewing a set of webpages comprising the at least one predetermined keyword, wherein the filtering generates a second set of activity data;
receive a fourth data stream comprising third party geolocation or demographic data associated with the plurality of consumers accessing at least a portion of the plurality of webpages comprising the at least one of the predetermined keywords;
determine one or more associations between the third party geolocation or demographic data and the at least one of the predetermined keywords in the crawled content;
transform the first data stream, the second data stream, the third data stream and the fourth data stream to be indexed in a first database, wherein the first data stream, the second data stream, the third data stream and the fourth data stream are collected from different sources;
associate the second set of user activity data and the third party geolocation or demographic data based at least in part on the respective locations for the at least one predetermined keyword in the first database for identifying and measuring a trend in consumer content demand;
determine, using the database, one or more vertical metrics of at least one of a product or a brand associated with the one or more vertical metrics;
collect, from a plurality of input devices associated with the plurality of user computing devices, data corresponding to movement of the plurality of input devices;
determine an amount of interaction time of the plurality of user computing devices with one or more webpages corresponding to the one or more vertical metrics;
filter the one or more vertical metrics in view of the amount of interaction time to generate a filtered list of vertical metrics, wherein the filter improves processing based on analysis of the interaction time as a function of the one or more vertical metrics; and
display a report comprising the amount of interaction time corresponding to each of the one or more vertical metrics.
15. The non-transitory computer readable storage medium of claim 14 , wherein the vertical relates to at least one of an automobile industry, a type of vehicle, a consumer product industry, a type of consumer product, an entertainment industry, a type of entertainment, a music industry, a type of music, a motion picture industry, a type of motion picture, a pharmaceuticals industry, a type of pharmaceutical, an apparel industry, a type of apparel, a financial products industry, a financial services industry, a type of financial product, or a type of financial service.