Clickstream analysis methods and systems related to improvements in online stores and media content
Methods and systems are provided herein for the analysis of information about online actions of a plurality of users. The analysis methods and systems allow for the creation of new online and offline business methods based on online consumer behavior. The methods and systems may obtain an input data set comprising information about online actions of a plurality of users, convert the input data set into data files having a common file format with each data file corresponding to a user of the plurality of users and comprising an identifier for the user and a plurality of Uniform Resource Locations (URLs) associated with online actions of the user, access online information relating to search terms and webpages, and determine one or more metrics of user behavior, including a “verticals” metric and a “search terms” metric.
1. A method, comprising:
obtaining, by one or more processors, an input data set from a plurality of data providers, the input data set comprising information about online actions of a plurality of users;
converting, by one or more processors, the input data set into data files having a common file format, each data file corresponding to a user of the plurality of users and comprising an identifier for the user and a plurality of Uniform Resource Locations (URLs) associated with online actions of the user;
decomposing and hashing, by one or more processors, the plurality of URLs associated with the online actions of the plurality of users to generate decomposed and hashed URL information;
accessing, by one or more processors, a list of verticals stored on an external communication facility, each of the verticals associated with a plurality of service providers;
extracting, by one or more processors, URL key-value pairs for each of the plurality of service providers from one or more webpages associated with the plurality of service providers;
identifying, by one or more processors, a URL of the plurality of URLs that matches an extracted URL key-value pair by comparing the decomposed and hashed URL information with the extracted URL key-value pairs;
obtaining, by one or more processors, webpage information of a webpage associated with the identified URL, the webpage information comprising product information of a product advertised on the webpage;
determining, by one or more processors, a first metric of user behavior associated the product advertised on the webpage based on the product information and the data files corresponding to the plurality of users; and
generating, by one or more processors, a separate data file comprising the first metric of user behavior.
2. The method of claim 1 , further comprising:
accessing, by one or more processors, an external data set comprising search terms associated with a plurality of webpages;
determining, by one or more processors, a format of a search URL associated with each of the plurality of webpages;
associating, by the one or more processors, the plurality of URLs with a webpage of the plurality of webpages based on the formats of the search URLs associated with the plurality of webpages; and
determining, by one or more processors, a second metric of user behavior based on the association of the plurality of URLs with the plurality of webpages.
3. The method of claim 2 , merging, by one or more processors, the second metric of user behavior into the separate data file.
4. The method of claim 1 , further comprising downloading and storing in a local memory a copy of the webpage associated with the identified URL.
5. The method of claim 1 , wherein a vertical comprises a high-level category, a sub-category, and a plurality of webpages associated with the plurality of service providers.
6. The method of claim 1 , further comprising modifying, by one or more processors, a promotion process to include an incentive for buying a brand of product based on the first metric of user behavior.
7. The method of claim 1 , further comprising:
selecting, by one or more processors, a panel of users from the plurality of users, the panel of users having one or more common demographic characteristics; and
retaining, by one or more processors, data files that correspond to a user of the panel of users and decomposing and hashing only the retained data files,
wherein first metric of user behavior is based on the data files corresponding to the panel of users.
8. The method of claim 1 , further comprising cleansing, by one or more processors, the data files in the common file format by modifying the plurality of URLs associated with the online actions of the plurality of users based on a plurality of URL rules.
9. A system, comprising:
one or more processors;
a memory storing instructions that, when executed, cause the one or more processors to:
obtain an input data set from a plurality of data providers, the input data set comprising information about online actions of a plurality of users;
convert the input data set into data files having a common file format, each data file corresponding to a user of the plurality of users and comprising an identifier for the user and a plurality of Uniform Resource Locations (URLs) associated with online actions of the user;
decompose and hash the plurality of URLs associated with the online actions of the plurality of users to generate decomposed and hashed URL information;
access a list of verticals stored on an external communication facility, each of the verticals associated with a plurality of service providers;
extract URL key-value pairs for each of the plurality of service providers from one or more webpages associated with the plurality of service providers;
identify a URL of the plurality of URLs that matches an extracted URL key-value pair by comparing the decomposed and hashed URL information with the extracted URL key-value pairs;
obtain webpage information of a webpage associated with the identified URL, the webpage information comprising product information of a product advertised on the webpage;
determine a first metric of user behavior associated the product advertised on the webpage based on the product information and the data files corresponding to the plurality of users; and
generate a separate data file comprising the first metric of user behavior.
10. The system of claim 9 , wherein the instructions, when executed, further cause the one or more processors to:
access an external data set comprising search terms associated with a plurality of webpages;
determine a format of a search URL associated with each of the plurality of webpages;
associate the plurality of URLs with a webpage of the plurality of webpages based on the formats of the search URLs associated with the plurality of webpages; and
determine a second metric of user behavior based on the association of the plurality of URLs with the plurality of webpages.
11. The system of claim 10 , wherein the instructions, when executed, further cause the one or more processors to merge the second metric of user behavior into the separate data file.
12. The system of claim 9 , wherein the instructions, when executed, further cause the one or more processors to download and store in a local memory a copy of the webpage associated with the identified URL.
13. The system of claim 9 , wherein a vertical comprises a high-level category, a sub-category, and a plurality of webpages associated with the plurality of service providers.
14. The system of claim 9 , wherein the instructions, when executed, further cause the one or more processors to modify a promotion process to include an incentive for buying a brand of product based on the first metric of user behavior.
15. A non-transitory computer readable medium having stored thereon computer executable instructions that, when executed by one or more processors of a computer, causes the one or more processors to:
obtain an input data set from a plurality of data providers, the input data set comprising information about online actions of a plurality of users;
convert the input data set into data files having a common file format, each data file corresponding to a user of the plurality of users and comprising an identifier for the user and a plurality of Uniform Resource Locations (URLs) associated with online actions of the user;
decompose and hash the plurality of URLs associated with the online actions of the plurality of users to generate decomposed and hashed URL information;
access a list of verticals stored on an external communication facility, each of the verticals associated with a plurality of service providers;
extract URL key-value pairs for each of the plurality of service providers from one or more webpages associated with the plurality of service providers;
identify a URL of the plurality of URLs that matches an extracted URL key-value pair by comparing the decomposed and hashed URL information with the extracted URL key-value pairs;
obtain webpage information of a webpage associated with the identified URL, the webpage information comprising product information of a product advertised on the webpage;
determine a first metric of user behavior associated the product advertised on the webpage based on the product information and the data files corresponding to the plurality of users; and
generate a separate data file comprising the first metric of user behavior.
16. The non-transitory computer readable medium of claim 15 , wherein the instructions, when executed, further cause the one or more processors to:
access an external data set comprising search terms associated with a plurality of webpages;
determine a format of a search URL associated with each of the plurality of webpages;
associate the plurality of URLs with a webpage of the plurality of webpages based on the formats of the search URLs associated with the plurality of webpages; and
determine a second metric of user behavior based on the association of the plurality of URLs with the plurality of webpages.
17. The non-transitory computer readable medium of claim 16 , wherein the instructions, when executed, further cause the one or more processors to merge the second metric of user behavior into the separate data file.
18. The non-transitory computer readable medium of claim 15 , wherein the instructions, when executed, further cause the one or more processors to download and store in a local memory a copy of the webpage associated with the identified URL.
19. The non-transitory computer readable medium of claim 15 , wherein a vertical comprises a high-level category, a sub-category, and a plurality of webpages associated with the plurality of service providers.
20. The non-transitory computer readable medium of claim 15 , wherein the instructions, when executed, further cause the one or more processors to modify a promotion process to include an incentive for buying a brand of product based on the first metric of user behavior.