System and method for gathering ecommerce data
View Patent ↗The techniques introduced here provide a method of gathering ecommerce data. The techniques described here allow a system to return information about a product from several non-related ecommerce sites in response to a single search query. Using the techniques described here, a data mining system determines from the search query a product ID and retrieves from a database one or more product links that correspond to the product ID. Using the product links retrieved from the database, the data mining system traverses the links and parses the web-pages corresponding each of the links to determine up to date product information. The product information can then be returned to the application that initiated the request.
1. A method comprising:
receiving at a data mining system, a product URL;
determining, using a hardware processor of the data mining system, whether the product URL has a pattern of alphanumeric characters, numbers or symbols that is defined by a regular expression, wherein the regular expression defines a rule that represents URLs as a sequence of alphanumeric characters, numbers or symbols;
if the product URL has the pattern defined by the regular expression, generating a unique link ID associated with the product URL by extracting the alphanumeric characters, numbers or symbols from the product URL based on the regular expression;
retrieving from a database a product ID corresponding to the unique link ID;
retrieving from the database one or more product links that correspond to the product ID;
traversing the one or more product links;
parsing, by the processor, a web-page corresponding to each of the one or more product links to determine product information; and
displaying the product information, via a display device, to a user.
2. The method of claim 1 wherein determining a unique link ID includes:
determining from the product URL a domain and retrieving the regular expression that corresponds to the domain from the database;
processing the product URL using the regular expression;
combining the domain with the processed product URL to create a unique link; and
retrieving a unique link ID associated with the unique link from the database.
3. The method of claim 1 further comprising writing the returned product information to the database, wherein the product link is used as a key for the product information.
4. The method of claim 1 further comprising writing the returned product information to a local instance of a shared cache and replicating the product information to other instances of the shared cache.
5. The method of claim 1 wherein the product information includes a screenshot of a web-page referenced by the product link.
6. The method of claim 1 wherein the product information includes at least one of, a SKU, a product name, a manufacturer name, a part number, a physical description, or a sales price.
7. A data mining system comprising:
a web server coupled with an application through a network, the web server configured to receive from the application a URL associated with a product, wherein, in operation, the web server determines whether the URL has a pattern of alphanumeric characters, numbers or symbols that is defined by a regular expression, wherein the regular expression defines a rule that represents URLs as a sequence of alphanumeric characters, numbers or symbols, if the product URL has the pattern defined by the regular expression, generates a unique product ID from the URL by extracting the alphanumeric characters, numbers or symbols from the URL based on the regular expression, retrieves from a database one or more product links that are associated with the unique product ID, and creates a job request to gather information from the one or more product links;
an acquisition engine configured to traverse the one or more product links in response to the job request, wherein the acquisition engine, in operation, parses a web-page corresponding to each of the one or more product links and returns information relating to the product from the one or more product links.
8. The data mining system of claim 7 further comprising a job server configured to queue the job request until the acquisition engine is available to process the job request.
9. The data mining system of claim 7 further comprising a datastore including a product information database, wherein the information relating to the product is written to the product information database.
10. The data mining system of claim 7 further comprising a shared cache, wherein the information relating the product is written to the shared cache.
11. The data mining system of claim 7 further comprising an application programming interface (API) configured to provide an interface with the application to access the information gathered by the data mining system.
12. A method comprising:
receiving at a data mining system, through an application programming interface, a search query including a product URL;
determining, at the data mining system, whether the product URL has a pattern of alphanumeric characters, numbers or symbols that is defined by a regular expression, wherein the regular expression defines a rule that represents URLs as a sequence of alphanumeric characters, numbers or symbols;
if the product URL has the pattern defined by the regular expression, producing using a hardware processor of the data mining system, a unique link by extracting the alphanumeric characters, numbers or symbols from the product URL based on the regular expression;
retrieving from a database a unique link ID that corresponds to the unique link;
retrieving from the database a product ID corresponding to the unique link ID;
retrieving from the database one or more product links that correspond to the product ID;
traversing the one or more product links;
parsing, by the processor, a web-page corresponding to each of the one or more product links to determine product information; and
displaying the product information, via a display device, to a user.
13. The method of claim 12 wherein producing a unique link by running the product URL through a regular expression includes:
determining from the product URL a domain and retrieving the regular expression that corresponds to the domain from the database;
processing the product URL using the regular expression; and combining the domain with the processed product URL to create a unique link.
14. The method of claim 12 further comprising writing the returned product information to a product database.
15. The method of claim 12 further comprising writing the returned product information to a local instance of a shared cache and replicating the product information to other instances of the shared cache.
16. The method of claim 12 further comprising taking a screenshot of a web-page referenced by each product link of the one or more product links and returning the screenshot with the product information.
17. The method of claim 12 wherein the product information includes at least one of, a SKU, a product name, a manufacturer name, a part number, a physical description, or a sales price.
18. A computer server, comprising:
a processor;
a network interface, coupled to the processor, through which the computer server can communicate with ecommerce sites;
a data storage storing a database; and
a memory storing instructions which, when executed by the processor, cause the computer server to perform a process including:
receiving, via the network interface, a product URL of a product web page from an ecommerce site;
determining whether the product URL has a pattern of alphanumeric characters, numbers or symbols that is defined by a regular expression, wherein the regular expression defines a rule that represents URLs as a sequence of alphanumeric characters, numbers or symbols;
if the product URL has the pattern defined by the regular expression, reducing to a prime URL from the product URL by extracting the alphanumeric characters, numbers or symbols from the product URL using the regular expression, wherein the product URL is not in the simplest form, wherein the prime URL is a URL that cannot be expressed in any simpler form and still leads to the product web page; and
retrieving, form the database, a product ID associated with the prime URL, wherein the product ID identifies a product being sold in the product web page.
19. The computer server of claim 18 , wherein the creating a prime URL by running the product URL through a regular expression further includes:
determining, by the processor, a domain of the ecommerce site from the product URL of the product web page;
retrieving, from the database, a base URL and a regular expression associated with the domain; and
creating a prime URL by running the product URL through a regular expression and adding the base URL.
20. The computer server of claim 18 , wherein the process further includes:
associating the product ID to a plurality of product URLs, wherein the plurality of product URLs have the same prime URL.