DATA REFINING ENGINE FOR HIGH PERFORMANCE ANALYSIS SYSTEM AND METHOD
Price and product attributes from webpages are analyzed over time to identify price changes specific to products on individual webpages and for products across all webpages as well as to identify longitudinal correlations between price changes and product attributes. Users may search the data and set alerts.
1 . A computer implemented method of processing information from webpages, the method comprising:
receiving a first and a second set of price and product attributes for a first product, which attributes comprise:
a first identifier of a first identifier-type derived from a URI which links to a webpage offering the product for sale, a second identifier of a second identifier-type assigned to all instances of the product as offered for sale at any URI, and a first category in a category taxonomy;
performing a first URI-specific price analysis of price values in the first and second sets of price and product attributes to identify changes in price for the first product and associating the result with the first identifier of the first identifier-type and saving the result as a first URI-specific core price result;
receiving a third and fourth set of price and product attributes for a second product, which attributes comprise:
a third identifier of the first identifier-type, a fourth identifier of the second identifier-type, and a second category in the category taxonomy;
performing a second URI-specific price analysis of price values in the third and fourth sets of price and product attributes to identify changes in price for the second product and associating the result with the third identifier of the first identifier-type and saving the result as a second URI-specific core price result;
when the second identifier and the fourth identifiers are the same, performing a first non-URI-specific price analysis utilizing the first and second URI-specific core price results to identify changes in price according to the second identifier-type and saving the result as a first non-URI-specific core price result;
saving and indexing the output of the URI-specific and non-URI-specific price analyses in a first file structure and making the first file structure available to be searched substantially as the sets of product and price attributes are received;
performing a meta-analysis utilizing the URI-specific and non-URI-specific core price results to identify what product and price attributes across the datasets are associated with the changes in price; and
saving and indexing the output of the meta-analysis as a second file structure and making the second file structure available to be searched.
2 . The method of claim 1 , further comprising merging new product attribute records into prior product attribute records and saving new price attribute records along with prior price attribute records.
3 . The method of claim 1 , wherein the URI-specific price analysis comprises determining the high, low, average, mean, magnitude and number of price changes over at least one time period for the price and product attributes associated with the same identifier of the first identifier-type.
4 . The method of claim 1 , wherein the non-URI-specific price analysis comprises determining the high, low, average, mean, magnitude and number of price changes over at least one time period for the price and product attributes associated the same identifier of the second identifier-type.
5 . The method of claim 4 , wherein at least one of the first and second identifier-types are further associated with at least one of a store, a merchant, and a location and wherein the non-URI-specific price analysis produces results associated therewith.
6 . The method of claim 1 , wherein the price attributes comprise at least one of a time, a product name, a price, a quantity, a unit of measurement, a merchant name, a store name, a bundle detail, and a location.
7 . The method of claim 1 , wherein the product attributes comprise at least one of a title, a brand, a category in the category taxonomy, a color, a product type, and a size.
8 . The method of claim 1 , further comprising receiving a user query and executing the query relative to the first and/or second file structures.
9 . The method of claim 1 , further comprising receiving a user query, a schedule for executing the query, executing the query at the scheduled time on the first and/or second file structures, and alerting the user regarding the result of the query.
10 . The method of claim 1 , wherein the first and second file structures may be searched by at least one of the first identifier-type, the second identifier-types, or a category in the category taxonomy.
11 . The method of claim 1 , wherein the first and second categories are the same.
12 . The method of claim 1 , wherein the meta-analysis determines the volatility of price changes over time for each of the first and second products.
13 . The method of claim 12 , wherein the volatility is determined by counting the number of price changes in a time period according to at least one of the first identifier-type, the second identifier-type, a brand, a region, a price band, and a category in the category taxonomy.
14 . The method of claim 1 , wherein the meta-analysis determines whether one of the products is a substitute for the other.
15 . The method of claim 14 , wherein whether one of the products is a substitute for the other is determined by determining if the first and second products are in the same category in the category taxonomy and by determining whether the first and second products are within a price band within the category.
16 . The method of claim 15 , further comprising determining if the first and second products share at least fifty-percent of the same product attributes.
17 . The method of claim 1 , wherein the meta-analysis determines predictions regarding the future prices for the products.
18 . The method of claim 17 , wherein the predictions are determined by obtaining the last price of at least one of the products from the URI-specific core price associated therewth, calculating or obtaining first and second linear regression parameters, multiplying the second linear regression parameter by the last price and adding this to the first linear regression parameter.
19 . The method of claim 1 , wherein the price and product attributes comprise at least one of a store, merchant, or brand and the meta-analysis determines products associated therewith and competitors thereof.
20 . The method of claim 1 , wherein the meta-analysis determines whether a price change for the first product leads or follows a price change for the second product.
21 . The method of claim 1 , wherein the meta-analysis determines whether the first or second product is a premium product relative to the other.
22 . The method of claim 1 , wherein the meta-analysis determines the price ranges in which the products are offered for sale.
23 . A webpage information processing computing apparatus, the apparatus comprising a processor and a memory storing instructions that, when executed by the processor, configure the apparatus to:
receive a first and a second set of price and product attributes for a first product, which attributes comprise:
a first identifier of a first identifier-type derived from a URI which links to a webpage offering the product for sale, a second identifier of a second identifier-type assigned to all instances of the product as offered for sale at any URI, and a first category in a category taxonomy;
perform a first URI-specific price analysis of price values in the first and second sets of price and product attributes to identify changes in price for the first product and associating the result with the first identifier of the first identifier-type and save the result as a first URI-specific core price result;
receive a third and fourth set of price and product attributes for a second product, which attributes comprise:
a third identifier of the first identifier-type, a fourth identifier of the second identifier-type, and a second category in the category taxonomy;
perform a second URI-specific price analysis of price values in the third and fourth sets of price and product attributes to identify changes in price for the second product and associate the result with the third identifier of the first identifier-type and save the result as a second URI-specific core price result;
when the second identifier and the fourth identifiers are the same, perform a first non-URI-specific price analysis utilizing the first and second URI-specific core price results to identify changes in price according to the second identifier-type and save the result as a first non-URI-specific core price result;
save and index the output of the URI-specific and non-URI-specific price analyses in a first file structure and make the first file structure available to be searched substantially as the sets of product and price attributes are received;
perform a meta-analysis utilizing the URI-specific and non-URI-specific core price results to identify what price and product attributes across the datasets are associated with the changes in price; and
save and index the output of the meta-analysis as a second file structure and make the second file structure available to be searched.
24 . A non-transient computer-readable storage medium having stored thereon instructions that, when executed by a processor, configure the processor to:
receive a first and a second set of price and product attributes for a first product, which attributes comprise:
a first identifier of a first identifier-type derived from a URI which links to a webpage offering the product for sale, a second identifier of a second identifier-type assigned to all instances of the product as offered for sale at any URI, and a first category in a category taxonomy;
perform a first URI-specific price analysis of price values in the first and second sets of price and product attributes to identify changes in price for the first product and associating the result with the first identifier of the first identifier-type and save the result as a first URI-specific core price result;
receive a third and fourth set of price and product attributes for a second product, which attributes comprise:
a third identifier of the first identifier-type, a fourth identifier of the second identifier-type, and a second category in the category taxonomy;
perform a second URI-specific price analysis of price values in the third and fourth sets of price and product attributes to identify changes in price for the second product and associate the result with the third identifier of the first identifier-type and save the result as a second URI-specific core price result;
when the second identifier and the fourth identifiers are the same, perform a first non-URI-specific price analysis utilizing the first and second URI-specific core price results to identify changes in price according to the second identifier-type and save the result as a first non-URI-specific core price result;
save and index the output of the URI-specific and non-URI-specific price analyses in a first file structure and make the first file structure available to be searched substantially as the sets of product and price attributes are received;
perform a meta-analysis utilizing the URI-specific and non-URI-specific core price results to identify what price and product attributes across the datasets are associated with the changes in price; and
save and index the output of the meta-analysis as a second file structure and make the second file structure available to be searched.