Data refining engine for high performance analysis system and method

1. A computer implemented method of storing information and searching the stored information in close-to realtime, the method comprising:

at a first computer comprising a processor and a memory, which memory comprises:

an attribute datastore for storing price and product attributes for a set of products, which price and product attributes are obtained from webpages accessed via Uniform Resource Identifiers (“URIs”),

a history datastore for storing historical price and product attributes for the set of products, and

an analytics datastore for storing the result of an analysis of the price and product attributes in the attribute datastore and the history datastore;

at the first computer receiving a set of price and product attributes obtained from a web site for a first product, which attributes comprise a first category in a category taxonomy;

at the first computer determining a geographic area of the website;

at the first computer determining a replica of the attribute datastore and a replica of the analytics datastore, based on the geographic area of the website and the first category;

by the first computer storing the price and product attributes in the determined replica of the attribute datastore;

performing the analysis and storing the result of the analysis in the determined replica of the analytics datastore;

with respect to the price and product attributes stored in the determined replica of the attribute datastore, obtaining a set of values of high cardinality entries in the price and product attributes and hashing each such value to form a set of index key hash values;

forming an inverted bitmap index of a subset of the result of the analysis of the price and product attributes stored in the determined replica of the analytics datastore;

receiving a query from a client device coupled to the first computer over a network;

hashing a high cardinality search term in the query, if any, and comparing the hashed high cardinality search term with the set of index key hash values to determine that the replica of the attribute datastore comprises a set of records responsive to the query or, if the query does not comprise a high cardinality search term, searching for terms in the query according to the inverted bitmap index to identify the corresponding replica in the analytics datastore and the set of records responsive to the query therein;

in response to the query, returning the set of records responsive to the query to the client device, for display to a user via a user interface on the client device.

2. The method according to claim 1 , wherein the price and product attributes further comprise:

a first identifier of a first identifier-type derived from a URI which links to a webpage offering the product for sale, and

a second identifier of a second identifier-type assigned to all instances of the product as offered for sale at any URI;

and wherein the analysis comprises:

performing a first URI-specific price analysis of price values in the first and second sets of price and product attributes to identify changes in price for the first product and associating the result with the first identifier of the first identifier-type and saving the result as a first URI-specific core price result;

receiving a third and fourth set of price and product attributes for a second product, which attributes comprise:

a third identifier of the first identifier-type,

a fourth identifier of the second identifier-type, and

a second category in the category taxonomy;

performing a second URI-specific price analysis of price values in the third and fourth sets of price and product attributes to identify changes in price for the second product and associating the result with the third identifier of the first identifier-type and saving the result as a second URI-specific core price result;

when the second identifier and the fourth identifiers are the same, performing a first non-URI-specific price analysis utilizing the first and second URI-specific core price results to identify changes in price according to the second identifier-type and saving the result as a first non-URI-specific core price result;

saving and indexing the output of the URI-specific and non-URI-specific price analyses in the analytics datastore;

performing a meta-analysis utilizing the URI-specific and non-URI-specific core price results to identify what product and price attributes across the datasets are associated with the changes in price; and

saving the output of the meta-analysis in the analytics datastore.

3. The method of claim 2 , wherein the URI-specific price analysis comprises determining the high, low, average, mean, magnitude and number of price changes over at least one time period for the price and product attributes associated with the same identifier of the first identifier-type.

4. The method of claim 2 , wherein the non-URI-specific price analysis comprises determining the high, low, average, mean, magnitude and number of price changes over at least one time period for the price and product attributes associated the same identifier of the second identifier-type.

5. The method of claim 4 , wherein at least one of the first and second identifier-types are further associated with at least one of a store, a merchant, and a location and wherein the non-URI-specific price analysis produces results associated therewith.

6. The method of claim 2 , wherein the price attributes comprise at least one of a time, a product name, a price, a quantity, a unit of measurement, a merchant name, a store name, a bundle detail, and a location.

7. The method of claim 2 , wherein the product attributes comprise at least one of a title, a brand, a category in the category taxonomy, a color, a product type, and a size.

8. The method of claim 2 , further comprising receiving a user query and executing the query relative to the analytics datastore.

9. The method of claim 2 , further comprising receiving a user query, a schedule for executing the query, executing the query at the scheduled time on the analytics datastore, and alerting the user regarding the result of the query.

10. The method of claim 2 , wherein the first and second file structures may be searched by at least one of the first identifier-type, the second identifier-types, or a category in the category taxonomy.

11. The method of claim 2 , wherein the first and second categories are the same.

12. The method of claim 2 , wherein the meta-analysis determines the volatility of price changes over time for each of the first and second products.

13. The method of claim 12 , wherein the volatility is determined by counting the number of price changes in a time period according to at least one of the first identifier-type, the second identifier-type, a brand, a region, a price band, and a category in the category taxonomy.

14. The method of claim 2 , wherein the meta-analysis determines whether one of the products is a substitute for the other.

15. The method of claim 14 , wherein whether one of the products is a substitute for the other is determined by determining if the first and second products are in the same category in the category taxonomy and by determining whether the first and second products are within a price band within the category.

16. The method of claim 15 , further comprising determining if the first and second products share at least fifty-percent of the same product attributes.

17. The method of claim 2 , wherein the meta-analysis determines predictions regarding the future prices for the products.

18. The method of claim 17 , wherein the predictions are determined by obtaining the last price of at least one of the products from the URI-specific core price associated therewith, calculating or obtaining first and second linear regression parameters, multiplying the second linear regression parameter by the last price and adding this to the first linear regression parameter.

19. The method of claim 2 , wherein the price and product attributes comprise at least one of a store, merchant, or brand and the meta-analysis determines products associated therewith and competitors thereof.

20. The method of claim 2 , wherein the meta-analysis determines whether a price change for the first product leads or follows a price change for the second product.

21. The method of claim 2 , wherein the meta-analysis determines whether the first or second product is a premium product relative to the other.

22. The method of claim 2 , wherein the meta-analysis determines the price ranges in which the products are offered for sale.

23. An apparatus for storing information and searching the stored information in close-to realtime, the apparatus comprising:

a computer processor and memory, which memory comprises:

an attribute datastore for storing price and product attributes for a set of products, which price and product attributes are obtained from webpages accessed via Uniform Resource Identifiers (“URIs”),

a history datastore for storing historical price and product attributes for the set of products, and

an analytics datastore for storing the result of an analysis of the price and product attributes in the attribute datastore and the history datastore;

a data ingestion module;

an attribute datastore index module;

an analytics datastore index module;

a core price module and an insight module;

a user contact module; wherein

the data ingestion module is to:

receive a set of price and product attributes obtained from a web site for a first product, which attributes comprise a first category in a category taxonomy;

determine a geographic area of the website;

determine a replica of the attribute datastore and a replica of the analytics datastore, based on the geographic area of the website and the first category;

store the price and product attributes in the determined replica of the attribute datastore;

store the result of an analysis of the price and product attributes in the determined replica of the attribute datastore and the history datastore;

wherein the attribute datastore index module is to, with respect to the price and product attributes stored in the determined replica of the attribute datastore, obtain a set of values of high cardinality entries in the price and product attributes and hash each such value to form a set of index key hash values;

wherein the core price module and an insight module are to perform the analysis of the price and product attributes in the attribute datastore and the history datastore;

wherein the analytics datastore index module is to form an inverted bitmap index of a subset of the result of the analysis of the price and product attributes stored in the determined replica of the analytics datastore;

wherein the user contact module is to:

receive a query from a user device, the user device coupled to the apparatus over a network, hash a high cardinality search term in the query, if any, and compare the hashed high cardinality search term with the set of index key hash values to determine that the replica of the attribute datastore comprises a set of records responsive to the query or, if the query does not comprise a high cardinality search term, search for terms in the query according to the inverted bitmap index to identify the corresponding replica in the analytics datastore and the set of records responsive to the query therein, and

return the set of records responsive to the query to the user device, for display to a user via a user interface on the user device.