System and method for analyzing and correcting retail data
View Patent ↗A computer system and method is disclosed that analyzes and corrects retail data. The system and method includes several client workstations and one or more servers coupled together over a network. A database stores various data used by the system. A business logic server uses competitive and complementary fusion to analyze and correct some of the data sources stored in database server. The data fusion process itself is an iterative one—utilizing both competitive and complementary fusion methods. In competitive fusion, two or more data sources that provide overlapping attributes are compared against each other. More accurate/reliable sources are used to correct less accurate/reliable sources. In complementary fusion, relationships modeled where data sources overlap are projected to areas of the data framework in which fewer sources exist—enhancing the accuracy/reliability of those fewer sources even in the absence of the other sources upon which the models were based.
1. A system comprising:
one or more computers being operable to store:
retail data associated with a first entity, the retail data including data from a first data source and a second data source, the retail data further including product identifiers;
retail data associated with a second entity, the retail data including product identifiers, the retail data including data from a third data source; and
a plurality of factor calculations;
wherein one or more of said computers contains business logic that is operable to:
identify and retrieve, based on the product identifiers, a plurality of overlapping attribute segments to use for comparing the data from the first and second data sources,
compare the plurality of the overlapping attribute segments,
calculate a plurality of factors for each of the overlapping attribute segments, each factor representing a bias present in the second data source, and
use the factors to adjust the values in the retail data from the third data source, thereby reduce bias present in the third source.
2. The system of claim 1 , wherein the one or more servers are further operable to use the factors to update the retail data from the third data source to reduce incompleteness.
3. The system of claim 1 , wherein the one or more servers are further operable to calculate the factor for an overlapping attribute segment by dividing a first data source volume amount by a corresponding second data source volume amount.
4. The system of claim 1 , wherein the one or more computers are further operable apply the factor for the attribute segment to the retail data from the third data source by multiplying a plurality of third data source volume amounts by the factor for the corresponding attribute segment.
5. The system of claim 1 , wherein the one or more computers are further operable to only calculate the factors for each attribute that is identified in the server to be relevant.
6. The system of claim 1 , wherein the one or more computers are further operable to only calculate the factors for attribute segments that are determined to be significant.
7. The system of claim 1 , wherein the one or more computers are further operable to save each factor in a database.
8. The system of claim 2 , wherein the one or more computers are further operable to calculate a blended factor when at least two measures are available for a same factor, said blended factor being used to reduce bias in the second or third data source.
9. The system of claim 8 , wherein the one or more computers are further operable to calculate the blended factor by giving a more accurate data source a higher relative weight and by giving a less accurate data source a lower relative weight.
10. The system of claim 1 , wherein the one or more computers are further operable to calculate a blended factor when at least two measures are available for a same factor, said blended factor being used to reduce bias in the second data source.
11. The system of claim 10 , wherein the one or more computers are further operable to calculate the blended factor by giving the more accurate data source a higher relative weight and by giving the less accurate data source a lower relative weight.
12. The system of claim 1 , further comprising:
a user station coupled to the one or more computers over a network, wherein the user station includes a display, and is operable to present a graphical user interface that allows a user to administer a plurality of settings used for analyzing and correcting the data sources.
13. The system of claim 1 , further comprising:
a number of user stations coupled to the one or more computers over a network, wherein each of the user stations includes a respective one of a number of displays, and is operable to present a graphical user interface that allows a user to view the corrected data in a multidimensional format.
14. The system of claim 1 , wherein the product identifiers in the first or second data sources include universal product codes.
15. The system of claim 1 , wherein the one or more computers further contain business logic operable to:
use the factors to adjust values in the retail data from the second data source, thereby reduce bias present in the second source.
16. The system of claim 1 , wherein the first data source includes point-of-sale data from a first retailer, the second data source includes panel data from the first retailer, and the third data source includes panel data from a second retailer.
17. The system of claim 1 , wherein the one or more computers are further operable to store retail data from a fourth data source corresponding to the second entity, and wherein the one or more computers further contains business logic that is operable to:
use the factors to adjust values in the retail data from the fourth data source, thereby reducing bias present in the fourth data source.
18. The system of claim 17 , wherein the fourth data source includes product shipment data.
19. The system of claim 1 , wherein the one or more computers are further operable to store retail data from a fourth data source corresponding to the second entity, wherein the fourth data source contains product identifiers, and wherein the one or more computers further contains business logic that is operable to:
identify and retrieve, based on the product identifiers, a plurality of overlapping attribute segments to use for comparing the data from the first and second data sources,
compare the plurality of the overlapping attribute segments,
calculate a plurality of factors for each of the overlapping attribute segments, each factor representing a bias present in the second data source, and
use the factors to adjust the values in the retail data from the third data source, thereby reduce bias present in the third source.
20. The system of claim 1 , wherein retail data consists of one or more of:
point of sale data, consumer panel data, shipment data, channel report data reported sale data, store database data, geo-demographic data, category studies, industry reports, delivery data, retailer-supplied data, audit data, causal data, promotional data, population census data, geographic data, store universe data, supply chain data, loyalty card data, and survey-based data.
21. The system of claim 20 , wherein retail data consists of one or more of:
point of sale data and consumer panel data.
22. A method comprising:
using a computer, identifying a first entity with corresponding first and second retail data sources, the first and second data sources including product identifiers;
identifying a second entity and a corresponding third data source, the third data source including product identifiers;
based on the product identifiers of the first, second, and third data sources, identifying a plurality of overlapping attribute segments among the first, second, and third data sources to use for comparing the data sources;
calculating at least one factor as a function of at least one of the overlapping attribute segments among the first and second data sources; and
using the at least one factor to create modified values in the third data source, said modified values having a reduced bias compared to the original, non-modified values.
23. The method of claim 22 , wherein the first data source is point-of-sale data and the second data source is consumer panel data.
24. The method of claim 22 , wherein the first data source does not include shipment data.
25. The method of claim 24 , wherein the second data source does not include shipment data.
26. The method of claim 22 , wherein the data sources are compared based on a common time dimension.
27. The method of claim 22 , wherein the data sources are compared based on a common venue dimension.
28. The method of claim 22 , wherein the data sources are compared based on a common product dimension.
29. The method of claim 22 , further comprising:
determining that additional data sources are available for comparison;
using the additional data sources to calculate additional factors; and
applying the additional factors to the third data source.
30. The method of claim 22 , wherein the using the at least one factor step is repeated multiple times to further improve the accuracy of the second data source.
31. The method of claim 26 , wherein the first data source is point-of-sale data that was captured with a reader, and wherein the data sources are compared based on a common time dimension, and wherein the factors are used to update a first group of values in the second data source to reduce bias.
32. The method of claim 22 , wherein retail data consists of one or more of:
point of sale data, consumer panel data, shipment data, channel report data reported sale data, store database data, geo-demographic data, category studies, industry reports, delivery data, retailer-supplied data, audit data, causal data, promotional data, population census data, geographic data, store universe data, supply chain data, loyalty card data, and survey-based data.
33. The method of claim 32 , wherein retail data consists of one or more of:
point of sale data and consumer panel data.