System and method for merging slowly changing data
The disclosure generally describes computer-implemented methods, software, and systems for accessing volumes of data records structured to include sets dimensions, each dimension labelled in a manner specific to respective entities; identifying candidates data records keyed by managed keys that span a subset of dimensions even though at least one dimension from the subset of dimensions is labelled differently between the different volumes; comparing the candidate data records from the different volumes to determine whether a particular managed key is valid based on contents of the candidate data records from the different volumes; in response to determining that the particular managed key is valid, combining the candidate data records keyed by the valid managed key to be merged and accessible as one continuous entry; and in response to determining that the particular managed key is invalid, combining the candidate data records from the different volumes as separate entries.
1. A computer-implemented method comprising:
accessing a first volume of multiple data records from a first data server managed by a first entity, the first volume covering a first period and including a first set of dimensions, each dimension labelled specifically to the first entity;
accessing a second volume of multiple data records from a second data server managed by a second entity that is different from the first entity, the second volume covering a second period and including a second set of dimensions, each dimension labelled specifically to the second entity, and the second period overlapping the first period during an overlapping period;
identifying a plurality of candidate data records keyed by a plurality of managed keys, wherein each of the plurality of managed keys spans a subset of dimensions included by the first volume and the second volume, and wherein at least one dimension from the subset of dimensions is labelled differently between the first volume and the second volume;
determining that a managed key from the plurality of managed keys is valid based on a comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume, wherein the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume both cover the overlapping period; and
consolidating the first volume with the second volume by joining the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume according to the valid managed key such that the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume keyed by the valid managed key are merged and accessible as one continuous entry.
2. The method of claim 1 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume comprises:
comparing contents of the plurality of candidate data records from the first volume keyed by the managed key with the contents of the plurality of candidate data records from the second volume keyed by the managed key.
3. The method of claim 2 , wherein the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volumes cover more than a threshold duration within the overlapping period.
4. The method of claim 2 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume utilizes a fuzzy logic.
5. The method of claim 4 , wherein the fuzzy logic comprises:
comparing contents of the plurality of candidate data records from the first volume with the contents of the plurality of candidate data records from the second volume to determine that the contents of the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume substantially match, wherein the at least one dimension is labelled differently.
6. The method of claim 5 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are identical for more than a threshold percentage of a duration within the overlapping period.
7. The method of claim 5 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from second volume substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are textually more than a threshold percent identical.
8. The method of claim 5 , further comprising:
scoring a similarity between the contents of the plurality of candidate data records from the first volume with the contents of the plurality of the candidate data records from the second volume, wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume substantially match when the scored similarity exceeds a threshold value.
9. The method of claim 1 , wherein the subset of dimensions includes three or more dimensions.
10. A computer system comprising one or more processors that are configured to perform operations of:
accessing a first volume of multiple data records from a first data server managed by a first entity, the first volume covering a first period and including a first set of dimensions, each dimension labelled specifically to the first entity;
accessing a second volume of multiple data records from a second data server managed by a second entity that is different from the first entity, the second volume covering a second period and including a second set of dimensions, each dimension labelled specifically to the second entity, and the second period overlapping the first period during an overlapping period;
identifying a plurality of candidate data records keyed by a plurality of managed keys, wherein each of the plurality of managed keys spans a subset of dimensions included by the first volume and the second volume, and wherein at least one dimension from the subset of dimensions is labelled differently between the first volume and the second volume;
determining that a managed key is valid based on a comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume, wherein the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume both cover the overlapping period; and
consolidating the first volume with the second volume by joining the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume according to the valid managed key such that the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume keyed by the valid managed key are merged and accessible as one continuous entry.
11. The computer system of claim 10 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume comprises:
comparing contents of the plurality of candidate data records from the first volume keyed by the managed key with the contents of the plurality of candidate data records from the second volume keyed by the managed key.
12. The computer system of claim 11 , the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volumes cover more than a threshold duration within the overlapping period.
13. The computer system of claim 11 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume utilizes a fuzzy logic.
14. The computer system of claim 13 , wherein the fuzzy logic comprises:
comparing contents of the plurality of candidate data records from the first volume with the contents of the plurality of candidate data records from the second volume to determine that the contents of the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume substantially match, wherein the at least one dimension is labelled differently.
15. The computer system of claim 14 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are identical for more than a threshold percentage of a duration within the overlapping period.
16. The computer system of claim 14 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from second volume substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are textually more than a threshold percent identical.
17. The computer system of claim 14 , wherein the operations further comprise:
scoring a similarity between the contents of the plurality of candidate data records from the first volume with the contents of the plurality of the candidate data records from the second volume, wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume substantially match when the scored similarity exceeds a threshold value.
18. The computer system of claim 10 , wherein the subset of dimensions includes three or more dimensions.