System and method for merging slowly changing data
The disclosure generally describes computer-implemented methods, software, and systems for accessing volumes of data records structured to include sets dimensions, each dimension labelled in a manner specific to respective entities; identifying candidates data records keyed by managed keys that span a subset of dimensions even though at least one dimension from the subset of dimensions is labelled differently between the different volumes; comparing the candidate data records from the different volumes to determine whether a particular managed key is valid based on contents of the candidate data records from the different volumes; in response to determining that the particular managed key is valid, combining the candidate data records keyed by the valid managed key to be merged and accessible as one continuous entry; and in response to determining that the particular managed key is invalid, combining the candidate data records from the different volumes as separate entries.
1. A computer-implemented method comprising:
identifying a plurality of candidate data records from a first volume and a plurality of candidate data records from a second volume, wherein the plurality of candidate data records are keyed by a plurality of managed keys,
wherein each of the plurality of managed keys spans a subset of dimensions, wherein the dimensions indicate attributes of the plurality of candidate data records,
wherein at least one dimension from the subset of dimensions is labelled differently between the first volume and the second volume,
wherein the first volume of multiple data records are hosted on a first data server managed by a first entity,
wherein the first volume covers a first period and includes a first set of dimensions, each dimension being labelled specifically to the first entity,
wherein the second volume of multiple data records are hosted on a second data server managed by a second entity, and
wherein the second volume covers a second period and includes a second set of dimensions, each dimension being labelled specifically to the second entity,
determining that a managed key from the plurality of managed keys is invalid based on a comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume, wherein the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume both cover a period where the first period and the second period overlap; and
consolidating the first volume with the second volume by combining the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume according to the invalid managed key such that the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume keyed by the invalid managed key are accessible as one continuous entry.
2. The method of claim 1 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume comprises:
comparing contents of the plurality of candidate data records from the first volume keyed by the managed key with the contents of the plurality of candidate data records from the second volume keyed by the managed key.
3. The method of claim 2 , wherein the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volumes cover more than a threshold duration within the overlapping period.
4. The method of claim 2 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume utilizes a fuzzy logic.
5. The method of claim 4 , wherein the fuzzy logic comprises:
comparing contents of the plurality of candidate data records from the first volume with the contents of the plurality of candidate data records from the second volume to determine that the contents of the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume do not substantially match, wherein the at least one dimension is labelled differently.
6. The method of claim 5 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume do not substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are identical for no more than a threshold percentage of a duration within the overlapping period.
7. The method of claim 5 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from second volume do not substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are textually no more than a threshold percent identical.
8. The method of claim 5 , further comprising:
scoring a similarity between the contents of the plurality of candidate data records from the first volume with the contents of the plurality of the candidate data records from the second volume, wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume do not substantially match when the scored similarity does not exceed a threshold value.
9. The method of claim 1 , wherein the subset of dimensions includes three or more dimensions.
10. A computer system comprising one or more processors that are configured to perform operations of:
identifying a plurality of candidate data records from a first volume and a plurality of candidate data records from a second volume, wherein the plurality of candidate data records are keyed by a plurality of managed keys,
wherein each of the plurality of managed keys spans a subset of dimensions, wherein the dimensions indicate attributes of the plurality of candidate data records,
wherein at least one dimension from the subset of dimensions is labelled differently between the first volume and the second volume,
wherein the first volume of multiple data records are hosted on a first data server managed by a first entity,
wherein the first volume covers a first period and includes a first set of dimensions, each dimension being labelled specifically to the first entity,
wherein the second volume of multiple data records are hosted on a second data server managed by a second entity, and
wherein the second volume covers a second period and includes a second set of dimensions, each dimension being labelled specifically to the second entity,
determining that a managed key from the plurality of managed keys is invalid based on a comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume, wherein the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume both cover a period where the first period and the second period overlap; and
consolidating the first volume with the second volume by combining the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume according to the invalid managed key such that the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume keyed by the invalid managed key are accessible as one continuous entry.
11. The computer system of claim 10 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume comprises:
comparing contents of the plurality of candidate data records from the first volume keyed by the managed key with the contents of the plurality of candidate data records from the second volume keyed by the managed key.
12. The computer system of claim 11 , wherein the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volumes cover more than a threshold duration within the overlapping period.
13. The computer system of claim 11 , wherein the comparison of the plurality of candidate data records from the first volume with the plurality of candidate data records from the second volume utilizes a fuzzy logic.
14. The computer system of claim 13 , wherein the fuzzy logic comprises:
comparing contents of the plurality of candidate data records from the first volume with the contents of the plurality of candidate data records from the second volume to determine that the contents of the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume do not substantially match, wherein the at least one dimension is labelled differently.
15. The computer system of claim 14 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume do not substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are identical for no more than a threshold percentage of a duration within the overlapping period.
16. The computer system of claim 14 , wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from second volume do not substantially match when the plurality of candidate data records from the first volume and the plurality of candidate data records from the second volume are textually no more than a threshold percent identical.
17. The computer system of claim 14 , wherein the operations further comprise:
scoring a similarity between the contents of the plurality of candidate data records from the first volume with the contents of the plurality of the candidate data records from the second volume, wherein the contents of the plurality of candidate data records from the first volume and the contents of the plurality of candidate data records from the second volume do not substantially match when the scored similarity does not exceed a threshold value.
18. The computer system of claim 10 , wherein the subset of dimensions includes three or more dimensions.