System and method for merging slowly changing data
The disclosure generally describes computer-implemented methods, software, and systems for accessing volumes of data records structured to include sets dimensions, each dimension labelled in a manner specific to respective entities; identifying candidates data records keyed by managed keys that span a subset of dimensions even though at least one dimension from the subset of dimensions is labelled differently between the different volumes; comparing the candidate data records from the different volumes to determine whether a particular managed key is valid based on contents of the candidate data records from the different volumes; in response to determining that the particular managed key is valid, combining the candidate data records keyed by the valid managed key to be merged and accessible as one continuous entry; and in response to determining that the particular managed key is invalid, combining the candidate data records from the different volumes as separate entries.
1. A computer-implemented method comprising:
obtaining data records from a first volume and data records from a second volume, wherein each of the data records is keyed by a respective key,
wherein each key spans a subset of dimensions, wherein each of the dimensions indicates a respective attribute of the data records,
wherein at least one dimension from the subset of dimensions is labelled differently between the first volume and the second volume,
determining that a particular key is invalid based on a comparison of the data records from the first volume with the data records from the second volume, wherein the data records from the first volume and the data records from the second volume cover an overlapping period of time; and
combining the data records from the first volume with the data records from the second volume according to the invalid key such that the data records from the first volume and the data records from the second volume keyed by the invalid key are accessible as a single entry.
2. The method of claim 1 , wherein the comparison of the data records from the first volume with the data records from the second volume comprises:
comparing contents of the data records from the first volume with the contents of the data records from the second volume.
3. The method of claim 2 , wherein the data records from the first volume and the data records from the second volumes cover more than a threshold duration within the overlapping period.
4. The method of claim 2 , wherein the comparison of the data records from the first volume with the data records from the second volume utilizes a fuzzy logic.
5. The method of claim 4 , wherein the fuzzy logic comprises:
determining that the contents of the data records from the first volume and the data records from the second volume do not substantially match, wherein the at least one dimension is labelled differently.
6. The method of claim 5 , wherein the contents of the data records from the first volume and the contents of the data records from the second volume do not substantially match when the data records from the first volume and the data records from the second volume are identical for no more than a threshold percentage of a duration within the overlapping period.
7. The method of claim 5 , wherein the contents of the data records from the first volume and the contents of the data records from second volume do not substantially match when the data records from the first volume and the data records from the second volume are textually no more than a threshold percent identical.
8. The method of claim 5 , further comprising:
scoring a similarity between the contents of the data records from the first volume with the contents of the data records from the second volume, wherein the contents of the data records from the first volume and the contents of the data records from the second volume do not substantially match when the scored similarity does not exceed a threshold value.
9. The method of claim 1 , wherein the subset of dimensions includes three or more dimensions.
10. A computer system comprising one or more processors that are configured to perform operations of:
obtaining data records from a first volume and data records from a second volume, wherein each of the data records is keyed by a respective key,
wherein each key spans a subset of dimensions, wherein each of the dimensions indicates a respective attribute of the data records,
wherein at least one dimension from the subset of dimensions is labelled differently between the first volume and the second volume,
determining that a particular key is invalid based on a comparison of the data records from the first volume with the data records from the second volume, wherein the data records from the first volume and the data records from the second volume cover an overlapping period of time; and
combining the data records from the first volume with the data records from the second volume according to the invalid key such that the data records from the first volume and the data records from the second volume keyed by the invalid key are accessible as a single entry.
11. The computer system of claim 10 , wherein the comparison of the data records from the first volume with the data records from the second volume comprises:
comparing contents of the data records from the first volume with the contents of the data records from the second volume.
12. The computer system of claim 11 , wherein the data records from the first volume and the data records from the second volumes cover more than a threshold duration within the overlapping period.
13. The computer system of claim 11 , wherein the comparison of the data records from the first volume with the data records from the second volume utilizes a fuzzy logic.
14. The computer system of claim 13 , wherein the fuzzy logic comprises:
determining that the contents of the data records from the first volume and the data records from the second volume do not substantially match, wherein the at least one dimension is labelled differently.
15. The computer system of claim 14 , wherein the contents of the data records from the first volume and the contents of the data records from the second volume do not substantially match when the data records from the first volume and the data records from the second volume are identical for no more than a threshold percentage of a duration within the overlapping period.
16. The computer system of claim 14 , wherein the contents of the data records from the first volume and the contents of the data records from second volume do not substantially match when the data records from the first volume and the data records from the second volume are textually no more than a threshold percent identical.
17. The computer system of claim 14 , wherein the operations further comprise:
scoring a similarity between the contents of the data records from the first volume with the contents of the data records from the second volume, wherein the contents of the data records from the first volume and the contents of the data records from the second volume do not substantially match when the scored similarity does not exceed a threshold value.
18. The computer system of claim 10 , wherein the subset of dimensions includes three or more dimensions.