Bitwise data comparison utilizing dynamic temporal segmentation

1. A method for synchronizing records stored at a first data storage location with records stored at a second, remote data storage location utilizing bitwise fingerprints to reduce network resource usage and bitwise operations to reduce processing resource usage, the method comprising

(a) for records in a first data set stored at a first data store that are each associated with a numerical ID unique to the first data set,

(i) determining a block size representing a number of sequential numerical IDs which will be associated with a single block,

(ii) defining a plurality of blocks based on the determined block size and the unique IDs of the records in the first data set, each block of the defined plurality of blocks being associated with a block identifier comprising a unique numerical block identifier, wherein the unique numerical block identifier for a block and the block size determine sequential numerical IDs associated with that block, and

(iii) for each respective block of the defined plurality of blocks, generating a respective bitwise fingerprint for that respective block, each respective bit of the respective bitwise fingerprint indicating the existence or nonexistence of a record in the first data set having the unique ID associated with that respective bit;

(b) for records in a second data set stored at a second data store that are each associated with a numerical ID unique to the second data set,

(i) determining a block size representing a number of sequential numerical IDs which will be associated with a single block,

(ii) defining a plurality of blocks based on the determined block size and the unique IDs of the records in the first data set, each block of the defined plurality of blocks being associated with a block identifier comprising a unique numerical block identifier, wherein the unique numerical block identifier for a block and the block size determine the sequential numerical IDs associated with that block, and

(iii) for each respective block of the defined plurality of blocks, generating a respective bitwise fingerprint for that respective block, each respective bit of the respective bitwise fingerprint indicating the existence or nonexistence of a record in the second data set having the unique ID associated with that respective bit;

(c) comparing the existence of records in the first and second data sets by performing bitwise operations on the generated bitwise fingerprints by, for each respective numerical block identifier associated with a block of the first data set and a block of the second data set,

(i) determining, by performing an equality comparison between

(A) the generated bitwise fingerprint for the block of the first data set corresponding to the respective numerical block identifier, and

(B) the generated bitwise fingerprint for the block of the second data set corresponding to the respective numerical block identifier, whether the same records associated with the block corresponding to the respective numerical block identifier exist in both the first data set and the second data set,

(ii) for cases where the generated bitwise fingerprint for the block of the first data set corresponding to the respective numerical block identifier and the generated bitwise fingerprint for the block of the second data set corresponding to the respective numerical identifier do not match,

(A) generating an AND fingerprint corresponding to the respective numerical block identifier by performing an AND operation between

(I) the generated bitwise fingerprint for the block of the first data set corresponding to the respective numerical block identifier, and

(II) the generated bitwise fingerprint for the block of the second data set corresponding to the respective numerical block identifier,

(B) determining records that exist in the first data set but not the second data set by generating a first data set XOR fingerprint for the block of the first data set corresponding to the respective numerical block identifier by performing an XOR operation between

(I) the generated bitwise fingerprint for the block of the first data set corresponding to the respective numerical block identifier, and

(II) the generated AND fingerprint corresponding to the respective numerical block identifier,

(III) wherein each respective bit of the generated first data set XOR fingerprint indicates that a record having the unique ID associated with that respective bit exists in the first data set but not the second data set, and

(C) determining records that exist in the second data set but not the first data set by generating a second data set XOR fingerprint for the block of the second data set corresponding to the respective numerical block identifier by performing an XOR operation between

(I) the generated bitwise fingerprint for the block of the second data set corresponding to the respective numerical block identifier, and

(II) the generated AND fingerprint corresponding to the respective numerical block identifier,

(III) wherein each respective bit of the generated second data set XOR fingerprint indicates that a record having the unique ID associated with that respective bit exists in the second data set but not the first data set; and

(d) updating the second data set by transferring, from the first data store to the second data store, records from the first data set that were indicated, based on having a unique ID associated with a respective bit in a generated first data set XOR fingerprint, to exist in the first data set but not the second data set.

2. The method of claim 1 , wherein the block size is sixty three sequential numerical IDs.

3. The method of claim 1 , wherein the block size is selected to be (2∧n−1) sequential numerical IDs, where n is any number greater than 1.

4. The method of claim 1 , wherein the method is utilized to transfer records from a clinical data warehouse data store to a clinical performance management data store.

5. The method of claim 1 , wherein the first data set comprises healthcare records.

6. A method for synchronizing records stored at a first data storage location with records stored at a second, remote data storage location utilizing bitwise fingerprints to reduce network resource usage and bitwise operations to reduce processing resource usage, the method comprising

(a) maintaining, at a first data store, for each respective record in a first data set stored at the first data store,

(i) a respective dynamic temporal segmentation last update date, the respective dynamic temporal segmentation last update date for the respective record being generated based on the last two update dates for the respective record,

(ii) wherein each respective record is associated with a numerical ID unique to the first data set,

(b) for records in the first data set,

(i) determining a block size representing a number of sequential numerical IDs which will be associated with a single block, and

(ii) defining a plurality of numerical block identifiers based on the determined block size and the unique IDs of the records in the first data set, wherein the unique numerical block identifier and the block size determine sequential numerical IDs associated with that unique numerical block identifier,

(iii) for each respective numerical block identifier,

(A) determining a respective set of dynamic temporal segmentation last update dates for records in the first data set having a unique ID associated with the respective numerical block identifier, and

(B) for each respective dynamic temporal segmentation last update date in the respective set of dynamic temporal segmentation last update dates,

(1) defining a respective temporal block associated with the respective numerical block identifier and the respective dynamic temporal segmentation last update date, each respective temporal block being associated with a temporal block identifier comprising an indication of the corresponding respective numerical block identifier and an indication of the corresponding respective dynamic temporal segmentation last update date, and

(2) for each defined respective temporal block, generating a respective bitwise fingerprint for that respective temporal block, each respective bit of the respective bitwise fingerprint indicating whether a record in the first data set having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date matching the respective dynamic temporal segmentation last update date;

(c) maintaining, at a second data store, for each respective record in a second data set stored at the second data store,

(i) a respective dynamic temporal segmentation last update date, the respective dynamic temporal segmentation last update date for the respective record being generated based on the last two update dates for the respective record,

(ii) wherein each respective record is associated with a numerical ID unique to the first data set,

(d) for records in the second data set,

(i) determining a block size representing a number of sequential numerical IDs which will be associated with a single block, and

(ii) defining a plurality of numerical block identifiers based on the determined block size and the unique IDs of the records in the first data set, wherein the unique numerical block identifier and the block size determine sequential numerical IDs associated with that unique numerical block identifier,

(iii) for each respective numerical block identifier,

(A) determining a respective set of dynamic temporal segmentation last update dates for records in the second data set having a unique ID associated with the respective numerical block identifier, and

(B) for each respective dynamic temporal segmentation last update date in the respective set of dynamic temporal segmentation last update dates,

(1) defining a respective temporal block associated with the respective numerical block identifier and the respective dynamic temporal segmentation last update date, each respective temporal block being associated with a temporal block identifier comprising an indication of the corresponding respective numerical block identifier and an indication of the corresponding respective dynamic temporal segmentation last update date, and

(2) for each defined respective temporal block, generating a respective bitwise fingerprint for that respective temporal block, each respective bit of the respective bitwise fingerprint indicating whether a record in the first data set having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date matching the respective dynamic temporal segmentation last update date;

(e) comparing the timeliness of records in the first and second data sets by performing bitwise operations on the generated bitwise fingerprints by, for each respective temporal block identifier for the first data set or the second data set,

(i) determining, by performing an equality comparison between

(A) the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier, and

(B) the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier, whether any of the records associated with the temporal block corresponding to the respective temporal block identifier have different dynamic temporal segmentation last update date in the first and second data sets, and

(ii) for cases where the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier and the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal identifier do not match,

(A) generating an AND fingerprint corresponding to the respective temporal block identifier by performing an AND operation between

(I) the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier, and

(II) the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier,

(B) determining records that have a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the first data set but not the second data set by generating a first data set XOR fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier by performing an XOR operation between

(I) the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier, and

(II) the generated AND fingerprint corresponding to the respective temporal block identifier,

(III) wherein each respective bit of the generated first data set XOR fingerprint indicates that a record having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the first data set but not the second data set, and

(C) determining records that have a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the second data set but not the first data set by generating a second data set XOR fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier by performing an XOR operation between,

(I) the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier, and

(II) the generated AND fingerprint corresponding to the respective temporal block identifier,

(III) wherein each respective bit of the generated second data set XOR fingerprint indicates that a record having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the second data set but not the first data set, and

(f) comparing, for each unique ID indicated by a generated first data set XOR fingerprint for a first respective temporal block and a generated second data set XOR fingerprint for a second respective temporal block to correspond to records in the first and second data sets that have different dynamic temporal segmentation last update dates,

(i) the dynamic temporal segmentation last update date corresponding to the first respective temporal block, and

(ii) the dynamic temporal segmentation last update date corresponding to the second respective temporal block, and for one or more unique IDs for which it is determined based on the comparison that one of the data sets includes a more recently updated record having that unique ID, transferring the more recently updated record having that unique ID to the other data store.

7. The method of claim 6 , wherein the block size is sixty three sequential numerical IDs.

8. The method of claim 6 , wherein the block size is selected to be (2∧n−1) sequential numerical IDs, where n is any number greater than 1.

9. The method of claim 6 , wherein the method is utilized to transfer records from a clinical data warehouse data store to a clinical performance management data store.

10. The method of claim 6 , wherein the first data set comprises healthcare records.

11. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a decade.

12. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a year.

13. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a quarter.

14. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a month.

15. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a week.

16. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a day.

17. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, an hour.

18. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a minute.

19. The method of claim 6 , wherein dynamic temporal segmentation last update dates are configured to utilize, as one potential level of temporal segmentation, a second.

20. One or more computer readable media containing computer executable instructions for performing a method for synchronizing records stored at a first data storage location with records stored at a second, remote data storage location utilizing bitwise fingerprints to reduce network resource usage and bitwise operations to reduce processing resource usage, the method comprising

(a) maintaining, at a first data store, for each respective record in a first data set stored at the first data store,

(i) a respective dynamic temporal segmentation last update date, the respective dynamic temporal segmentation last update date for the respective record being generated based on the last two update dates for the respective record,

(ii) wherein each respective record is associated with a numerical ID unique to the first data set,

(b) for records in the first data set,

(i) determining a block size representing a number of sequential numerical IDs which will be associated with a single block,

(ii) defining a plurality of numerical block identifiers based on the determined block size and the unique IDs of the records in the first data set, wherein the unique numerical block identifier and the block size determine sequential numerical IDs associated with that unique numerical block identifier, and

(iii) for each respective numerical block identifier,

(A) determining a respective set of dynamic temporal segmentation last update dates for records in the first data set having a unique ID associated with the respective numerical block identifier, and

(B) for each respective dynamic temporal segmentation last update date in the respective set of dynamic temporal segmentation last update dates,

(1) defining a respective temporal block associated with the respective numerical block identifier and the respective dynamic temporal segmentation last update date, each respective temporal block being associated with a temporal block identifier comprising an indication of the corresponding respective numerical block identifier and an indication of the corresponding respective dynamic temporal segmentation last update date, and

(2) for each defined respective temporal block, generating a respective bitwise fingerprint for that respective temporal block, each respective bit of the respective bitwise fingerprint indicating whether a record in the first data set having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date matching the respective dynamic temporal segmentation last update date;

(c) maintaining, at a second data store, for each respective record in a second data set stored at the second data store,

(i) a respective dynamic temporal segmentation last update date, the respective dynamic temporal segmentation last update date for the respective record being generated based on the last two update dates for the respective record,

(ii) wherein each respective record is associated with a numerical ID unique to the first data set,

(d) for records in the second data set,

(i) determining a block size representing a number of sequential numerical IDs which will be associated with a single block,

(ii) defining a plurality of numerical block identifiers based on the determined block size and the unique IDs of the records in the first data set, wherein the unique numerical block identifier and the block size determine sequential numerical IDs associated with that unique numerical block identifier, and

(iii) for each respective numerical block identifier,

(A) determining a respective set of dynamic temporal segmentation last update dates for records in the second data set having a unique ID associated with the respective numerical block identifier, and

(B) for each respective dynamic temporal segmentation last update date in the respective set of dynamic temporal segmentation last update dates,

(1) defining a respective temporal block associated with the respective numerical block identifier and the respective dynamic temporal segmentation last update date, each respective temporal block being associated with a temporal block identifier comprising an indication of the corresponding respective numerical block identifier and an indication of the corresponding respective dynamic temporal segmentation last update date, and

(2) for each defined respective temporal block, generating a respective bitwise fingerprint for that respective temporal block, each respective bit of the respective bitwise fingerprint indicating whether a record in the first data set having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date matching the respective dynamic temporal segmentation last update date;

(e) comparing the timeliness of records in the first and second data sets by performing bitwise operations on the generated bitwise fingerprints by, for each respective temporal block identifier for the first data set or the second data set,

(i) determining, by performing an equality comparison between

(A) the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier, and

(B) the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier, whether any of the records associated with the temporal block corresponding to the respective temporal block identifier have different dynamic temporal segmentation last update date in the first and second data sets, and

(ii) for cases where the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier and the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal identifier do not match,

(A) generating an AND fingerprint corresponding to the respective temporal block identifier by performing an AND operation between

(I) the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier, and

(II) the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier,

(B) determining records that have a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the first data set but not the second data set by generating a first data set XOR fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier by performing an XOR operation between

(I) the generated bitwise fingerprint for the temporal block of the first data set corresponding to the respective temporal block identifier, and

(II) the generated AND fingerprint corresponding to the respective temporal block identifier,

(III) wherein each respective bit of the generated first data set XOR fingerprint indicates that a record having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the first data set but not the second data set, and

(C) determining records that have a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the second data set but not the first data set by generating a second data set XOR fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier by performing an XOR operation between

(I) the generated bitwise fingerprint for the temporal block of the second data set corresponding to the respective temporal block identifier, and

(II) the generated AND fingerprint corresponding to the respective temporal block identifier,

(III) wherein each respective bit of the generated second data set XOR fingerprint indicates that a record having the unique ID associated with that respective bit has a dynamic temporal segmentation last update date corresponding to the respective temporal block identifier in the second data set but not the first data set, and

(f) comparing, for each unique ID indicated by a generated first data set XOR fingerprint for a first respective temporal block and a generated second data set XOR fingerprint for a second respective temporal block to correspond to records in the first and second data sets that have different dynamic temporal segmentation last update dates,

(i) the dynamic temporal segmentation last update date corresponding to the first respective temporal block, and

(ii) the dynamic temporal segmentation last update date corresponding to the second respective temporal block, and for one or more unique IDs for which it is determined based on the comparison that one of the data sets includes a more recently updated record having that unique ID, transferring the more recently updated record having that unique ID to the other data store.