Aggregating high volumes of temporal data from multiple overlapping sources
There is provided a technique that includes (1) receiving a data element, (2) comparing the data element to a corresponding data element in a pend update history repository that contains changes that have occurred to the data element over time, thus yielding a comparison, (3) determining, based on the comparison, that the data element has changed, (4) checking that there is no update process running that might contain more up-to-date data for the data element, (5) relating the data element to a cutoff number that is associated with a particular range of time, and (6) updating corresponding data elements in (a) a full universe data repository that stores a current value for the data element and a previous value for the data element, and (b) a delta data repository that stores a record that indicates (i) that the data element is being updated, and (ii) the cutoff number.
1. A method of aggregating data from multiple sources in a computer network, said method comprising:
receiving a data element from a source among said multiple sources;
comparing said data element to a corresponding current data element in a pend update history repository that contains changes that have occurred to said data element over time, thus yielding a comparison;
determining, based on said comparison, that said data element has changed, thus yielding first updated data for said data element from said source, wherein said first updated data has a first timestamp;
executing a first update process that includes:
(A) monitoring whether a second update process is running having second updated data for said data element, wherein said second updated data has a second timestamp that is earlier than said first timestamp;
(B) holding said first updated data until said second update process has finished; and
(C) releasing said first updated data and performing operations of:
(1) obtaining a cutoff number that identifies a particular range of time; and
(2) updating fields for said data element, based on said first updated data, in (a) a full universe data repository that stores (i) said first updated data for said data element, and (ii) a previous value for said data element, and (b) a delta data repository that stores (i) said first updated data for said data element, and (ii) said cutoff number;
performing said receiving, said comparing, said determining and said executing for other data elements from other sources among said multiple sources; and
accessing said delta data repository, and utilizing said cutoff number to determine that an update for said data element was made within said particular range of time.
2. The method of claim 1 ,
wherein said method also includes:
determining, in accordance with rules that consider said source, whether to update said data element.
3. The method of claim 2 ,
wherein said source is a first source,
wherein said pend update history repository holds a current value for said data element from a second source, and
wherein said rules also consider said second source.
4. The method of claim 3 ,
wherein other values for said data element are available from a plurality of sources, and
wherein said rules consider said plurality of sources.
5. A system for aggregating data from multiple sources in a computer network, said system comprising:
a processor; and
a memory that contains instructions that are readable by said processor to cause said processor to:
receive a data element from a source among said multiple sources;
compare said data element to a corresponding current data element in a pend update history repository that contains changes that have occurred to said data element over time, thus yielding a comparison;
determine, based on said comparison, that said data element has changed, thus yielding first updated data for said data element from said source, wherein said first updated data has a first timestamp;
execute a first update process that includes operations to:
(A) monitor whether a second update process is running having second updated data for said data element, wherein said second updated data has a second timestamp that is earlier than said first timestamp;
(B) hold said first updated data until said second update process has finished; and
(C) release said first updated data and perform operations to:
(1) obtain a cutoff number that identifies a particular range of time; and
(2) update fields for said data element, based on said first updated data, in (a) a full universe data repository that stores (i) said first updated data for said data element, and (ii) a previous value for said data element, and (b) a delta data repository that stores (i) said first updated data for said data element, and (ii) said cutoff number;
perform said receive, said compare, said determine and said execute for other data elements from other sources among said multiple sources; and
access said delta data repository, and utilize said cutoff number to determine that an update for said data element was made within said particular range of time.
6. The system of claim 5 ,
wherein said instructions also cause said processor to:
determine, in accordance with rules that consider said source, whether to update said data element.
7. The system of claim 6 ,
wherein said source is a first source,
wherein said pend update history repository holds a current value for said data element from a second source, and
wherein said rules also consider said second source.
8. The system of claim 7 ,
wherein other values for said data element are available from a plurality of sources, and
wherein said rules consider said plurality of sources.
9. A storage device that is non-transitory, comprising instructions that are readable by a processor to perform a method of aggregating data from multiple sources in a computer network, by causing said processor to:
receive a data element from a source among said multiple sources;
compare said data element to a corresponding current data element in a pend update history repository that contains changes that have occurred to said data element over time, thus yielding a comparison;
determine, based on said comparison, that said data element has changed, thus yielding first updated data for said data element from said source, wherein said first updated data has a first timestamp;
execute a first update process that includes operations to:
(A) monitor whether a second update process is running having second updated data for said data element, wherein said second updated data has a second timestamp that is earlier than said first timestamp;
(B) hold said first updated data until said second update process has finished; and
(C) release said first updated data and perform operations to:
(1) obtain a cutoff number that identifies a particular range of time; and
(2) update fields for said data element, based on said first updated data, in (a) a full universe data repository that stores (i) said first updated data for said data element, and (ii) a previous value for said data element, and (b) a delta data repository that stores (i) said updated data for said data element, and (ii) said cutoff number; and
access said delta data repository, and utilize said cutoff number to determine that an update for said data element was made within said particular range of time.
10. The storage device of claim 9 ,
wherein said instructions also cause said processor to:
determine, in accordance with rules that consider said source, whether to update said data element.
11. The storage device of claim 10 ,
wherein said source is a first source,
wherein said pend update history repository holds a current value for said data element from a second source, and
wherein said rules also consider said second source.
12. The storage device of claim 11 ,
wherein other values for said data element are available from a plurality of sources, and wherein said rules consider said plurality of sources.