Partial database update based on lightweight join
A machine is configured to access an authoritative database of data fields, where each data field has a corresponding authoritative value and a corresponding update time. The machine accesses a propagation time that was last updated in response to a most recent propagation of updated authoritative values from the authoritative database to a recipient database. The machine performs a database join that joins only any unpropagated authoritative values having update times that exceed the propagation time to corresponding reference checksums of corresponding data fields. The machine generates candidate checksums of only the corresponding data fields whose unpropagated authoritative values have update times that exceed the propagation time. The machine propagates, from the authoritative database to the recipient database, the unpropagated authoritative values, and the machine replaces the reference checksums with the generated candidate checksums.
1 . A method comprising:
accessing, by one or more processors, data fields that each include a corresponding value and a corresponding update time;
accessing, by the one or more processors, a propagation time that indicates a most recent provision of one or more values of the data fields to a recipient;
performing, by the one or more processors, a database join that joins only values whose update times exceed the propagation time to corresponding reference checksums of their corresponding data fields;
generating, by the one or more processors, candidate checksums of only the data fields whose values have update times that exceed the propagation time;
providing, by the one or more processors and to the recipient, the values whose update times exceed the propagation time; and
replacing, by the one or more processors, the reference checksums with the candidate checksums for the data fields whose values have update times that exceed the propagation time.
2 . The method of claim 1 , further comprising:
updating the propagation time in response to the replacing of the reference checksums with the candidate checksums.
3 . The method of claim 1 , wherein:
the database join that joins only the values whose update times exceed the propagation time to their corresponding reference checksums omits joining any values whose update times fail to exceed the propagation time.
4 . The method of claim 1 , wherein:
the database join that joins only the values whose update times exceed the propagation time to their corresponding reference checksums is less computationally intensive than joining all values of the data fields to their corresponding reference checksums of their corresponding data fields.
5 . The method of claim 1 , further comprising:
detecting that a first value of a first data field is updated after the propagation time; and
wherein:
the providing of the values whose update times exceed the propagation time includes propagating the updated first value of the first data field to the recipient in real-time.
6 . The method of claim 1 , further comprising:
detecting that a first value of a first data field is updated after the propagation time; and
wherein:
the replacing of the reference checksums with the candidate checksums includes replacing a first reference checksum of the first data field with a first candidate checksum of the first data field in real-time.
7 . The method of claim 1 , wherein:
the recipient includes a device configured based on a reverse extract-transform-load (reverse-ETL) streaming protocol; and
the providing of the values whose update times exceed the propagation time causes the device to update a database with the provided values based on the reverse-ETL streaming protocol.
8 . A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
accessing data fields that each include a corresponding value and a corresponding update time;
accessing a propagation time that indicates a most recent provision of one or more values of the data fields to a recipient;
performing a database join that joins only values whose update times exceed the propagation time to corresponding reference checksums of their corresponding data fields;
generating candidate checksums of only the data fields whose values have update times that exceed the propagation time;
providing, to the recipient, the values whose update times exceed the propagation time; and
replacing the reference checksums with the candidate checksums for the data fields whose values have update times that exceed the propagation time.
9 . The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise:
updating the propagation time in response to the replacing of the reference checksums with the candidate checksums.
10 . The non-transitory machine-readable medium of claim 8 , wherein:
the database join that joins only the values whose update times exceed the propagation time to their corresponding reference checksums omits joining any values whose update times fail to exceed the propagation time.
11 . The non-transitory machine-readable medium of claim 8 , wherein:
the database join that joins only the values whose update times exceed the propagation time to their corresponding reference checksums is less computationally intensive than joining all values of the data fields to their corresponding reference checksums of their corresponding data fields.
12 . The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise:
detecting that a first value of a first data field is updated after the propagation time; and
wherein:
the providing of the values whose update times exceed the propagation time includes propagating the updated first value of the first data field to the recipient in real-time.
13 . The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise:
detecting that a first value of a first data field is updated after the propagation time; and
wherein:
the replacing of the reference checksums with the candidate checksums includes replacing a first reference checksum of the first data field with a first candidate checksum of the first data field in real-time.
14 . The non-transitory machine-readable medium of claim 8 , wherein:
the recipient includes a device configured based on a reverse extract-transform-load (reverse-ETL) streaming protocol; and
the providing of the values whose update times exceed the propagation time causes the device to update a database with the provided values based on the reverse-ETL streaming protocol.
15 . A system comprising:
one or more processors; and
a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the system to perform operations comprising:
accessing data fields that each include a corresponding value and a corresponding update time;
accessing a propagation time that indicates a most recent provision of one or more values of the data fields to a recipient;
performing a database join that joins only values whose update times exceed the propagation time to corresponding reference checksums of their corresponding data fields;
generating candidate checksums of only the data fields whose values have update times that exceed the propagation time;
providing, to the recipient, the values whose update times exceed the propagation time; and
replacing the reference checksums with the candidate checksums for the data fields whose values have update times that exceed the propagation time.
16 . The system of claim 15 , wherein the operations further comprise:
updating the propagation time in response to the replacing of the reference checksums with the candidate checksums.
17 . The system of claim 15 , wherein:
the database join that joins only the values whose update times exceed the propagation time to their corresponding reference checksums omits joining any values whose update times fail to exceed the propagation time.
18 . The system of claim 15 , wherein:
the database join that joins only the values whose update times exceed the propagation time to their corresponding reference checksums is less computationally intensive than joining all values of the data fields to their corresponding reference checksums of their corresponding data fields.
19 . The system of claim 15 , wherein the operations further comprise:
detecting that a first value of a first data field is updated after the propagation time; and
wherein:
the providing of the values whose update times exceed the propagation time includes propagating the updated first value of the first data field to the recipient in real-time.
20 . The system of claim 15 , wherein the operations further comprise:
detecting that a first value of a first data field is updated after the propagation time; and
wherein:
the replacing of the reference checksums with the candidate checksums includes replacing a first reference checksum of the first data field with a first candidate checksum of the first data field in real-time.