Source-adapted data retrieval for multi-tenant system
A method includes: maintaining, at an integration computing device, a scheduling parameter corresponding to a source computing device maintaining a plurality of source data records with record identifiers; receiving, from the source computing device, a plurality of modification indicators, each modification indicator generated at the source computing device in response to modification of a source data record, and containing the record identifier of the source data record; storing the modification indicators in a repository; according to the scheduling parameter, periodically obtaining a subset of the source data records from the source computing device, by: retrieving the modification indicators from the repository; generating a bulk record request containing the plurality of record identifiers from the retrieved modification indicators; transmitting the bulk record request to the source computing device; and responsive to transmitting the bulk record request, receiving the subset of source data records from the source computing device.
1. A method, comprising:
maintaining, at an integration computing device, a scheduling parameter corresponding to a source computing device maintaining a plurality of source data records having respective record identifiers, wherein the scheduling parameter includes at least one of (i) a request frequency, or (ii) a threshold number of records;
receiving, at the integration computing device from the source computing device, a plurality of modification indicators generated at the source computing device in response to modification of corresponding source data records, the modification indicators containing the record identifiers of the corresponding source data records;
storing the modification indicators in a repository at the integration computing device;
according to the scheduling parameter, periodically obtaining, at the integration computing device, a subset of the source data records from the source computing device by:
retrieving the modification indicators from the repository;
generating a bulk record request containing the plurality of record identifiers from the retrieved modification indicators;
transmitting the bulk record request to the source computing device; and
responsive to transmitting the bulk record request, receiving the subset of source data records from the source computing device.
2. The method of claim 1 , further comprising
after receiving the subset of source data records, storing the subset of source data records in a central source data repository.
3. The method of claim 1 , further comprising:
after receiving the subset of source data records, publishing the subset of source data records to a multi-tenant subsystem for further processing.
4. The method of claim 1 , further comprising:
after retrieving the modification indicators, removing the modification indicators from the repository.
5. The method of claim 1 , further comprising:
maintaining distinct scheduling parameters for each of a plurality of source computing devices each maintaining respective sets of source data records; and
receiving modification indicators from each source computing device and storing each modification indicator in the repository in association with an identifier of the corresponding source computing device.
6. The method of claim 5 , wherein the modification indicators include respective source device identifiers.
7. The method of claim 5 , further comprising:
repeating the obtaining, retrieving, generating, and transmitting for each of the source computing devices according to the respective scheduling parameters.
8. The method of claim 1 , wherein the scheduling parameter includes a request frequency; and wherein the method further comprises, at the integration computing device:
storing, in response to sending the bulk record request, a timestamp corresponding to the request.
9. The method of claim 1 , wherein generating the bulk record request includes detecting and removing duplicate record identifiers from the retrieved modification indicators.
10. A computing device, comprising:
a memory storing a scheduling parameter corresponding to a source computing device maintaining a plurality of source data records having respective record identifiers, wherein the scheduling parameter includes at least one of (i) a request frequency, or (ii) a threshold number of records;
a communications interface; and
a processor coupled with the memory and the communications interface, the processor configured to:
receive, via the communications interface from the source computing device, a plurality of modification indicators generated at the source computing device in response to modification of corresponding source data records, the modification indicators containing the record identifiers of the corresponding source data records;
store the modification indicators in a repository in the memory;
according to the scheduling parameter, periodically obtain a subset of the source data records from the source computing device by:
retrieving the modification indicators from the repository;
generating a bulk record request containing the plurality of record identifiers from the retrieved modification indicators;
transmitter the bulk record request to the source computing device; and
responsive to transmitting the bulk record request, receiving the subset of source data records from the source computing device.
11. The computing device of claim 10 , wherein the processor is further configured to:
after receiving the subset of source data records, store the subset of source data records in a central source data repository.
12. The computing device of claim 10 , wherein the processor is further configured to:
after receiving the subset of source data records, publish the subset of source data records to a multi-tenant subsystem for further processing.
13. The computing device of claim 10 , wherein the processor is further configured to:
after retrieving the modification indicators, remove the modification indicators from the repository.
14. The computing device of claim 10 , wherein the processor is further configured to:
maintain distinct scheduling parameters for each of a plurality of source computing devices each maintaining respective sets of source data records; and
receive modification indicators from each source computing device and store each modification indicator in the repository in association with an identifier of the corresponding source computing device.
15. The computing device of claim 14 , wherein the modification indicators include respective source device identifiers.
16. The computing device of claim 14 , wherein the processor is further configured to:
repeat the obtaining, retrieving, generating, and transmitting for each of the source computing devices according to the respective scheduling parameters.
17. The computing device of claim 10 , wherein the scheduling parameter includes a request frequency; and wherein the processor is further configured to:
store, in response to sending the bulk record request, a timestamp corresponding to the request.
18. The computing device of claim 10 , wherein the processor is further configured, to generate the bulk record request, to detect and remove duplicate record identifiers from the retrieved modification indicators.
19. A non-transitory computer-readable medium storing instructions executable by a processor of an integration computing device to:
maintain a scheduling parameter corresponding to a source computing device maintaining a plurality of source data records having respective record identifiers, wherein the scheduling parameter includes at least one of (i) a request frequency, or (ii) a threshold number of records;
receive, from the source computing device, a plurality of modification indicators generated at the source computing device in response to modification of corresponding source data records, the modification indicators containing the record identifiers of the corresponding source data records;
store the modification indicators in a repository at the integration computing device;
according to the scheduling parameter, periodically obtain, at the integration computing device, a subset of the source data records from the source computing device by:
retrieving the modification indicators from the repository;
generating a bulk record request containing the plurality of record identifiers from the retrieved modification indicators;
transmitting the bulk record request to the source computing device; and
responsive to transmitting the bulk record request, receiving the subset of source data records from the source computing device.