Systems and methods for providing error recovery in data transmissions
Receive a user request to perform a service request associated with user data. Select, based on the user request, a first service provider interface (SPI) integrated into a first service, the first SPI being configured to obtain and buffer the user data from a datastore associated with the first service into a first buffer of the first SPI. Obtain, by the first SPI, a first page of records of the user data from the datastore. Buffer, by the first SPI, the first page of records in the first buffer of the first SPI as the records of the first page of records are being obtained from the datastore. Provide, by the first SPI, the first page of records to a central controller engine. Store, by the central controller engine, the first page of records. Provide the first page of records to a user.
1. A computing system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the computing system to perform:
receiving, by a central controller engine, a request associated with user data, the user data including first data, the request including a request to stream the first data to a user device, the first data including a set of records;
selecting, by the central controller engine based on the request, a first service provider interface (SPI) integrated into a first service, the first SPI being configured to operate on the first data in a first datastore associated with the first service, the first SPI including a common pagination component configured to obtain the first data as a data stream from the first datastore, buffer a first page of records of the first data as received into a first buffer of the first SPI, the first page of records of the first data being a first portion of the first set of records, and buffer a second page of records of the first data as received into a second buffer of the first SPI, the second page of records of the first data being a second portion of the first set of records;
obtaining, by the common pagination component of the first SPI, the first page of records of the first data as a data stream from the first datastore;
buffering, by the common pagination component of the first SPI, the first page of records of the first data in the first buffer as the records of the first page of records of the first data are being streamed from the first datastore;
buffering, by the common pagination component of the first SPI, the second page of records of the first data in the second buffer as the records of the second page of records of the first data are being streamed from the first datastore;
providing, by the common pagination component of the first SPI, the first page of records of the first data to the central controller engine;
receiving, by the central controller engine, the first page of records of the first data;
informing, by the central controller engine, the first SPI when the first page of records of the first data has been fully received to inform the first SPI that it can begin providing the second page of records of the first data to the central controller engine;
storing, by the central controller engine, the first page of records of the first data; and
providing, by the central controller engine, the first page of records of the first data as a data stream to the user device.
2. The computing system of claim 1 , wherein the first page of records of the first data is provided to the central controller engine by the common pagination component of the first SPI in response to a first fetch request from the central controller engine.
3. The computing system of claim 1 , wherein the instructions further cause the system to perform:
providing, by the common pagination component of the first SPI, the second page of records of the first data to the central controller engine in response to the central controller engine informing the first SPI that the first page of records of the first data has been fully received;
storing, by the central controller engine, the second page of records of the first data; and
providing, by the central controller engine, the second page of records of the first data as a data stream to the user.
4. The computing system of claim 3 , wherein the second page of records of the first data is provided to the central controller engine by the common pagination component of the first SPI in response to a second fetch request from the central controller engine.
5. The computing system of claim 3 , wherein the providing the second page of records of the first data is based on an identifier of the second buffer of the first SPI, the identifier being included in the first page of records of the first data.
6. The computing system of claim 1 , wherein the obtaining the first page of records of the first data from the first datastore comprises:
pushing, by the first service, the first page of records of the first data from the first datastore to the first buffer of the first SPI.
7. The system of claim 1 , wherein the providing the first page of records of the first data to the central controller engine comprises:
pulling, by the central controller engine, the first page of records of the first data from the first buffer of the first SPI to a datastore of the central controller engine.
8. The computing system of claim 3 , wherein the instructions further cause the system to perform:
detecting an error associated with the second page of records of the first data; and
reinitiating the providing, by the common pagination component of the first SPI, the second page of records of the first data to the central controller engine.
9. The computing system of claim 1 , wherein the user data includes second data, and wherein the instructions further cause the system to perform:
selecting, by the central controller engine based on the request, a second SPI integrated into a second service, the second SPI being configured to operate on the second data in a second datastore associated with the second service, the second SPI including the common pagination component configured to obtain the second data as a data stream from the second datastore, buffer a first page of records of the second data as received into a first buffer of the second SPI, the first page of records of the second data being a third portion of the first set of records, and buffer a second page of records of the second data as received into a second buffer of the second SPI, the second page of records of the second data being a fourth portion of the first set of records;
obtaining, by the common pagination component of the SPI, a first page of records of the second data as a data stream from the second datastore;
buffering, by the common pagination component of the second SPI, the first page of records of the second data in the first buffer of the second SPI as the records of the first page of records of the second data are being streamed from the second datastore;
buffering, by the common pagination component of the second SPI, the second page of records of the second data in the second buffer as the records of the second page of records of the second data are being streamed from the second datastore;
providing, by the common pagination component of the second SPI, the first page of records of the second data to the central controller engine;
storing, by the central controller engine, the first page of records of the second data; and
providing, by the central controller engine, the first page of records of the second data as a data stream to the user.
10. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
receiving, by a central controller engine, a request associated with user data, the user data including first data, the request including a request to stream the first data to a user device, the first data including a set of records;
selecting, by the central controller engine based on the request, a first service provider interface (SPI) integrated into a first service, the first SPI being configured to operate on the first data in a first datastore associated with the first service, the first SPI including a common pagination component configured to obtain the first data as a data stream from the first datastore, buffer a first page of records of the first data as received into a first buffer of the first SPI, the first page of records of the first data being a first portion of the first set of records, and buffer a second page of records of the first data as received into a second buffer of the first SPI, the second page of records of the first data being a second portion of the first set of records;
obtaining, by the common pagination component of the first SPI, the first page of records of the first data as a data stream from the first datastore;
buffering, by the common pagination component of the first SPI, the first page of records of the first data in the first buffer as the records of the first page of records of the first data are being streamed from the first datastore;
buffering, by the common pagination component of the first SPI, the second page of records of the first data in the second buffer as the records of the second page of records of the first data are being streamed from the first datastore;
providing, by the common pagination component of the first SPI, the first page of records of the first data to the central controller engine;
receiving, by the central controller engine, the first page of records of the first data;
informing, by the central controller engine, the first SPI when the first page of records of the first data has been fully received to inform the first SPI that it can begin providing the second page of records of the first data to the central controller engine;
storing, by the central controller engine, the first page of records of the first data; and
providing, by the central controller engine, the first page of records of the first data as a data stream to the user device.
11. The method of claim 10 , wherein the first page of records of the first data is provided to the central controller engine by the common pagination component of the first SPI in response to a first fetch request from the central controller engine.
12. The method of claim 10 , further comprising:
providing, by the common pagination component of the first SPI, the second page of records of the first data to the central controller engine in response to the central controller engine informing the first SPI that the first page of records of the first data has been fully received;
storing, by the central controller engine, the second page of records of the first data; and
providing, by the central controller engine, the second page of records of the first data as a data stream to the user.
13. The method of claim 12 , wherein the second page of records of the first data is provided to the central controller engine by the common pagination component of the first SPI in response to a second fetch request from the central controller engine.
14. The method of claim 12 , wherein the providing the second page of records of the first data is based on an identifier of the second buffer of the first SPI, the identifier being included in the first page of records of the first data.
15. The method of claim 10 , wherein the obtaining the first page of records of the first data from the first datastore comprises:
pushing, by the first service, the first page of records of the first data from the first datastore to the first buffer of the first SPI.
16. The method of claim 10 , wherein the providing the first page of records of the first data to the central controller engine comprises:
pulling, by the central controller engine, the first page of records of the first data from the first buffer of the first SPI to a datastore of the central controller engine.
17. The method of claim 10 , further comprising:
selecting, by the central controller engine based on the request, a second SPI integrated into a second service, the second SPI being configured to operate on the second data in a second datastore associated with the second service, the second SPI including the common pagination component configured to obtain the second data as a data stream from the second datastore, buffer a first page of records of the second data as received into a first buffer of the second SPI, the first page of records of the second data being a third portion of the first set of records, and buffer a second page of records of the second data as received into a second buffer of the second SPI, the second page of records of the second data being a fourth portion of the first set of records;
obtaining, by the common pagination component of the SPI, a first page of records of the second data as a data stream from the second datastore;
buffering, by the common pagination component of the second SPI, the first page of records of the second data in the first buffer of the second SPI as the records of the first page of records of the second data are being streamed from the second datastore;
buffering, by the common pagination component of the second SPI, the second page of records of the second data in the second buffer as the records of the second page of records of the second data are being streamed from the second datastore;
providing, by the common pagination component of the second SPI, the first page of records of the second data to the central controller engine;
storing, by the central controller engine, the first page of records of the second data; and
providing, by the central controller engine, the first page of records of the second data as a data stream to the user.
18. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
receiving, by a central controller engine, a request associated with user data, the user data including first data, the request including a request to stream the first data to a user device, the first data including a set of records;
selecting, by the central controller engine based on the request, a first service provider interface (SPI) integrated into a first service, the first SPI being configured to operate on the first data in a first datastore associated with the first service, the first SPI including a common pagination component configured to obtain the first data as a data stream from the first datastore, buffer a first page of records of the first data as received into a first buffer of the first SPI, the first page of records of the first data being a first portion of the first set of records, and buffer a second page of records of the first data as received into a second buffer of the first SPI, the second page of records of the first data being a second portion of the first set of records;
obtaining, by the common pagination component of the first SPI, the first page of records of the first data as a data stream from the first datastore;
buffering, by the common pagination component of the first SPI, the first page of records of the first data in the first buffer as the records of the first page of records of the first data are being streamed from the first datastore;
buffering, by the common pagination component of the first SPI, the second page of records of the first data in the second buffer as the records of the second page of records of the first data are being streamed from the first datastore;
providing, by the common pagination component of the first SPI, the first page of records of the first data to the central controller engine;
receiving, by the central controller engine, the first page of records of the first data;
informing, by the central controller engine, the first SPI when the first page of records of the first data has been fully received to inform the first SPI that it can begin providing the second page of records of the first data to the central controller engine;
storing, by the central controller engine, the first page of records of the first data; and
providing, by the central controller engine, the first page of records of the first data as a data stream to the user device.
19. The non-transitory computer readable medium of claim 18 comprising further instructions that, when executed, cause one or more processors to perform:
selecting, by the central controller engine based on the request, a second SPI integrated into a second service, the second SPI being configured to operate on the second data in a second datastore associated with the second service, the second SPI including the common pagination component configured to obtain the second data as a data stream from the second datastore, buffer a first page of records of the second data as received into a first buffer of the second SPI, the first page of records of the second data being a third portion of the first set of records, and buffer a second page of records of the second data as received into a second buffer of the second SPI, the second page of records of the second data being a fourth portion of the first set of records;
obtaining, by the common pagination component of the SPI, a first page of records of the second data as a data stream from the second datastore;
buffering, by the common pagination component of the second SPI, the first page of records of the second data in the first buffer of the second SPI as the records of the first page of records of the second data are being streamed from the second datastore;
buffering, by the common pagination component of the second SPI, the second page of records of the second data in the second buffer as the records of the second page of records of the second data are being streamed from the second datastore;
providing, by the common pagination component of the second SPI, the first page of records of the second data to the central controller engine;
storing, by the central controller engine, the first page of records of the second data; and
providing, by the central controller engine, the first page of records of the second data as a data stream to the user.