Data forwarder prioritizing live data
In accordance with implementations of the present disclosure, a backup of live data received by a data forwarder is generated at the data forwarder while the live data is provided to a real-time data pipeline for forwarding from the data forwarder. A a first portion of the live data is recovered from the backup to a stale data pipeline of the data forwarder. A request to forward the live data to a destination node is received by the data forwarder. In response to the request data is forwarded to the destination node, where the first portion of the live data from the stale data pipeline is added to a second portion of the live data from the real-time data pipeline in the response based on determining headroom remains to reach an amount of the data identified to include in the response.
1. A computer-implemented method comprising:
generating, at a data forwarder, a backup of live data received by the data forwarder while the live data is provided to a real-time data pipeline for forwarding from the data forwarder;
recovering a first portion of the live data from the backup to a stale data pipeline of the data forwarder;
receiving, by the data forwarder, a request to forward the live data to a destination node; and
forwarding data to the destination node in response to the request, wherein the first portion of the live data from the stale data pipeline is added to a second portion of the live data from the real-time data pipeline in the response based on determining headroom remains to reach an amount of the data identified to include in the response.
2. The computer-implemented method of claim 1 , wherein the request is a pull request from the destination node.
3. The computer-implemented method of claim 1 , wherein the recovered first portion is included in the response based on a data buffer of the real-time data pipeline including less than the amount of the data identified to include in the response.
4. The computer-implemented method of claim 1 , wherein the amount of data is requested by the destination node.
5. The computer-implemented method of claim 1 , wherein the amount of data is a number of transactions specified in the request.
6. The computer-implemented method of claim 1 , wherein the first portion of the live data from the stale data pipeline is selected to match the amount of the data identified to include in the response, the amount of data specified in the request.
7. The computer-implemented method of claim 1 , wherein the first portion of the live data from the stale data pipeline is included in the response based on the data forwarder responding to pull requests faster than new live data is received by an input interface of the data forwarder.
8. The computer-implemented method of claim 1 , wherein the backup of the received live data is on non-volatile storage, and the real-time data pipeline uses only volatile storage.
9. The computer-implemented method of claim 1 , wherein the first portion of the live data is included in the response based on being dropped from the real-time data pipeline.
10. The computer-implemented method of claim 1 , wherein the first portion of the live data is dropped from the real-time data pipeline based on the real-time data pipeline being at a capacity.
11. The computer-implemented method of claim 1 , further comprising logging one or more acknowledgments of the first portion of the live data, the one or more acknowledgements being received by the data forwarder based on the forwarding.
12. The computer-implemented method of claim 1 , wherein the first portion of the live data is included in the response based on a comparison between the backup of the live data and one or more acknowledgments of at least some of the live data, the one or more acknowledgements received at the data forwarder.
13. The computer-implemented method of claim 1 , after the forwarding of the first portion of the live data and based on determining the first portion of the live data is unacknowledged, recovering the first portion of the live data from the backup to the stale data pipeline for additional forwarding.
14. The computer-implemented method of claim 1 , further comprising removing the first portion of the live data from the backup based on receiving an acknowledgement in response to the forwarding.
15. The computer-implemented method of claim 1 , wherein the recovering of the first portion of the live data from the backup comprises:
closing a transaction log in the backup of the live data;
opening a new transaction log for the backup for subsequently received live data;
retrieving the first portion of the live data from the closed transaction log; and
generating the backup of the subsequently received live data in the new transaction log.
16. The computer-implemented method of claim 1 , wherein the real-time data pipeline comprises a live data queue and the stale data pipeline comprises a stale data queue.
17. The computer-implemented method of claim 1 , wherein the backup comprises transactions and transaction identifiers corresponding to the live data.
18. The computer-implemented method of claim 1 , further comprising:
receiving an acknowledgement in response to the forwarding; and
recording the received acknowledgement in an acknowledgement log.
19. The computer-implemented method of claim 1 , wherein the recovering of the first portion of the live data from the backup comprises:
closing an acknowledgement log comprising recorded acknowledgments of forwarded data;
opening a new acknowledgement log for recording subsequent acknowledgments of forwarded data;
identifying the first portion in the backup based on a comparison between the closed acknowledgment log and the backup; and
recording the subsequent acknowledgments in the new acknowledgment log.
20. The computer-implemented method of claim 1 , wherein the live data is received by the data forwarder from a source node in a plurality of network communications, and the response is forwarded to the destination node over one or more network communications.
21. A system for forwarding data, the system comprising:
one or more data processors; and
one or more computer-readable storage media containing instructions which when executed on the one or more data processors, cause the one or more processors to perform operations including:
generating, at a data forwarder, a backup of live data received by the data forwarder while the live data is provided to a real-time data pipeline for forwarding from the data forwarder;
recovering a first portion of the live data from the backup to a stale data pipeline of the data forwarder;
receiving, by the data forwarder, a request to forward the live data to a destination node; and
forwarding data to the destination node in response to the request, wherein the first portion of the live data from the stale data pipeline is added to a second portion of the live data from the real-time data pipeline in the response based on determining headroom remains to reach an amount of the data identified to include in the response.
22. The system of claim 21 , wherein the request is a pull request from the destination node.
23. The system of claim 21 , wherein the recovered first portion is included in the response based on a data buffer of the real-time data pipeline including less than the amount of the data identified to include in the response.
24. The system of claim 21 , wherein the amount of data is requested by the destination node.
25. The system of claim 21 , wherein the amount of data is a number of transactions specified in the request.
26. The system of claim 21 , wherein the first portion of the live data from the stale data pipeline is selected to match the amount of the data identified to include in the response, the amount of data specified in the request.
27. One or more non-transitory computer-storage media storing computer-useable instructions that, when executed by a computing device, perform a method for forwarding data, the method comprising:
generating, at a data forwarder, a backup of live data received by the data forwarder while the live data is provided to a real-time data pipeline for forwarding from the data forwarder;
recovering a first portion of the live data from the backup to a stale data pipeline of the data forwarder;
receiving, by the data forwarder, a request to forward the live data to a destination node; and
forwarding data to the destination node in response to the request, wherein the first portion of the live data from the stale data pipeline is added to a second portion of the live data from the real-time data pipeline in the response based on determining headroom remains to reach an amount of the data identified to include in the response.
28. The computer-storage media of claim 27 , wherein the request is a pull request from the destination node.
29. The computer-storage media of claim 27 , wherein the recovered first portion is included in the response based on a data buffer of the real-time data pipeline including less than the amount of the data identified to include in the response.
30. The computer-storage media of claim 27 , wherein the amount of data is requested by the destination node.