Mechanism for backfilling records dropped during transfer from distributed node system
A method for backfilling records dropped by a distributed node system during a transfer to an external database includes initially determining whether an attempt to transfer a data record failed or succeeded. In response to determining that the transfer succeeded, the data record is appended at its original location include a first transfer status identifier. In response to determining that the transfer failed, the data record is appended to include a second transfer status identifier and queued copy of the data record is created by copying the data record to a delayed re-try queue. The method further includes verifying that the transfer status identifier of the queued copy is identical to the transfer status identifier of a newest version of the data record within the distributed node system and, responsive to the verification, attempting a subsequent transfer of the queued copy of the data record to the external database.
1. A distributed node system comprising:
a node configured to:
attempt a transfer of a data record residing on the node to an external database;
in response to determining that the transfer succeeded, append the data record on the node to include a first transfer status identifier indicative of successful transfer;
in response to determining that the transfer failed, append the data record on the node to include a second transfer status identifier indicative of failed transfer and create a queued copy of the data record by copying the data record to a delayed re-try queue; and
a backfill processor configured to attempt a subsequent transfer of the queued copy of the data record to the external database in response to confirming that the transfer status identifier of the queued copy is identical to the transfer status identifier of a newest version of the data record within the distributed node system.
2. The distributed node system of claim 1 , wherein the backfill removes the queued copy of the data record from the delayed re-try queue without transferring the queued copy to the external database in response to confirming that the transfer status identifier of the queued copy is not identical to the transfer status identifier of the newest version of the data record.
3. The distributed node system of claim 1 , wherein the distributed node system lacks read access to the external database.
4. The distributed node system of claim 1 , wherein the node determines whether the transfer attempt succeeded or failed based on feedback received from the external database and without reading data out of the external database.
5. The distributed node system of claim 4 , wherein the node attempts to re-transfer the data record one or more additional times in response to the feedback and prior to adding the data record to the delayed re-try queue.
6. The distributed node system of claim 1 , wherein the distributed node system includes a plurality of nodes tasked with storing a plurality of data records, wherein different versions of each record of the plurality of records are stored on different nodes of the plurality.
7. The distributed node system of claim 1 , wherein the distributed node system implements logic to track and identify a newest copy of each of the data records at each point in time.
8. The distributed node system of claim 1 , wherein the node attempts to transfer the data record in response to receiving a notification that the data record has been updated.
9. The distributed node system of claim 1 , wherein different versions of a same record have identical filenames within the distributed node system.
10. A method of backfilling records dropped by a distributed node system during a transfer to an external database, the method comprising:
attempting to transfer a data record from a given node of the distributed node system to an external database;
determining whether the transfer succeeded or failed;
in response to determining that the transfer succeeded, appending the data record on the given node to include a first transfer status identifier;
in response to determining that the transfer failed appending the data record on the given node to include a second transfer status identifier and creating a queued copy of the data record by copying the data record to a delayed re-try queue; and
in response to confirming that the transfer status identifier of the queued copy is identical to the transfer status identifier of a newest version of the data record within the distributed node system, attempting a subsequent transfer of the queued copy of the data record to the external database.
11. The method of claim 10 , further comprising:
in response to confirming that the transfer status identifier of the queued copy is not identical to the transfer status identifier of the newest version of the data record, removing the queued copy of the data record from the delayed re-try queue without transferring the queued copy to the external database.
12. The method of claim 10 , wherein the distributed node system lacks read access to the external database.
13. The method of claim 10 , wherein determining whether the transfer failed or succeeded is based on feedback from the external database and without reading data out of the external database.
14. The method of claim 13 , further comprising: attempting transfer of the data record to the external database one or more additional times in response to receiving the feedback and prior to creating the queued copy of the data record.
15. The method of claim 10 , wherein the distributed node system includes a plurality of nodes tasked with storing a plurality of data records, wherein different versions of each record of the plurality of records are stored on different nodes of the plurality.
16. The method of claim 10 , wherein the distributed node system implements logic to track and identify a newest copy of each of the data records at each point in time.
17. The method of claim 10 , wherein different versions of a same record have identical filenames.
18. One or more tangible computer-readable storage media encoding computer-executable instructions for executing a compute process comprising:
attempting to transfer a data record from a given node of the distributed node system to an external database;
determining whether the transfer succeeded or failed;
in response to determining that the transfer succeeded, appending the data record on the given node to include a first transfer status identifier;
in response to determining that the transfer failed appending the data record on the given node to include a second transfer status identifier and creating a queued copy of the data record by copying the data record to a delayed re-try queue; and
in response to confirming that the transfer status identifier of the queued copy is identical to the transfer status identifier of a newest version of the data record within the distributed node system, attempting a subsequent transfer of the queued copy of the data record to the external database.
19. The one or more tangible computer-readable storage media of claim 18 , wherein the computer process further comprises:
in response to confirming that the transfer status identifier of the queued copy is not identical to the transfer status identifier of the newest version of the data record, removing the queued copy of the data record from the delayed re-try queue without transferring the queued copy to the external database.
20. The one or more tangible computer-readable storage media of claim 18 , wherein determining whether the transfer failed or succeeded is based on feedback from the external database and without reading data out of the external database.