Method for state based snapshot difference with restart capability
A data management device includes a buffer and a processor. The processor may select an unprocessed full key and generate a buffer entry based on a difference between an entry of a local snapshot specified by the unprocessed full key and an entry of a previous local snapshot specified by the unprocessed full key. The processor may make a first determination that a lookup key entry associated with the unprocessed full key is different than a lookup key entry associated with a processed full key. The process may add the generated buffer entry to the buffer after processing the buffer in response to the first determination.
1. A data management device, comprising:
a buffer; and
a processor programmed to:
select an unprocessed full key,
generate a buffer entry based on a difference between an entry of a local snapshot specified by the unprocessed full key and an entry of a previous local snapshot specified by the unprocessed full key,
make a first determination that a lookup key entry associated with the unprocessed full key is different than a lookup key entry associated with a processed full key, and
in response to the first determination, add the generated buffer entry to the buffer after processing the buffer.
2. The data management device of claim 1 , wherein the processor is further programmed to:
select a second unprocessed full key;
generate a second buffer entry based on a difference between a second entry of the local snapshot specified by the second unprocessed full key and a second entry of the previous local snapshot specified by the second unprocessed full key;
make a second determination that a lookup key entry associated with the second unprocessed full key is the same as the lookup key entry associated with the processed full key; and
in response to the second determination, add the second buffer entry to the buffer before processing the buffer.
3. The data management device of claim 1 , wherein processing the buffer comprises:
updating a cookie of the buffer based on the processed full key;
filtering the buffer; and
sending a copy of the buffer to the remote storage.
4. The data management device of claim 3 , wherein processing the buffer further comprises:
making a second determination that the remote storage did not process the copy of the buffer;
in response to the second determination, sending a second copy of the buffer to the remote storage;
making a third determination that the remote storage processed the second copy of the buffer; and
emptying the buffer in response to the third determination.
5. The data management device of claim 3 , wherein processing the buffer further comprises:
making a second determination that the remote storage processed the copy of the buffer; and
emptying the buffer in response to the second determination.
6. The data management device of claim 3 , wherein filtering the buffer comprises:
matching entries of the buffer to a criteria; and
modifying any matched entries of the buffer.
7. The data management device of claim 6 , wherein the criteria is based on a key type specified by entries of the buffer, wherein modifying the matched entries of the buffer comprises changing the key type specified by the matched entries of the buffer.
8. The data management device of claim 7 , wherein the key type specified by the matched entries of the buffer is modified from a key type indirect to a key type direct.
9. The data management device of claim 8 , wherein the key type indirect specifies that data associated with the key is stored in a cloud storage.
10. The data management device of claim 1 , wherein the processor is further programmed to:
make a second determination that the remote storage crashed; and
in response to the second determination:
exchange cookies with the remote storage,
identify the last processed full key based on the cookies,
select a second unprocessed full key based on the last processed full key, and
generate a second buffer entry based on a difference between a second entry of the local snapshot specified by the second unprocessed full key and a second entry of the previous local snapshot specified by the second unprocessed full key.
11. The data management device of claim 10 , wherein the processor is further programmed to:
make a third determination that a lookup key entry associated with the second unprocessed full key is the same as the lookup key entry associated with the processed full key; and
in response to the third determination, add the second buffer entry to the buffer before processing the buffer.
12. The data management device of claim 10 , wherein the processor is further programmed to:
make a third determination that a lookup key entry associated with the second unprocessed full key is different from the lookup key entry associated with the processed full key; and
in response to the third determination, add the generated buffer entry to a buffer after processing the buffer.
13. The data management device of claim 10 , wherein identify the last processed full key based on the cookies comprises:
identifying a second lookup key specified by the cooking;
identifying a plurality of full keys based on the second lookup key;
generating a plurality of buffer entries based on the plurality of full keys;
making a fourth determination that a lookup key associated with a buffer entry of the buffer entries is the same as the second lookup key; and
using a full key associated with the lookup key associated with the buffer entry as the last processed full key.
14. A method of operating a data management device, comprising:
selecting, by the data management device, an unprocessed full key;
generating, by the data management device, a buffer entry based on a difference between an entry of a local snapshot specified by the unprocessed full key and an entry of a previous local snapshot specified by the unprocessed full key;
make a first determination, by the data management device, that a lookup key entry associated with the unprocessed full key is different than a lookup key entry associated with a processed full key; and
in response to the first determination, adding, by the data management device, the generated buffer entry to a buffer after processing the buffer.
15. The method of claim 14 , further comprising:
selecting, by the data management device, a second unprocessed full key;
generating, by the data management device, a second buffer entry based on a difference between a second entry of the local snapshot specified by the second unprocessed full key and a second entry of the previous local snapshot specified by the second unprocessed full key;
making, by the data management device, a second determination that a lookup key entry associated with the second unprocessed full key is the same as the lookup key entry associated with the processed full key; and
in response to the second determination, adding, by the data management device, the second buffer entry to the buffer before processing the buffer.
16. The method of claim 14 , wherein processing the buffer comprises:
updating, by the data management device, a cookie of the buffer based on the processed full key;
filtering, by the data management device, the buffer; and
sending, by the data management device, a copy of the buffer to the remote storage.
17. The method of claim 14 , wherein filtering the buffer comprises:
matching, by the data management device, entries of the buffer to a criteria; and
modifying, by the data management device, any matched entries of the buffer.
18. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for operating a data management device, the method comprising:
selecting, by the data management device, an unprocessed full key;
generating, by the data management device, a buffer entry based on a difference between an entry of a local snapshot specified by the unprocessed full key and an entry of a previous local snapshot specified by the unprocessed full key;
make a first determination, by the data management device, that a lookup key entry associated with the unprocessed full key is different than a lookup key entry associated with a processed full key; and
in response to the first determination, adding, by the data management device, the generated buffer entry to a buffer after processing the buffer.
19. The non-transitory computer readable medium of claim 18 , wherein the method further comprises:
selecting, by the data management device, a second unprocessed full key;
generating, by the data management device, a second buffer entry based on a difference between a second entry of the local snapshot specified by the second unprocessed full key and a second entry of the previous local snapshot specified by the second unprocessed full key;
making, by the data management device, a second determination that a lookup key entry associated with the second unprocessed full key is the same as the lookup key entry associated with the processed full key; and
in response to the second determination, adding, by the data management device, the second buffer entry to the buffer before processing the buffer.
20. The non-transitory computer readable medium of claim 18 , wherein processing the buffer comprises:
updating, by the data management device, a cookie of the buffer based on the processed full key;
filtering, by the data management device, the buffer; and
sending, by the data management device, a copy of the buffer to the remote storage.