Systems and methods for securing data in motion
View Patent ↗Systems and methods for reading and writing a set of data using a journaling service are provided. The journaling service may be used to identify and record data storage operations associated with one or more shares of data stored in one or more share locations. The journaling service may use logs to record each of the read and write requests to the share locations. In some embodiments, the log may be a queue data structure that stores information associated with failed data storage operations. In some embodiments, the journaling service may leverage both memory and disk storage in order to maintain the journaling queue. In some embodiments, the journaling queue may maintain information associated with the state of each share location. In some embodiments, this information may be used by the journaling service to determine when to monitor and record information regarding data storage operations associated with the share locations.
1. A method thr reading and writing a set of data, comprising:
splitting the set of data into one or more data shares using an information dispersal algorithm;
storing the one or more data shares in share locations;
determining that at least one of the share locations is unavailable for data storage operations; and
storing incoming data storage operations associated with each of the unavailable share locations in respective journaling queues unique to each of the unavailable share locations.
2. The method of claim 1 , further comprising:
determining that at least one of the unavailable share locations has been made available; and
executing the data storage operations stored in the journaling queue unique to the at least one unavailable share location that has been made available.
3. The method of claim 2 , further comprising flushing the executed data storage operations.
4. The method of claim 1 , wherein the journaling queues are stored in memory, further comprising:
establishing a queue limit associated with an amount of data storage operations;
determining that the queue limit is exceeded for at least one of the journaling queues; and
for each of the journaling queues exceeding the queue limit, flushing that journaling queue from memory to disk storage.
5. The method of claim 1 , further comprising:
establishing a maximum amount of time that a share location is unavailable for data storage operations;
determining that the maximum amount of time is exceeded for at least one of the unavailable share locations;
for each of the unavailable share locations exceeding the maximum amount of time, refusing incoming data storage operations.
6. A method for reading and writing a set of data, comprising:
splitting the set of data into one or more data shares using an information dispersal algorithm;
storing the one or more data shares in share locations;
establishing a maximum number of failed data storage operations;
storing failed incoming data storage operations associated with each of the share locations in respective journaling queues;
determining that the failed incoming data storage operations stored in at least one of the journaling queues exceeds the established maximum number; and
discarding the failed incoming data storage operations stored in the at least one of the journaling queues that exceeds the established maximum number.
7. The method of claim 6 , further comprising discarding incoming data storage operations associated with each of the respective journaling queues that exceed the established maximum number.
8. The method of claim 6 , further comprising:
storing each of the one or more data shares in two or more share locations; and
for each share location associated with at least one of the journaling queues that exceeds the established maximum number, rebuilding at least one of the one or more data shares using at least one of the two or more share locations associated with at least one of the journaling queues that has not exceeded the established maximum number.
9. A method for reading and writing a set of data, comprising:
splitting the set of data into one or more data shares using an information dispersal algorithm;
storing the one or more data shares in share locations, including a rebuilding share location;
receiving a request to perform a data storage operation on a file;
determining that the file is restored in the rebuilding share location; and
based on the determination, storing the data storage operation in a journaling queue associated with the rebuilding share location.
10. The method of claim 9 , further comprising:
determining that the file is not restored in the rebuilding share location; and
based on the determination that the file is not restored in the rebuilding share location, discarding the data storage operation.
11. The method of claim 9 , further comprising:
determining that the rebuilding share location is available for data storage operations; and
based on the determination that the rebuilding share location is available for data storage operations, executing the data storage operation stored in the journaling queue associated with the rebuilding share location.
12. A system for reading and writing a set of data, the system comprising processing circuitry configured to:
split the set of data into one or more data shares using an information dispersal algorithm;
store the one or more data shares in share locations;
determine that at least one of the share locations is unavailable for data storage operations; and
store incoming data storage operations associated with each of the unavailable share locations in respective journaling queues unique to each of the unavailable share locations.
13. The system of claim 12 , wherein the processing circuitry is further configured to:
determine that at least one of the unavailable share locations has been made available; and
execute the data storage operations stored in the journaling queue unique to the at least one of the unavailable share locations that has been made available.
14. The system of claim 13 , wherein the processing circuitry is further configured to flush the executed data storage operations.
15. The system of claim 12 , further comprising a memory and disk storage, and wherein the processing circuitry is further configured to:
store the journaling queues in the memory;
establish a queue limit associated with an amount of data storage operations;
determine that the queue limit is exceeded for at least one of the journaling queues; and
for each of the journaling queues exceeding the queue limit, flush that journaling queue from the memory to the disk storage.
16. The system of claim 12 , wherein the processing circuitry is further configured to:
establish a maximum amount of time that a share location is unavailable for data storage operations;
determine that the maximum amount of time is exceeded for at least one of the unavailable share locations;
for each of the unavailable share locations exceeding the maximum amount of time, refuse incoming data storage operations.
17. A system for reading and writing a set of data, the system comprising processing circuitry configured to:
split the set of data into one or more data shares using an information dispersal algorithm;
store the one or more data shares in share locations;
establish a maximum number of failed data storage operations;
store failed incoming data storage operations associated with each of the share locations in respective journaling queues;
determine that the failed incoming data storage operations stored in at least one of the journaling queues exceeds the established maximum number; and
discard the failed incoming data storage operations stored in the at least one of the journaling queues that exceeds the established maximum number.
18. The system of claim 17 , wherein the processing circuitry is further configured to discard incoming data storage operations associated with each of the respective journaling queues that exceed the established maximum number.
19. The system of claim 17 , wherein the processing circuitry is further configured to:
store each of the one or more data shares in two or more share locations; and
for each share location associated with at least one of the journaling queues that exceeds the established maximum number, rebuild at least one of the one or more data shares using at least one of the two or more share locations associated with at least one of the journaling queues that has not exceeded the established maximum number.
20. A system for reading and writing a set of data, the system comprising processing circuitry configured to:
split the set of data into one or more data shares using an information dispersal algorithm;
store the one or more data shares in share locations, including a rebuilding share location;
receive a request to perform a data storage operation on a file;
determine that the file is restored in the rebuilding share location; and
based on the determination that the file is restored in the rebuilding share location, store the data storage operation in a journaling queue associated with the rebuilding share location.
21. The system of claim 20 , wherein the processing circuitry is further configured to:
determine that the file is not restored in the rebuilding share location; and
based on the determination that the file is not restored in the rebuilding share location, discard the data storage operation.
22. The system of claim 20 , wherein the processing circuitry is further configured to:
determine that the rebuilding share location is available for data storage operations; and
based on the determination that the rebuilding share location is available for data storage operations, execute the data storage operations stored in the journaling queue associated with the rebuilding share location.