Storage method for loss-tolerant data stream recording
A distributed storage of sequential video frames is designed without compromising data reliability. Capitalizing on the loss-tolerant nature of video data, user experience is maintained by leveraging higher capacity drives with lower latency, higher throughput and more efficient utilization of provisioned storage. No RAID, replication or erasure computations are necessary. There are a few optimization to this method which improves reliability, such as reliably storing video data in partitioned drives, making data storage configuration map known to each partition, implementing a strategy for avoiding loss of a common moment, preventing potential loss of reference frames, leveraging different sized drives, avoiding pitfalls associated with data getting stored disproportionately to large drives etc. Though video streaming data has been used as an illustrative example of loss-tolerant data, the methods and systems can be applied to any type of loss-tolerant data.
1. A computer-implemented method for dynamic distributed allocation of loss-tolerant video data across a plurality of drives for reliable storage and playback, the video data comprising video surveillance data streams of sequential video frames coming from a plurality of cameras, the method comprising:
for each video stream coming from an individual camera, storing the i th video frame f i to a drive D j , where j=i mod n, where there are a total of ‘n’ number of drives;
providing a configuration map that keeps track of which video frame coming from which camera is stored in which drive; and
during playback, using the configuration map to reconstitute a video stream that came in from a particular camera by collating stored video frames of that video stream retrieved from the ‘n’ number of drives based on original time sequence of those video frames.
2. The method of claim 1 , wherein the method further comprises:
when a particular drive among the plurality of drives is unavailable, skipping that drive while storing the video frames in the remaining (n−1) number of available drives.
3. The method of claim 2 , wherein drive unavailability is caused by total or partial failure of the drive requiring a rebuild, or, the drive being off-line for scheduled maintenance.
4. The method of claim 2 , wherein the method further comprises:
updating the configuration map to indicate which drive is unavailable.
5. The method of claim 2 , wherein the method further comprises:
introducing a properly functioning drive to replace the unavailable drive to restore the number of drives available for data recording to ‘n,’ and initialize the replacement drive so that video frames start recording therein.
6. The method of claim 5 , wherein the method further comprises:
updating the configuration map so that during playback, video frames recorded in the replacement drive are retrieved for reconstituting the video stream.
7. The method of claim 1 , wherein the ‘n’ number of drives are partitioned to improve reliability in video data storage and playback.
8. The method of claim 7 , wherein the partition is known to the operating system.
9. The method of claim 8 , wherein the configuration map is also known to the operating system.
10. The method of claim 7 , wherein the configuration map is known to each of the ‘n’ drives through a database located in a consistent location on each partition providing information about where each video frame from each video stream is stored.
11. The method of claim 1 , wherein a random offset unique to each video frame is introduced to prevent potential loss of a common moment due to failure of a single drive that would otherwise always store video data pertaining to that moment in each storage cycle in the absence of the random offset.
12. The method of claim 1 , wherein the number of drives ‘n’ is not divisible by the frequency of full frames in order to prevent failure of a single drive leading to loss of full-frames in successive storage cycles.
13. The method of claim 1 , wherein all of the ‘n’ number of drives do not have the same size.
14. The method of claim 13 , wherein an algorithm is used to allocate newly incoming video frames to individual drives at a rate that is proportional to the relative size of a particular drive within the collection of ‘n’ number of drives.
15. The method of claim 1 , wherein the video data is loss-tolerant, such that a loss of a predetermined percentage (ϵ%) of video frames does not substantially affect the quality of the video stream being reconstituted during playback.
16. The method of claim 15 , wherein none of the individual drives stores video data that is more than (ϵ%) of the total amount of available video storage capacity distributed among the ‘n’ number of drives.