Partitioning, processing, and protecting media data
A technique for managing data objects in a storage cluster includes splitting a media data object into multiple portions at boundaries within the media data object. The technique further includes transforming the portions of the media data object into segments that provide individually processable units and distributing the segments among multiple computing nodes of the storage cluster for storage therein.
1. A method of managing media data, comprising:
splitting a media data object into multiple portions at boundaries within the media data object;
transforming the portions into segments that provide individually processable units of media data; and
distributing the segments among multiple computing nodes of a storage cluster for storage therein,
wherein splitting the media data includes providing an overlap region between two consecutive segments such that the consecutive segments contain respective regions having identical media data, the overlap region having a configurable size based at least in part on a type of processing to be performed individually on the consecutive segments.
2. The method of claim 1 , wherein splitting the media data object into portions includes defining multiple portions that contain video data corresponding to respective intervals of time.
3. The method of claim 2 , wherein the defined portions that contain video data contain no audio data.
4. The method of claim 2 , wherein splitting the media data object into portions further includes providing at least one portion that contains audio data but no video data.
5. The method of claim 2 , wherein splitting the media data object into portions further includes defining multiple portions that contain audio data corresponding to respective intervals of time but contain no video data.
6. The method of claim 5 , wherein the media data object includes audio data for multiple audio tracks, and wherein each of the portions that contain audio data includes audio data for all of the multiple audio tracks for the respective interval of time.
7. The method of claim 2 , wherein splitting the media data object into portions further includes providing at least one portion that contains subtitle data.
8. The method of claim 2 , wherein the two consecutive segments are of a same type, the type being one of video, audio, or subtitle.
9. The method of claim 8 , further comprising defining a size of the overlap region based on at least one of (i) a specified duration of time and (ii) a specified number of frames.
10. The method of claim 9 , wherein the media data object includes video data having a frame rate, and wherein defining the size of the overlap region is based on a longer of (i) the specified duration of time and (ii) the specified number of frames.
11. The method of claim 9 , wherein the size of the overlap regions is a user-definable setting.
12. The method of claim 8 , wherein splitting the media data object into portions at boundaries within the media data object includes:
identifying a first IDR (Instantaneous Decoder Refresh) frame at a first location in video data of the media data object;
identifying a second IDR frame at a second location in the video data, the second IDR frame corresponding to a later point in time than the first IDR frame;
ending a first portion of the media data object at the second location; and
beginning a second portion of the media data object at the first location, the first portion and the second portion thereby defining the overlapping region, which extends between the first location and the second location.
13. The method of claim 12 , wherein the video data of the media data object includes at least one intervening IDR frame between the first IDR frame and the second IDR frame.
14. The method of claim 2 , wherein transforming the portions into segments includes rendering the portions as standalone, playable media content.
15. The method of claim 14 , further comprising storing an AI (artificial intelligence) filter, configured to process one or more of the segments, among the computing nodes of the storage cluster.
16. The method of claim 15 , further comprising executing the Al filter on a single segment without reference to any other segments.
17. The method of claim 16 , wherein the Al filter includes a neural network configured to identify a specified class of objects or behavior.
18. The method of claim 14 , wherein rendering the portions as standalone, playable media content includes creating respective containers for the portions, the containers including metadata based on respective contents of the media data object.
19. The method of claim 2 , wherein the media data object includes multiple chunks, each chunk including contiguous, time-ordered data for one of (i) video data, (ii) audio data, or (iii) subtitle data, and wherein the method further comprises storing a metadata index that associates chunks with respective byte ranges within the media data object, the metadata index thereby enabling access to chunks based on byte range.
20. The method of claim 2 , wherein the media data object includes multiple chunks, each chunk including contiguous, time-ordered data for one of (i) video data, (ii) audio data, or (iii) subtitle data, and wherein the method further comprises storing a metadata index that associates chunks with respective time ranges within the media data object, the metadata index thereby enabling access to chunks based on time range.
21. The method of claim 1 , further comprising reconstructing the media data object from the distributed segments.
22. A computerized apparatus, comprising control circuitry that includes a set of processors coupled to memory, the control circuitry constructed and arranged to:
split a media data object into multiple portions at boundaries within the media data object;
transform the portions into segments that provide individually processable units of media data; and
distribute the segments among multiple computing nodes of a storage cluster for storage therein,
wherein the control circuitry constructed and arranged to split the media data is further constructed and arranged to provide an overlap region between two consecutive segments such that the consecutive segments contain respective regions having identical media data, the overlap region having a configurable size based at least in part on a type of processing to be performed individually on the consecutive segments.
23. A computer program product including a set of non- transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of managing media data, the method comprising:
splitting a media data object into multiple portions at boundaries within the media data object;
transforming the portions into segments that provide individually processable units of media data; and
distributing the segments among multiple computing nodes of a storage cluster for storage therein,
wherein splitting the media data includes providing an overlap region between two consecutive segments such that the consecutive segments contain respective regions having identical media data, the overlap region having a configurable size based at least in part on a type of processing to be performed individually on the consecutive segments.
24. The method of claim 2 , further comprising storing segment metadata that associates segments of the media data object with locations in the storage cluster where the respective segments are stored.
25. The method of claim 1 , wherein the two consecutive segments include video data and the type of processing to be performed individually on the consecutive segments includes video analytics processing.
26. The method of claim 25 , wherein the video analytics processing is performed using an Al (artificial intelligence) filter, and wherein the configurable size is defined based on a warm-up time of the Al filter.