Concurrent decompression of multiple video streams with constrained decompression resources
View Patent ↗A method and system for gracefully reducing demands for shared decompression resources required to present multiple video streams concurrently such that their streaming video content is simultaneously available for further processing or display. In particular, it relates to the use of presentation time stamps for incoming frames with reference to a time base clock and a threshold to determine if the decompression resources are falling behind in their ability to process incoming frames in real time. If this occurs then frames are dropped, i.e. discarded without being decompressed, with preference given to dropping incremental frames rather than key frames.
1. A method of reducing demands for shared decompression resources required to present a plurality of video streams concurrently, the method including:
at a decompression processor in a computing device, receiving a key frame in each of the plurality of video streams for decompression;
obtaining presentation time stamps, abbreviated PTSs, for the key frames, benchmarking a first PTS against one or more time base clocks to obtain benchmark reference values, and decompressing the key frames, wherein the presentation time stamps give presentation times at which the frames are to be presented or displayed;
receiving additional frames in the plurality of video streams;
obtaining additional PTSs for at least some subsequent additional frames;
when particular additional PTSs for particular additional frames, as compared to the one or more time base clocks and the benchmark reference values, indicate that the particular additional PTSs have fallen behind their respective presentation times, by more than a threshold, at which the particular additional frames should be presented, responsively initiating dropping of incremental frames in the respective video streams at least until receipt of a further key frame to distribute the dropped incremental frames over at least a portion of the plurality of video streams; and
decompressing at least some incremental frames that are not dropped in the plurality of video streams.
2. The method of claim 1 , further including establishing a new benchmark reference value by benchmarking the further key frame against the time base clock.
3. The method of claim 1 , further including:
using the decompression processor to decompress eight or more video streams.
4. The method of claim 1 , wherein:
a first video stream includes first and second types of incremental frames;
the first type of incremental frames depends on one additional frame; and
the second type of incremental frames depends on two additional frames;
further including dropping only the second type of incremental frames while continuing to process the first type of incremental frames.
5. The method of claim 1 , wherein:
the video stream includes first and second types of incremental frames;
the first type of incremental frames are predictive P-frames compliant with an MPEG standard; and
the second type of incremental frames are bidirectional B-frames compliant with an MPEG standard;
further including dropping only the B-frames while continuing to process the P-frames.
6. The method of claim 1 , wherein at least some of the PTSs are extracted from the header data before decompression.
7. The method of claim 1 , wherein some of the PTSs are calculated at least in part based on other fields in header data.
8. The method of claim 1 , wherein some of the PTSs are based on a frame rate of the video stream.
9. The method of claim 1 , further applied to more than eight video steams concurrently.
10. The method of claim 1 , further applied at least partially in parallel by multiple processors to more than eight video streams concurrently.
11. The method of claim 9 , wherein:
the determinations to drop frames are distributed among the video streams according to a cost function.
12. A computer system for reducing demands for shared decompression resources required to present a plurality of video streams concurrently, the system including:
a processor and a non-transitory computer readable storage medium storing computer instructions configured to cause the processor to:
receive a key frame in each of the plurality of video streams for decompression by a decompression processor;
obtain presentation time stamps, abbreviated PTSs, for each of the key frames, benchmark the PTSs against one or more time base clocks to obtain benchmark reference values, and decompress the key frames, wherein the presentation time stamps give presentation times at which the frames are to be presented or displayed;
receive additional frames in the plurality of video streams;
obtain additional PTSs for at least some subsequent additional frames;
when particular additional PTSs for particular additional frames, as compared to the one or more time base clocks and the benchmark reference values, indicate that the particular additional PTSs have fallen behind their presentation times, by more than a threshold, at which the particular additional frames should be presented, responsively initiate dropping of incremental frames in the respective video streams at least until receipt of a further key frame; and
decompress at least some incremental frames that are not dropped in the plurality of video streams.
13. The system of claim 12 , further configured to cause the processor to:
establish a new benchmark reference value by benchmarking the further key frame against the time base clock.
14. The system of claim 12 , further configured to cause the processor to:
apply the decompression processor to decompression of eight or more video streams.
15. The system of claim 12 , wherein:
the video stream includes first and second types of incremental frames;
the first type of incremental frames depends on one additional frame;
the second type of incremental frames depends on two additional frames; and
further configured to cause the processor to drop only the second type of incremental frames while continuing to process the first type of incremental frames.
16. The system of claim 12 , wherein:
the video stream includes first and second types of incremental frames;
the first type of incremental frames are predictive P-frames compliant with an MPEG standard;
the second type of incremental frames are bidirectional B-frames compliant with an MPEG standard; and
further configured to cause the processor to drop only the B-frames while continuing to process the P-frames.
17. The system of claim 12 , wherein:
at least some of the PTSs are extracted from header data before decompression.
18. The system of claim 12 , wherein:
some of the PTSs are calculated at least in part based on other fields in header data.
19. The system of claim 12 , wherein:
some of the PTSs are based on a frame rate of the video stream.
20. The system of claim 12 , further configured to cause the processor to process more than eight video streams concurrently.
21. The system of claim 12 , further configured with multiple processors to process at least partially in parallel more than eight video streams concurrently.
22. The system of claim 20 , wherein:
the determinations to drop frames are distributed among the video streams according to a cost function.
23. A non-transitory computer readable storage medium storing computer instructions configured to cause a processor to:
receive a key frame in each of a plurality of video streams for decompression by a decompression processor;
obtain presentation time stamps, abbreviated PTSs, for each of the key frames, benchmark the PTSs against one or more time base clocks to obtain benchmark reference values, and decompress the key frames, wherein the presentation time stamps give presentation times at which the frames are to be presented or displayed;
receive additional frames in the plurality of video streams;
obtain additional PTSs for at least some subsequent additional frames;
when particular additional PTSs for particular additional frames, as compared to the one or more time base clocks and the benchmark reference values, indicate that the particular additional PTSs have fallen behind their presentation times, by more than a threshold, at which the particular additional frames should be presented, responsively initiate dropping of incremental frames in the respective video streams at least until receipt of a further key frame; and
decompress at least some incremental frames that are not dropped in the plurality of video streams.
24. The non-transitory computer readable storage medium of claim 23 , further configured to cause the processor to:
apply the decompression processor to decompression of eight or more video streams.
25. The non-transitory computer readable storage medium of claim 23 , wherein:
the video stream includes first and second types of incremental frames;
the first type of incremental frames depends on one additional frame;
the second type of incremental frames depends on two additional frames; and
further configured to cause the processor to drop only the second type of incremental frames while continuing to process the first type of incremental frames.