Scalable real-time video compositing systems and methods
In some embodiments, a server system composites in real-time, in response to a user video search query, a standard-compliant (e.g. MPEG-4/H.264) SD or HD video stream encoding a rectangular (x-y) composite video preview panel array (grid) of video search results. Each panel/tile in the rectangular panel array displays a temporal section (e.g. the first 90 seconds, looped-back) of a video identified in response to the user query. Generating the composite video panel array in real-time is achieved by compositing the component video panels in the compressed domain, after each panel undergoes a compressed-domain pre-compositing preparation process that facilitates dynamic compositing of the panels into a single video stream. The pre-compositing preparation includes transcoding to a format having a down-scaled common resolution, common GOP structure, and one-slice-per-row slice structure.
1. A computer server comprising:
a plurality of special-purpose hardware transcoder integrated circuits, each configured to transcode video streams; and
at least one central processing unit connected to the plurality of transcoder integrated circuits and configured to execute a set of instructions to
configure at least one of the hardware transcoder integrated circuits to perform a pre-compositing preparation by transcoding a plurality of independent input video streams to generate corresponding transcoded video streams, wherein at least one of the input independent video streams includes slices encoding more than one row of video blocks, and wherein transcoding the plurality of independent input video streams comprises
downscaling each of the independent video streams,
enforcing a common group-of-picture (GOP) structure across the transcoded video streams, and
enforcing a one-slice-per-row slice structure for each picture of the transcoded video streams;
in response to receiving a user video search query from a client computer system, identifying a plurality of transcoded video streams responsive to the user video search query;
composite in the transform domain the plurality of transcoded video streams responsive to the user video search query into a composite intermediate video stream encoding a preview video panel array for the identified plurality of transcoded video streams, wherein a GOP structure of the composite intermediate video stream substantially coincides with the common GOP structure of the identified plurality of transcoded video streams;
configure at least one of the plurality of transcoder integrated circuits to transcode the composite intermediate video stream to generate an output video stream encoding the preview video panel array;
encapsulate the output video stream in a container to generate an encapsulated output video stream; and
send a response to the user video search query to the client computer system, wherein the response to the user video search query includes the encapsulated output video stream.
2. The server of claim 1 , wherein the at least one central processing unit is further configured to execute the set of instructions to identify a display hardware configuration of the client computer system, and to select an array size of the preview video panel array according to the display hardware configuration.
3. The server of claim 2 , wherein the at least one central processing unit is configured to select the array size of the preview video panel array according to a screen diagonal size of the client computer system.
4. The server of claim 1 , wherein the at least one central processing unit is further configured to execute the set of instructions to identify a display hardware configuration of the client computer system, and to select, according to the display hardware configuration, a pre-constructed video library from a plurality of video libraries, the pre-constructed video library comprising the plurality of input video streams, for retrieving the input video streams from the pre-constructed video library.
5. The server of claim 1 , wherein the response to the user video search query comprises a plurality of audio streams, each audio stream corresponding to one of the panels of the preview video panel array.
6. The server of claim 1 , wherein the at least one central processing unit is further configured to execute the set of instructions to send to the client computer system an audio stream corresponding to a selected panel of the preview video panel array in response to receiving from the client computer system an indicator of a user selection from the preview video panel array of the selected panel.
7. The server of claim 1 , wherein the preview video panel array shows a loopback display of a predetermined temporal section of each of the input video streams.
8. The server of claim 1 , wherein the output video stream includes slices encoding more than one row of video blocks.
9. The server of claim 1 , wherein the at least one processor is configured to perform the pre-compositing preparation for at least some of the independent input video streams prior to receiving the user video search query, to generate a video library prepared for compositing.
10. A method comprising employing a server computer system comprising at least one processor to:
perform a pre-compositing preparation by transcoding a plurality of independent input video streams to generate corresponding transcoded video streams, wherein at least one of the input independent video streams includes slices encoding more than one row of video blocks, and wherein transcoding the plurality of independent input video streams comprises
downscaling each of the independent video streams,
enforcing a common group-of-picture (GOP) structure across the transcoded video streams, and
enforcing a one-slice-per-row slice structure for each picture of the plurality of transcoded video streams;
in response to receiving a user video search query from a client computer system, identify a plurality of transcoded video streams responsive to the user video search query;
composite in the transform domain the plurality of transcoded video streams responsive to the user video search query into a composite intermediate video stream encoding a preview video panel array for the identified plurality of transcoded video streams, wherein a GOP structure of the composite intermediate video stream substantially coincides with the common GOP structure of the identified plurality of transcoded video streams;
transcode the composite intermediate video stream to generate an output video stream encoding the preview video panel array; and
send a response to the user video search query to the client computer system, wherein the response to the user video search query includes the output video stream.
11. A computer system comprising at least one processor, the computer system being configured to:
perform a pre-compositing preparation by transcoding a plurality of independent input video streams to generate corresponding transcoded video streams, wherein at least one of the input independent video streams includes slices encoding more than one row of video blocks, and wherein transcoding the plurality of independent input video streams comprises
downscaling each of the independent video streams,
enforcing a common group-of-picture (GOP) structure across the transcoded video streams, and
enforcing a one-slice-per-row slice structure for each picture of the plurality of transcoded video streams;
in response to receiving a user video search query from a client computer system, identify a plurality of transcoded video streams responsive to the user video search query;
composite in the transform domain the plurality of transcoded video streams responsive to the user video search query into a composite intermediate video stream encoding a preview video panel array for the identified plurality of transcoded video streams, wherein a GOP structure of the composite intermediate video stream substantially coincides with the common GOP structure of the identified plurality of transcoded video streams;
transcode the composite intermediate video stream to generate an output video stream encoding the preview video panel array; and
send a response to the user video search query to the client computer system, wherein the response to the user video search query includes the output video stream.
12. The computer system of claim 11 , wherein the computer system is further configured to identify a display hardware configuration of the client computer system, and to select an array size of the preview video panel array according to the display hardware configuration.
13. The computer system of claim 12 , wherein the computer system is configured to select the array size of the preview video panel array according to a screen diagonal size of the client computer system.
14. The computer system of claim 11 , wherein the computer system is further configured to identify a display hardware configuration of the client computer system, and to select, according to the display hardware configuration, a pre-constructed video library from a plurality of video libraries, the pre-constructed video library comprising the plurality of input video streams, for retrieving the input video streams from the pre-constructed video library.
15. The computer system of claim 11 , wherein the response to the user video search query comprises a plurality of audio streams, each audio stream corresponding to one of the panels of the preview video panel array.
16. The computer system of claim 11 , wherein the computer system is further configured to send to the client computer system an audio stream corresponding to a selected panel of the preview video panel array in response to receiving from the client computer system an indicator of a user selection from the preview video panel array of the selected panel.
17. The computer system of claim 11 , wherein the preview video panel array shows a loopback display of a predetermined temporal section of each of the input video streams.
18. The computer system of claim 11 , wherein the output video stream includes slices encoding more than one row of video blocks.
19. The computer system of claim 11 , wherein the computer system is configured to perform the pre-compositing preparation for at least some of the independent input video streams prior to receiving the user video search query, to generate a video library prepared for compositing.