Method and system for composing video images from a plurality of endpoints
View Patent ↗The present application discloses a method and apparatus for a multipoint video conferencing system that can offer a wide variety of layouts and the ability to support a large number of conferees. A received stream of NALs from a plurality of endpoints can be processed by an exemplary NAL Processor Module (NPM) and be arranged into a stream of NALs that are referred to in this disclosure as organized NALs (ONALs). Two or more streams of compressed video organized into ONALs can be received by an exemplary NAL Editor Module (NEM), wherein each stream was created from a video image that was generated at an endpoint participating in the conference. An exemplary NEM can comprise a compressed composed frame memory (CCFM) module. Each CCFM can be divided into segments according to the layout of the continuous presence (CP) image that is to be displayed by one or more receiving endpoints.
1. An apparatus for creating a composed video image from video image data representing two or more video images, comprising:
a compressed video data interface; and
a processor that:
retrieves compressed video image data for the two or more video images from the compressed video data interface, wherein the compressed video image data for each image comprises organized network abstraction layer (ONAL) units, each ONAL unit originating from only one network abstraction layer (NAL) unit and originating from only one row of macroblocks (MBs) within a frame of the compressed video image data of its video image;
converts the ONAL units of each of the video images onto ONAL units to be stored in a frame memory of composed video image data, wherein each one of the ONAL units of the composed video image data is originated from one ONAL unit within the retrieved compressed video image data; and
transmits the composed frame memory of the composed video image data to the compressed video data interface.
2. The apparatus of claim 1 , wherein the processor is further programmed to modify the frame memory for the composed video image data to prevent deblocking filtering of the composed video image data.
3. The apparatus of claim 2 , wherein the modifying adds one or more border lines to the composed video image.
4. The apparatus of claim 1 , wherein the composed video image data is compressed according to the H.264 standard.
5. The apparatus of claim 1 , wherein the apparatus is embedded within an multipoint control unit.
6. A multipoint control unit (MCU) for composing a composed video image from video image data from two or more endpoints in a video conference, comprising:
a network interface for receiving and sending video image data between the MCU and the two or more endpoints;
a compressed video data interface;
a data bus for communicating video image data between the network interface and the compressed video data interface; and
a processor that:
retrieves, from the compressed video data interface, compressed video image data originating from the two or more endpoints, wherein the compressed video image data of each of the endpoints comprises organized network abstraction layer (ONAL) units, each ONAL unit originating from only one network abstraction layer (NAL) unit and originating from only one row of macroblocks (MBs) within a frame of the compressed video image data from its endpoint;
maps a composed frame memory for storing the ONAL units of each of the video images and converts the ONAL units of each of the video images onto ONAL units of the composed frame memory storing the composed video image data, wherein the ONAL units of the composed video image data originate from only one ONAL unit of one of the video images;
composes the frame memory for the composed video image data according to the map;
transmits the composed frame memory of the composed video image data to the compressed video data interface; and
send the composed video image data to at least one of the two or more endpoints via the network interface.
7. The apparatus of claim 6 , wherein the processor is further programmed to modify the frame memory for the composed video image data to prevent deblocking filtering of the composed video image data.
8. The apparatus of claim 7 , wherein the modifying adds one or more border lines to the composed video image.
9. The apparatus of claim 6 , wherein the composed video image data is compressed according to the H.264 standard.
10. The apparatus of claim 6 , wherein the processor is further programed to:
retrieve, from the compressed video data interface, NAL-formatted compressed video image data from the two or more endpoints, wherein the video image data comprises NAL units;
convert the NAL units of the NAL-formatted compressed video image data units into ONAL units; and
transmit the ONAL units to the compressed video data interface.
11. The apparatus of claim 10 , wherein converting the NAL units to ONAL units comprises modifying a portion of the compressed video data.
12. The apparatus of claim 11 , wherein the modified portion of the compressed video data is representative of motion information.
13. A method for composing a videoconferencing layout, the method comprising:
receiving a compressed video input stream from each of two or more endpoints, wherein each of the compressed video input streams complies with the H.264 standard and is organized into NAL units;
converting each video input stream of NAL units into a stream of organized NAL (ONAL) units, wherein each ONAL unit comprises compressed video data that originated from only one NAL unit and only one row of the compressed video input stream;
storing each stream of organized NAL units in a compressed composed frame memory, wherein information stored in each ONAL unit reflects the location of the ONAL unit's compressed video data in the desired composed videoconferencing layout;
manipulating each of the stored ONAL units in the compressed composed frame memory to reflect each ONAL unit's location in the desired composed videoconferencing layout; and
transmitting the manipulated ONAL units as a compressed output stream reflecting the desired composed videoconferencing layout.
14. The method of claim 13 , wherein storing each stream of ONAL units in a compressed composed frame memory further comprises modifying the frame memory for the composed video image data to prevent deblocking filtering of the composed video image data.
15. The method of claim 14 , wherein the modifying comprises adding one or more border lines to the composed videoconferencing layout.
16. The method of claim 13 , wherein converting each received video input stream of NAL units into a stream of ONAL units further comprises modifying a portion of the compressed video data.
17. The method of claim 16 , wherein the modified portion of the compressed video data is relevant to motion information.