Methods and systems for improving error resilience in video delivery
View Patent ↗A multimedia transceiver apparatus continuously generates a model of a video scene, and then uses the generated model to estimate missing sections (e.g., image frames) of the video stream by morphing and/or modifying available uncorrupted video data based on the model. By estimating the missing sections, the subjective quality of video under error conditions (e.g., image frame loss, image frame corruption, video signal interruption, etc.) is improved.
1. A multimedia transceiver apparatus comprising:
a frame prediction circuit configured to,
generate a parameterized model description for each of a plurality of image frames in a sequence of image frames of a video scene, the plurality of image frames including at least a current image frame and a previous image frame, and the parameterized model description including position and orientation information for objects in an image frame;
predict a parameterized model description for a subsequent image frame in the sequence of image frames based on the parameterized model descriptions for the plurality of image frames, and
estimate the subsequent image frame based on the current image frame and the predicted parameterized model description for the subsequent image frame;
a signal loss detector configured to detect an interruption in a received video signal; and
an output selector configured to output the estimated subsequent image frame in response to the detected interruption in the received video signal.
2. The multimedia transceiver apparatus of claim 1 , wherein the frame prediction circuit comprises:
a scene analysis circuit configured to,
separate the current image frame into an object portion and a background portion,
detect unique objects within the object portion of the current image frame, and
generate unique object information for the current image frame based on the detected unique objects within the object portion, the unique object information including position information for the detected unique objects; and wherein
the frame prediction circuit is configured to generate the parameterized model description for the current image frame based on the unique object information.
3. The multimedia transceiver apparatus of claim 1 , wherein the frame prediction circuit is configured to predict the parameterized model description for the subsequent image frame using one of forward prediction and bidirectional prediction.
4. The multimedia transceiver apparatus of claim 1 , wherein the frame prediction circuit further comprises:
an image frame estimation circuit configured to,
determine a difference between positions of objects in the current image frame and predicted positions of objects identified in the parameterized model description of the subsequent image frame, and
estimate the subsequent image frame based on the determined difference.
5. The multimedia transceiver apparatus of claim 4 , wherein the frame prediction circuit further comprises:
a frame buffer configured to buffer the current image frame, and to output the current image frame to the image frame estimation circuit.
6. The multimedia transceiver apparatus of claim 1 , further comprising:
an output device configured to output the estimated subsequent image frame to a user.
7. The multimedia transceiver apparatus of claim 1 , wherein the signal loss detector is configured to detect interruption in the received video signal if the subsequent image frame is missing or corrupted in the sequence of image frames.
8. The multimedia transceiver apparatus of claim 1 , wherein the output selector is configured to output the subsequent image frame if the signal loss detector does not detect interruption in the received video signal.
9. The multimedia transceiver apparatus of claim 1 , wherein the objects in the image frame include at least one of body and facial landmarks in a human portion of the image frame.
10. The multimedia transceiver apparatus of claim 1 , wherein the multimedia transceiver apparatus is a multimedia server.
11. A multimedia transceiver apparatus comprising:
a model creation circuit configured to generate a parameterized model description for a current image frame in a sequence of image frames of a video scene, the parameterized model description including position and orientation information for objects in the current image frame;
an encoder configured to encode the current image frame for transmission; and
a transmitter configured to transmit the encoded current image frame and the parameterized model description for the current image frame.
12. The multimedia transceiver apparatus of claim 11 , further comprising:
a scene analysis circuit configured to,
separate the current image frame into an object portion and a background portion,
detect unique objects within the object portion of the current image frame, and
generate unique object information for the current image frame based on the detected unique objects within the object portion, the unique object information including position information for the detected unique objects; and wherein
the model creation circuit is configured to generate the parameterized model description for the current image frame based on the unique object information.
13. The multimedia transceiver apparatus of claim 11 , wherein the objects in the current image frame include at least one of body and facial landmarks in a human portion of the current image frame.
14. The multimedia transceiver apparatus of claim 11 , wherein the multimedia transceiver apparatus is a multimedia server.
15. A multimedia transceiver apparatus comprising:
an image frame estimation circuit configured to,
receive a parameterized model description for a current image frame in a sequence of image frames of a video scene, the parameterized model description including position and orientation information for objects in the current image frame, and
estimate the current image frame based on a previous image frame and the received parameterized model description for the current image frame;
a signal loss detector configured to detect interruption in a received video signal; and
an output selector configured to output the estimated current image frame in response to the detected interruption in the received video signal.
16. The multimedia transceiver apparatus of claim 15 , wherein the image frame estimation circuit is further configured to,
determine a difference between positions of objects in the previous image frame and predicted positions of objects identified in the parameterized model description of the current image frame, and
estimate the current image frame based on the determined difference.
17. The multimedia transceiver apparatus of claim 15 , further comprising:
an output device configured to output the estimated current image frame to a user.
18. The multimedia transceiver apparatus of claim 15 , wherein the signal loss detector is configured to detect interruption in the received video signal if the current image frame is missing or corrupted in the sequence of image frames.
19. The multimedia transceiver apparatus of claim 15 , wherein the output selector is configured to output the current image frame if the signal loss detector does not detect interruption in the received video signal.
20. The multimedia transceiver apparatus of claim 15 , wherein the objects in the current image frame include at least one of body and facial landmarks in a human portion of the current image frame.