Architecture for combining media processing with networking
View Patent ↗Systems and methods for processing media streams for transport over a network based on network conditions. An integrated circuit comprises a media processing unit coupled to receive feedback from a network processing unit. The media processing unit converts a media stream from a compressed input stream to a compressed output stream such that the compressed output stream has characteristics that are best suited for the network conditions. Network conditions can include, for example, characteristics of the network (e.g., latency or bandwidth) or characteristics of the remote playback devices (e.g., playback resolution). Changes in the network conditions can result in a change in the conversion process.
1. An integrated circuit for converting media streams, comprising:
a frame converter circuit configured to:
extract motion vectors and a first set of compressed domain error terms from frames of an input media stream in a first compressed format, the first compressed format including input frames that are interframe compressed and intraframe compressed, and the first set of compressed domain error terms representing information related to transients not encoded by the motion vectors; and
convert the input media stream in the first compressed format into an intermediate media stream in a second format without decompressing the intraframe compressed frames of the input media stream and by applying the extracted motion vectors and adding the first set of compressed domain error terms to the interframe compressed frames, the intermediate media stream in the second format contains intraframe compressed frames but not interframe compressed frames;
an image processor configured to receive the intermediate media stream from the frame converter and perform image enhancement on the intermediate media stream by removing noise from the frames of the intermediate media stream in order to generate processed intermediate media stream; and
an encoder configured to receive the processed intermediate media stream and encode the processed intermediate media stream into an output media stream in a third compressed format by applying the extracted motion vectors to the processed intermediate media stream and adding second set of compressed domain error terms, the third compressed format being different from the first compressed format and including frames that are interframe compressed and intraframe compressed.
2. The integrated circuit of claim 1 , wherein the image processor is configured to perform the image processing on the intermediate frames based on a network condition for transmitting the output media stream.
3. The integrated circuit of claim 1 , wherein the image processor is configured to perform image processing based on display capabilities of a networked device receiving the output media stream via a network.
4. The integrated circuit of claim 1 , wherein the image processor is configured to:
determine scaling parameters based on an external signal and
scale frames in the intermediate media stream for encoding based on the scaling parameters.
5. The integrated circuit of claim 4 , wherein the scaler is further configured to scale the frames based on the extracted motion vectors.
6. The integrated circuit of claim 1 wherein the encoder is further configured to generate encoding parameters and quantization parameters for encoding the output media stream based on a network condition for transmitting the output media stream.
7. The integrated circuit of claim 1 , wherein encoder is further configured to generate macroblock mode decision parameters for encoding frames of the output media stream.
8. The integrated circuit of claim 1 , wherein the image processing comprises modifying at least one of bitrate, frame rate and resolution.
9. A method for converting media streams in an integrated circuit, comprising:
extracting motion vectors and a first set of compressed domain error terms from frames of an input media stream in a first compressed format, the first compressed format including input frames that are interframe compressed and intraframe compressed, and the first set of compressed domain error terms representing information related to transients not encoded by the motion vectors;
converting the input media stream in the first compressed format into an intermediate media stream in a second format without decompressing the intraframe compressed frames of the input media stream and by applying the extracted motion vectors and adding the first set of compressed domain error terms to the interframe compressed frames, the intermediate media stream in the second format contains intraframe compressed frames but not interframe compressed frames;
receiving the intermediate media stream by an image processor;
performing image enhancement on the intermediate frames by removing noise from the frames of the intermediate media stream in order to generate processed intermediate media stream by the image processor;
receiving the processed intermediate media stream by an encoder; and
encoding, at the encoder, the processed intermediate media stream into an output media stream in a third compressed format by applying the extracted motion vectors to the processed intermediate media stream and adding a second set of compressed domain error terms, the third compressed format being different from the first compressed format and including frames that are interframe compressed and intraframe compressed.
10. The method of claim 9 , wherein the image processing on the intermediate frames is based on a network condition for transmitting the output media stream.
11. The method of claim 9 , wherein the image processing is based on display capabilities of a networked device receiving the output media stream via a network.
12. The method of claim 9 , further comprising:
determining scaling parameters based on an external signal; and
scaling frames in the intermediate media stream for encoding based on the scaling parameters.
13. The method of claim 12 , wherein the scaling of the frames is further based on the extracted motion vectors.
14. The method of claim 9 , further comprising generating encoding parameters and quantization parameters for encoding the output media stream based on a network condition for transmitting the output media stream.
15. The method of claim 9 , further comprising generating encoding parameters and quantization parameters for encoding the output media stream based on a network condition for transmitting the output media stream.
16. The method of claim 9 , further comprising generating macroblock mode decision parameters for encoding frames of the output media stream.
17. A media device, comprising:
a frame converter circuit configured to:
extract motion vectors and a first set of compressed domain error terms from frames of an input media stream in a first compressed format, the first compressed format including input frames that are interframe compressed and intraframe compressed and the first set of compressed domain error terms representing information related to transients not encoded by the motion vectors; and
convert the input media stream in the first compressed format into an intermediate media stream in a second format without decompressing the intraframe compressed frames of the input media stream and by applying the extracted motion vectors and adding the first set of compressed domain error terms to the interframe compressed frames, the intermediate media stream in the second format contains intraframe compressed frames but not interframe compressed frames;
an image processor configured to receive the intermediate media stream from the frame converter and perform image enhancement on the intermediate media stream by removing noise from the frames of the intermediate media stream in order to generate processed intermediate media stream; and
an encoder configured to receive the processed intermediate media stream and encode the processed intermediate media stream into an output media stream in a third compressed format by applying the extracted motion vectors to the processed intermediate media stream and adding a second set of compressed domain error terms, the third compressed format being different from the first compressed format and including frames that are interframe compressed and intraframe compressed.