Architecture for combining media processing with networking
View Patent ↗Systems and methods for processing media streams for transport over a network based on network conditions. An integrated circuit comprises a media processing unit coupled to receive feedback from a network processing unit. The media processing unit converts a media stream from a compressed input stream to a compressed output stream such that the compressed output stream has characteristics that are best suited for the network conditions. Network conditions can include, for example, characteristics of the network (e.g., latency or bandwidth) or characteristics of the remote playback devices (e.g., playback resolution). Changes in the network conditions can result in a change in the conversion process.
1. An integrated circuit for converting media streams for transport over a network, comprising:
a media processing unit, coupled to the network to receive information on network conditions, the media processing unit configured to:
extract motion parameters from an input media stream in a first compressed format responsive to receiving the input media stream, the first compressed format including input frames that are interframe compressed and intraframe compressed;
convert the input media stream in the first compressed format into an intermediate media stream in a second format by applying the extracted motion parameters to the input frames and generating intermediate frames that are intraframe compressed but not interframe compressed;
perform processing on the intermediate frames based on the network conditions information; and
encode the processed intermediate media stream into an output media stream in a third compressed format for transport over the network by applying the extracted motion parameters to the processed intermediate media stream, the third compressed format including processed frames that are interframe compressed and intraframe compressed.
2. The integrated circuit of claim 1 , wherein the network conditions information comprises information related to capabilities of a screen in a networked device receiving the output media stream over the network.
3. The integrated circuit of claim 2 , wherein the display capabilities comprise an output resolution of the networked device.
4. The integrated circuit of claim 2 , wherein the display capabilities comprise a decoding capability of the networked device.
5. The integrated circuit of claim 1 , wherein the network conditions information comprise information related to throughput of the network.
6. The integrated circuit of claim 1 , wherein the media processing unit comprises:
a controller configured to determine scaling parameters based on the network conditions information; and
a scaler configured to scale frames in the intermediate media stream for encoding based on the scaling parameters.
7. The integrated circuit of claim 6 , wherein the scaler is further configured to scale the frames based on the motion parameters.
8. The integrated circuit of claim 1 wherein the media processing unit is further configured to generate encoding parameters and quantization parameters for encoding the output media stream based on the network conditions information.
9. The integrated circuit of claim 1 , wherein the media processing unit is configured to generate macroblock mode decision parameters for encoding frames of the output media stream.
10. The integrated circuit of claim 1 , further comprising:
a network conditions detector, coupled to the media processing unit, the network conditions detector configured to insert probe packets into an outgoing packet stream to the network, and to analyze response packets in an incoming packet stream from the network, to determine network conditions.
11. The integrated circuit of claim 10 , wherein the response packets are generated by a networked device receiving the output media stream.
12. The integrated circuit of claim 10 , wherein the network conditions detector measures a round trip time for a probe packet and a response packet in response to the probe packet.
13. A method for converting media streams in an integrated circuit for transport over a network, comprising:
receiving information on network conditions of the network;
receiving an input media stream in a first compressed format including input frames that are interframe compressed and intraframe compressed;
extracting motion parameters from an input media stream responsive to receiving the input media stream;
converting, by a media processing unit in the integrated circuit, the input media stream in the first compressed format into an intermediate media stream in a second format by applying the extracted motion parameters to the input frames and generating intermediate frames that are intraframe compressed but not interframe compressed;
performing, by the media processing unit, processing on the intermediate frames based on the network conditions information; and
encoding, by the media processing unit, the processed intermediate media stream into an output media stream in a third compressed format for transport over the network by applying the extracted motion parameters to the processed intermediate media stream, the third compressed format including processed frames that are interframe compressed and intraframe compressed.
14. The method of claim 13 , wherein the network conditions information comprises information related to display capabilities of a networked device.
15. The method of claim 14 , wherein the display capabilities comprise an output resolution of the networked device.
16. The method of claim 14 , wherein the display capabilities comprise a decoding capability of the networked device.
17. The method of claim 13 , wherein the network conditions information comprise information related to throughput of the network.
18. The method of claim 13 , further comprising:
determining scale parameters based on the network conditions information; and
scaling frames in the intermediate media stream for encoding based on the scaling parameters.
19. The method of claim 18 , wherein the scaling of the frames is based further on the extracted motion parameters.
20. The method of claim 13 , further comprising generating encoding parameters and quantization parameters for encoding the output media stream based on the network conditions information.
21. The method of claim 13 , further comprising generating macroblock mode decision parameters for encoding frames of the output media stream.
22. The method of claim 13 , further comprising:
inserting probe packets into an outgoing packet stream to the network; and
analyzing response packets in an incoming packet stream from the network, to determine network conditions.
23. The method of claim 22 , further comprising:
prompting a response packet from a networked device receiving the output media stream, the response packet containing information related to conditions of the networked device.
24. The method of claim 22 , further comprising:
measuring a round trip time for a probe packet and a response packet in response to the probe packet.
25. A system for converting media streams for transport over a network, comprising:
an integrated circuit coupled to the network and comprising a media processing unit coupled to receive information on network conditions of a network processing unit, the media processing unit configured to:
extract motion parameters from an input media stream in a first compressed format responsive to receiving the input media stream, the first compressed format including input frames that are interframe compressed and intraframe compressed;
convert the input media stream in the first compressed format into an intermediate media stream in a second format by applying the extracted motion parameters to the input frames and generating intermediate frames that are intraframe compressed but not interframe compressed;
perform processing on the intermediate frames based on the network conditions information; and
encode the processed intermediate media stream into an output media stream in a third compressed format for transport over the network by applying the extracted motion parameters to the processed intermediate media stream, the third compressed format including processed frames that are interframe compressed and intraframe compressed ; and
a networked device coupled to the network to receive the output media stream, the networked device configured to send response packets to the network processing unit in response to probe packets received from the integrated circuit, the response packets containing information related to display capabilities of the networked device.
26. The integrated circuit of claim 1 , wherein applying the extracted motion parameters to the processed intermediate media stream comprises reusing or refining the extracted motion parameters.
27. The method of claim 13 , wherein applying the extracted motion parameters to the processed intermediate media stream comprises reusing or refining the extracted motion parameters.
28. The system of claim 25 , wherein applying the extracted motion parameters to the processed intermediate media stream comprises reusing or refining the extracted motion parameters.
29. The integrated circuit of claim 1 , wherein the processing on the intermediate frames comprises modifying at least one of bitrate, frame rate and resolution.
30. The method of claim 13 , wherein the processing on the intermediate frames comprises modifying at least one of bitrate, frame rate and resolution.
31. The system of claim 25 , wherein the processing on the intermediate frames comprises modifying at least one of bitrate, frame rate and resolution.