Hybrid encoding/decoding for remote gaming
Rendered image data is encoded by a server computing device and transmitted to a remote client device that executes an interactive application program. The client device decodes and displays the image data and, when the user interacts with the application program, the client device provides input control signals to the server computing device. When input control signals are received by the server, the latency incurred for encoding and/or decoding the image data is reduced. Therefore, the user does not experience inconsistencies in the frame rate of images displayed on the client when the user interacts with the application program. The reduction in latency is achieved by dynamically switching from a hardware implemented encoding technique to a software implemented encoding technique. Latency may also be reduced by dynamically switching from a hardware implemented decoding technique to a software implemented decoding technique.
1. A method for encoding image data for transmission to a client device, the method comprising:
encoding first image data using dedicated circuitry to produce encoded first image data;
transmitting the encoded first image data to the client device;
receiving, from the client device, an input control signal that is generated during execution of an interactive application program;
switching, in response to receiving the input control signal, from encoding the first image data using the dedicated circuitry to encoding second image data using at least one software encoder engine to produce encoded second image data; and
transmitting the encoded second image data to the client device.
2. The method of claim 1 , wherein encoding the second image data using at least one software encoder engine comprises:
instantiating a first software encoder engine that encodes a first portion of the second image data;
instantiating a second software encoder engine that encodes a second portion of the second image data;
executing, by a first processing core, the first software encoder engine to produce a first portion of the encoded second image data; and
executing, by a second processing core, the second software encoder engine to produce a second portion of the encoded second image data.
3. The method of claim 2 , wherein the first portion of the second image data partially overlaps the second portion of the second image data within a frame.
4. The method of claim 1 , wherein the encoded second image data comprises a first encoded portion and a second encoded portion, and further comprising:
decoding the first encoded portion of the encoded second image data to produce a first decoded portion of the second image data;
decoding the second encoded portion of the encoded second image data to produce a second decoded portion of the second image data; and
combining the first decoded portion with the second decoded portion to produce decoded second image data; and
displaying the decoded second image data at the client device.
5. The method of claim 4 , wherein the first decoded portion partially overlaps the second decoded portion within a frame and combining comprises blending the first decoded portion with the second decoded portion.
6. The method of claim 1 , further comprising:
decoding the encoded first image data using dedicated circuitry residing within the client device to produce decoded first image data;
displaying the decoded first image data at the client device;
determining that the client device is operating in a mode that minimizes power consumption;
decoding the encoded second image data using the dedicated circuitry residing within the client device to produce decoded second image data; and
displaying the decoded second image data at the client device.
7. The method of claim 1 , further comprising:
decoding the encoded first image data using the dedicated circuitry residing within the client device to produce decoded first image data;
displaying the decoded first image data at the client device;
determining that the client device is not operating in a mode that minimizes power consumption;
in response, switching from decoding the encoded first image data using the dedicated circuitry residing within the client device to decoding the encoded second image data using at least one software decoder engine on a second general purpose processor residing within the client device to produce decoded second image data; and
displaying the decoded second image data at the client device.
8. The method of claim 1 , further comprising:
rendering a first scene associated with the interactive application program to generate the first image data; and
rendering, based on the input control signal, a second scene associated with the interactive application program.
9. The method of claim 1 , wherein the interactive application program comprises a game application program.
10. The method of claim 1 , further comprising:
determining that a time duration has expired without receiving a subsequent input control signal from the client device;
in response, switching, in response to the time duration expiration, from encoding the second image data using the at least one software encoder engine to encoding third image data using the dedicated circuitry to produce encoded third image data; and
transmitting the encoded third image data to the client device.
11. A system, comprising:
a server machine that includes:
a processor; and
a memory coupled to the processor and storing an interactive application program, and a software encoder engine;
wherein the server machine is configured to:
encode first image data using dedicated circuitry to produce encoded first image data;
transmit the encoded first image data to a client device;
receive, from the client device, an input control signal that is generated during execution of the interactive application program;
switch, in response to receiving the input control signal, from encoding the first image data using the dedicated circuitry to encode second image data using the software encoder engine to produce encoded second image data; and
transmit the encoded second image data to the client device.
12. The system of claim 11 , wherein the server machines is configured to encode the second image data using the software encoder engine by:
instantiating a first software encoder engine that encodes a first portion of the second image data;
instantiating a second software encoder engine that encodes a second portion of the second image data;
executing, by a first processing core, the first software encoder engine to produce a first portion of the encoded second image data; and
executing, by a second processing core, the second software encoder engine to produce a second portion of the encoded second image data.
13. The system of claim 12 , wherein the first portion of the second image data partially overlaps the second portion of the second image data within a frame.
14. The system of claim 11 , wherein the encoded second image data comprises a first encoded portion and a second encoded portion, and further comprising the client device that includes:
a client processor; and
a client memory coupled to the client processor, wherein the client memory includes a software decoder engine that, when executed by the client processor, causes the client processor to generate encoded image data by:
decoding the first encoded portion of the encoded second image data to produce a first decoded portion of the second image data;
decoding the second encoded portion of the encoded second image data to produce a second decoded portion of the second image data; and
combining the first decoded portion with the second decoded portion to produce decoded second image data; and
displaying the decoded second image data at the client device.
15. The system of claim 14 , wherein the first decoded portion partially overlaps the second decoded portion within a frame, and combining comprises blending the first decoded portion with the second decoded portion.
16. The system of claim 11 , wherein the encoded second image data comprises a first encoded portion and a second encoded portion, and further comprising the client device that includes:
a client processor; and
a client memory coupled to the client processor, wherein the client memory includes a software decoder engine that, when executed by the client processor, causes the client processor to generate decoded image data by:
decoding the encoded first image data using dedicated circuitry residing within the client device to produce decoded first image data;
displaying the decoded first image data at the client device;
determining that the client device is operating in a mode that minimizes power consumption;
decoding the encoded second image data using the dedicated circuitry residing within the client device to produce decoded second image data; and
displaying the decoded second image data at the client device.
17. The system of claim 11 , wherein the encoded second image data comprises a first encoded portion and a second encoded portion, and further comprising the client device that includes:
a client processor; and
a client memory coupled to the client processor, wherein the client memory includes a software decoder engine that, when executed by the client processor, causes the client processor to generate decoded image data by:
decoding the encoded first image data using the dedicated circuitry residing within the client device to produce decoded first image data;
displaying the decoded first image data at the client device;
determining that the client device is not operating in a mode that minimizes power consumption;
in response, switching from decoding the encoded first image data using the dedicated circuitry residing within the client device to decoding the encoded second image data using at least one software decoder engine on a second general purpose processor residing within the client device to produce decoded second image data; and
displaying the decoded second image data at the client device.
18. The system of claim 11 , wherein there server machine is further configured to:
render a first scene associated with the interactive application program; and
render, based on the input control signal, a second scene associated with the interactive application program.
19. The system of claim 11 , wherein the interactive application program comprises a game application program.
20. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to transmit encoded image data to a client device, by performing the steps of:
causing first image data to be encoded using dedicated circuitry to produce encoded first image data;
causing the encoded first image data to be transmitted to the client device;
receiving, from the client device, an input control signal that is generated during execution of an interactive application program;
causing, in response to receiving the input control signal, encoding to be switched from encoding the first image data using the dedicated circuitry to encoding second image data using at least one software encoder engine to produce encoded second image data; and
causing the encoded second image data to be switched to the client device.