IP Library Granted Patent US 9,968,847
Granted Patent B2
US 9,968,847 · App. 15/591,062 · Granted May 15, 2018

System and method for compressing video frames or portions thereof based on feedback information from a client device

Inventors: Stephen G. Perlman (Palo Alto, CA); Roger van der Laan (Menlo Park, CA); Timothy Cotter (Sunnyvale, CA); Scott Furman (Menlo Park, CA); Robert McCool (Menlo Park, CA); Ian Buckley (San Carlos, CA)
Assignee: Sony Interactive Entertainment America LLC
A63F13/355A63F13/25A63F13/335A63F13/358A63F13/71H04L65/605H04L67/38H04N19/146H04N21/234363H04N21/25833H04N21/2662H04N21/658H04N21/6583H04N21/6587A63F2300/407A63F2300/534A63F2300/538H04N19/172H04N19/188
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,968,847
App. No.
15/591,062
Granted
May 15, 2018
Kind
B2
Abstract

A computer-implemented system and method are described for performing video compression by an online video game service is provided. One example method includes receiving user input by a server and the user input is received from a client device over a network established between the client device and the server. The user input is from an input device associated with the client device and the client device and the input device are used to access the server to play a video game. The method further includes executing the video game on the server while receiving user input from the input device, and the video game produces a video output including a sequence of video images of the video game. Encoding each of the sequence of video images in a video encoding resource of the server to generate a sequence of encoded video frames and transmitting the sequence of encoded video frames by the server over the communication channel to the client device for decoding to produce a corresponding sequence of decoded video frames. The sequence of decoded video frames is configured to be rendered on a display device associated with the client device. The method further includes detecting a change in a communication channel characteristic of the communication channel, and the detected change is based on feedback information received by the server. The feedback information is received from the client device and dynamically adjusting a compression ratio for said encoding of the sequence of encoded video frames based on the detected change in the communication channel characteristic.

Claims (48)

1. A computer-implemented method for performing video compression by an online video game service, comprising:

receiving user input by a server, the user input received from a client device over a network established between the client device and the server, the user input being from an input device associated with the client device, the client device and the input device used to access the server to play a video game, the network including a communication channel between the client device and the server;

executing the video game on the server while receiving user input from the input device, the video game producing a video output including a sequence of video images of the video game;

encoding each of the sequence of video images in a video encoding resource of the server to generate a sequence of encoded video frames;

transmitting the sequence of encoded video frames by the server over the communication channel to the client device for decoding to produce a corresponding sequence of decoded video frames, the sequence of decoded video frames are configured to be rendered on a display device associated with the client device;

detecting a change in a communication channel characteristic of the communication channel, wherein the detected change is based on feedback information received by the server, the feedback information being received from the client device; and

dynamically adjusting a compression ratio for said encoding of the sequence of encoded video frames based on the detected change in the communication channel characteristic.

2. The computer-implemented method of claim 1 , wherein, the adjusted compression ratio is based on selection of a first encoding format or a second encoding format.

3. The computer-implemented method of claim 1 , wherein the feedback information comprises an indication that the encoded sequence of video frames have been successfully received or decoded at the client device.

4. The computer-implemented method of claim 1 , wherein the feedback information comprises an indication that at least one of the encoded sequence of video frames have not been successfully received or decoded at the client device.

5. The computer-implemented method of claim 1 , wherein the sequence of encoded video frames comprise images of the video game rendered by the display device associated with the client device.

6. The computer-implemented method of claim 1 , wherein the feedback information includes at least one of an indication that some or all of the sequence of encoded video frames have been decoded at the client device or an indication that some or all of the images associated with each of the sequence of encoded video frames have been displayed by the display device associated with the client device.

7. The computer-implemented method of claim 1 , wherein the video game is a multiplayer video game and the client device is one of a plurality of client devices and wherein the communication channel is one of a plurality of communication channels established between the server or additional servers associated with the server and a respective each one of the plurality of client devices;

wherein detecting the change in the communication channel characteristic of the communication channel includes detecting a first communication channel between the server and a respective first client device of the plurality of client devices is inoperative, wherein the first communication channel is one of the plurality of communication channels; and

notifying at least one of the plurality of client devices that the first client device is not available.

8. The computer-implemented method of claim 7 , further comprising, pausing the execution of the multiplayer video game by the server for the plurality of client devices, when the first communication channel is inoperative.

9. The computer-implemented method of claim 1 , further comprising:

determining that a maximum data rate capacity of the communication channel has decreased based on at least one of a detected increase in packet loss or an increase in latency while the sequence of encoded video frames are being transmitted in a streaming format to the client device; and

responsively and dynamically reducing the data rate of the sequence of encoded video frames until at least one of the packet loss or latency have reached corresponding predetermined acceptable values.

10. A computer-implemented method for performing video compression by an online video game service, comprising:

receiving user input by a server, the user input received from a client device over a network established between the client device and the server, the user input being from an input device associated with the client device, the client device and the input device used to access the server to play a video game, the network including a communication channel between the client device and the server;

executing the video game on the server while receiving user input from the input device, the video game producing a video output including a sequence of video images of the video game;

encoding each of the sequence of video images in a video encoding resource of the server to generate a sequence of encoded video frames;

transmitting the sequence of encoded video frames by the server over the communication channel to the client device for decoding to produce a corresponding sequence of decoded video frames, the sequence of decoded video frames are configured to be rendered on a display device associated with the client device;

detecting a change in a communication channel characteristic of the communication channel, wherein the detected change is based on feedback information received by the server, the feedback information being received from the client device; and

dynamically adjusting an image resolution of the sequence of encoded video frames based on the detected change in the communication channel characteristic; and

transmitting an instruction to the client device to scale a display of each one of the sequence of decoded video frames having said adjusted image resolution to maintain an image size corresponding to the display device associated with the client device.

11. The computer-implemented method of claim 10 , wherein, the adjusted image resolution of the sequence of encoded video frames is based on selection of a first encoding format or a second encoding format.

12. The computer-implemented method of claim 10 , wherein the feedback information comprises an indication that the encoded sequence of video frames have been successfully received or decoded at the client device.

13. The computer-implemented method of claim 10 , wherein the sequence of encoded video frames comprise images of the video game displayed by the display device associated with the client device.

14. The computer-implemented method of claim 10 , wherein the video game is a multiplayer video game and the client device is one of a plurality of client devices and wherein the communication channel is one of a plurality of communication channels established between the server and a respective each one of the plurality of client devices;

wherein detecting the change in the communication channel characteristic of the communication channel includes detecting a first communication channel between the server and a respective first client device of the plurality of client devices is inoperative or falling below an acceptable level, wherein the first communication channel is one of the plurality of communication channels; and

notifying at least one of the plurality of client devices that the first client device is not available or will not be available.

15. The computer-implemented method of claim 10 , further comprising:

determining that a maximum data rate capacity of the communication channel has decreased based on an indication of an increased number of the encoded sequence of video frames that have not been successfully received or decoded at the client device while the sequence of encoded video frames are being transmitted in a streaming format to the client device; and

responsively and dynamically reducing the data rate of the sequence of encoded video frames until the number of the encoded sequence of video frames that have not been successfully received or decoded at the client device have reached a corresponding predetermined acceptable value.

16. A computer-implemented method for performing video compression by an online video game service, comprising:

receiving user input by a server, the user input received from a client device over a network established between the client device and the server, the user input being from an input device associated with the client device, the client device and the input device used to access the server to play a video game, the network including a communication channel between the client device and the server;

executing the video game on the server while receiving user input from the input device, the video game producing a video output including a sequence of video images of the video game;

encoding each of the sequence of video images in a video encoding resource of the server to generate a sequence of encoded video frames;

transmitting the sequence of encoded video frames by the server over the communication channel to the client device for decoding to produce a corresponding sequence of decoded video frames, the sequence of decoded video frames are configured to be rendered on a display device associated with the client device;

determining a maximum data rate capacity of the communication channel by sending an increasingly greater data rate stream of the sequence of encoded video frames to the client device until at least one of a packet loss or an increased latency indicates the maximum data rate capacity of the communication channel has been exceeded; and

continuing to transmit the sequence of encoded video frames during a session of playing the video game.

17. The computer-implemented method of claim 16 , further comprising:

determining that the maximum data rate capacity of the communication channel has decreased based on at least one of a detected increase in packet loss or an increase in latency while the sequence of encoded video frames are being streamed to the client device; and

responsively and dynamically reducing the data rate or increasing a compression ratio of the sequence of encoded video frames until at least one of the packet loss or the latency have reached corresponding predetermined acceptable values.

18. The computer-implemented method of claim 17 , wherein responsively and dynamically reducing the data rate or increasing the compression ratio of the sequence of encoded video frames includes modifying a frame rate of the sequence of encoded video frames.

19. The computer-implemented method of claim 17 , wherein responsively and dynamically reducing the data rate or increasing the compression ratio of the sequence of encoded video frames includes modifying an image resolution of the sequence of encoded video frames.

Assignments (1)
MERGER Recorded Mar 30, 2020
From: SONY INTERACTIVE ENTERTAINMENT AMERICA LLC
To: SONY INTERACTIVE ENTERTAINMENT LLC
Reel/Frame 053323/0567 →
Continuity (7)
Continuation 15097076 · Apr 12, 2016
Continuation 12538054 · Aug 7, 2009
Continuation In Part 12359150 · Jan 23, 2009
Continuation 11999475 · Dec 5, 2007
Continuation In Part 10315460 · Dec 10, 2002
Provisional Application 61210888 · Mar 23, 2009
Related Publication 20170239566A1 · Aug 24, 2017