Granted Patent B2

US 12,296,257 · App. 17/692,071 · Granted May 13, 2025

Inventor: Timothy Cotter (Sunnyvale, CA)

Assignee: Sony Interactive Entertainment LLC

A63F13/355A63F13/86A63F13/87A63F13/795H04N21/2365

Abstract

Claims (81)

1. A method, comprising:

executing a plurality of instances of a video game by a plurality of game servers to generate a plurality of video streams, wherein the plurality of instances of the video game is executed responsive to user input of a plurality of users participating in a multi-player gaming session, wherein a plurality of gaming communication channels between the plurality of game servers provide for delivery of data related to gaming activity from each of the plurality of instances of the video game to other instances in the plurality of instances of the video game;

receiving at a compositing server from the plurality of game servers the plurality of video streams each of which is uncompressed and formatted for presentation on a corresponding display, wherein the plurality of video streams is accessed before being encoded for streaming to a plurality of client devices, wherein the plurality of video streams are delivered over a plurality of output communication paths directly connecting the plurality of game servers to the compositing server;

scaling by the compositing server each of the plurality of video streams to generate a plurality of thumbnail windows;

generating by the compositing server a composited video stream to combine the plurality of thumbnail windows into a single video stream;

encoding by an encoder the composited video stream, wherein the encoding is performed at an acceptable latency for the multi-player gaming session; and

streaming the composited video stream that is encoded to a client device for simultaneous viewing of the plurality of video streams corresponding with the plurality of thumbnail windows.

2. The method of claim 1 , wherein the generating a composited video stream includes:

overlaying each of the plurality of thumbnail windows at a corresponding location in the user interface.

3. The method of claim 1 , further comprising:

executing one or more applications on a plurality of servers of a cloud gaming system to generate the plurality of video streams.

4. The method of claim 3 ,

wherein each of the plurality of video streams includes live video.

5. The method of claim 1 , further comprising:

receiving from the client device a selection of a thumbnail window of the plurality of thumbnail windows in the user interface;

discontinuing streaming of the composited video stream to the client device;

generating an encoded video stream by encoding a video stream associated with the selected thumbnail window, wherein the video stream is unscaled; and

streaming the encoded video stream to the client device.

6. The method of claim 1 , further comprising:

streaming the composited video stream as the single video stream to a plurality of client devices.

7. The method of claim 1 , further comprising:

receiving from the client device a selection of a thumbnail window of the plurality of thumbnail windows in the user interface,

wherein each of the plurality of thumbnail windows is associated with a multi-player gaming session of a multi-player gaming application,

wherein the selection of the thumbnail window includes a request to join the multi-player gaming session,

instantiating an instance of the multi-player gaming application on a new server that is executed in response to control inputs received from the client device to generate a new video stream;

encoding the new video stream; and

streaming the new video stream that is encoded to the client device.

8. A non-transitory computer-readable medium storing a computer program for performing a method, the computer-readable medium comprising:

program instructions for executing a plurality of instances of a video game by a plurality of game servers to generate a plurality of video streams, wherein the plurality of instances of the video game is executed responsive to user input of a plurality of users participating in a multi-player gaming session, wherein a plurality of gaming communication channels between the plurality of game servers provide for delivery of data related to gaming activity from each of the plurality of instances of the video game to other instances in the plurality of instances of the video game;

program instructions for receiving at a compositing server from the plurality of game servers the plurality of video streams each of which is uncompressed and formatted for presentation on a corresponding display, wherein the plurality of video streams is accessed before being encoded for streaming to a plurality of client devices, wherein the plurality of video streams are delivered over a plurality of output communication paths directly connecting the plurality of game servers to the compositing server;

program instructions for scaling by the compositing server each of the plurality of video streams to generate a plurality of thumbnail windows;

program instructions for generating by the compositing server a composited video stream to combine the plurality of thumbnail windows into a single video stream;

program instructions for encoding by an encoder the composited video stream, wherein the encoding is performed at an acceptable latency for the multi-player gaming session; and

program instructions for streaming the composited video stream that is encoded to a client device for simultaneous viewing of the plurality of video streams corresponding with the plurality of thumbnail windows.

9. The non-transitory computer-readable medium of claim 8 , wherein the generating a composited video stream includes:

overlaying each of the plurality of thumbnail windows at a corresponding location in the user interface.

10. The non-transitory computer-readable medium of claim 8 , further comprising:

executing one or more applications on a plurality of servers of a cloud gaming system to generate the plurality of video streams.

11. The non-transitory computer-readable medium of claim 10 ,

wherein each of the plurality of video streams includes live video.

12. The non-transitory computer-readable medium of claim 8 , further comprising:

receiving from the client device a selection of a thumbnail window of the plurality of thumbnail windows in the user interface;

discontinuing streaming of the composited video stream to the client device;

generating an encoded video stream by encoding a video stream associated with the selected thumbnail window, wherein the video stream is unscaled; and

streaming the encoded video stream to the client device.

13. The non-transitory computer-readable medium of claim 8 , further comprising:

streaming the composited video stream as the single video stream to a plurality of client devices.

14. The non-transitory computer-readable medium of claim 8 , further comprising:

receiving from the client device a selection of a thumbnail window of the plurality of thumbnail windows in the user interface,

wherein each of the plurality of thumbnail windows is associated with a multi-player gaming session of a multi-player gaming application,

wherein the selection of the thumbnail window includes a request to join the multi-player gaming session,

instantiating an instance of the multi-player gaming application on a new server that is executed in response to control inputs received from the client device to generate a new video stream;

encoding the new video stream; and

streaming the new video stream that is encoded to the client device.

15. A computer system comprising:

a processor;

memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method, comprising:

executing a plurality of instances of a video game by a plurality of game servers to generate a plurality of video streams, wherein the plurality of instances of the video game is executed responsive to user input of a plurality of users participating in a multi-player gaming session, wherein a plurality of gaming communication channels between the plurality of game servers provide for delivery of data related to gaming activity from each of the plurality of instances of the video game to other instances in the plurality of instances of the video game;

receiving at a compositing server from the plurality of game servers the plurality of video streams each of which is uncompressed and formatted for presentation on a corresponding display, wherein the plurality of video streams is accessed before being encoded for streaming to a plurality of client devices, wherein the plurality of video streams are delivered over a plurality of output communication paths directly connecting the plurality of game servers to the compositing server;

scaling by the compositing server each of the plurality of video streams to generate a plurality of thumbnail windows;

generating by the compositing server a composited video stream to combine the plurality of thumbnail windows into a single video stream;

encoding by an encoder the composited video stream, wherein the encoding is performed at an acceptable latency for the multi-player gaming session; and

streaming the composited video stream that is encoded to a client device for simultaneous viewing of the plurality of video streams corresponding with the plurality of thumbnail windows.

16. The computer system of claim 15 , wherein in the method the generating a composited video stream includes:

overlaying each of the plurality of thumbnail windows at a corresponding location in the user interface.

17. The computer system of claim 15 , the method further comprising:

executing one or more applications on a plurality of servers of a cloud gaming system to generate the plurality of video streams.

18. The computer system of claim 15 , the method further comprising:

receiving from the client device a selection of a thumbnail window of the plurality of thumbnail windows in the user interface;

discontinuing streaming of the composited video stream to the client device;

generating an encoded video stream by encoding a video stream associated with the selected thumbnail window, wherein the video stream is unscaled; and

streaming the encoded video stream to the client device.

19. The computer system of claim 15 , the method further comprising:

streaming the composited video stream as the single video stream to a plurality of client devices.

20. The computer system of claim 15 , the method further comprising:

receiving from the client device a selection of a thumbnail window of the plurality of thumbnail windows in the user interface,

wherein each of the plurality of thumbnail windows is associated with a multi-player gaming session of a multi-player gaming application,

wherein the selection of the thumbnail window includes a request to join the multi-player gaming session,

instantiating an instance of the multi-player gaming application on a new server that is executed in response to control inputs received from the client device to generate a new video stream;

encoding the new video stream; and

streaming the new video stream that is encoded to the client device.

Assignments (4)

Continuity (3)

References Cited (13)