Granted Patent B2

US 12,226,690 · App. 18/081,931 · Granted Feb 18, 2025

Inventors: Dov Zimring (Belmont, CA); Paul Leventis (Mountain View, CA); Benjamin Frenkel (Mountain View, CA); Matthew Rodgers (San Jose, CA); Clinton Smullen (Mountain View, CA); Robert McCool (Menlo Park, CA)

Assignee: GOOGLE LLC

A63F13/358A63F13/323A63F13/335A63F13/40G06F9/45558A63F2300/534A63F2300/535

Abstract

Claims (95)

1. A method, comprising:

receiving an input event during a current game state of a game session executing on a gaming device;

determining a first frame that was displayed by the gaming device at a time of the input event, the first frame being one of a plurality of output frames transmitted for the game session prior to receiving the input event;

determining a first game state associated with the first frame, the first game state occurring prior to the current game state;

processing a gameplay output based on the input event in accordance with the determined first game state; and

transmitting a response frame for display by the gaming device based on the gameplay output.

2. The method of claim 1 , wherein determining the first game state comprises:

determining a game time index associated with the first frame; and

determining a game state that existed at a time corresponding to the game time index associated with the first frame.

3. The method of claim 1 , wherein processing the gameplay output comprises

providing the input event to a game engine that determines gameplay outputs based on input events and game states.

4. The method of claim 1 , wherein processing the gameplay output comprises:

introducing a new virtual object or modifying an existing virtual object;

including the new virtual object or the modified existing virtual object in the response frame; and

generating a subsequent game state in accordance with the new or modified virtual object.

5. The method of claim 1 , wherein the current game state is associated with a second frame reflecting a game state which is inconsistent with an intended game state reflecting the input event; and

wherein processing the gameplay output comprises reconciling the current game state with the intended game state.

6. The method of claim 5 , wherein reconciling the current game state with the intended game state comprises modifying a frame depicting the current game state to generate a subsequent frame depicting the intended game state.

7. The method of claim 1 , further comprising:

receiving, at a server during the game session, a plurality of display events from the gaming device, each respective display event being associated with frame identification data for a respective frame that was displayed by the gaming device during the display event;

wherein determining the first frame comprises:

determining which of the plurality of display events was received closest in time to the input event; and

identifying the first frame based on the frame identification data for the determined display event.

8. The method of claim 1 , wherein the input event is associated with a first timestamp indicating a timing of the input event; and

wherein determining the first frame comprises:

assigning a second timestamp to the input event indicating a time the input event was received at a server;

determining a first one-way transit time of the input event from the gaming device to the server based on a comparison of the first and second timestamps;

approximating a second one-way transit time from the server to the gaming device based on the first one-way transit time; and

identifying the first frame based on the approximated second one-way transit time.

9. The method of claim 1 , wherein the input event is associated with a first timestamp indicating a timing of the input event; and

wherein determining the first frame comprises:

including the first timestamp in metadata of the response frame; and

transmitting the response frame with the first timestamp to the gaming device for use in determining a controller-to-gaming device latency.

10. The method of claim 1 , further comprising, prior to receiving the input event:

rendering an output frame;

including a first timestamp in metadata of the output frame, the first timestamp indicating

a rendering timing or a transmission timing of the output frame; and

transmitting the output frame with the first timestamp to the gaming device for use in

determining a one-way server-to-gaming device latency;

wherein determining the first frame comprises:

receiving a display event from the gaming device at a server, the display event including an indication of the one-way server-to-gaming device latency; and

identifying the first frame based on the indicated one-way server-to-gaming device latency.

11. The method of claim 1 , further comprising, prior to receiving the input event:

rendering an output frame;

including a first timestamp in metadata of the output frame, the first timestamp indicating a rendering timing or a transmission timing of the output frame; and

transmitting the output frame with the first timestamp to the gaming device;

wherein determining the first frame comprises:

receiving a display event from the gaming device, the display event including the first timestamp and a second timestamp indicating a display timing of the output frame;

determining a one-way server-to-gaming device latency value based on a comparison of the first and second timestamps; and

identifying, as the first frame, the output frame based on the determined one-way server-to-gaming device latency value.

12. The method of claim 1 , further comprising, prior to receiving the input event:

rendering an output frame;

tagging the output frame with a first audio signal;

transmitting the output frame with the first audio signal to the gaming device; and

associating a transmission timing of the output frame with a first timestamp;

wherein determining the first frame comprises:

receiving, with the input event, (i) an acknowledgement by the gaming device of the first audio signal, and (ii) a second timestamp indicating a transmission timing of the input event from the gaming device to a server;

determining a one-way server-to-gaming device latency value based on a comparison of the first and second timestamps;

determining a one-way gaming device-to-server latency value based on a comparison of the one-way server-to-gaming device latency value and a round-trip-time (RTT) latency value between the server and the gaming device; and

identifying, as the first frame, the output frame based on the determined one-way gaming device-to-server latency value.

13. The method of claim 1 , further comprising, prior to receiving the input event:

rendering an output frame;

tagging the output frame with a first image pattern;

transmitting the output frame with the first image pattern to the gaming device; and

associating a transmission timing of the output frame with a first timestamp;

wherein determining the first frame comprises:

receiving, with the input event, (i) an acknowledgement, the image pattern acknowledgement by the gaming device of the first image pattern; and (ii) a second timestamp indicating a transmission timing of the input event from the gaming device to a server;

determining a one-way server-to-gaming device latency value based on a comparison of the first and second timestamps;

determining a one-way gaming device-to-server latency value based on a comparison of the one-way server-to-gaming device latency value and a round-trip-time (RTT) latency value between the server and the gaming device; and

identifying, as the first frame, the output frame based on the determined one-way gaming device-to-server latency value.

14. A server, comprising:

one or more processors; and

memory storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for:

receiving an input event during a current game state of a game session executing on a gaming client;

determining a first frame that was displayed by the gaming client at a time of the input event, the first frame being one of a plurality of output frames transmitted for the game session prior to receiving the input event;

determining a first game state associated with the first frame, the first game state occurring prior to the current game state;

processing a gameplay output based on the input event in accordance with the determined first game state; and

transmitting a response frame for display by the gaming client based on the gameplay output.

15. The server of claim 14 , wherein determining the first game state comprises:

determining a game time index associated with the first frame; and

determining a game state that existed at a time corresponding to the game time index associated with the first frame.

16. The server of claim 14 , wherein processing the gameplay output comprises providing the input event to a game engine that determines gameplay outputs based on input events and game states.

17. The server of claim 14 , wherein processing the gameplay output comprises:

introducing a new virtual object or modifying an existing virtual object;

including the new virtual object or the modified existing virtual object in the response frame; and

generating a subsequent game state in accordance with the new or modified virtual object.

18. The server of claim 14 , wherein the current game state is associated with a second frame reflecting a game state which is inconsistent with an intended game state reflecting the input event; and

wherein processing the gameplay output comprises reconciling the current game state with the intended game state.

19. The server of claim 18 , wherein reconciling the current game state with the intended game state comprises modifying a frame depicting the current game state to generate a subsequent frame depicting the intended game state.

20. A non-transitory computer readable storage medium storing instructions that, when executed by one or more processors of a gaming server, cause the gaming server to:

receive an input event during a current game state of a game session executing on a gaming client;

determine a first frame that was displayed by the gaming client at a time of the input event, the first frame being one of a plurality of output frames transmitted for the game session prior to receiving the input event;

determine a first game state associated with the first frame, the first game state occurring prior to the current game state;

process a gameplay output based on the input event in accordance with the determined first game state; and

transmit a response frame for display by the gaming client based on the gameplay output.

Assignments (1)

Continuity (4)

References Cited (56)