Determining targeted display time of a video frame for a client based on server side timing information
A method for cloud gaming. The method including generating a video frame when executing a video game at a server. The method including performing a scan-out process to deliver the video frame to an encoder configured to compress the video frame, wherein the scan-out process begins at a flip-time of the video frame. The method including transmitting the video frame that is compressed to a client. The method including determining at the client a target display time for the video frame. The method including scheduling at the client a display time for the video frame based on the target display time.
1. A method of cloud gaming, comprising:
aligning a server VSYNC signal of a server and a client VSYNC signal of a client;
receiving at the client from the server a video frame that is encoded;
receiving at the client from the server timing information for the video frame;
determining a target display time of the video frame based on the timing information, wherein the target display time is a targeted occurrence of the server VSYNC signal;
translating the target display time to a targeted occurrence of the client VSYNC signal; and
scheduling a display time for the video frame based on the targeted occurrence of the client VSYNC signal.
2. The method of claim 1 ,
wherein the timing information includes a time at which scan-out of the video frame started at the server.
3. The method of claim 1 ,
wherein the timing information identifies the targeted occurrence of the server VSYNC signal.
4. The method of claim 1 ,
wherein the display time is a fractional occurrence of the targeted occurrence of the client VSYNC signal.
5. The method of claim 1 , further comprising:
receiving in the timing information a flip-time and a simulation time associated with the video frame; and
determining the targeted occurrence of the server VSYNC signal based on the flip-time and the simulation time,
wherein scan-out of the video frame to an encoder occurs at the flip-time,
wherein the simulation time indicates a time taken to generate the video frame.
6. The method of claim 1 , further comprising:
providing an offset between the server VSYNC signal and the client VSYNC signal,
wherein the offset is determined based on a predetermined number of video frames that are received right before a corresponding next occurrence of the client VSYNC signal,
wherein the server VSYNC signal and the client VSYNC signal are aligned to a frequency.
7. The method of claim 1 , further comprising:
sending the timing information in a data control packet using a graphics pipeline unit (GPU) application programming interface (API).
8. A non-transitory computer-readable medium storing a computer program for cloud gaming, the computer-readable medium comprising:
program instructions for aligning a server VSYNC signal of a server and a client VSYNC signal of a client;
program instructions for receiving at the client from the server a video frame that is encoded;
program instructions for receiving at the client from the server timing information for the video frame;
program instructions for determining a target display time of the video frame based on the timing information, wherein the target display time is a targeted occurrence of the server VSYNC signal;
program instructions for translating the target display time to a targeted occurrence of the client VSYNC signal; and
program instructions for scheduling a display time for the video frame based on the targeted occurrence of the client VSYNC signal.
9. The non-transitory computer-readable medium of claim 8 ,
wherein in the method the timing information includes a time at which scan-out of the video frame started at the server.
10. The non-transitory computer-readable medium of claim 8 ,
wherein in the method the timing information identifies the targeted occurrence of the server VSYNC signal.
11. The non-transitory computer-readable medium of claim 8 ,
wherein in the method the display time is a fractional occurrence of the targeted occurrence of the client VSYNC signal.
12. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for receiving in the timing information a flip-time and a simulation time associated with the video frame; and
program instructions for determining the targeted occurrence of the server VSYNC signal based on the flip-time and the simulation time,
wherein scan-out of the video frame to an encoder occurs at the flip-time,
wherein the simulation time indicates a time taken to generate the video frame.
13. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for providing an offset between the server VSYNC signal and the client VSYNC signal,
wherein the offset is determined based on a predetermined number of video frames that are received right before a corresponding next occurrence of the client VSYNC signal,
wherein the server VSYNC signal and the client VSYNC signal are aligned to a frequency.
14. The non-transitory computer-readable medium of claim 8 , further comprising:
program instructions for sending the timing information in a data control packet using a graphics pipeline unit (GPU) application programming interface (API).
15. A computer system comprising:
a processor; and
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for cloud gaming comprising:
aligning a server VSYNC signal of a server and a client VSYNC signal of a client;
receiving at the client from the server a video frame that is encoded;
receiving at the client from the server timing information for the video frame;
determining a target display time of the video frame based on the timing information, wherein the target display time is a targeted occurrence of the server VSYNC signal;
translating the target display time to a targeted occurrence of the client VSYNC signal; and
scheduling a display time for the video frame based on the targeted occurrence of the client VSYNC signal.
16. The computer system of claim 15 ,
wherein in the method the timing information includes a time at which scan-out of the video frame started at the server.
17. The computer system of claim 15 ,
wherein in the method the timing information identifies the targeted occurrence of the server VSYNC signal.
18. The computer system of claim 15 ,
wherein in the method the display time is a fractional occurrence of the targeted occurrence of the client VSYNC signal.
19. The computer system of claim 15 , the method further comprising:
receiving in the timing information a flip-time and a simulation time associated with the video frame; and
determining the targeted occurrence of the server VSYNC signal based on the flip-time and the simulation time,
wherein scan-out of the video frame to an encoder occurs at the flip-time,
wherein the simulation time indicates a time taken to generate the video frame.
20. The computer system of claim 15 , the method further comprising:
providing an offset between the server VSYNC signal and the client VSYNC signal,
wherein the offset is determined based on a predetermined number of video frames that are received right before a corresponding next occurrence of the client VSYNC signal,
wherein the server VSYNC signal and the client VSYNC signal are aligned to a frequency.