Granted Patent B2

US 12,447,412 · App. 17/944,589 · Granted Oct 21, 2025

Inventor: Shaobin Lin (Shenzhen, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

A63F13/85A63F13/537A63F13/58A63F13/86H04N21/44008

Abstract

Claims (95)

1. A commentary video generation method performed by a commentary server, the commentary video generation method comprising:

obtaining a game instruction frame, the game instruction frame comprising at least one game operation instruction, and the game operation instruction being used for controlling a virtual object to perform an in-game behavior in a game;

generating a commentary data stream based on the game instruction frame, the commentary data stream comprising at least one piece of commentary audio describing a game event, and the game event being triggered during the virtual object performing the in-game behavior;

rendering a game screen based on the game instruction frame and a global game screen to generate a game video stream, the game video stream corresponding to a target game viewing angle of the global game scene based on a target game screen; and

generating a commentary video stream corresponding to the target game viewing angle based on combining the commentary data stream and the game video stream, the game video frame and the commentary audio corresponding to the same game event in the commentary video stream being aligned in time.

2. The commentary video generation method according to claim 1 , wherein the combining comprises:

determining a target game video frame in the game video stream, wherein the target game video frame is any one game video frame in the game video stream;

determining a game time corresponding to the target game video frame as a target game time, wherein the target game time is the time elapsing from the start of the game to the target game video frame;

determining the game instruction frame generated within the target game time as a target game instruction frame and determining a target frame number of the target game instruction frame;

determining the game event corresponding to the target frame number as a target game event and using the commentary audio for describing the target game event as target commentary audio;

aligning the target commentary audio with the target game video frame in time; and

generating the commentary video stream based on the target commentary audio and the target game video frame that are aligned in time.

3. The commentary video generation method according to claim 2 , wherein the game instruction frame corresponds to a first frame rate; and

the determining a target frame number of the target game instruction frame comprises:

determining the target frame number of the target game instruction frame based on the target game time and the first frame rate.

4. The commentary video generation method according to claim 2 , wherein the determining a game time comprises:

utilizing an image recognition model to perform image recognition on the game time in the target game video frame to obtain an image recognition result; and

determining the game time corresponding to the target game video frame based on the image recognition result and using the determined game time as the target game time.

5. The commentary video generation method according to claim 4 , wherein a frame rate of the game video stream is a second frame rate; and

the determining the game time corresponding to the target game video frame comprises:

determining a basic game time corresponding to the target game video frame based on the image recognition result;

determining a game time offset based on historical recognition times of the basic game time and the second frame rate, wherein the historical recognition times of the basic game time refer to the number of times that the basic game time is recognized within a historical recognition period; and

determining a sum of the basic game time and the game time offset as the target game time.

6. The commentary video generation method according to claim 1 , wherein the generating a commentary data stream comprises:

obtaining a preset game event set, wherein the game event set comprises a plurality of preset game events;

controlling the virtual object to perform the in-game behavior in the game based on the game instruction frame;

determining attribute information of virtual objects in the game after the in-game behavior is performed;

selecting at least one candidate game event matching the attribute information from the plurality of preset game events;

selecting the target game event from the at least one candidate game event; and

generating a commentary text based on the target game event and performing text-to-speech processing on the commentary text to generate the commentary data stream.

7. The commentary video generation method according to claim 6 , wherein the selecting at least one candidate game event comprises:

matching the attribute information with preset attribute information of the preset game events in the game event set, to obtain target preset attribute information matching the attribute information; and

determining the candidate game event based on the preset game event corresponding to the target preset attribute information.

8. The commentary video generation method according to claim 7 , wherein the determining the candidate game event comprises:

determining the preset game event corresponding to the target preset attribute information, and using the preset game event that meets a preset commentary condition in the preset game event corresponding to the target preset attribute information as the candidate game event, wherein the preset commentary condition comprises at least one of a game angle condition or an event repeat condition, the game angle condition means that the preset game event is within a game viewing angle, and the event repeat condition means that the number of times that the preset game event occurs within a preset duration is less than a threshold of times.

9. The commentary video generation method according to claim 6 , wherein the selecting the target game event comprises:

obtaining event weights corresponding to the candidate game events;

determining event scores corresponding to the candidate game events based on importance of the candidate game events in the game, wherein the importance is related to at least one of the following: a location where the candidate game event occurs, a virtual object type that triggers the candidate game event, and the number of virtual objects that trigger the candidate game event;

weighting the event scores by the event weights to obtain an event weighted scores corresponding to the candidate game events; and

determining the candidate game event with the highest event weighted score as the target game event.

10. The commentary video generation method according to claim 1 , wherein the rendering a game screen comprises:

rendering the game screen based on the game instruction frame to obtain the global game screen;

determining the target game viewing angle in game viewing angles; and

extracting the target game screen from the global game screen based on the target game viewing angle, and generating the game video stream corresponding to the target game viewing angle based on the target game screen, wherein a plurality of game viewing angles correspond to a plurality of game video streams; and

the combining the commentary data stream with the game video stream to generate the commentary video stream comprises:

combining the plurality of game video streams with the commentary data stream to generate a plurality of commentary video streams corresponding to the plurality of game viewing angles.

11. A commentary video generation apparatus, comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:

obtaining code configured to cause the at least one processor to obtain a game instruction frame, the game instruction frame comprising at least one game operation instruction, and the game operation instruction being used for controlling a virtual object to perform an in-game behavior in a game;

first generation code configured to cause the at least one processor to generate a commentary data stream based on the game instruction frame, the commentary data stream comprising at least one piece of commentary audio describing a game event, and the game event being triggered during the virtual object performing the in-game behavior;

second generation code configured to cause the at least one processor to render a game screen based on the game instruction frame and a global game screen to generate a game video stream, the game video stream corresponding to a target game viewing angle of the global game scene based on a target game screen; and

third generation code configured to cause the at least one processor to generate a commentary video stream corresponding to the target game viewing angle based on combining the commentary data stream and the game video stream, the game video frame and the commentary audio corresponding to the same game event in the commentary video stream being aligned in time.

12. The commentary video generation apparatus according to claim 11 , wherein the third generation code further comprises:

first determining code configured to cause the at least one processor to:

determine a target game video frame in the game video stream, wherein the target game video frame is any one game video frame in the game video stream; and

determine a game time corresponding to the target game video frame as a target game time, wherein the target game time is the time elapsing from the start of the game to the target game video frame;

second determining code configured to cause the at least one processor to determine the game instruction frame generated within the target game time as a target game instruction frame and determine a target frame number of the target game instruction frame; and

time alignment code configured to cause the at least one processor to:

determine the game event corresponding to the target frame number as a target game event, and use the commentary audio for describing the target game event as target commentary audio; align the target commentary audio with the target game video frame in time; and

generate the commentary video stream based on the target commentary audio and the target game video frame that are aligned in time.

13. The commentary video generation apparatus according to claim 12 , wherein the game instruction frame corresponds to a first frame rate, and

the second determining code is further configured to cause the at least one processor to determine the target frame number of the target game instruction frame based on the target game time and the first frame rate.

14. The commentary video generation apparatus according to claim 12 , wherein the first determining code is further configured to cause the at least one processor to:

utilize an image recognition model to perform image recognition on the game time in the target game video frame to obtain an image recognition result; and

determine the game time corresponding to the target game video frame based on the image recognition result and use the determined game time as the target game time.

15. The commentary video generation apparatus according to claim 14 , wherein a frame rate of the game video stream is a second frame rate; and

the first determining code is further configured to cause the at least one processor to:

determine a basic game time corresponding to the target game video frame based on the image recognition result;

determine a game time offset based on historical recognition times of the basic game time and the second frame rate; and

determine a sum of the basic game time and the game time offset as the target game time, wherein the historical recognition times of the basic game time refer to the number of times that the basic game time is recognized within a historical recognition period.

16. The commentary video generation apparatus according to claim 11 , wherein the first generation code further comprises:

third determining code configured to cause the at least one processor to:

obtain a preset game event set, wherein the game event set comprises a plurality of preset game events;

control the virtual object to perform the in-game behavior in the game based on the game instruction frame; and

determine attribute information of virtual objects in the game after the in-game behavior is performed;

fourth determining code configured to cause the at least one processor to select at least one candidate game event matching the attribute information from the plurality of preset game events;

screening code configured to cause the at least one processor to select the target game event from the at least one candidate game event; and

first generation code configured to cause the at least one processor to generate a commentary text based on the target game event and perform text-to-speech processing on the commentary text to generate the commentary data stream.

17. The commentary video generation apparatus according to claim 16 , wherein the fourth determining code is further configured to cause the at least one processor to

match the attribute information with preset attribute information of the preset game events in the game event set to obtain target preset attribute information matching the attribute information; and

determine the candidate game event based on the preset game event corresponding to the target preset attribute information.

18. The commentary video generation apparatus according to claim 17 , wherein the fourth determining code is further configured to cause the at least one processor to determine the preset game event corresponding to the target preset attribute information, and use the preset game event that meets a preset commentary condition in the preset game event corresponding to the target preset attribute information as the candidate game event, wherein the preset commentary condition comprises at least one of a game angle condition or an event repeat condition, the game angle condition means that the preset game event is within a game viewing angle, and the event repeat condition means that the number of times that the preset game event occurs within a preset duration is less than a threshold of times.

19. A non-transitory computer-readable storage medium, storing computer code that when executed by at least one processor causes the at least one processor to:

obtain a game instruction frame, the game instruction frame comprising at least one game operation instruction, and the game operation instruction being used for controlling a virtual object to perform an in-game behavior in a game;

generate a commentary data stream based on the game instruction frame, the commentary data stream comprising at least one piece of commentary audio describing a game event, and the game event being triggered during the virtual object performing the in-game behavior;

render a game screen based on the game instruction frame and a global game screen to generate a game video stream, the game video stream corresponding to a target game viewing angle of the global game scene based on a target game screen; and

generate a commentary video stream corresponding to the target game viewing angle based on combining the commentary data stream and the game video stream, the game video frame and the commentary audio corresponding to the same game event in the commentary video stream being aligned in time.

20. The non-transitory computer-readable storage medium according to claim 19 , wherein the combine the commentary data stream with the game video stream comprises:

determining a target game video frame in the game video stream, wherein the target game video frame is any one game video frame in the game video stream;

determining a game time corresponding to the target game video frame as a target game time, wherein the target game time is the time elapsing from the start of the game to the target game video frame;

determining the game instruction frame generated within the target game time as a target game instruction frame and determining a target frame number of the target game instruction frame;

determining the game event corresponding to the target frame number as a target game event and using the commentary audio for describing the target game event as target commentary audio;

aligning the target commentary audio with the target game video frame in time; and

generating the commentary video stream based on the target commentary audio and the target game video frame that are aligned in time.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (11)