Granted Patent B2

US 12,744,958 · App. 19/099,533 · Granted Sep 22, 2026

Inventors: Jianwei Li (Los Angeles, CA); Xiao Yang (Los Angeles, CA)

Assignee: Lemon Inc.

H04N21/43072G06V40/168H04N21/439H04N21/44008H04N21/8547

Abstract

Claims (82)

1 . A video determination method applied to a client, the method comprising:

acquiring, in response to an effect trigger operation, a first facial image comprising a first object;

determining a first audio, and determining a key video frame sequence corresponding to the first audio;

determining, based on the key video frame sequence and the first facial image, a first facial feature in the first facial image that is presented when the first audio is played; and

determining a first effect audio and video based on the first facial feature and the first audio,

wherein the key video frame sequence comprises at least one video frame, and a facial feature of a user in the video frame is inconsistent with a second facial feature.

2 . The method according to claim 1 , wherein the effect trigger operation comprises at least one of the following:

triggering an effect prop;

a frame-in picture comprising the first object;

triggering an effect wake-up word using audio information; and

a current body movement being consistent with a first effect movement.

3 . The method according to claim 1 , wherein the determining a first audio comprises:

displaying at least one second audio, and determining the first audio based on a trigger operation on the at least one second audio within first duration; or

receiving an uploaded third audio as the first audio.

4 . The method according to claim 1 , further comprising:

determining the key video frame sequence corresponding to the first audio based on a pre-selected first language type corresponding to the first audio.

5 . The method according to claim 1 , wherein the determining a key video frame sequence corresponding to the first audio comprises:

retrieving the key video frame sequence corresponding to the first audio from a pre-determined key video frame sequence library, wherein the first audio is determined from at least one second audio that is displayed in a display interface, and the key video frame sequence library comprises a corresponding key video frame sequence obtained after the at least one second audio is processed; or

processing the first audio to obtain the key video frame sequence corresponding to the first audio.

6 . The method according to claim 5 , wherein the facial feature comprises a mouth shape feature, and the determining the key video frame sequence comprises:

obtaining a second facial image;

obtaining, based on the second audio or the first audio, and the second facial image, a first audio and video in which a mouth shape feature in the second facial image is consistent with a mouth shape feature presented when a third audio or the first audio is played; and

using, as the key video frame sequence, a plurality of audio and video frames of the first audio and video in which mouth shape features are inconsistent with a first mouth shape feature.

7 . The method according to claim 6 , wherein the facial feature further comprises a facial organ feature, and the determining the key video frame sequence comprises:

processing, based on a pre-trained facial driving model, the first audio and video and a third facial image, so as to obtain a second audio and video in which a facial organ feature in the third facial image changes;

sequentially determining a facial organ feature in each audio and video frame of the second audio and video;

using, as a key video frame, an audio and video frame in which the facial organ feature is inconsistent with a first facial organ feature; and

determining, based on timestamps of a plurality of key video frames, the key video frame sequence corresponding to the first audio and video.

8 . The method according to claim 1 , wherein the determining, based on the key video frame sequence and the first facial image, a first facial feature in the first facial image that is presented when the first audio is played comprises:

determining, based on reference feature point data of each key video frame in the key video frame sequence and basic feature point data of the first facial image, first feature point data corresponding to the key video frame; and

determining, based on the first feature point data, the first facial image, and the corresponding basic feature point data, the first facial feature in the first facial image that is presented when the first audio is played,

wherein the reference feature point data corresponds to facial organ feature point data or mouth shape feature point data.

9 . The method according to claim 8 , wherein the determining, based on reference feature point data of each key video frame in the key video frame sequence and basic feature point data of the first facial image, first feature point data corresponding to the key video frame comprises:

determining, for each key video frame, the reference feature point data of the current key video frame and the basic feature point data of the first facial image, and determining difference feature data from the current key video frame; and

determining, based on the difference feature data from each key video frame and the basic feature point data, the first feature point data corresponding to the first facial image in each key video frame.

10 . The method according to claim 8 , wherein the determining, based on the first feature point data, the first facial image, and the corresponding basic feature point data, the first facial feature in the first facial image that is presented when the first audio is played comprises:

inputting the first feature point data, the first facial image, and the corresponding basic feature point data into a pre-trained effect audio and video generation model to obtain the first facial feature of the first object.

11 . The method according to claim 10 , further comprising:

determining sound-picture synchronized videos that correspond to at least one fourth audio in different language types;

determining, based on the sound-picture synchronized video and fourth facial images of different second objects, a first key video frame sequence of the different second objects in the corresponding sound-picture synchronized video; and

obtaining a fifth facial image of a third object, the first key video frame sequence, and facial feature data that corresponds to the fifth facial image, to construct a training sample for training the effect audio and video generation model.

12 . An electronic device, comprising:

one or more processors; and

a storage apparatus configured to store one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to:

acquire, in response to an effect trigger operation, a first facial image comprising a first object;

determine a first audio, and determining a key video frame sequence corresponding to the first audio;

determine, based on the key video frame sequence and the first facial image, a first facial feature in the first facial image that is presented when the first audio is played; and

determine a first effect audio and video based on the first facial feature and the first audio;

wherein the key video frame sequence comprises at least one video frame, and a facial feature of a user in the video frame is inconsistent with a second facial feature.

13 . A non-transitory storage medium comprising computer-executable instructions that, when executed by a computer processor, cause the computer processor to:

acquire, in response to an effect trigger operation, a first facial image comprising a first object;

determine a first audio, and determining a key video frame sequence corresponding to the first audio;

determine, based on the key video frame sequence and the first facial image, a first facial feature in the first facial image that is presented when the first audio is played; and

determine a first effect audio and video based on the first facial feature and the first audio;

wherein the key video frame sequence comprises at least one video frame, and a facial feature of a user in the video frame is inconsistent with a second facial feature.

14 . The device according to claim 12 , wherein the effect trigger operation comprises at least one of the following:

triggering an effect prop;

a frame-in picture comprising the first object;

triggering an effect wake-up word using audio information; and

a current body movement being consistent with a first effect movement.

15 . The device according to claim 12 , wherein the one or more programs causing the one or more processors to determine the first audio further cause the one more processors to:

display at least one second audio, and determining the first audio based on a trigger operation on the at least one second audio within first duration; or

receive an uploaded third audio as the first audio.

16 . The device according to claim 12 , the one or more programs further cause the one more processors to:

determine the key video frame sequence corresponding to the first audio based on a pre-selected first language type corresponding to the first audio.

17 . The device according to claim 12 , wherein the one or more programs causing the one more processors to determine the key video frame sequence corresponding to the first audio further cause the one more processors to:

retrieve the key video frame sequence corresponding to the first audio from a pre-determined key video frame sequence library, wherein the first audio is determined from at least one second audio that is displayed in a display interface, and the key video frame sequence library comprises a corresponding key video frame sequence obtained after the at least one second audio is processed; or

process the first audio to obtain the key video frame sequence corresponding to the first audio.

18 . The device according to claim 17 , wherein the facial feature comprises a mouth shape feature, and the one or more programs causing the one more processors to determine the key video frame sequence further cause the one more processors to:

obtain a second facial image;

obtain, based on the second audio or the first audio, and the second facial image, a first audio and video in which a mouth shape feature in the second facial image is consistent with a mouth shape feature presented when a third audio or the first audio is played; and

use, as the key video frame sequence, a plurality of audio and video frames of the first audio and video in which mouth shape features are inconsistent with a first mouth shape feature.

19 . The device according to claim 18 , wherein the facial feature further comprises a facial organ feature, and the one or more programs causing the one more processors to determine the key video frame sequence further cause the one more processors to:

process, based on a pre-trained facial driving model, the first audio and video and a third facial image, so as to obtain a second audio and video in which a facial organ feature in the third facial image changes;

sequentially determine a facial organ feature in each audio and video frame of the second audio and video;

use, as a key video frame, an audio and video frame in which the facial organ feature is inconsistent with a first facial organ feature; and

determine, based on timestamps of a plurality of key video frames, the key video frame sequence corresponding to the first audio and video.

20 . The device according to claim 12 , wherein the one or more programs causing the one more processors to determine, based on the key video frame sequence and the first facial image, the first facial feature in the first facial image that is presented when the first audio is played further cause the one more processors to:

determine, based on reference feature point data of each key video frame in the key video frame sequence and basic feature point data of the first facial image, first feature point data corresponding to the key video frame; and

determine, based on the first feature point data, the first facial image, and the corresponding basic feature point data, the first facial feature in the first facial image that is presented when the first audio is played,

wherein the reference feature point data corresponds to facial organ feature point data or mouth shape feature point data.

Assignments (2)

Priority Claims (1)

Continuity (1)

References Cited (80)