Granted Patent B2

US 12,493,353 · App. 17/841,697 · Granted Dec 9, 2025

Inventors: Jingru Wang (Beijing, CN); Fanhao Kong (Beijing, CN)

Assignee: Beijing BOE Technology Development Co., Ltd.

G06F3/017G06V40/172G06V40/28

Abstract

Claims (59)

1 . A control method, of an electronic device, wherein the electronic device comprises a processor, a memory, and a display, the method comprising:

obtaining, by the processor, a first image;

performing, by the processor, face recognition and gesture recognition on the first image;

determining, by the processor, to control the electronic device through a gesture when a first target face is recognized from the first image and a first target gesture is recognized from the first image;

returning to, by the processor, the obtaining by the processor the first image when the first target face is not recognized from the first image or the first target gesture is not recognized from the first image;

determining, by the processor, to control the electronic device through the gesture when a control hand is recognized from at least one frame of the first image; and

playing, by the processor, a video when a third target face is recognized from at least one frame of the first image and the control hand is not recognized from the at least one frame of the first image.

2 . The method according to claim 1 , further comprising: providing, by the processor, first prompt information on a display interface when the first target face is recognized from the first image and the first target gesture is recognized from the first image, wherein the first prompt information is used for prompting a user that the processor determines to control the electronic device through the gesture.

3 . The method according to claim 1 , further comprising:

providing, by the processor, second prompt information on a display interface when the first target face is recognized from the first image and the first target gesture is not recognized from the first image, wherein the second prompt information is used for prompting a user to adjust a gesture; or

providing, by the processor, third prompt information on a display interface when the first target face is not recognized from the first image, wherein the third prompt information is used for prompting a user to adjust an angle of a face facing an acquisition device.

4 . The method according to claim 1 , further comprising:

obtaining, by the processor, a second image after the processor determines to control the electronic device through the gesture, and performing, by the processor, the face recognition and the gesture recognition on the second image; and

determining, by the processor, to control the electronic device through a corresponding second target gesture when a second target face is recognized from the second image and the second target gesture is recognized from the second image;

or,

returning to, by the processor, the obtaining, by the processor, the second image when the second target face is not recognized from the second image or the second target gesture is not recognized from the second image; and

determining, by the processor, not to control the electronic device through any gesture when the second target face is not recognized from second images of consecutive multiple frames within a set time period.

5 . The method according to claim 1 , wherein the gesture recognition on the first image comprises: determining, by the processor, a second tracking identification of at least one hand detected from the first image, wherein different hands have different second tracking identifications;

determining, by the processor, a position and a gesture of the at least one hand in a plurality of consecutive frames of the first image according to the second tracking identification of the at least one hand, wherein the plurality of consecutive frames of the first image comprises a current frame of the first image and at least one frame of the first image before the current frame of the first image; and

recognizing, by the processor, the control hand in the at least one hand according to the position and the gesture of the at least one hand in the plurality of consecutive frames of the first image.

6 . The method according to claim 5 , wherein recognizing, by the processor, the control hand in the at least one hand according to the position and the gesture of the at least one hand in the plurality of consecutive frames of the first image comprises:

when detecting a same hand in consecutive M frames of the first image, and recognizing that the hand is in a first gesture in X frames of the first image among the M frames of the first image, and that a distance of movement of the hand in any two adjacent frames of the first image among the M frames of the first image is less than or equal to a first threshold, recognizing, by the processor, the hand as the control hand; wherein X is less than or equal to M, and both X and M are integers.

7 . The method according to claim 6 , wherein recognizing, by the processor, the control hand in the at least one hand according to the position and the gesture of the at least one hand in the plurality of consecutive frames of the first image further comprises: when detecting a plurality of control hands during a recognition process of a frame of the first image, selecting, by the processor, the control hand with the largest average area of trajectory in the consecutive M frames of the first image from the plurality of control hands.

8 . The method according to claim 1 , wherein the face recognition on the first image comprises: determining, by the processor, a first tracking identification of at least one face detected from the first image, wherein different faces have different first tracking identifications;

determining, by the processor, whether the at least one face is a front face in a plurality of consecutive frames of the first image according to the first tracking identification of the at least one face, wherein the plurality of consecutive frames of the first image comprises a current frame of the first image and at least one frame of the first image before the current frame of the first image; and

when detecting a same face that is a front face in the plurality of consecutive frames of the first image, returning, by the processor, a result that the third target face is recognized.

9 . The method according to claim 1 , further comprising: after the processor determines to control the electronic device through the gesture, opening, by the processor, a first-level functional interface on a display interface, and controlling, by the processor, the first-level functional interface according to a gesture of the control hand.

10 . The method according to claim 9 , wherein controlling, by the processor, the first-level functional interface according to the gesture of the control hand comprises at least one of the following:

when recognizing that the control hand is in a second gesture, controlling, by the processor, movement of a cursor in the first-level functional interface according to movement of the second gesture; or

when recognizing that the control hand is in a third gesture, turning on, by the processor, a function indicated by an icon corresponding to a mapping position of the control hand in the display interface.

11 . The method according to claim 10 , wherein turning on, by the processor, the function indicated by the icon corresponding to the mapping position of the control hand in the display interface comprises: opening, by the processor, a second-level functional interface indicated by the icon;

the method further comprises at least one of the following:

controlling, by the processor, a display size adjustment operation of the second-level functional interface according to a gesture change of the control hand;

controlling, by the processor, a page turning operation in the second-level functional interface according to the gesture of the control hand and a moving direction of the gesture; or

controlling, by the processor, an operation of returning from the second-level functional interface to the first-level functional interface according to a gesture of the control hand and a duration of the gesture.

12 . An electronic device, comprising: a processor, a memory, and a display,

wherein the display is connected to the processor and is adapted to provide a display interface;

the memory is used to store a computer program, and when the processor executes the computer program, the processor performs:

obtaining a first image;

performing face recognition and gesture recognition on the first image;

determining to control the electronic device through a gesture when a first target face is recognized from the first image and a first target gesture is recognized from the first image;

returning to the obtaining the first image when the first target face is not recognized from the first image or the first target gesture is not recognized from the first image;

determining to control the electronic device through a gesture when a control hand is recognized from at least one frame of the first image; and

playing a video when a third target face is recognized from at least one frame of the first image and the control hand is not recognized from the at least one frame of the first image.

13 . The electronic device according to claim 12 , wherein the gesture recognition on the first image comprises: determining a second tracking identification of at least one hand detected from the first image, wherein different hands have different second tracking identifications;

determining a position and a gesture of the at least one hand in a plurality of consecutive frames of the first image according to the second tracking identification of the at least one hand, wherein the plurality of consecutive frames of the first image comprises a current frame of the first image and at least one frame of the first image before the current frame of the first image; and

recognizing the control hand in the at least one hand according to the position and the gesture of the at least one hand in the plurality of consecutive frames of the first image.

14 . The electronic device according to claim 13 , wherein the recognizing the control hand in the at least one hand according to the position and the gesture of the at least one hand in the plurality of consecutive frames of the first image comprises:

when detecting a same hand in consecutive M frames of the first image, and recognizing that the hand is in a first gesture in X frames of the first image among the M frames of the first image, and that a distance of movement of the hand in any two adjacent frames of the first image among the M frames of the first image is less than or equal to a first threshold, recognizing the hand as the control hand; wherein X is less than or equal to M, and both X and M are integers.

15 . A non-transitory computer-readable storage medium, storing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform:

obtaining a first image;

performing face recognition and gesture recognition on the first image;

determining to control the electronic device through a gesture when a first target face is recognized from the first image and a first target gesture is recognized from the first image;

returning to the obtaining the first image when the first target face is not recognized from the first image or the first target gesture is not recognized from the first image-;

determining to control the electronic device through a gesture when a control hand is recognized from at least one frame of the first image; and

playing a video when a third target face is recognized from at least one frame of the first image and the control hand is not recognized from the at least one frame of the first image.

16 . The non-transitory computer-readable storage medium according to claim 15 , wherein the gesture recognition on the first image comprises: determining a second tracking identification of at least one hand detected from the first image, wherein different hands have different second tracking identifications;

determining a position and a gesture of the at least one hand in a plurality of consecutive frames of the first image according to the second tracking identification of the at least one hand, wherein the plurality of consecutive frames of the first image comprises a current frame of the first image and at least one frame of the first image before the current frame of the first image; and

recognizing the control hand in the at least one hand according to the position and the gesture of the at least one hand in the plurality of consecutive frames of the first image.

Assignments (2)

Continuity (2)

References Cited (31)