Electronic device, speech recognition method therefor, and medium
Embodiments of this application provide a speech recognition method. The speech recognition method includes: obtaining a facial depth image and a to-be-recognized voice of a user, where the facial depth image is an image collected by using a depth camera; recognizing a mouth shape feature from the facial depth image, and recognizing a voice feature from a to-be-recognized audio; and fusing the voice feature and the mouth shape feature into an audio-video feature, and recognizing, based on the audio-video feature, a voice uttered by the user.
1 . A speech recognition method for an electronic device, comprising:
obtaining a facial depth image of a user and a to-be-recognized voice of the user, wherein the facial depth image of the user is an image collected by using a depth camera;
recognizing a mouth shape feature from the facial depth image of the user, and recognizing a voice feature from the to-be-recognized voice of the user; and
fusing the voice feature and the mouth shape feature into an audio-video feature, and recognizing, based on the audio-video feature, a voice uttered by the user;
wherein recognizing the mouth shape feature from the facial depth image of the user comprises:
performing noise reduction on the facial depth image of the user, and binarizing the facial depth image of the user based on depth information of each pixel in a facial depth image obtained after noise reduction, to obtain a binarized image; and
determining contours of lips in the binarized image, and segmenting the lips based on the contours of the lips.
2 . The method according to claim 1 , wherein obtaining the facial depth image of the user and the to-be-recognized voice of the user comprises:
emitting an infrared light pulse through a face of the user and obtaining an infrared light pulse reflected by the face of the user;
calculating a time difference based on a change between phases of the emitted and reflected infrared light pulses;
calculating depth information of the face of the user with reference to a propagation speed of the infrared light pulse reflected by the face of the user and the time difference; and
saving the depth information as the facial depth image of the user.
3 . The method according to claim 2 , wherein the depth information is a value formed by performing uniform quantizing on a maximum depth distance and a minimum depth distance between the face of the user and the electronic device.
4 . The method according to claim 1 , wherein a mouth shape feature corresponding to the lips is obtained by using a convolutional neural network.
5 . The method according to claim 1 , wherein recognizing the mouth shape feature from the facial depth image of the user comprises:
separating a lip region sub-image from the facial depth image of the user; and
inputting the lip region sub-image into a convolutional neural network to obtain a mouth shape feature corresponding to the lip region sub-image.
6 . The method according to claim 1 , wherein recognizing the voice feature from the to-be-recognized voice of the user comprises:
separating a first frequency band range part and a second frequency band range part from the to-be-recognized voice of the user by using a linear predictor, wherein a frequency of a frequency band comprised in the first frequency band range part is lower than a frequency of a frequency band comprised in the second frequency band range part; and
fusing at least one of a first voice feature extracted from the first frequency band range part and a second voice feature extracted from the second frequency band range part as the voice feature from the to-be-recognized voice of the user.
7 . The method according to claim 6 , wherein the first voice feature and the second voice feature are fused to obtain the voice feature from the to-be-recognized voice of the user.
8 . The method according to claim 6 , wherein
the first voice feature is extracted from the first frequency band range part by using a voice feature extraction algorithm used for simulating a perception capability of a cochlea of a human ear; and
the voice feature extraction algorithm used for simulating a perception capability of a cochlea of a human ear is a mel-frequency cepstral coefficient (MFCC) extraction method, and the first voice feature is a MFCC.
9 . The method according to claim 6 , wherein
the second voice feature is extracted from the second frequency band range part through wavelet transform, wherein the second voice feature is a time-frequency feature obtained through wavelet transform.
10 . The method according to claim 9 , wherein separating the first frequency band range part and the second frequency band range part from the to-be-recognized voice of the user by using the linear predictor comprises:
separating the first frequency band range part from the to-be-recognized voice of the user by using the linear predictor, and using, as the second frequency band range part, a remaining part of the to-be-recognized voice of the user obtained after the first frequency band range part is separated.
11 . The method according to claim 6 , wherein the first frequency band range part represents a feature of a sound channel of a sound-uttering object that utters audio, and the second frequency band range part represents a feature of a sound source of the sound-uttering object.
12 . The method according to claim 1 , wherein fusing the voice feature and the mouth shape feature into the audio-video feature comprises:
obtaining a voice feature matrix and a mouth shape feature matrix respectively corresponding to the voice feature and the mouth shape feature, wherein a dimension of the voice feature matrix is the same as a dimension of the mouth shape feature matrix; and
calculating an audio-video feature corresponding to the voice feature matrix and the mouth shape feature matrix through concatenation.
13 . The method according to claim 12 , wherein weighting calculation is performed on a value of the voice feature matrix and a value of the mouth shape feature matrix to obtain the audio-video feature corresponding to the voice feature matrix and the mouth shape feature matrix.
14 . The method according to claim 1 , wherein recognizing the voice uttered by the user comprises:
obtaining, through a convolutional neural network, a voice feature value corresponding to the audio-video feature, wherein the voice feature value is used to represent a probability of information indicated by the voice uttered by the user; and
recognizing, based on the probability of the information, the voice uttered by the user.
15 . The method according to claim 14 , wherein deduplication calculation is performed on the voice feature value by using a time series classification algorithm, to remove repeated information comprised in the voice feature value.
16 . An electronic device, comprising:
a depth camera configured to obtain a facial depth image of a user;
a processor; and
a memory coupled to the processor and storing instructions, which when executed by the processor, cause the electronic device to control the depth camera and perform operations, the operations comprising:
obtaining the facial depth image of the user and a to-be-recognized voice of the user, wherein the facial depth image of the user is an image collected by using a depth camera;
recognizing a mouth shape feature from the facial depth image of the user, and recognizing a voice feature from the to-be-recognized voice of the user; and
fusing the voice feature and the mouth shape feature into an audio-video feature, and recognizing, based on the audio-video feature, a voice uttered by the user;
wherein recognizing the mouth shape feature from the facial depth image of the user comprises:
performing noise reduction on the facial depth image of the user, and binarizing the facial depth image of the user based on depth information of each pixel in a facial depth image obtained after noise reduction, to obtain a binarized image; and
determining contours of lips in the binarized image, and segmenting the lips based on the contours of the lips.
17 . The electronic device according to claim 16 , wherein obtaining the facial depth image of the user and the to-be-recognized voice of the user comprises:
emitting an infrared light pulse through a face of the user and obtaining an infrared light pulse reflected by the face of the user;
calculating a time difference based on a change between phases of the emitted and reflected infrared light pulses;
calculating depth information of the face of the user with reference to a propagation speed of the infrared light pulse reflected by the face of the user and the time difference; and
saving the depth information as the facial depth image of the user.
18 . A non-transitory computer-readable medium storing instructions, which when run on a computer, cause the computer to perform operations, the operations comprising:
obtaining a facial depth image of a user and a to-be-recognized voice of the user, wherein the facial depth image of the user is an image collected by using a depth camera;
recognizing a mouth shape feature from the facial depth image of the user, and recognizing a voice feature from the to-be-recognized voice of the user; and
fusing the voice feature and the mouth shape feature into an audio-video feature, and recognizing, based on the audio-video feature, a voice uttered by the user;
wherein recognizing the mouth shape feature from the facial depth image of the user comprises:
performing noise reduction on the facial depth image of the user, and binarizing the facial depth image of the user based on depth information of each pixel in a facial depth image obtained after noise reduction, to obtain a binarized image; and
determining contours of lips in the binarized image, and segmenting the lips based on the contours of the lips.