Speech recognition device and operating method thereof
Provided are a method and device for speech recognition. The speech recognition method includes: receiving a speech signal generated by an utterance of a user; identifying a named entity from the received speech signal; determining a speech signal portion, which corresponds to the identified named entity, from the received speech signal; generating a first acoustic embedding vector corresponding to the speech signal portion, based on an acoustic embedding model; determining a second acoustic embedding vector that is one of a plurality of acoustic embedding vectors corresponding to a plurality of named entities included in an acoustic embedding database (DB), based on distances between the plurality of acoustic embedding vectors and the first acoustic embedding vector; determining a corrected named entity corresponding to the second acoustic embedding vector; and providing a result of speech recognition with respect to the speech signal, based on the corrected named entity.
1. A speech recognition method comprising:
receiving a speech signal generated by an utterance of a user;
identifying a named entity from the received speech signal;
determining a speech signal portion, which corresponds to the identified named entity, from the received speech signal;
generating a first acoustic embedding vector corresponding to the speech signal portion, based on an acoustic embedding model;
determining an acoustic embedding vector closet to the first acoustic embedding vector from among a plurality of acoustic embedding vectors corresponding to a plurality of named entities included in an acoustic embedding database (DB), as a second acoustic embedding vector, based on the acoustic embedding model;
determining a corrected named entity corresponding to the second acoustic embedding vector, from among the plurality of named entities included in the acoustic embedding DB;
displaying the corrected named entity and a result of speech recognition with respect to the speech signal, based on the corrected named entity;
determining at least one candidate embedding vector including an acoustic embedding vector second closest to the first acoustic embedding vector and an acoustic embedding vector third closest to the first acoustic embedding vector from among the plurality of acoustic embedding vectors, based on the acoustic embedding model;
displaying at least one candidate named entity corresponding to the at least one candidate embedding vector from among the plurality of named entities included in the acoustic embedding DB; and
based on receiving a user input for selecting one of the at least one candidate named entity, displaying a result of the speech recognition corresponding to the selected candidate named entity and storing the first acoustic embedding vector in the acoustic embedding DB to correspond to the selected candidate named entity,
wherein the acoustic embedding model is trained using training data, the training data comprising texts representing one or more named entities, respective phoneme labels corresponding to the texts, and one or more speech signals corresponding to utterances of the one or more named entities; and wherein the training comprises:
performing a first training of the acoustic embedding model using the training data including the texts, the respective phoneme labels, and the one or more speech signals; and
performing a second training of the acoustic embedding model using a subset of the training data, the subset including the texts and the one or more speech signals.
2. The speech recognition method of claim 1 , wherein the determining of the second acoustic embedding vector based on the distances between the plurality of acoustic embedding vectors and the first acoustic embedding vector comprises:
determining the second acoustic embedding vector which is closest in distance to the first acoustic embedding vector, from among the plurality of acoustic embedding vectors.
3. The speech recognition method of claim 1 , wherein the acoustic embedding DB includes a named entity corresponding to two or more embedding vectors that are converted from different speech signals.
4. The speech recognition method of claim 3 , wherein the different speech signals are speech signals according to pronunciations of the named entity in different linguistic spheres.
5. The speech recognition method of claim 1 , wherein the determining of the speech signal portion corresponding to the identified named entity, from the received speech signal, comprises:
identifying time periods of phonemes represented by the received speech signal;
determining a time period corresponding to the identified named entity, based on the identified time periods of the phonemes; and
determining a portion of the received speech signal, which corresponds to the time period, to be the speech signal portion corresponding to the identified named entity.
6. The speech recognition method of claim 1 , wherein, as the result of the speech recognition, an application providing a content regarding the corrected named entity is executed, and
the speech recognition method further comprises:
when a user input for the provided content is received, storing the first acoustic embedding vector in the acoustic embedding DB to correspond to the corrected named entity.
7. The speech recognition method of claim 1 , further comprising:
displaying a menu for selecting the named entity identified from the received speech signal, in addition to the result of the speech recognition based on the corrected named entity;
when a user input for selecting the identified named entity is received, storing the first acoustic embedding vector in the acoustic embedding DB to correspond to the identified named entity; and
providing the result of the speech recognition with respect to the speech signal, based on the identified named entity.
8. A speech recognition device comprising:
a microphone;
at least one memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions to:
receive, via the microphone, a speech signal generated by an utterance of a user;
identify a named entity from the received speech signal;
determine a speech signal portion, which corresponds to the identified named entity, from the received speech signal;
generate a first acoustic embedding vector corresponding to the speech signal portion, based on an acoustic embedding model;
determine an acoustic embedding vector closest to the first acoustic embedding vector from among a plurality of acoustic embedding vectors, corresponding to a plurality of named entities included in an acoustic embedding database (DB), as a second acoustic embedding vector, based on the acoustic embedding model;
determine a corrected named entity, which corresponds to the second acoustic embedding vector, from among the plurality of named entities included in the acoustic embedding DB;
display the corrected named entity and a result of speech recognition with respect to the speech signal, based on the corrected named entity;
display at least one candidate named entity corresponding to the at least one candidate embedding vector from among the plurality of named entities included in the acoustic embedding DB; and
based on receiving a user input for selecting one of the at least one candidate named entity, display a result of the speech recognition corresponding to the selected candidate named entity and store the first acoustic embedding vector in the acoustic embedding DB to correspond to the selected candidate named entity,
wherein the acoustic embedding model is trained using training data, the training data comprising texts representing one or more named entities, respective phoneme labels corresponding to the texts, and one or more speech signals corresponding to utterances of the one or more named entities; and wherein the training comprises:
performing a first training of the acoustic embedding model using the training data including the texts, the respective phoneme labels, and the one or more speech signals; and
performing a second training of the acoustic embedding model using a subset of the training data, the subset including the texts and the one or more speech signals.
9. The speech recognition device of claim 8 , wherein the at least one processor is configured to execute the one or more instructions to:
determine, the second acoustic embedding vector which is closest in distance to the first acoustic embedding vector, from among the plurality of acoustic embedding vectors.
10. The speech recognition device of claim 8 , wherein the acoustic embedding DB includes a named entity corresponding to two or more embedding vectors that are converted from different speech signals.
11. The speech recognition device of claim 10 , wherein the different speech signals are speech signals according to pronunciations of the named entity in different linguistic spheres.
12. The speech recognition device of claim 8 , wherein the at least one processor is configured to execute the one or more instructions to determine the speech signal portion, which corresponds to the identified named entity, from the received speech signal by:
identifying time periods of phonemes represented by the received speech signal;
determining a time period corresponding to the identified named entity, based on the identified time periods of the phonemes; and
determining a portion of the received speech signal, which corresponds to the time period, to be the speech signal portion corresponding to the identified named entity.
13. The speech recognition device of claim 8 , wherein, as the result of the speech recognition, an application providing a content regarding the corrected named entity is executed, and
the at least one processor is further configured to execute the one or more instructions to:
when a user input for the provided content is received, store the first acoustic embedding vector in the acoustic embedding DB to correspond to the corrected named entity.
14. The speech recognition device of claim 8 , wherein the at least one processor is further configured to execute the one or more instructions to:
display a menu for selecting the named entity identified from the received speech signal, in addition to the result of the speech recognition based on the corrected named entity;
when a user input for selecting the identified named entity is received, store the first acoustic embedding vector in the acoustic embedding DB to correspond to the identified named entity; and
provide the result of the speech recognition with respect to the speech signal, based on the identified named entity.