Device for recognizing speech input from user and operating method thereof

1. A method of performing speech recognition by a device, the method comprising:

by using a vocabulary list comprising a plurality of named entities, generating a weighted finite state transducer model by performing training regarding a probability that a subword extracted from each of the plurality of named entities is predictable as a word or word sequence representing a named entity;

receiving a speech input from a user;

by using a first artificial intelligence (AI) decoding model, obtaining a feature vector indicating a probability of predicting a subword from the received speech input, and obtaining a first string comprising a plurality of predicted strings based on the feature vector;

inputting the feature vector to a second AI decoding model that uses the weighted finite state transducer model that is trained to predict the word or word sequence representing the named entity, from a trained subword extracted from a plurality of trained named entities;

by using the second AI decoding model, obtaining a second string based on the feature vector, the second string comprising a recognized word sequence that corresponds to at least one of a plurality of predetermined named entities, and an unrecognized word sequence that does not correspond to any one of the plurality of predetermined named entities; and

outputting a text corresponding to the received speech input by substituting the unrecognized word sequence of the second string with a word sequence included in the first string.

2. The method of claim 1 , wherein the generating of the weighted finite state transducer model comprises:

obtaining the vocabulary list comprising the plurality of named entities;

segmenting a word or a string constituting the plurality of named entities into trained subwords that are in units of phonemes or syllables; and

obtaining a confidence score comprising a posterior probability of predicting one of the plurality of named entities from the trained subwords, through state transition using frequencies of the trained subwords and arrangement orders of the trained subwords.

3. The method of claim 2 , wherein the weighted finite state transducer model comprises a lexicon finite state transducer and a grammar finite state transducer,

wherein the lexicon finite state transducer comprises mapping information which is a probability of predicting a first word or a first word sequence from each of the trained subwords, and

wherein the grammar finite state transducer comprises weight information for, when the first word or the first word sequence is input, predicting a second word sequence capable of being arranged after the first word or the first word sequence.

4. The method of claim 1 , wherein the first AI decoding model is an end-to-end automatic speech recognition (ASR) model.

5. The method of claim 1 , wherein the generating of the weighted finite state transducer model by:

classifying the plurality of named entities according to a plurality of domains different from each other; and

generating a plurality of weighted finite state transducer models for the respective plurality of domains by using the classified plurality of named entities.

6. The method of claim 5 , further comprising:

identifying words corresponding to named entities included in an application executed by the device or a web page provided via the device; and

determining a domain into which the application or the web page, by comparing the identified words with a plurality of named entities which are comprised in a vocabulary list of each of the plurality of weighted finite state transducer models generated for the respective plurality of domains.

7. The method of claim 1 , further comprising:

receiving update information for a vocabulary list from a server, the update information comprising at least one of addition of a new named entity, deletion of the named entity, and modification of the named entity; and

updating the vocabulary list by using the update information,

wherein the generating of the weighted finite state transducer model comprises generating the weighted finite state transducer model by using the updated vocabulary list.

8. The method of claim 1 , further comprising:

recognizing that the device enters a new region, by obtaining position information of the device;

transmitting entry-into-new-region information of the device to a server of an application service provider; and

receiving a point-of-interest (POI) vocabulary list from the server of the application service provider, the POI vocabulary list comprising named entities regarding at least one of place names, attractions, tourist spots, or famous restaurants of the new region,

wherein the generating of the weighted finite state transducer model comprises generating the weighted finite state transducer model by training the weighted finite state transducer model using the plurality of named entities included in the received POI vocabulary list.

9. The method of claim 1 , wherein the generating of the weighted finite state transducer model comprises generating the weighted finite state transducer model by training the weighted finite state transducer model using the plurality of named entities that reflect characteristics of the user that are obtained from at least one of an application executed by the device more than a predetermined frequency, log data of a messenger application, and a search term history in a content streaming application.

10. A device for performing speech recognition, the device comprising:

a speech inputter configured to receive a speech input from a user;

at least one memory configured to store one or more computer-readable instructions; and

at least one processor configured to execute the one or more computer-readable instructions to:

by using a vocabulary list comprising a plurality of named entities, generate a weighted finite state transducer model by performing training regarding a probability that a subword extracted from each of the plurality of named entities is predictable as a word or word sequence representing a named entity;

receive the speech input from the speech inputter;

by using a first artificial intelligence (AI) decoding model, obtain a feature vector indicating a probability of predicting a subword from the received speech input, and obtain a first string comprising a plurality of predicted strings based on the probability of the feature vector;

input the feature vector to a second AI decoding model that uses the weighted finite state transducer model that is trained to predict the word or word sequence representing the named entity, from a trained subword extracted from a plurality of named entities;

by using the second AI decoding model, obtain a second string based on the feature vector, the second string comprising a recognized word sequence that corresponds to at least one of a plurality of predetermined named entities, and an unrecognized word sequence that does not correspond to any one of the plurality of predetermined named entities; and

obtain a text corresponding to the received speech input by substituting the unrecognized word sequence of the second string with a word sequence included in the first string.

11. The device of claim 10 , wherein the at least one processor is further configured to execute the one or more computer-readable instructions to generate the weighted finite state transducer model by:

obtaining the vocabulary list comprising the plurality of named entities;

segmenting a word or a string constituting the plurality of named entities into trained subwords that are in units of phonemes or syllables; and

obtaining a confidence score comprising a posterior probability of predicting one of the plurality of named entities from the trained subwords, through state transition using frequencies of the trained subwords and arrangement orders of the trained subwords.

12. The device of claim 11 , wherein the at least one processor is further configured to execute the one or more computer-readable instructions to generate the weighted finite state transducer model by:

performing filtering to remove the named entity from the plurality of named entities, based the named entity overlapping with a word pre-stored in the at least one memory of the device.

13. The device of claim 11 , wherein the weighted finite state transducer model comprises a lexicon finite state transducer and a grammar finite state transducer,

wherein the lexicon finite state transducer comprises mapping information which is a probability of predicting a first word or a first word sequence from each of the trained subwords, and

wherein the grammar finite state transducer comprises weight information for, when the first word or the first word sequence is input, predicting a second word sequence capable of being arranged after the first word or the first word sequence.

14. The device of claim 10 , wherein the first AI decoding model is an end-to-end automatic speech recognition (ASR) model.

15. The device of claim 10 , wherein the at least one processor is further configured to execute the one or more computer-readable instructions to:

classifying the plurality of named entities according to a plurality of domains different from each other; and

generating a plurality of weighted finite state transducer models for the respective plurality of domains by using the classified plurality of named entities.

16. The device of claim 15 , wherein the at least one processor is further configured to execute the one or more computer-readable instructions to:

identify words corresponding to named entities included in an application executed by the device or a web page accessed by the device; and

determine a domain into which the application or the web page, by comparing the identified words with the plurality of predetermined named entities which are comprised in a vocabulary list of each of the plurality of weighted finite state transducer models generated for the respective plurality of domains.

17. The device of claim 10 , further comprising a communication interface configured to transmit data to and receive data from a server,

wherein the at least one processor is further configured to execute the one or more computer-readable instructions to:

receive update information for a vocabulary list from the server by using the communication interface, the update information comprising at least one of addition of a new named entity, deletion of the named entity, and modification of the named entity;

update the vocabulary list by using the update information; and

generate the weighted finite state transducer model by training the weighted finite state transducer model using the updated vocabulary list.

18. The device of claim 10 , further comprising:

a position sensor configured to obtain position information of the device; and

a communication interface configured to transmit data to and receive data from a speech assistant server or an external server,

wherein the at least one processor is further configured to execute the one or more computer-readable instructions to:

recognize that the device enters a new region, based on the position information of the device;

in response to recognizing that the device enters the new region, receive a point-of-interest (POI) vocabulary list from an application service provider sever through the communication interface, the POI vocabulary list comprising named entities regarding at least one of place names, attractions, tourist spots, or famous restaurants of the new region; and

generate the weighted finite state transducer model by training the weighted finite state transducer model using the plurality of named entities included in the received POI vocabulary list.

19. The device of claim 10 , wherein the at least one processor is further configured to execute the one or more computer-readable instructions to generate the weighted finite state transducer model by training the weighted finite state transducer model using the plurality of named entities that reflect characteristics of the user that are obtained from at least one of an application executed by the device more than a predetermined frequency, log data of a messenger application, and a search term history in a content streaming application.

20. A non-transitory computer-readable storage medium storing a program that is executable by a computer to perform the method of claim 1 .