Apparatus, method, and program for utilizing language model
Disclosed is a model adaptation technology of a language model with higher adaptability. An aspect of the present disclosure relates to an apparatus includes a first neural network unit that transforms an input symbol and outputs an intermediate state; and a second neural network unit that transforms input auxiliary information and the intermediate state and predicts a symbol following the input symbol, wherein the second neural network unit includes a plurality of hidden layers receiving, as input, the intermediate state and auxiliary information, and pieces of the auxiliary information input to each hidden layer are different from each other.
1. A computer-implemented method for predicting an object of a language based on adapting aspects of a language model, the method comprising:
receiving an input symbol of a plurality of input symbols of the language in series;
receiving a set of auxiliary data, wherein the set of auxiliary data is distinct from the input symbol, and the set of auxiliary data represents a latent topic of the plurality of input symbols of the language according to a bag-of-words model;
generating, based on the received input symbol, an intermediate state symbol using a first neural network;
generating, based on the generated intermediate state symbol and the received set of auxiliary data, an output symbol following the input symbol as a result of predicting the object using a second neural network, thereby adapting the output symbol to the latent topic of the plurality of input symbols of the language according to the bag-of-words model, wherein the second neural network comprises a plurality of hidden layers, and each hidden layer of the plurality of hidden layers receives at least a part of the generated intermediate state symbol and a distinct auxiliary data of the set of auxiliary data as input.
2. The computer-implemented method of claim 1 ,
wherein the input symbol being an input symbol for learning,
wherein the set of auxiliary data being a set of auxiliary data for learning,
wherein the intermediate state symbol being an intermediate state symbol for learning,
wherein the first neural network and the second neural network are distinct; and
the method further comprising:
receiving a correct symbol for output, wherein the correct symbol for output relates to the input symbol;
comparing the correct symbol and the symbol following the input symbol for training; and
updating, based on the comparison, one or more parameters in the first neural network and the second neural network for learning.
3. The computer-implemented method of claim 1 , wherein the set of auxiliary data is based at least on data excluding the input symbol that influences selecting of the symbol following the input symbol.
4. The computer-implemented method of claim 1 , wherein the set of auxiliary data includes at least one of:
a topic based on a series of symbols chronologically preceding and including the input symbol,
a context of the series of symbols, or
an environment of acquiring the series of symbols.
5. The computer-implemented method of claim 1 , wherein a symbol comprises a vector representing an object for processing by a neural network as a language model, and wherein the object includes one of:
a word,
a phoneme,
a morpheme, or
a character.
6. The computer-implemented method of claim 1 , wherein each hidden layer comprises a linear hidden network based on auxiliary data that is distinct from other hidden layers in the plurality of hidden layers.
7. The computer-implemented method of claim 1 , the method further comprising:
generating, based on the generated symbol following the input symbol as the prediction, a set of prediction output vectors using softmax;
determining an output object based on the generated set of prediction output vectors; and
providing the output object.
8. A system for predicting an object of a language based on adapting aspects of a language model, the system comprises:
a processor, and
a memory storing computer-executable instructions that when executed by the processor cause the system to:
receive an input symbol of a plurality of input symbols of the language in series;
receive a set of auxiliary data, wherein the auxiliary data is distinct from the input symbol, and the set of auxiliary data represents a latent topic of the plurality of input symbols of the language according to a bag-of-words model;
generate, based on the received input symbol, an intermediate state symbol using a first neural network;
generate, based on the generated intermediate state symbol and the received set of auxiliary data, an output symbol following the input symbol as a result of predicting the object using a second neural network, thereby adapting the output symbol to the latent topic of the plurality of input symbols of the language according to the bag-of-words model, wherein the second neural network comprises a plurality of hidden layers, and each hidden layer of the plurality of hidden layers receives at least a part of the generated intermediate state symbol and a distinct auxiliary data of the set of auxiliary data as input.
9. The system of claim 8 ,
wherein the input symbol being an input symbol for learning,
wherein the set of auxiliary data being a set of auxiliary data for learning,
wherein the intermediate state symbol being an intermediate state symbol for learning,
wherein the first neural network and the second neural network are distinct; and
the method further comprising:
receiving a correct symbol for output, wherein the correct symbol for output relates to the input symbol;
comparing the correct symbol and the symbol following the input symbol for training; and
updating, based on the comparison, one or more parameters in the first neural network and the second neural network for learning.
10. The system of claim 8 , wherein the set of auxiliary data is based at least on data excluding the input symbol that influences selecting of the symbol following the input symbol.
11. The system of claim 8 , wherein the set of auxiliary data includes at least one of:
a topic based on a series of symbols chronologically preceding and including the input symbol,
a context of the series of symbols, or
an environment of acquiring the series of symbols.
12. The system of claim 8 , wherein a symbol comprises a vector representing an object for processing by a neural network as a language model, and wherein the object includes one of:
a word,
a phoneme,
a morpheme, or
a character.
13. The system of claim 8 , wherein each hidden layer comprises a linear hidden network based on auxiliary data that is distinct from other hidden layers in the plurality of hidden layers.
14. The system of claim 8 , the computer-executable instructions when executed further causing the system to:
generate, based on the generated symbol following the input symbol as the prediction, a set of prediction output vectors;
determine an output object based on the generated set of prediction output vectors; and
provide the output object.
15. A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to:
receiving an input symbol of a plurality of input symbols of the language in series;
receiving a set of auxiliary data, wherein the auxiliary data is distinct from the input symbol, and the set of auxiliary data represents a latent topic of the plurality of input symbols of the language according to a bag-of-words model;
generating, based on the received input symbol, an intermediate state symbol using a first neural network;
generating, based on the generated intermediate state symbol and the received set of auxiliary data, a symbol following the input symbol as a result of predicting an object using a second neural network, thereby adapting the output symbol to the latent topic of the plurality of input symbols of the language according to the bag-of-words model, wherein the second neural network comprises a plurality of hidden layers, and each hidden layer of the plurality of hidden layers receives at least a part of the generated intermediate state symbol and a distinct auxiliary data of the set of auxiliary data as input.
16. The computer-readable non-transitory recording medium of claim 15 ,
wherein the input symbol being an input symbol for learning,
wherein the set of auxiliary data being a set of auxiliary data for learning,
wherein the intermediate state symbol being an intermediate state symbol for learning,
wherein the first neural network and the second neural network are distinct; and
the method further comprising:
receiving a correct symbol for output, wherein the correct symbol for output relates to the input symbol;
comparing the correct symbol and the symbol following the input symbol for training; and
updating, based on the comparison, one or more parameters in the first neural network and the second neural network for learning.
17. The computer-readable non-transitory recording medium of claim 15 , wherein the set of auxiliary data is based at least on data excluding the input symbol that influences selecting of the symbol following the input symbol.
18. The computer-readable non-transitory recording medium of claim 15 , wherein the set of auxiliary data includes at least one of:
a topic based on a series of symbols chronologically preceding and including the input symbol,
a context of the series of symbols, or
an environment of acquiring the series of symbols.
19. The computer-readable non-transitory recording medium of claim 15 , wherein a symbol comprises a vector representing an object for processing by a neural network as a language model, and wherein the object includes one of:
a word,
a phoneme,
a morpheme, or
a character.
20. The computer-readable non-transitory recording medium of claim 15 , the computer-executable instructions when executed further causing the system to:
generate, based on the generated symbol following the input symbol as the prediction, a set of prediction output vectors;
determine an output object based on the generated set of prediction output vectors; and
provide the output object.