Electronic apparatus for artificial intelligence model compression and method thereof
An electronic apparatus is provided. The electronic apparatus includes a memory configured to store one instruction or more and a processor configured to obtain output data by inputting input data to an artificial intelligence model including a plurality of layers by executing the instruction, and the artificial intelligence model is configured to output the output data based on operation through the plurality of layers and the processor is configured to encode operation data output from one of the plurality of layers and store the encoded operation data in the memory, obtain recovery data corresponding to the operation data by decoding the encoded operation data stored in the memory, and provide the obtained recovery data to another layer from among the plurality of layers.
1 . An electronic apparatus comprising:
memory configured to store one instruction or more; and
at least one processor configured to obtain output data by inputting input data to an artificial intelligence model including a plurality of layers by executing the one instruction or more,
wherein the artificial intelligence model is configured to output the output data based on operation through the plurality of layers,
wherein the one instruction or more, when executed by the at least one processor causes the electronic apparatus to:
determine, based on a capacity of the memory, that compression of data output from a first layer of the artificial intelligence model is required in case of the data being stored in the memory without compression and being provided to a second layer of the artificial intelligence model after retrieval from the memory,
add a first hidden layer and a second hidden layer between the first layer and the second layer based on the determination, wherein the first hidden layer has a dimension that is lower than that of the first layer, and the second hidden layer has a dimension that is equal to that of the first layer,
encode operation data output from the first layer without the operation data having been or being stored in the memory, based on a weight of the first hidden layer retrieved from the memory when the operation data is output from the first layer, and store the encoded operation data in the memory,
retrieve the encoded operation data and a weight of the second hidden layer from the memory, and obtain restoration data corresponding to the operation data by decoding, based on the weight of the second hidden layer that is retrieved from the memory, the encoded operation data that is retrieved from the memory, and
provide the obtained restoration data to the second layer without the obtained restoration data having been or being stored in the memory,
wherein the artificial intelligence model is configured to learn the weight of the first hidden layer and the weight of the second hidden layer so as to reconstruct data of the first layer into a layer having fewer nodes than a number of nodes of the first layer,
wherein the artificial intelligence model is further configured to learn the weight of the first hidden layer and the weight of the second hidden layer so that an error between the operation data output by the first layer and the obtained restoration data output by the second hidden layer is minimized,
wherein the artificial intelligence model is further configured to, based on the first hidden layer and the second hidden layer being added after learning that of a subset of the plurality of layers, which does not include the first hidden layer and the second hidden layer, is completed, obtain the weight of the first hidden layer and the weight of the second hidden layer through learning of an entire system including the subset of the plurality of layers, the first hidden layer, and the second hidden layer while fixing weights of the plurality of layers, and
wherein the learned weight of the first hidden layer and the weight of the second hidden layer are stored in the memory.
2 . The electronic apparatus as claimed in claim 1 , wherein the artificial intelligence model is further configured to obtain the weight of the first hidden layer and the weight of the second hidden layer through learning of an entire system including the plurality of layers, the first hidden layer, and the second hidden layer.
3 . The electronic apparatus as claimed in claim 1 ,
wherein the plurality of layers include a third layer, and a third hidden layer, and a fourth hidden layer between the second layer and the third layer, and
wherein the artificial intelligence model is further configured to:
learn a weight of the third hidden layer and a weight of the fourth hidden layer to generate the fourth hidden layer that reconstructs the second layer, and
learn so that the first hidden layer and the third hidden layer have a same weight, and the second hidden layer and the fourth hidden layer have a same weight.
4 . The electronic apparatus as claimed in claim 3 , wherein the one instruction or more, when executed by the at least one processor, further causes the electronic apparatus to:
perform encoding of the operation data and encoding of input data of the third hidden layer through one encoder, and
perform decoding of the encoded operation data and decoding of input data of the fourth hidden layer through one decoder.
5 . A method performed by an electronic apparatus, the method comprising:
determining, based on a capacity of a memory, that compression of data output from a first layer of an artificial intelligence model including a plurality of layers is required in case of the data being stored in the memory without compression and being provided to a second layer of the artificial intelligence model after retrieval from the memory;
adding a first hidden layer and a second hidden layer between the first layer and the second layer based on the determination, wherein the first hidden layer has a dimension that is lower than that of the first layer, and the second hidden layer has a dimension that is equal to that of the first layer;
encoding, operation data output from the first layer without the operation data having been or being stored in the memory, based on a weight of the first hidden layer retrieved from the memory when the operation data is output from the first layer, and storing the encoded operation data in the memory;
retrieving the encoded operation data and a weight of the second hidden layer from the memory, and obtaining restoration data corresponding to the operation data by decoding, based on the weight of the second hidden layer that is retrieved from the memory, the encoded operation data that is retrieved from the memory; and
providing the obtained restoration data to the second layer without the obtained restoration data having been or being stored in the memory,
wherein the artificial intelligence model is configured to learn the weight of the first hidden layer and the weight of the second hidden layer so as to reconstruct data of the first layer into a layer having fewer nodes than a number of nodes of the first layer,
wherein the artificial intelligence model is further configured to learn the weight of the first hidden layer and the weight of the second hidden layer so that an error between the operation data output by the first layer and the obtained restoration data output by the second hidden layer is minimized,
wherein the artificial intelligence model is further configured to, based on the first hidden layer and the second hidden layer being added after learning of a subset of the plurality of layers, which does not include the first hidden layer and the second hidden layer, is completed, obtain the weight of the first hidden layer and the weight of the second hidden layer through learning of an entire system including the subset of the plurality of layers, the first hidden layer, and the second hidden layer while fixing weights of the plurality of layers, and
wherein the learned weight of the first hidden layer and the weight of the second hidden layer are stored in the memory.
6 . The method as claimed in claim 5 , wherein the artificial intelligence model is further configured to obtain the weight of the first hidden layer and the weight of the second hidden layer through learning of an entire system including the plurality of layers, the first hidden layer, and the second hidden layer.
7 . The method as claimed in claim 5 ,
wherein the plurality of layers include a third layer, and a third hidden layer, and a fourth hidden layer between the second layer and the third layer, and
wherein the artificial intelligence model is further configured to:
learn a weight of the third hidden layer and a weight of the fourth hidden layer to generate the fourth hidden layer that reconstructs the second layer; and
learn so that the first hidden layer and the third hidden layer have a same weight, and the second hidden layer and the fourth hidden layer have a same weight.
8 . The method as claimed in claim 7 , further comprising:
performing encoding of the operation data and encoding of input data of the third hidden layer through one encoder; and
performing decoding of the encoded operation data and decoding of input data of the fourth hidden layer through one decoder.
9 . A non-transitory computer readable recording medium storing at least one instruction that, when executed by at least one processor of an electronic apparatus, cause an electronic apparatus to perform operations to:
determine, based on a capacity of a memory, that compression of data output from a first layer of an artificial intelligence model including a plurality of layers is required in case of the data being stored in the memory without compression and being provided to a second layer of the artificial intelligence model after retrieval from the memory;
add a first hidden layer and a second hidden layer between the first layer and the second layer based on the determination, wherein the first hidden layer has a dimension that is lower than that of the first layer, and the second hidden layer has a dimension that is equal to that of the first layer;
encode operation data output from the first layer without the operation data having been or being stored in the memory, based on a weight of the first hidden layer retrieved from the memory when the operation data is output from the first layer, and store the encoded operation data in the memory;
retrieve the encoded operation data and a weight of the second hidden layer from the memory, and obtain restoration data corresponding to the operation data by decoding, based on the weight of the second hidden layer that is retrieved from the memory, the encoded operation data that is retrieved from the memory; and
provide the obtained restoration data to the second layer without the obtained restoration data having been or being stored in the memory,
wherein the artificial intelligence model is configured to learn the weight of the first hidden layer and the weight of the second hidden layer so as to reconstruct data of the first layer into a layer having fewer nodes than a number of nodes of the first layer,
wherein the artificial intelligence model is further configured to learn the weight of the first hidden layer and the weight of the second hidden layer so that an error between the operation data output by of the first layer and the obtained restoration data output by the second hidden layer is minimized,
wherein the artificial intelligence model is further configured to, based on the first hidden layer and the second hidden layer being added after learning of a subset of the plurality of layers, which does not include the first hidden layer and the second hidden layer, is completed, obtain the weight of the first hidden layer and the weight of the second hidden layer through learning of an entire system including the subset of the plurality of layers, the first hidden layer, and the second hidden layer while fixing weights of the plurality of layers, and
wherein the learned weight of the first hidden layer and the weight of the second hidden layer are stored in the memory.
10 . The non-transitory computer readable recording medium of claim 9 , wherein the encoded operation data is encoded in a vector having one dimension less than a vector including the operation data.
11 . The non-transitory computer readable recording medium of claim 10 , wherein the vector including the operation data comprises a same number of dimensions as a vector including the restoration data.
12 . The non-transitory computer readable recording medium of claim 9 , wherein the artificial intelligence model is received from an external device without the first hidden layer and without the second hidden layer.
13 . The non-transitory computer readable recording medium of claim 9 , wherein at least two layers of the plurality of layers of the artificial intelligence model do not include a hidden layer therebetween.
14 . The electronic apparatus as claimed in claim 1 , wherein at least two layers of the plurality of layers of the artificial intelligence model do not include a hidden layer therebetween.
15 . The method as claimed in claim 5 , wherein at least two layers of the plurality of layers of the artificial intelligence model do not include a hidden layer therebetween.