Electronic device for processing neural network model and method of operating the same
An electronic device is provided. The electronic device includes a memory, and a processor including a resource management unit and a neural processing unit. The processor may be configured to obtain an execution request for a specific function operating based on a specific neural network model, identify an available bandwidth of the memory through the resource management unit, and quantize the specific neural network model based on the available bandwidth of the memory through the neural processing unit.
1. An electronic device comprising:
memory storing one or more computer programs; and
one or more processors, including a resource management unit and a neural processing unit, communicatively coupled to the memory,
wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:
obtain an execution request for a specific function operating based on a specific neural network model,
identify an available bandwidth of the memory through the resource management unit,
quantize the specific neural network model based on the available bandwidth of the memory through the neural processing unit,
identify a section including the available bandwidth of the memory from a predefined table, and
select a bit depth corresponding to the identified section as an activation bit depth.
2. The electronic device of claim 1 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to identify the available bandwidth of the memory based on a predetermined period.
3. The electronic device of claim 1 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to identify the available bandwidth of the memory based on obtaining the execution request for the specific function.
4. The electronic device of claim 1 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to identify the available bandwidth of the memory based on obtaining a processing request for a specific layer of the specific neural network model.
5. The electronic device of claim 1 ,
wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:
identify a first section including the available bandwidth of the memory from the predefined table based on the available bandwidth of the memory exceeding a first threshold, and
select a first bit depth corresponding to the first section as the activation bit depth, and
wherein the first bit depth is a 16-bit depth.
6. The electronic device of claim 5 ,
wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:
identify a second section including the available bandwidth of the memory from the predefined table based on the available bandwidth of the memory being less than or equal to the first threshold and exceeding a second threshold, and
select a second bit depth corresponding to the second section as the activation bit depth, and
wherein the second bit depth is an 8-bit depth.
7. The electronic device of claim 6 ,
wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:
identify a third section including the available bandwidth of the memory from the predefined table based on the available bandwidth of the memory being less than or equal to the second threshold and exceeding a third threshold, and
select a third bit depth corresponding to the third section as the activation bit depth, and
wherein the third bit depth is a 6-bit depth.
8. The electronic device of claim 1 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:
quantize the specific neural network model by performing dynamic quantization on a specific activation bit depth of the specific neural network model based on the activation bit depth while maintaining a specific weight bit depth of the specific neural network model through the neural processing unit, and
perform a multiply and accumulate (MAC) operation on the quantized specific neural network model using the specific activation bit depth.
9. The electronic device of claim 1 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:
quantize the specific neural network model by performing dynamic quantization on a specific activation bit depth of a specific layer of the specific neural network model based on the activation bit depth while maintaining a specific weight bit depth of the specific layer of the specific neural network model through the neural processing unit, and
perform a multiply and accumulate (MAC) operation on the specific layer of the quantized specific neural network model using the specific activation bit depth.
10. A method of operating an electronic device, the method comprising:
obtaining an execution request for a specific function operating based on a specific neural network model;
identifying an available bandwidth of a memory of the electronic device through a resource management unit included in a processor of the electronic device;
quantizing the specific neural network model based on the available bandwidth of the memory through a neural processing unit included in the processor;
identifying a section including the available bandwidth of the memory from a predefined table; and
selecting a bit depth corresponding to the identified section as an activation bit depth.
11. The method of claim 10 , wherein the identifying of the available bandwidth of the memory includes identifying the available bandwidth of the memory based on a predetermined period.
12. The method of claim 10 , wherein the identifying of the available bandwidth of the memory includes identifying the available bandwidth of the memory based on obtaining the execution request for the specific function.
13. The method of claim 10 , wherein the identifying of the available bandwidth of the memory includes identifying the available bandwidth of the memory based on obtaining a processing request for a specific layer of the specific neural network model.
14. The method of claim 10 ,
wherein the selecting of the activation bit depth includes:
identifying a first section including the available bandwidth of the memory from the predefined table based on the available bandwidth of the memory exceeding a first threshold, and
selecting a first bit depth corresponding to the first section as the activation bit depth, and
wherein the first bit depth is a 16-bit depth.
15. The method of claim 14 ,
wherein the selecting of the activation bit depth includes:
identifying a second section including the available bandwidth of the memory from the predefined table based on the available bandwidth of the memory being less than or equal to the first threshold and exceeding a second threshold, and
selecting a second bit depth corresponding to the second section as the activation bit depth, and
wherein the second bit depth is an 8-bit depth.
16. The method of claim 15 ,
wherein the selecting of the activation bit depth includes:
identifying a third section including the available bandwidth of the memory from the predefined table based on the available bandwidth of the memory being less than or equal to the second threshold and exceeding a third threshold, and
selecting a third bit depth corresponding to the third section as the activation bit depth, and
wherein the third bit depth is a 6-bit depth.