Adaptive mixed digital and analog computational storage systems
Various embodiments of this disclosure are directed to a mixed digital and analog domain approach to computational storage or memory applications. The mixed approach enables certain compute operations to be advantageously performed in the analog domain, achieving power saving. In some embodiments, an analog compute core is implemented based on a first set of memory elements that are made available with a second set of memory elements for digital data storage. A controller coupled to both sets of memory elements is able to selectively direct computational tasks to either the analog compute core or a digital processor coupled with the controller, based on one or more parameters including power, precision, and workload. In certain embodiments involving neural network tasks, the controller is configured to route certain tasks to the analog compute core based on neural network based factors such as network layer positioning and input signal type.
1. A memory device, comprising:
a digital processor configured to perform computations in a digital domain;
an analog in-memory compute unit configured to perform computations in an analog domain, the analog in-memory compute unit comprising an array of memory elements configured to store weights for a neural network, wherein input data for the neural network comprises image data or audio data, the image data comprising a plurality of color channels and the audio data comprising a plurality of audio channels; and
control circuitry configured to:
receive input data; and
for each color channel of the plurality of channels or for each audio channel of the plurality of audio channels:
determine precision sensitivity of the received input data, wherein precision sensitivity is measured by a signal-to-noise ratio (SNR) value;
determine whether the digital processor or the analog in-memory compute unit is to execute inference operations for the color channel or the audio channel based on power efficiency of the digital processor and the analog in-memory compute unit and precision sensitivity of the received input data; and
direct the received input data for the color channel or for the audio channel to either the digital processor or the analog in-memory compute unit based on the determination, wherein the directing comprises:
routing the received input data having a SNR value greater than a threshold to the digital processor; and
routing the received input data having a SNR value less than the threshold to the analog in-memory compute unit.
2. The memory device of claim 1 , wherein for a plurality of stages of a computation related to the neural network, the control circuitry is further configured to determine based on parameters to:
direct a pre-configured number of initial stages of the computation to the digital processor; and
direct stages after the pre-configured number of initial stages of the computation to the analog in-memory compute unit.
3. The memory device of claim 1 , further comprising a non-volatile memory array; and
wherein for a plurality of stages of a computation related to the neural network, the control circuitry is further configured to:
retrieve data from the non-volatile memory array;
for each stage of a plurality of stages of a computation related to the neural network:
determine, based at least on a power parameter, whether the digital processor or the analog in-memory compute unit is to execute the stage of computation; and
direct the retrieved data from the non-volatile memory array for the stage of computation to either the digital processor or the analog in-memory compute unit based on the determination.
4. The memory device of claim 1 , further comprising a non-volatile memory array; and
wherein the control circuitry is further configured to store, in the non-volatile memory array, intermediate results of the inference operations for a color channel or audio channel directed to the analog in-memory compute unit as segregated from intermediate results of the inference operations for a color channel or audio channel directed to the digital processor.
5. The memory device of claim 4 , wherein the control circuitry is further configured to:
perform address translation functions for data stored in the non-volatile memory array; and
apply a different storage management policy to the intermediate results of the inference operations directed to the analog in-memory compute unit than a storage management policy for the intermediate results of the inference operations directed to the digital processor.
6. The memory device of claim 5 , wherein the storage management policy includes a policy relating to one or more of: garbage collection, wear leveling, and error correction.
7. The memory device of claim 1 , further comprising an analog interface for receiving analog data from a sensor, wherein the received analog data from the sensor is routed to the analog in-memory compute unit.
8. The memory device of claim 1 , wherein the analog in-memory compute unit is fabricated with the non-volatile memory array in a single package.
9. The memory device of claim 8 , wherein the digital processor and the control circuitry are fabricated within the single package.
10. A method of performing inference in a neural network within a data storage device, the method comprising:
retrieving, from a non-volatile memory array in a data storage device comprising a digital processing circuitry and an analog in-memory compute unit, data associated with a plurality of computational tasks associated with an inference operation of a neural network; and
for each computational task:
determining precision sensitivity of the retrieved data, wherein precision sensitivity is measured by a signal-to-noise ratio (SNR) value;
determining whether the digital processing circuitry or the analog in-memory compute unit is to execute the computational task, based at least on at least one parameter, wherein the at least one parameter is precision sensitivity of the retrieved data, wherein precision sensitivity corresponds to a network layer position of the computational task within the neural network, and wherein precision sensitivity is measured by a signal-to-noise ratio (SNR) value; and
based on the determining, directing the computational task and the associated retrieved data to the digital processing circuitry or the analog in-memory compute unit; and storing, in the non-volatile memory array, intermediate results of the plurality of computational tasks directed to the analog in-memory compute unit as segregated from intermediate results of the plurality of computational tasks directed to the digital processing circuitry, wherein the directing comprises:
routing the computational task and the associated retrieved data having a SNR value greater than a threshold to the digital processing circuitry; and
routing the computational task and the associated retrieved data having a SNR value less than the threshold to the analog in-memory compute unit.
11. The method of claim 10 , wherein the at least one parameter route a computational task associated with one or more initial layers of the neural network to the digital processing circuitry.
12. The method of claim 10 , wherein the at least one parameter route a computational task associated with one or more layers of the neural network requiring a pre-determined level of signal to noise ratio (SNR) to the digital processing circuitry.
13. The method of claim 10 , further comprising:
applying different storage policies to the intermediate results of computational tasks directed to the analog in-memory compute unit and the intermediate results of computational tasks directed to the digital processing circuitry.
14. The method of claim 10 , wherein the determining is further based on a parameter relating to a precision sensitivity of a sub-component of input data for the inference operation.
15. A memory device, comprising:
a digital processor;
a first memory array configured to serve as an analog in-memory compute unit;
a second memory array configured to digitally store data; and
control circuitry configured to:
for each stage of a plurality of stages of a neural network computation:
retrieve data from the second memory array;
based on the retrieved data, determine whether a current stage can be performed in both digital and analog domains;
in response to the current stage determined to be capable of being performed in both digital and analog domains, determine whether the digital processor or the analog in-memory compute unit is to execute the current stage based on:
available storage resources of the digital processor and the analog in-memory compute unit;
storage workload of the digital processor and the analog in-memory compute unit;
compute power available of the digital processor and the analog in-memory compute unit; and
precision sensitivity of the retrieved data, wherein precision sensitivity is measured by a signal-to-noise ratio (SNR) value;
direct the current stage of the neural network computation and the retrieved data to the analog in-memory compute unit or the digital processor based on the determining that the digital processor or the analog in-memory compute unit is to execute the current stage, wherein the directing comprises:
routing the retrieved data having a SNR value greater than a threshold to the digital processor; and
routing the retrieved data having a SNR value less than the threshold to the analog in-memory compute unit.
16. The memory device of claim 15 , wherein the first and second memory arrays comprise the same type of memory elements and are fabricated in a single package, and wherein the digital processor and the control circuitry are fabricated within the same single package.
17. The memory device of claim 15 , wherein stages of a neural network computation relate to training the neural network.
18. The memory device of claim 15 , wherein stages of the neural network computation relate to an inference operation using a neural network.
19. The memory device of claim 15 , wherein the control circuitry is further configured to downscale data associated with a current stage of the neural network computation to be performed in the analog in-memory compute unit, prior to sending the current stage to the analog in-memory compute unit.
20. The memory device of claim 15 , wherein the control circuitry is further configured to upscale data associated with a current stage of the neural network computation to be performed in the digital processor, prior to sending the current stage to the digital processor.
21. A memory device, comprising:
a digital processor configured to perform computations in a digital domain;
an analog in-memory compute unit configured to perform computations in an analog domain, the analog in-memory compute unit comprising an array of memory elements configured to store weights for a neural network;
an analog input interface;
a digital input interface; and
control circuitry configured to:
receive analog sensor data from the analog input interface;
receive digital sensor data from the digital input interface;
determine precision sensitivity of the received analog sensor data and the received digital sensor data, wherein precision sensitivity is measured by a signal-to-noise ratio (SNR) value;
if the SNR value of the received analog sensor data is less than a threshold, direct the received analog sensor data to the analog in-memory compute unit to perform at least one computation related to the neural network; and
if the SNR value of the received digital sensor data is greater than the threshold, direct the received digital sensor data to the digital processor to perform at least one other computation related to the neural network.