Compute in memory architecture and dataflows for depth-wise separable convolution

1. An apparatus, comprising:

a compute-in-memory (CIM) array configured to:

store a depthwise convolution kernel in a first one or more columns of the CIM array;

store a fused convolution kernel in a second one or more columns of the CIM array;

store pre-activations in a plurality of rows of the CIM array;

process the pre-activations with the depthwise convolution kernel in order to generate depthwise output;

modify one or more of the pre-activations based on the depthwise output to generate modified pre-activations; and

process the modified pre-activations with the fused convolution kernel to generate fused output.

2. The apparatus of claim 1 , further comprising:

a comparator configured to compare the depthwise output to a threshold; and

a gain circuit connected to each respective row of the plurality of rows of the CIM array and configured to modify a pre-activation stored in its respective row if the depthwise output is less than the threshold.

3. The apparatus of claim 1 , wherein:

the pre-activations correspond to one channel-wise group of pre-activation data, and

the depthwise output corresponds to one channel of the depthwise convolution kernel.

4. The apparatus of claim 1 , wherein in order to modify the one or more of the pre-activations based on the depthwise output, the CIM array is further configured to set the one or more of the pre-activations values to zero.

5. The apparatus of claim 1 , wherein the CIM array is further configured to disable one or more rows of the CIM array associated with the modified pre-activations.

6. The apparatus of claim 1 , further comprising a fusion circuit configured to fuse the depthwise convolution kernel and a pointwise convolution kernel in order to generate the fused convolution kernel.

7. The apparatus of claim 1 , further comprising:

a sequential accumulator connected to each column of the first one or more columns of the CIM array in which the depthwise convolution kernel is stored; and

a comparator configured to receive accumulated output from at least one sequential accumulator connected to the first one or more columns of the CIM array.

8. The apparatus of claim 1 , further comprising:

a sequential accumulator connected to each column of the second one or more columns of the CIM array in which the fused convolution kernel is stored;

an analog-to-digital converter (ADC) configured to receive accumulated output from at least one sequential accumulator connected to the second one or more columns of the CIM array and output digital data; and

a nonlinear operation circuit configured to process the digital data and generate activation data.

9. The apparatus of claim 8 , further comprising a scaling circuit configured to scale the digital data output from the ADC from a first bit width to a second bit width prior to providing the digital data to the nonlinear operation circuit.

10. The apparatus of claim 1 , wherein the CIM array comprises a plurality of static random access memory (SRAM) bitcells.

11. A method, comprising:

storing a depthwise convolution kernel in a first one or more columns of a CIM array;

storing a fused convolution kernel in a second one or more columns of the CIM array;

storing pre-activations in one or more input data buffers associated with a plurality of rows of the CIM array;

processing the pre-activations with the depthwise convolution kernel in order to generate depthwise output;

modifying one or more of the pre-activations based on the depthwise output to generate modified pre-activations; and

processing the modified pre-activations with the fused convolution kernel to generate fused output.

12. The method of claim 11 , further comprising: comparing the depthwise output to a threshold, wherein modifying the one or more of the pre-activations comprises setting a gain for the one or more of the pre-activations to zero.

13. The method of claim 11 , wherein modifying the one or more of the pre-activations comprises disabling the rows of the CIM array associated with the one or more of the pre-activations.

14. The method of claim 11 , wherein the pre-activations comprise one channel-wise group of pre-activations of a plurality of channel-wise groups of pre-activations.

15. The method of claim 14 , wherein processing the modified pre-activations with the fused convolution kernel to generate fused output is performed after each channel-wise group of pre-activations of the plurality of channel-wise groups of pre-activations has been processed by the depthwise convolution kernel.

16. The method of claim 11 , further comprising converting the fused output to digital output data via an analog-to-digital converter (ADC).

17. The method of claim 16 , further comprising processing the digital output data with a nonlinear operation to generate activation data.

18. The method of claim 17 , further comprising providing the activation data to a host processing system.

19. The method of claim 17 , further comprising providing the activation data to an activation buffer for processing another layer of a model using the CIM array.

20. The method of claim 17 , further comprising scaling the digital output data prior to processing the digital output data with the nonlinear operation.

21. The method of claim 11 , further comprising: fusing the depthwise convolution kernel and a pointwise convolution kernel in order to generate the fused convolution kernel.

22. The method of claim 11 , wherein the CIM array comprises a plurality of static random access memory (SRAM) bitcells.

23. The method of claim 12 , further comprising scaling the one or more of the pre-activations to reduce range compression of the fused output.

24. The method of claim 23 , wherein the scaling is performed only on the one or more of the pre-activations having a depthwise output greater than the threshold.

25. A non-transitory computer-readable medium comprising instructions that, when executed by a processor of a processing system, cause the processing system to perform a method, the method comprising:

storing a depthwise convolution kernel in a first one or more columns of a CIM array;

storing a fused convolution kernel in a second one or more columns of the CIM array;

storing pre-activations in one or more input data buffers associated with a plurality of rows of the CIM array;

processing the pre-activations with the depthwise convolution kernel in order to generate depthwise output;

modifying one or more of the pre-activations based on the depthwise output to generate modified pre-activations; and

processing the modified pre-activations with the fused convolution kernel to generate fused output.

26. The non-transitory computer-readable medium of claim 25 , wherein the method further comprises: comparing the depthwise output to a threshold, wherein modifying the one or more of the pre-activations comprises setting a gain for the one or more of the pre-activations to zero.

27. The non-transitory computer-readable medium of claim 25 , wherein modifying the one or more of the pre-activations comprises disabling the rows of the CIM array associated with the one or more of the pre-activations.

28. The non-transitory computer-readable medium of claim 25 , wherein the pre-activations comprise one channel-wise group of pre-activations of a plurality of channel-wise groups of pre-activations.

29. The non-transitory computer-readable medium of claim 28 , wherein processing the modified pre-activations with the fused convolution kernel to generate fused output is performed after each channel-wise group of pre-activations of the plurality of channel-wise groups of pre-activations has been processed by the depthwise convolution kernel.