Accelerated mathematical engine
Various embodiments of the disclosure relate to an accelerated mathematical engine. In certain embodiments, the accelerated mathematical engine is applied to image processing such that convolution of an image is accelerated by using a two-dimensional matrix processor comprising sub-circuits that include an ALU, output register and shadow register. This architecture supports a clocked, two-dimensional architecture in which image data and weights are multiplied in a synchronized manner to allow a large number of mathematical operations to be performed in parallel.
1. A matrix processor comprising:
a first input circuit configured to receive sensor data, wherein the sensor data is organized into individual first vectors, wherein an initial first vector is provided along a first direction of the matrix processor;
a second input circuit configured to receive one or more filters of a plurality of filters, wherein the one or more filters are organized into individual second vectors, wherein an initial second vector is provided along a second direction of the matrix processor; and
a plurality of sub-circuits arranged as a matrix along the first direction and second direction, wherein the sub-circuits are configured to convolve the sensor data and filters, wherein to convolve the sensor data and filters, the sub-circuits are configured to:
determine first partial dot products based on respective values included in the initial first vector and the initial second vector, wherein the first partial dot products are stored in the sub-circuits,
determine subsequent partial dot products based on respective values included in subsequent first vectors and subsequent second vectors,
wherein the sub-circuits are configured to add the first partial dot products stored in the accumulators with the subsequent partial dot products,
and wherein each sub-circuit stores a respective output pixel associated with the convolution.
2. The matrix processor of claim 1 , wherein the sub-circuits comprise respective multipliers, adders, and accumulators.
3. The matrix processor of claim 2 , wherein the first partial dot products are stored using the accumulators.
4. The matrix processor of claim 1 , wherein the first input circuit is associated with a first formatter configured to fetch the sensor data and organize the sensor data in the individual first vectors.
5. The matrix processor of claim 1 , wherein a portion of an output result associated with the convolution is shifted from a bottom row of the sub-circuits to output flip-flops.
6. The matrix processor of claim 1 , wherein at least a portion of the sub-circuits share a particular encoder, and wherein the particular encoder is a booth encoder.
7. The matrix processor of claim 1 , wherein the matrix processor implements a state machine configured to identify redundant data.
8. The matrix processor of claim 7 , wherein identifying redundant data is based on input comprising respective sizes associated with individual filters of the plurality of filters and/or individual strides of one or more strides.
9. The matrix processor of claim 1 , wherein each sub-circuit stores a respective output pixel associated with the convolution of a particular filter and a portion of the sensor data.
10. The matrix processor of claim 1 , wherein the matrix processor comprises an array of tiles, and wherein the tiles comprise respective subsets of the sub-circuits.
11. The matrix processor of claim 1 , wherein the sensor data comprises image data, LIDAR data, ultrasonic data, or radar data.
12. A system comprising:
a first logic circuit configured to format sensor data into individual first vectors, wherein an initial first vector is provided along a first direction of a matrix processor;
a second logic circuit configured to format one or more filters into individual second vectors, wherein an initial second vector is provided along a second direction of the matrix processor; and
the matrix processor comprising a plurality of sub-circuits arranged as a matrix along the first direction and second direction, wherein the sub-circuits are configured to convolve the sensor data and filters,
wherein to convolve the sensor data and filters, the sub-circuits are configured to accumulate partial dot products associated with individual first vectors and individual second vectors, wherein each sub-circuit stores a respective output pixel associated with the convolution.
13. The system of claim 12 , wherein accumulating partial products comprises:
determining first partial dot products based on respective values included in the initial first vector and the initial second vector, wherein the first partial dot products are stored in the sub-circuits; and
determining subsequent partial dot products based on respective values included in subsequent first vectors and subsequent second vectors,
wherein the sub-circuits are configured to add the first partial dot products stored in the accumulators with the subsequent partial dot products.
14. The system of claim 12 , wherein the sub-circuits comprise respective multipliers, adders, and accumulators.
15. The system of claim 12 , wherein individual partial dot products are stored using the accumulators.
16. The system of claim 12 , wherein each sub-circuit stores a respective output pixel associated with the convolution of a particular filter and a portion of the sensor data.
17. The system of claim 12 , wherein the matrix processor comprises an array of tiles, and wherein the tiles comprise respective subsets of the sub-circuits.
18. A method implemented by a matrix processor, the method comprising:
receiving, from a first logic circuit, sensor data, wherein the sensor data is organized into individual first vectors, wherein an initial first vector is provided along a first direction of the matrix processor;
receiving, from a second logic circuit, one or more filters of a plurality of filters, wherein the one or more filters are organized into individual second vectors, wherein an initial second vector is provided along a second direction of the matrix processor;
using a plurality of sub-circuits of the matrix processor to convolve the sensor data and filters, wherein the sub-circuits are arranged as a matrix along the first direction and second direction, and wherein to convolve the sensor data and filters the sub-circuits:
determine first partial dot products based on respective values included in the initial first vector and the initial second vector, wherein the first partial dot products are stored in the sub-circuits,
determine subsequent partial dot products based on respective values included in subsequent first vectors and subsequent second vectors,
wherein the sub-circuits are configured to add the first partial dot products stored in the accumulators with the subsequent partial dot products,
and wherein each sub-circuit stores a respective output pixel associated with the convolution.
19. The method of claim 18 , wherein the sub-circuits comprise respective multipliers, adders, and accumulators, and wherein the first partial dot products are stored using the accumulators.
20. The method of claim 18 , wherein each sub-circuit stores a respective output pixel associated with the convolution of a particular filter and a portion of the sensor data.