System and methods for processing spatial data
A system for processing spatial data may be designed to receive neural network outputs corresponding to a first spatial data set, and translate the neural network outputs corresponding to the first spatial data set based on the motion between a second spatial data set and the first spatial data set. The system may perform zero-gap run length encoding on the neural network outputs to store the neural network outputs in memory. The system may also perform on-the-fly skip zero decoding and bilinear interpolation to translate the neural network outputs.
1. A method for processing spatial data comprising:
configuring at least one processor to implement at least one neural network comprising at least a first neural network portion and a second neural network portion different than the first neural network portion;
processing a first spatial data set in the first neural network portion of the neural network to generate a first plurality of neural network outputs for the first spatial data set;
generating a plurality of predicted neural network outputs of the first neural network portion for a second spatial data set different than the first spatial data set, based at least in part on motion estimation performed in the at least one processor between at least portions of the first and second spatial data sets, the plurality of predicted neural network outputs being generated utilizing at least a portion of the first plurality of neural network outputs generated by the first neural network portion for the first spatial data set; and
processing in the second neural network portion at least respective subsets of both (i) the first plurality of neural network outputs generated by the first neural network portion for the first spatial data set and (ii) the plurality of predicted neural network outputs of the first neural network portion generated for the second spatial data set;
wherein the processing of the first spatial data set in the first neural network portion and the generating of the plurality of predicted neural network outputs further comprise:
receiving the first spatial data set and dividing the first spatial data set into a first plurality of receptive fields;
processing, by the first neural network portion, the first plurality of receptive fields to obtain the first plurality of neural network outputs, wherein each neural network output corresponds to a receptive field in the first plurality of receptive fields;
storing, in memory, the first plurality of neural network outputs;
receiving the second spatial data set and dividing the second spatial data set into a second plurality of receptive fields;
identifying, in the at least one processor, for each receptive field in the second plurality of receptive fields a prior location in the first spatial data set, wherein the prior location is between multiple receptive field locations in the first plurality of receptive fields;
obtaining, from memory, the neural network outputs corresponding to the receptive fields of the first plurality of receptive fields proximate to the identified prior location for each receptive field in the second plurality of receptive fields; and
calculating, in the at least one processor, the plurality of predicted neural network outputs for the second plurality of receptive fields by, for a given receptive field in the second plurality of receptive fields, interpolating between the neural network outputs corresponding to the receptive fields proximate to the prior location of the given receptive field in the first spatial data set;
wherein the method further comprises:
performing at least one computer vision task based at least in part on results of the processing, in the second neural network portion, of the at least respective subsets of both the first plurality of neural network outputs and the plurality of predicted neural network outputs.
2. The method of claim 1 , wherein storing the first plurality of neural network outputs includes run-length encoding the first plurality of neural network outputs.
3. The method of claim 2 , wherein calculating the plurality of predicted neural network outputs further includes performing on-the-fly skip-zero decoding.
4. The method of claim 3 , wherein performing on-the-fly skip-zero decoding for a given receptive field in the second spatial data set further includes storing the neural network outputs obtained from memory for the receptive fields proximate the identified prior location of the given receptive field in the second spatial data set in a plurality of decoding lanes, wherein each decoding lane stores zero-gap run-length encoded neural network outputs for a plurality of channels associated with a different receptive field of the first spatial data set located proximate to the identified prior location of the given receptive field of the second spatial data set.
5. The method of claim 1 , wherein interpolating between the neural network outputs includes performing bilinear interpolation.
6. The method of claim 1 , wherein at least one of the first neural network portion and the second neural network portion comprises a convolutional neural network portion.
7. The method of claim 1 , wherein the first spatial data set and the second spatial data set each comprise at least one video frame from a continuous stream of video frames.
8. The method of claim 7 , wherein the at least one video frame of the second spatial data set is subsequent relative to the at least one video frame of the first spatial data set.
9. The method of claim 1 , wherein at least parts of the first spatial data set and the second spatial data set are processed by at least one of an artificial intelligence (AI) processor and an AI accelerator of the at least one processor.
10. The method of claim 9 , wherein the at least one processor comprises an application-specific integrated circuit (ASIC).
11. The method of claim 9 , wherein the at least one processor comprises a field programmable gate array (FPGA).
12. The method of claim 1 , wherein the at least one processor comprises at least one of: a graphics processing unit or a vector processing unit.
13. A system for processing spatial data comprising:
a memory configured to receive and store a first spatial data set and a second spatial data set different than the first spatial data set, and at least one processor coupled to the memory, the at least one processor being configured to:
implement at least one neural network comprising at least a first neural network portion and a second neural network portion different than the first neural network portion;
process the first spatial data set in the first neural network portion of the neural network to generate a first plurality of neural network outputs for the first spatial data set;
generate a plurality of predicted neural network outputs of the first neural network portion for the second spatial data set different than the first spatial data set, based at least in part on motion estimation performed in the at least one processor between at least portions of the first and second spatial data sets, the plurality of predicted neural network outputs being generated utilizing at least a portion of the first plurality of neural network outputs generated by the first neural network portion for the first spatial data set; and
process in the second neural network portion at least respective subsets of both (i) the first plurality of neural network outputs generated by the first neural network portion for the first spatial data set and (ii) the plurality of predicted neural network outputs of the first neural network portion generated for the second spatial data set;
wherein the at least one processor, in the processing of the first spatial data set in the first neural network portion and the generating of the plurality of predicted neural network outputs, is further configured to:
divide the first spatial data set into a first plurality of receptive fields and divide the second spatial data set into a second plurality of receptive fields;
process the first plurality of receptive fields in the first neural network portion to obtain the first plurality of neural network outputs, wherein each neural network output corresponds to a receptive field in the first plurality of receptive fields;
store each neural network output in the memory;
identify, for each receptive field in the second plurality of receptive fields, a prior location in the first spatial data set, wherein the prior location is between multiple receptive field locations in the first spatial data set;
receive, from the memory, the neural network outputs corresponding to the receptive fields of the first spatial data set proximate to the identified prior location for each receptive field in the second plurality of receptive fields; and
calculate the plurality of predicted neural network outputs for the second plurality of receptive fields by interpolating between the neural network outputs corresponding to the receptive fields proximate to the prior locations in the first spatial data set;
wherein the at least one processor is further configured to:
perform at least one computer vision task based at least in part on results of the processing, in the second neural network portion, of the at least respective subsets of both the first plurality of neural network outputs and the plurality of predicted neural network outputs.
14. The system of claim 13 , wherein the at least one processor includes an artificial intelligence (AI) accelerator configured to perform zero-gap run-length encoding on the first plurality of neural network outputs and store the zero-gap run-length encoded first plurality of neural network outputs in the memory.
15. The system of claim 14 , wherein the at least one processor includes an interpolation processing unit configured to calculate the plurality of predicted neural network outputs, wherein calculating the plurality of predicted neural network outputs includes performing on-the-fly skip-zero decoding on the zero-gap run-length encoded first plurality of neural network outputs.
16. The system of claim 15 , wherein the interpolation processing unit is further configured to perform on-the-fly skip-zero decoding for a given receptive field in the second spatial data set by storing the neural network outputs obtained from the memory for the receptive fields proximate the identified prior location of the given receptive field in a plurality of decoding lanes, wherein each decoding lane stores the zero-gap run-length encoded neural network outputs for a plurality of channels associated with a different receptive field of the first spatial data set located proximate the identified prior location of the given receptive field of the second spatial data set.
17. The system of claim 13 , wherein interpolating between the neural network outputs includes configuring the at least one processor to perform bilinear interpolation.
18. The system of claim 13 , wherein the at least one processor includes an artificial intelligence (AI) processor comprising the first neural network portion as a convolutional neural network portion, wherein the AI processor is configured to process the first plurality of receptive fields to obtain the first plurality of neural network outputs, wherein each neural network output corresponds to a receptive field in the first plurality of receptive fields.
19. The system of claim 13 , wherein the memory comprises a buffer memory.
20. The system of claim 13 , wherein the memory comprises at least one of content addressable memory, cache memory, and flash memory.
21. The system of claim 13 , wherein generating predicted outputs of the first neural network portion for the second spatial data set further comprises bypassing processing of at least a part of the second spatial data set in the first neural network portion.