Processing sensor data for neural networks
Systems and techniques are disclosed for processing sensor data for neural networks. An example method can include receiving, from one or more sensors, one or more slices of a data frame, and wherein the one or more slices are captured sequentially by the one or more sensors; generating a mapping of operations identifying a set of operations to be applied by a neural network to the one or more slices as the one or more slices are received by the neural network and a set of dependencies for the set of operations; transmitting the one or more slices and the mapping of operations to the neural network; and processing, via the neural network, the one or more slices based on the mapping of operations.
1 . A system comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
receive, from one or more sensors, one or more slices of a data frame, and wherein, when the data frame comprises more than one slice, the slices are captured sequentially by the one or more sensors;
generate a mapping of operations identifying a set of operations to be applied by a neural network to the one or more slices as the one or more slices are received by the neural network and a set of dependencies for the set of operations;
transmit the one or more slices and the mapping of operations to the neural network; and
process, via the neural network, the one or more slices based on the mapping of operations;
wherein the mapping of operations comprise an explicit dependency graph generated before inference and transmitted with each sequentially captured sensor slice; and
wherein the neural network schedules execution for each slice upon receipt according to the dependency graph without waiting for a full data frame.
2 . The system of claim 1 , wherein the one or more sensors comprises at least one of a Light Detection and Ranging (LiDAR) sensor, a camera sensor, a radio detection and ranging (RADAR) sensor, or a combination thereof.
3 . The system of claim 1 , wherein the mapping of operations is generated by a front end system comprising a driver and a loader.
4 . The system of claim 3 , wherein the at least one processor is configured to control, via the driver, a flow of the one or more slices from the one or more sensors into the loader.
5 . The system of claim 3 , wherein the loader processes the one or more slices from the one or more sensors and transmits the one or more slices to the neural network, and wherein the loader processes the one or more slices via a normalization step, wherein the normalization step uses a normalizer from a previous data frame.
6 . The system of claim 1 , wherein the set of dependencies indicate at least one of a first operation from the set of operations that should be applied to the one or more slices before applying a second operation from the set of operations to the one or more slices and a sequence in which the set of operations should be applied to the one or more slices.
7 . The system of claim 1 , wherein the set of operations comprises at least one of convolution, pooling, downsampling, activation functions, fully connected layers, output prediction, loss calculation, backpropagation, or a combination thereof.
8 . A computer-implemented method comprising:
receiving, from one or more sensors, one or more slices of a data frame, and wherein, when the data frame comprises more than one slice, the slices are captured sequentially by the one or more sensors;
generating a mapping of operations identifying a set of operations to be applied by a neural network to the one or more slices as the one or more slices are received by the neural network and a set of dependencies for the set of operations;
transmitting the one or more slices and the mapping of operations to the neural network; and
processing, via the neural network, the one or more slices based on the mapping of operations;
wherein the mapping of operations comprise an explicit dependency graph generated before inference and transmitted with each sequentially captured sensor slice; and
wherein the neural network schedules execution for each slice upon receipt according to the dependency graph without waiting for a full data frame.
9 . The computer-implemented method of claim 8 , wherein the one or more sensors comprises at least one of a Light Detection and Ranging (LiDAR) sensor, a camera sensor, a radio detection and ranging (RADAR) sensor, or a combination thereof.
10 . The computer-implemented method of claim 8 , wherein the mapping of operations is generated by a front end system comprising a driver and a loader.
11 . The computer-implemented method of claim 10 , wherein the at least one processor is configured to control, via the driver, a flow of the one or more slices from the one or more sensors into the loader.
12 . The computer-implemented method of claim 10 , wherein the loader processes the one or more slices from the one or more sensors and transmits the one or more slices to the neural network, and wherein the loader processes the one or more slices via a normalization step, wherein the normalization step uses a normalizer from a previous data frame.
13 . The computer-implemented method of claim 8 , wherein the set of dependencies indicate at least one of a first operation from the set of operations that should be applied to the one or more slices before applying a second operation from the set of operations to the one or more slices and a sequence in which the set of operations should be applied to the one or more slices.
14 . The computer-implemented method of claim 8 , wherein the set of operations comprises at least one of convolution, pooling, downsampling, activation functions, fully connected layers, output prediction, loss calculation, backpropagation, or a combination thereof.
15 . A non-transitory computer-readable storage medium comprising at least one instruction for causing a computer or processor to:
receive, from one or more sensors, one or more slices of a data frame, and wherein, when the data frame comprises more than one slice, the one or more slices are captured sequentially by the one or more sensors;
generate a mapping of operations identifying a set of operations to be applied by a neural network to the one or more slices as the one or more slices are received by the neural network and a set of dependencies for the set of operations; and
transmit the one or more slices and the mapping of operations to the neural network; and process, via the neural network, the one or more slices based on the mapping of operations;
wherein the mapping of operations comprise an explicit dependency graph generated before inference and transmitted with each sequentially captured sensor slice; and
wherein the neural network schedules execution for each slice upon receipt according to the dependency graph without waiting for a full data frame.
16 . The non-transitory computer-readable storage medium of claim 15 , wherein the one or more sensors comprises at least one of a Light Detection and Ranging (LiDAR) sensor, a camera sensor, a radio detection and ranging (RADAR) sensor, or a combination thereof.
17 . The non-transitory computer-readable storage medium of claim 15 , wherein the mapping of operations is generated by a front end system comprising a driver and a loader.
18 . The non-transitory computer-readable storage medium of claim 17 , wherein the at least one processor is configured to control, via the driver, a flow of the one or more slices from the one or more sensors into the loader.
19 . The non-transitory computer-readable storage medium of claim 17 , wherein the loader processes the one or more slices from the one or more sensors and transmits the one or more slices to the neural network, and wherein the loader processes the one or more slices via a normalization step, wherein the normalization step uses a normalizer from a previous data frame.
20 . The non-transitory computer-readable storage medium of claim 15 , wherein the set of dependencies indicate at least one of a first operation from the set of operations that should be applied to the one or more slices before applying a second operation from the set of operations to the one or more slices and a sequence in which the set of operations should be applied to the one or more slices.