Machine learning models operating at different frequencies for autonomous vehicles
Systems and methods include machine learning models operating at different frequencies. An example method includes obtaining images at a threshold frequency from one or more image sensors positioned about a vehicle. Location information associated with objects classified in the images is determined based on the images. The images are analyzed via a first machine learning model at the threshold frequency. For a subset of the images, the first machine learning model uses output information from a second machine learning model, the second machine learning model being performed at less than the threshold frequency.
1. A method implemented by a system of one or more processors included in a vehicle, the method comprising:
implementing an image processing pipeline in which a neural network and a machine learning model are used to collectively output location information associated with objects depicted in sets of images, each set of images being obtained via cameras at, at least, a first frequency, wherein the cameras are positioned about the vehicle,
wherein the neural network:
receives individual sets of images at a second frequency which is lower than the first frequency and computes respective forward passes of individual sets of images at the second frequency,
wherein the neural network is configured to detect locations associated with objects,
wherein the machine learning model:
receives individual sets of images at the first frequency and computes respective forward passes of input including individual sets of images at the first frequency,
wherein the machine learning model is configured to update locations associated with the objects;
and wherein at least a subset of the inputs of the machine learning model includes output from the neural network.
2. The method of claim 1 , wherein the machine learning model computes motion flow information associated with objects.
3. The method of claim 1 , wherein output from the neural network includes information identifying the objects.
4. The method of claim 1 , wherein the machine learning model is a different neural network.
5. The method of claim 1 , wherein the machine learning model is a support vector machine.
6. The method of claim 1 , wherein processing of the neural network is performed during times at which processing of the machine learning model is not performed or wherein processing of the neural network is interrupted based on processing of the machine learning model.
7. A system configured for inclusion in a vehicle, the system comprising one or more processors and computer storage media storing instructions that when executed by the one or more processors, cause the one or more processors to:
implement an image processing pipeline in which a neural network and a machine learning model are used to collectively output location information associated with objects depicted in sets of images, each set of images being obtained via cameras at, at least, a first frequency, wherein the cameras are positioned about the vehicle,
wherein the neural network:
receives individual sets of images at a second frequency which is lower than the first frequency and computes respective forward passes of individual sets of images at the second frequency,
wherein the neural network is configured to detect locations associated with objects,
wherein the machine learning model:
receives individual sets of images at the first frequency and computes respective forward passes of input including individual sets of images at the first frequency,
wherein the machine learning model is configured to update locations associated with the objects;
and wherein at least a subset of the inputs of the machine learning model includes output from the neural network.
8. The system of claim 7 , wherein the machine learning model computes motion flow information associated with objects.
9. The system of claim 7 , wherein output from the neural network includes information identifying the objects.
10. The system of claim 7 , wherein the machine learning model is a different neural network.
11. The system of claim 7 , wherein the machine learning model is a support vector machine.
12. The system of claim 7 , wherein processing of the neural network is performed during times at which processing of the machine learning model is not performed or wherein processing of the neural network is interrupted based on processing of the machine learning model.
13. Non-transitory computer storage media storing instructions for execution by a system configured for inclusion in a vehicle, wherein the instructions cause the system to:
implement an image processing pipeline in which a neural network and a machine learning model are used to collectively output location information associated with objects depicted in sets of images, each set of images being obtained via cameras at, at least, a first frequency, wherein the cameras are positioned about the vehicle,
wherein the neural network:
receives individual sets of images at a second frequency which is lower than the first frequency and computes respective forward passes of individual sets of images at the second frequency,
wherein the neural network is configured to detect locations associated with objects,
wherein the machine learning model:
receives individual sets of images at the first frequency and computes respective forward passes of input including individual sets of images at the first frequency, and
wherein the machine learning model is configured to update locations associated with the objects;
wherein at least a subset of the inputs of the machine learning model includes output from the neural network.
14. The computer storage media of claim 13 , wherein the machine learning model computes motion flow information associated with objects.
15. The computer storage media of claim 13 , wherein output from the neural network includes information identifying the objects.
16. The computer storage media of claim 13 , wherein the machine learning model is a different neural network, or wherein the machine learning model is a support vector machine.
17. The computer storage media of claim 13 , wherein processing of the neural network is performed during times at which processing of the machine learning model is not performed or wherein processing of the neural network is interrupted based on processing of the machine learning model.