Machine-learned architecture for efficient object attribute and/or intention classification
A system for faster object attribute and/or intent classification may include an machine-learned (ML) architecture that processes temporal sensor data (e.g., multiple instances of sensor data received at different times) and includes a cache in an intermediate layer of the ML architecture. The ML architecture may be capable of classifying an object's intent to enter a roadway, idling near a roadway, or active crossing of a roadway. The ML architecture may additionally or alternatively classify indicator states, such as indications to turn, stop, or the like. Other attributes and/or intentions are discussed herein.
1. A method comprising:
receiving a first image associated with a first time;
receiving a second image associated with a second time prior to the first time, wherein the first image and the second image are associated with a same object;
determining, by a first set of machine-learned layers and based at least in part on the first image, a first output;
storing the first output in a memory;
retrieving a second output from the memory, the second output being associated with the second image and having been previously processed by the first set of machine-learned layers and stored in the memory;
determining, by one or more machine-learned layers and based at least in part on the first output and the second output, a first confidence score associated with a first pedestrian attribute and a second confidence score associated with a second pedestrian attribute,
wherein the first pedestrian attribute indicates one of: a pedestrian is idling, the pedestrian intends to enter a roadway, or the pedestrian is crossing a roadway; and the second pedestrian attribute indicates a different one of: the pedestrian is idling, the pedestrian intends to enter a roadway, or the pedestrian is crossing a roadway; and
controlling a vehicle based at least in part on at least one of the first confidence score or the second confidence score.
2. The method of claim 1 , wherein the one or more machine-learned layers comprise a second machine-learned layer, a third machine-learned layer, and a fourth machine learned layer; and determining the first confidence score associated with the first pedestrian attribute and the second confidence score associated with the second pedestrian attribute comprises:
determining, by the second machine-learned layer and based at least in part on the first output, a third output;
determining, by the second machine-learned layer and based at least in part on the second output, a fourth output;
concatenating, as a concatenated output, the third output and the fourth output;
providing the concatenated output as an input to the third machine-learned layer;
receiving a fifth output from the third machine-learned layer;
providing the fifth output to the fourth machine-learned layer as input; and
receiving the third confidence score and the fourth confidence score from the fourth machine-learned layer.
3. The method of claim 2 , wherein the first set of machine-learned layers comprises multiple layers of a neural network and the second machine-learned layer, third machine-learned layer, and the fourth machine-learned layer comprise different fully-connected layers.
4. The method of claim 1 , wherein:
the first image is a first portion of a first larger image;
the second image is a second portion of a second larger image; and
the first portion and the second portion are generated by a first machine-learned component based at least in part on the first larger image and the second larger image.
5. A system comprising:
one or more processors; and
a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving a first image associated with a first time;
receiving a second image associated with a second time prior to the first time;
determining, by a first set of machine-learned layers and based at least in part on the first image, a first output;
storing the first output in a memory;
retrieving a second output from the memory, the second output being associated with the second image;
determining, by one or more machine-learned layers and based at least in part on the first output and the second output, an attribute associated with an object in an environment, wherein the attribute indicates one of: a pedestrian is idling, the pedestrian intends to enter a roadway, or the pedestrian is crossing a roadway and the first output and the second output are different; and
controlling a vehicle based at least in part on the attribute.
6. The system of claim 5 , wherein the attribute associated with the object instead comprises:
an indication of a vehicle motion state;
an indication of an object indicator state;
an indication that the object is idling;
an indication that the object intends to enter a roadway; or
an indication that the object is crossing the roadway.
7. The system of claim 5 , wherein the memory is a cache and the cache stores n number of outputs of the first set of machine-learned layers, wherein n is a positive integer associated with n previous time steps.
8. The system of claim 5 , wherein the first output and the second output are each processed by the first set of machine-learned layers once.
9. The system of claim 5 , wherein the attribute indicates that a pedestrian is idling near a roadway, the pedestrian intends to enter the roadway, the pedestrian is crossing the roadway, the pedestrian is ready to enter the roadway, or the pedestrian is not related to the roadway.
10. The system of claim 5 , wherein the one or more machine-learned layers comprise a second machine-learned layer, a third machine-learned layer, and a fourth machine-learned layer; and determining the attribute comprises:
determining, by the second machine-learned layer and based at least in part on the first output, a third output;
determining, by the second machine-learned layer and based at least in part on the second output, a fourth output;
concatenating, as a concatenated output, the third output and the fourth output;
providing the concatenated output as an input to the third machine-learned layer;
receiving a fifth output from the third machine-learned layer;
providing the fifth output to the fourth machine-learned layer as input; and
receiving a confidence score associated with the attribute from the fourth machine-learned layer.
11. The system of claim 10 , wherein the first set of machine-learned layers comprises multiple layers of a neural network and the second machine-learned layer, third machine-learned layer, and the fourth machine-learned layer comprise different fully-connected layers.
12. The system of claim 5 , wherein:
the first image is a first portion of a first larger image;
the second image is a second portion of a second larger image; and
the first portion and the second portion are generated by a first machine-learned component based at least in part on the first larger image and the second larger image.
13. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a first image associated with a first time;
receiving a second image associated with a second time prior to the first time;
determining, by a first set of machine-learned layers and based at least in part on the first image, a first output;
storing the first output in a memory;
retrieving a second output from the memory, the second output being associated with the second image;
determining, by one or more machine-learned layers and based at least in part on the first output and the second output, an attribute associated with an object in an environment, wherein the attribute indicates one of: a pedestrian is idling, the pedestrian intends to enter a roadway, or the pedestrian is crossing a roadway and the first output and the second output are different; and
controlling a vehicle based at least in part on the attribute.
14. The one or more non-transitory computer-readable media of claim 13 , wherein the attribute associated with the object instead comprises:
an indication of a vehicle motion state;
an indication of an object indicator state;
an indication that the object is idling;
an indication that the object intends to enter a roadway; or
an indication that the object is crossing the roadway.
15. The one or more non-transitory computer-readable media of claim 13 , wherein the memory is a cache and the cache stores n number of outputs of the first set of machine-learned layers, wherein n is a positive integer associated with n previous time steps.
16. The one or more non-transitory computer-readable media of claim 13 , wherein the first output and the second output are each processed by the first set of machine-learned layers once.
17. The one or more non-transitory computer-readable media of claim 13 , wherein the attribute indicates that a pedestrian is idling near a roadway, the pedestrian intends to enter the roadway, the pedestrian is crossing the roadway, the pedestrian is ready to enter the roadway, or the pedestrian is not related to the roadway.
18. The one or more non-transitory computer-readable media of claim 13 , wherein the one or more machine-learned layers comprise a second machine-learned layer, a third machine-learned layer, and a fourth machine-learned layer; and determining the attribute comprises:
determining, by the second machine-learned layer and based at least in part on the first output, a third output;
determining, by the second machine-learned layer and based at least in part on the second output, a fourth output;
concatenating, as a concatenated output, the third output and the fourth output;
providing the concatenated output as an input to the third machine-learned layer;
receiving a fifth output from the third machine-learned layer;
providing the fifth output to the fourth machine-learned layer as input; and
receiving a confidence score associated with the attribute from the fourth machine-learned layer.
19. The one or more non-transitory computer-readable media of claim 18 , wherein the first set of machine-learned layers comprises multiple layers of a neural network and the second machine-learned layer, third machine-learned layer, and the fourth machine-learned layer comprise different fully-connected layers.