Multiple stage image based object detection and recognition
Systems, methods, tangible non-transitory computer-readable media, and devices for autonomous vehicle operation are provided. For example, a computing system can receive object data that includes portions of sensor data. The computing system can determine, in a first stage of a multiple stage classification using hardware components, one or more first stage characteristics of the portions of sensor data based on a first machine-learned model. In a second stage of the multiple stage classification, the computing system can determine second stage characteristics of the portions of sensor data based on a second machine-learned model. The computing system can generate an object output based on the first stage characteristics and the second stage characteristics. The object output can include indications associated with detection of objects in the portions of sensor data.
1. An autonomous vehicle control system for an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory, computer-readable media storing instructions that are executable to cause the one or more processors to perform operations comprising:
receiving sensor data descriptive of an environment of the autonomous vehicle;
determining in a first stage of a multiple stage classification, one or more first stage characteristics of the sensor data based in part on a first machine-learned model, wherein the first stage characteristics are determined by the first machine-learned model with a first level of confidence;
determining in a second stage of the multiple stage classification, one or more second stage characteristics of the sensor data based in part, on a second machine-learned model, wherein the second stage characteristics are determined by the second machine-learned model with a second level of confidence that is higher than the first level of confidence; and
generating an object output based in part on the second stage characteristics, the object output indicating detection of one or more objects in the sensor data.
2. The autonomous vehicle control system of claim 1 , wherein the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classification are indicative of a likelihood that the sensor data contains objects.
3. The autonomous vehicle control system of claim 1 , wherein the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classification are indicative of one or more portions of the sensor data being classified as background or foreground.
4. The autonomous vehicle control system of claim 1 , wherein the one or more second stage characteristics of the sensor data determined in the second stage of the multiple stage classification are indicative of an object classification for a type of object detected in the sensor data.
5. The autonomous vehicle control system of claim 1 , the operations further comprising:
generating in the first stage, a heat map associated with the sensor data, the heat map describing a probability of an object being contained within a respective area of the sensor data.
6. The autonomous vehicle control system of claim 1 , wherein an input to the second stage of the multiple stage classification is associated with one or more foreground portions of the sensor data.
7. The autonomous vehicle control system of claim 1 , the operations further comprising:
generating, in the first stage and based in part on the sensor data, visual descriptor output associated with the sensor data, the visual descriptor output comprising color hue information, color saturation information, brightness information, or histogram of oriented gradients information, wherein the one or more first stage characteristics are determined based in part on the visual descriptor output.
8. The autonomous vehicle control system of claim 1 , wherein the sensor data comprises one or more LIDAR features and one or more camera features.
9. The autonomous vehicle control system of claim 1 , the operations further comprising:
controlling a motion of the autonomous vehicle based in part on the object output.
10. A method comprising:
receiving sensor data descriptive of an environment of an autonomous vehicle;
determining in a first stage of a multiple stage classification, one or more first stage characteristics of the sensor data based in part on a first machine-learned model, wherein the first stage characteristics are determined by the first machine-learned model with a first level of confidence;
determining in a second stage of the multiple stage classification, one or more second stage characteristics of the sensor data based in part on a second machine-learned model, wherein the second stage characteristics are determined by the second machine-learned model with a second level of confidence that is higher than the first level of confidence; and
generating an object output based in part on the second stage characteristics, the object output indicating detection of one or more objects in the sensor data.
11. The method of claim 10 , wherein the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classification are indicative of a likelihood that the sensor data contains objects.
12. The method of claim 10 , wherein the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classification are indicative of one or more portions of the sensor data being background or foreground.
13. The method of claim 10 , wherein the one or more second stage characteristics of the sensor data determined in the second stage of the multiple stage classification are indicative of an object classification for a type of object detected in the sensor data.
14. The method of claim 10 , further comprising:
generating in the first stage, a heat map associated with the sensor data, the heat map describing a probability of an object being contained within a respective area of the sensor data.
15. The method of claim 10 , wherein an input to the second stage of the multiple stage classification is associated with one or more foreground portions of the sensor data.
16. The method of claim 10 , further comprising:
generating, in the first stage and based in part on the sensor data, visual descriptor output associated with the sensor data, the visual descriptor output comprising color hue information, color saturation information, brightness information, or histogram of oriented gradients information, wherein the one or more first stage characteristics are determined based in part on the visual descriptor output.
17. The method of claim 10 , wherein the sensor data comprises one or more LIDAR features and one or more camera features.
18. The method of claim 10 , further comprising:
controlling a motion of the autonomous vehicle based in part on the object output.
19. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
receiving sensor data descriptive of an environment of an autonomous vehicle;
determining in a first stage of a multiple stage classification, one or more first stage characteristics of the sensor data based in part on a first machine-learned model, wherein the first stage characteristics are determined by the first machine-learned model with a first level of confidence;
determining in a second stage of the multiple stage classification, one or more second stage characteristics of the sensor data based in part on a second machine-learned model,
wherein the second stage characteristics are determined by the second machine-learned model with a second level of confidence that is higher than the first level of confidence; and
generating an object output based in part on the second stage characteristics, the object output indicating detection of one or more objects in the sensor data.
20. The one or more tangible, non-transitory computer-readable media of claim 19 , wherein:
the one or more first stage characteristics of the sensor data determined in the first stage of the multiple stage classification are indicative of one or more portions of the sensor data being background or foreground; and
the one or more second stage characteristics of the sensor data determined in the second stage of the multiple stage classification are indicative of an object classification for a type of object detected in the sensor data.