Object detection with images
A computer is programmed to receive image data from a sensor; generate a feature pyramid from the image data, the feature pyramid including a plurality of features; apply a plurality of preliminary bounding boxes to the features to generate a plurality of preliminarily bounded features, each preliminarily bounded feature being a pairing of one of the preliminary bounding boxes and one of the features; execute a machine-learning program on the preliminarily bounded features to determine a plurality of classifications and a respective plurality of predicted bounding boxes; and actuate a component of a machine, e.g., a vehicle, based on the classifications and the predicted bounding boxes. The machine-learning program is a two-stage object detector having a first stage and a second stage. The first stage selects a subset of the preliminarily bounded features to pass to the second stage.
1 . A computer comprising a processor and a memory, the memory storing instructions executable by the processor to:
receive image data from a sensor;
generate a feature pyramid from the image data, the feature pyramid including a plurality of features;
apply a plurality of preliminary bounding boxes to the features to generate a plurality of preliminarily bounded features, each preliminarily bounded feature being a pairing of one of the preliminary bounding boxes and one of the features;
execute a machine-learning program on the preliminarily bounded features to determine a plurality of classifications and a respective plurality of predicted bounding boxes; and
actuate a machine component based on the classifications and the predicted bounding boxes;
wherein the machine-learning program is a two-stage object detector having a first stage and a second stage;
the first stage is programmed to select a subset of the preliminarily bounded features;
the second stage is programmed to determine the classifications of the features of the selected preliminarily bounded features and determine the predicted bounding boxes for the selected preliminarily bounded features;
the second stage is further programmed to, before determining the classifications and the predicted bounding boxes, apply a plurality of decoder transformer layers to the selected preliminarily bounded features;
each decoder transformer layer includes a cross-attention operation that, for a given one of the selected preliminarily bounded features, pools data about the features that neighbor the feature of the given one of the preliminarily bounded features in the feature pyramid; and
the cross-attention operation is a multiscale deformable operation.
2 . The computer of claim 1 , wherein the machine component includes at least one of a propulsion system, a brake system, or a steering system of a vehicle.
3 . The computer of claim 1 , wherein the preliminary bounding boxes have a plurality of different pixel areas.
4 . The computer of claim 1 , wherein the preliminary bounding boxes have a plurality of aspect ratios.
5 . The computer of claim 4 , wherein
the preliminary bounding boxes have a plurality of different pixel areas;
the second stage is further programmed to scale the selected preliminarily bounded features to a uniform size; and
scaling each selected preliminarily bounded feature to the uniform size includes selecting one of a plurality of transition networks according to the pixel area and aspect ratio of the preliminarily bounded feature and applying the selected transition network to the preliminarily bounded feature.
6 . The computer of claim 1 , wherein the second stage is further programmed to, before determining the classifications and the predicted bounding boxes, scale the selected preliminarily bounded features to a uniform size.
7 . The computer of claim 1 , wherein the cross-attention operation receives the feature pyramid as an input.
8 . The computer of claim 1 , wherein each decoder transformer layer includes a self-attention operation that, for a given one of the selected preliminarily bounded features, interacts the given one of the selected preliminarily bounded features with others of the selected preliminarily bounded features.
9 . The computer of claim 1 , wherein each decoder transformer layer includes a feedforward operation.
10 . The computer of claim 1 , wherein selecting the subset of the preliminarily bounded features includes executing a convolutional neural network including at least one hidden layer.
11 . The computer of claim 10 , wherein the hidden layer includes a bottleneck layer.
12 . The computer of claim 1 , wherein determining the classifications includes selecting the classifications from a plurality of prestored classifications.
13 . The computer of claim 12 , wherein the prestored classifications include a nonobject classification indicating that the classified feature is not from an object.
14 . The computer of claim 12 , wherein the prestored classifications include at least one vehicle classification indicating that the classified feature is from a vehicle and at least one pedestrian classification indicating that the classified feature is from a pedestrian.
15 . The computer of claim 1 , wherein determining the classifications includes executing a regression network.
16 . The computer of claim 1 , wherein determining the predicted bounding boxes includes executing a regression network.
17 . A method comprising:
receiving image data from a sensor;
generating a feature pyramid from the image data, the feature pyramid including a plurality of features;
applying a plurality of preliminary bounding boxes to the features to generate a plurality of preliminarily bounded features, each preliminarily bounded feature being a pairing of one of the preliminary bounding boxes and one of the features;
executing a machine-learning program on the preliminarily bounded features to determine a plurality of classifications and a respective plurality of predicted bounding boxes; and
actuating a machine component based on the classifications and the predicted bounding boxes;
wherein the machine-learning program is a two-stage object detector having a first stage and a second stage;
the first stage is programmed to select a subset of the preliminarily bounded features;
the second stage is programmed to determine the classifications of the features of the selected preliminarily bounded features and determine the predicted bounding boxes for the selected preliminarily bounded features;
the second stage is further programmed to, before determining the classifications and the predicted bounding boxes, apply a plurality of decoder transformer layers to the selected preliminarily bounded features;
each decoder transformer layer includes a cross-attention operation that, for a given one of the selected preliminarily bounded features, pools data about the features that neighbor the feature of the given one of the preliminarily bounded features in the feature pyramid; and
the cross-attention operation is a multiscale deformable operation.