Granted Patent B1

US 12,639,846 · App. 18/227,813 · Granted May 26, 2026

Inventor: Ethan Miller Pronovost (Redwood City, CA)

Assignee: Zoox, Inc.

G06T7/74G06T7/11G06T7/50G06V20/582G06T2207/20081G06T2207/30261

Abstract

Claims (95)

1 . A system comprising:

one or more processors; and

non-transitory memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

receiving sensor data associated with an environment surrounding a vehicle;

determining, based at least in part on the sensor data and map data, a top-down representation of the environment;

determining, as a patch generation operation, a plurality of patches, an individual patch of the plurality of patches having a patch size determined based at least in part on maintaining, increasing, or decreasing a default patch size based at least in part on at least one of the map data or the sensor data, wherein;

the individual patch indicates a portion of the top-down representation,

increasing the default patch size is associated with lowering a resolution for cross-attention associated with data included in the portion of the top-down representation, and

decreasing the default patch size is associated with heightening the resolution for cross-attention associated with data included in the portion of the top-down representation;

determining a vector representation of the portion of the top-down representation and the individual patch;

detecting, based at least in part on the sensor data, an object in the environment and a location of the object;

determining a predicted state of the object associated with a future time based at least in part on applying, by a transformer-based machine-learned model, cross-attention between object data associated with the object and the vector representation, the object data having the resolution for cross-attention; and

controlling the vehicle based at least in part on the predicted state.

2 . The system of claim 1 , wherein the operations further comprise determining, based at least in part on at least one of the map data or the sensor data, a second portion of the top-down representation for which the patch generation operation either increases a second size of a second patch associated with the second portion or excludes the second portion of the top-down representation of the environment from the patch generation operation.

3 . The system of claim 2 , wherein:

determining to increase the second size of the second patch is based at least in part on at least one of:

determining that the second portion is behind the vehicle;

determining that the second portion is at a distance from the vehicle that meets or exceeds a threshold distance; or

determining that the second portion is occluded to one or more sensors of the vehicle; and

determining to exclude the second portion from the patch generation operation is based at least in part on at least one of:

determining that the map data indicates that the second portion is outside a roadway; or

determining that the second portion is at a distance from the vehicle that meets or exceeds a threshold distance.

4 . The system of claim 1 , wherein the operations further comprise receiving a track indicating previous behavior of the object and applying cross-attention comprises:

determining a query vector based at least in part on at least one of the object data associated with the object or the previous behavior;

determining a score based at least in part on determining cross-attention between the query vector and a key vector that is based at least in part on the vector representation; and

determining the predicted state based at least in part on the score.

5 . The system of claim 1 , wherein the patch generation operation decreases the patch size based at least in part on determining that the individual patch is associated with an area of the environment that at least one of:

includes the object;

includes a roadway feature;

includes signage;

includes a region of interest;

is within a first threshold distance of the vehicle; or

is within a second threshold distance of a heading or path of the vehicle.

6 . The system of claim 1 , wherein the patch size is further based at least in part on a velocity of the vehicle.

7 . One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, perform operations comprising:

receiving a top-down representation of an environment;

receiving object data indicating an object in the environment;

determining, as a patch generation operation based at least in part on the top-down representation, a plurality of patches, an individual patch of the plurality of patches comprising a spatial subdivision of the top-down representation, wherein the individual patch has a patch size that is determined based at least in part on at least one of map data or the object data;

determining a vector representation of the spatial subdivision of the top-down representation;

determining a predicted state of the object in the environment based at least in part on applying, by a transformer-based machine-learned model, cross-attention between object data associated with the object and the individual patch; and

controlling a vehicle based at least in part on the predicted state.

8 . The one or more non-transitory computer-readable media of claim 7 , wherein the object data is based at least in part on sensor data and the operations further comprise determining, based at least in part on at least one of the map data or the object data, a second portion of the top-down representation for which the patch generation operation either increases a second size of a second patch associated with the second portion or excludes the second portion of the top-down representation of the environment from the patch generation operation.

9 . The one or more non-transitory computer-readable media of claim 8 , wherein:

determining to increase the second size of the second patch is based at least in part on at least one of:

determining that the second portion is behind the vehicle;

determining that the second portion is at a distance from the vehicle that meets or exceeds a threshold distance; or

determining that the second portion is occluded to one or more sensors of the vehicle; and

determining to exclude the second portion from the patch generation operation is based at least in part on at least one of:

determining that the map data indicates that the second portion is outside a roadway; or

determining that the second portion is at a distance from the vehicle that meets or exceeds a threshold distance.

10 . The one or more non-transitory computer-readable media of claim 7 , wherein the operations further comprise receiving a track indicating previous behavior of the object and applying cross-attention comprises:

determining a query vector based at least in part on at least one of the object data associated with the object or the previous behavior;

determining a score based at least in part on determining cross-attention between the query vector and a key vector that is based at least in part on the vector representation; and

determining the predicted state based at least in part on the score.

11 . The one or more non-transitory computer-readable media of claim 10 , wherein:

determining the query vector comprises:

determining, from among multiple patches associated with the top-down representation, that the individual patch is associated with a location of the object;

determining, by a first machine-learned model based at least in part on the previous behavior, a first embedding and multiplying the first embedding by a first set of learned weights;

determining the key vector comprises determining, by a second machine-learned model based at least in part on the vector representation and a position of the individual patch, a second embedding and multiplying the second embedding by a second set of learned weights; and

determining the predicted state is based at least in part on:

determining a context vector based at least in part on a dot product between a value vector and a set of scores including the score, wherein the value vector is determined based at least in part on the vector representation; and

determining, by a transformer decoder using the context vector, the predicted state.

12 . The one or more non-transitory computer-readable media of claim 10 , wherein:

a location of the object is at least part of a first predicted state determined based at least in part on a previous iteration of cross-attention that is based at least in part on a current location of the object;

the predicted state is a second predicted state associated with a second time, the second time being later than a first time associated with the first predicted state; and

the previous behavior comprises a hidden behavior state associated with the first time determined by the transformer-based machine-learned model as part of the previous iteration of cross-attention.

13 . The one or more non-transitory computer-readable media of claim 7 , wherein the patch generation operation decreases the patch size based at least in part on determining that the individual patch is associated with an area of the environment that at least one of:

includes the object;

includes a roadway feature;

includes signage;

includes a region of interest;

is within a first threshold distance of the vehicle; or

is within a second threshold distance of a heading or path of the vehicle.

14 . The one or more non-transitory computer-readable media of claim 7 , wherein the patch size is further based at least in part on a velocity of the vehicle.

15 . The one or more non-transitory computer-readable media of claim 7 , wherein determining the vector representation comprises determining, by a machine-learned model, an embedding based at least in part on at least one of data associated with the spatial subdivision of the top-down representation or a position of the individual patch relative to at least one of a current position of the vehicle or a position of the individual patch relative to another patch.

16 . The one or more non-transitory computer-readable media of claim 7 , wherein the predicted state of the object comprises at least one of a future position, orientation, classification, velocity, or acceleration of the object.

17 . A method comprising:

receiving a top-down representation of an environment;

receiving object data indicating an object in the environment;

determining, as a patch generation operation, a plurality of patches, an individual patch of the plurality of patches indicating a portion of the top-down representation, wherein the individual patch has a patch size that is determined based at least in part on at least one of map data or the object data and the individual patch comprises an embedding density for the individual patch;

determining, based at least in part on the embedding density, a vector representation of the portion of the top-down representation;

determining a predicted state of the object in the environment based at least in part on applying, by a transformer-based machine-learned model, cross-attention between the object data associated with the object and the individual patch; and

controlling a vehicle based at least in part on the predicted state.

18 . The method of claim 17 , further comprising determining, based at least in part on at least one of the map data or the object data, a second portion of the top-down representation for which the patch generation operation either increases a second size of a second patch associated with the second portion or excludes the second portion of the top-down representation of the environment from the patch generation operation.

19 . The method of claim 17 , further comprising receiving a track indicating previous behavior of the object and applying cross-attention comprises:

determining a query vector based at least in part on at least one of the object data associated with the object or the previous behavior;

determining a score based at least in part on determining cross-attention between the query vector and a key vector that is based at least in part on the vector representation; and

determining the predicted state based at least in part on the score.

20 . The method of claim 17 , wherein the patch generation operation decreases the patch size based at least in part on determining that the individual patch is associated with an area of the environment that at least one of:

includes the object;

includes a roadway feature;

includes signage;

includes a region of interest;

is within a first threshold distance of the vehicle; or

is within a second threshold distance of a heading or path of the vehicle.

Assignments (1)

References Cited (15)