Granted Patent B2

US 12,462,575 · App. 17/820,859 · Granted Nov 4, 2025

Inventors: John Emmons (Austin, TX); Danny Hung (Austin, TX); Ethan Knight (Austin, TX); Lane McIntosh (Austin, TX)

Assignee: Tesla, Inc.

G06V20/58G06N20/00

Abstract

Claims (36)

1 . A method implemented by a vehicle processor system, the method comprising:

obtaining images from a multitude of image sensors positioned about a vehicle;

determining features associated with the images, wherein the features are output based on a forward pass through a first portion of a machine learning model;

projecting, based on a second portion of the machine learning model, the features into a vector space associated with a virtual camera at a particular height;

aggregating, based on a plurality of video modules, the projected features with other projected features associated with prior images; and

determining, based on a plurality of heads of the machine learning model, a plurality of objects positioned according to the virtual camera,

wherein the machine learning model includes a first and a second branch, and wherein the first branch is associated with the virtual camera at the particular height and the second branch is associated with a different virtual camera at a different height.

2 . The method of claim 1 , wherein the first portion includes individual backbone networks for individual image sensors, and wherein each backbone network determines a portion of the features.

3 . The method of claim 2 , wherein the first branch includes an attention network which receives an aggregated input of the features from the backbone networks.

4 . The method of claim 1 , wherein the particular height is less than 2 meters.

5 . The method of claim 1 , wherein the different height is less than 21 meters and greater than 2 meters, wherein the second branch identifies non-vulnerable road user objects, and wherein respective velocities for the non-vulnerable road user objects are determined by the second branch.

6 . The method of claim 1 , wherein the features are projected into the vector space, and wherein the vector space is warped.

7 . The method of claim 6 , wherein the vector space is three-dimensional, and wherein the vector space is warped to enlarge a center of the vector space.

8 . The method of claim 6 , wherein the vector space is three-dimensional, and wherein objects are elongated in the vector space.

9 . A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform operations, wherein the system is included in an autonomous or semiautonomous vehicle, and wherein the operations comprise:

obtaining images from a multitude of image sensors positioned about a vehicle;

determining features associated with the images, wherein the features are output based on a forward pass through a first portion of a machine learning model;

projecting, based on a second portion of the machine learning model, the features into a vector space associated with a virtual camera at a particular height;

aggregating, based on a plurality of video modules, the projected features with other projected features associated with prior images; and

determining, based on a plurality of heads of the machine learning model, a plurality of objects positioned according to the virtual camera,

wherein the machine learning model includes a first and a second branch, and wherein the first branch is associated with the virtual camera at the particular height and the second branch is associated with a different virtual camera at a different height.

10 . The system of claim 9 , wherein the first portion includes individual backbone networks for individual image sensors, and wherein each backbone network determines a portion of the features.

11 . The system of claim 10 , wherein the first branch includes an attention network which receives an aggregated input of the features from the backbone networks.

12 . The system of claim 9 , wherein the particular height is less than 2 meters.

13 . The system of claim 9 , wherein the different height is less than 20 meters and greater than 2 meters.

14 . The system of claim 9 , wherein the features are projected into the vector space, and wherein the vector space is warped.

15 . The system of claim 14 , wherein the vector space is three-dimensional, and wherein the vector space is warped to enlarge a center of the vector space.

16 . The system of claim 14 wherein the vector space is three-dimensional, and wherein objects are elongated in the vector space.

17 . Non-transitory computer storage media storing instructions that when executed by a system of one or more processors which are included in an autonomous or semi-autonomous vehicle, cause the system to perform operations comprising:

obtaining images from a multitude of image sensors positioned about a vehicle;

determining features associated with the images, wherein the features are output based on a forward pass through a first portion of a machine learning model;

projecting, based on a second portion of the machine learning model, the features into a vector space associated with a virtual camera at a particular height;

aggregating, based on a plurality of video modules, the projected features with other projected features associated with prior images; and

determining, based on a plurality of heads of the machine learning model using the aggregated projected features and the other projected features, a plurality of objects positioned according to the virtual camera,

wherein the machine learning model includes a first and a second branch, and wherein the first branch is associated with the virtual camera at the particular height and the second branch is associated with a different virtual camera at a different height.

18 . The computer storage media of claim 17 , wherein the particular height is less than 2 meters, and wherein the different height is less than 20 meters and greater than 2 meters.

Assignments (2)

Continuity (4)

References Cited (400)