Granted Patent B1

US 12,417,644 · App. 18/545,429 · Granted Sep 16, 2025

Inventors: Davis Edward King (Billerica, MA); Yan Li (San Francisco, CA)

Assignee: Aurora Operations, Inc.

G06V20/584G05D1/0088G05D1/0251G05D1/0274G06F18/214G06V10/22

Abstract

Claims (74)

1. A method implemented by one or more processors, the method comprising:

generating a plurality of training instances, each of the plurality of training instances including a training instance input and a corresponding training instance output,

wherein the training instance input includes vision data capturing a traffic light, and includes a configuration indication that corresponds to a configuration that is assigned to the traffic light,

wherein the configuration that is assigned to the traffic light is one of multiple disparate configurations in a taxonomy of configurations,

wherein the configuration that is assigned to the traffic light defines an orientation of the traffic light and a bulb pattern of the traffic light, and

wherein the bulb pattern of the traffic light defines a placement of a plurality of bulbs of the traffic light relative to one another, and

wherein the corresponding training instance output includes a ground truth state of the traffic light; and

training a machine learning classifier, for subsequent use by an autonomous vehicle, based on the plurality of training instances, the machine learning classifier being selected, from among a plurality of candidate machine learning classifiers each associated with a corresponding one of the multiple disparate configurations in the taxonomy of configurations, to process an image of a given traffic light captured during the subsequent use in an environment of the autonomous vehicle and assigned the configuration.

2. The method of claim 1 , further comprising:

generating a plurality of additional training instances, each of the plurality of additional training instances including an additional training instance input and a corresponding additional training instance output,

wherein the additional training instance input includes additional vision data capturing an additional traffic light, and includes an additional configuration indication that corresponds to an additional configuration that is assigned to the additional traffic light, and

wherein the corresponding additional training instance output includes an additional ground truth state of the additional traffic light; and

training an additional machine learning classifier, for subsequent use by the autonomous vehicle, based on the plurality of additional training instances.

3. The method of claim 2 , wherein the additional configuration that is assigned to the additional traffic light is an additional one of the multiple disparate configurations in the taxonomy of configurations, and wherein the additional configuration that is assigned to the additional traffic light is distinct from the configuration assigned to the traffic light.

4. The method of claim 3 , further comprising:

subsequent to training the machine learning classifier and the additional machine learning classifier:

receiving, from one or more vision components of the autonomous vehicle, an image that captures a given traffic light; and

controlling the autonomous vehicle based on a current state of the given traffic light included in the image.

5. The method of claim 4 , further comprising:

determining whether a given configuration assigned to the given traffic light corresponds to the configuration or the additional configuration; and

in response to determining that the given configuration assigned to the given traffic light corresponds to the configuration:

selecting the machine learning classifier to process the received image that captures the given traffic light;

generating, based on processing the received image using the machine learning classifier, predicted output associated with multiple candidate states of the given traffic light; and

selecting, based on the predicted output, one of the multiple candidate states as a current state of the given traffic light.

6. The method of claim 4 , further comprising:

determining whether a given configuration assigned to the given traffic light corresponds to the configuration or the additional configuration; and

in response to determining that the given configuration assigned to the given traffic light corresponds to the additional configuration:

selecting the additional machine learning classifier to process the received image that captures the given traffic light;

generating, based on processing the received image using the additional machine learning classifier, predicted output associated with multiple candidate states of the given traffic light; and

selecting, based on the predicted output, one of the multiple candidate states as the current state of the given traffic light.

7. The method of claim 4 , further comprising:

identifying a given configuration assigned to the given traffic light;

generating, based on processing the received image using the machine learning classifier along with a given configuration indication that corresponds to the given configuration, predicted output associated with multiple candidate states of the given traffic light; and

selecting, based on the predicted output, one of the multiple candidate states as the current state of the given traffic light.

8. The method of claim 1 , wherein the configuration indication that corresponds to the configuration that is assigned to the traffic light comprises a semantic indicator associated with the configuration that is assigned to the traffic light.

9. The method of claim 1 , wherein the machine learning classifier is a convolutional neural network.

10. The method of claim 1 , wherein the ground truth state of the traffic light is one of multiple candidate states of the traffic light, and wherein the multiple candidate states of the traffic light comprise two or more of a first state indicative of a red state of the traffic light, a second state indicative of a yellow state of the traffic light, a third state indicative of a green state of the traffic light, a fourth state indicative of an off state of the traffic light, or a fifth state indicative of a not visible state of the traffic light.

11. A system, the system comprising:

at least one processor; and

memory comprising instructions that, when executed, cause the at least one processor to:

generate a plurality of training instances, each of the plurality of training instances including a training instance input and a corresponding training instance output,

wherein the training instance input includes vision data capturing a traffic light, and includes a configuration indication that corresponds to a configuration that is assigned to the traffic light,

wherein the configuration that is assigned to the traffic light is one of multiple disparate configurations in a taxonomy of configurations,

wherein the configuration that is assigned to the traffic light defines an orientation of the traffic light and a bulb pattern of the traffic light, and

wherein the bulb pattern of the traffic light defines a placement of a plurality of bulbs of the traffic light relative to one another, and

wherein the corresponding training instance output includes a ground truth state of the traffic light; and

train a machine learning classifier, for subsequent use by an autonomous vehicle, based on the plurality of training instances, the machine learning classifier being selected, from among a plurality of candidate machine learning classifiers each associated with a corresponding one of the multiple disparate configurations in the taxonomy of configurations, to process an image of a given traffic light captured during the subsequent use in an environment of the autonomous vehicle and assigned the configuration.

12. The system of claim 11 , wherein the instructions further cause the at least one processor to:

generate a plurality of additional training instances, each of the plurality of additional training instances including an additional training instance input and a corresponding additional training instance output,

wherein the additional training instance input includes additional vision data capturing an additional traffic light, and includes an additional configuration indication that corresponds to an additional configuration that is assigned to the additional traffic light, and

wherein the corresponding additional training instance output includes an additional ground truth state of the additional traffic light; and

train an additional machine learning classifier, for subsequent use by the autonomous vehicle, based on the plurality of additional training instances.

13. The system of claim 12 , wherein the additional configuration that is assigned to the additional traffic light is an additional one of the multiple disparate configurations in the taxonomy of configurations, and wherein the additional configuration that is assigned to the additional traffic light is distinct from the configuration assigned to the traffic light.

14. The system of claim 13 , wherein the instructions further cause the at least one processor to:

subsequent to training the machine learning classifier and the additional machine learning classifier:

receive, from one or more vision components of the autonomous vehicle, an image that captures a given traffic light; and

control the autonomous vehicle based on a current state of the given traffic light included in the image.

15. The system of claim 14 , wherein the instructions further cause the at least one processor to:

determine whether a given configuration assigned to the given traffic light corresponds to the configuration or the additional configuration; and

in response to determining that the given configuration assigned to the given traffic light corresponds to the configuration:

select the machine learning classifier to process the received image that captures the given traffic light;

generate, based on processing the received image using the machine learning classifier, predicted output associated with multiple candidate states of the given traffic light; and

select, based on the predicted output, one of the multiple candidate states as a current state of the given traffic light.

16. The system of claim 14 , wherein the instructions further cause the at least one processor to:

determine whether a given configuration assigned to the given traffic light corresponds to the configuration or the additional configuration; and

in response to determining that the given configuration assigned to the given traffic light corresponds to the additional configuration:

select the additional machine learning classifier to process the received image that captures the given traffic light;

generate, based on processing the received image using the additional machine learning classifier, predicted output associated with multiple candidate states of the given traffic light; and

select, based on the predicted output, one of the multiple candidate states as the current state of the given traffic light.

17. The system of claim 14 , wherein the instructions further cause the at least one processor to:

identify a given configuration assigned to the given traffic light;

generate, based on processing the received image using the machine learning classifier along with a given configuration indication that corresponds to the given configuration, predicted output associated with multiple candidate states of the given traffic light; and

select, based on the predicted output, one of the multiple candidate states as the current state of the given traffic light.

18. The system of claim 11 , wherein the ground truth state of the traffic light is one of multiple candidate states of the traffic light, and wherein the multiple candidate states of the traffic light comprise two or more of a first state indicative of a red state of the traffic light, a second state indicative of a yellow state of the traffic light, a third state indicative of a green state of the traffic light, a fourth state indicative of an off state of the traffic light, or a fifth state indicative of a not visible state of the traffic light.

Assignments (3)

Continuity (2)

References Cited (19)