System and method for determining a navigational action based on a captured image
A navigation system for a host vehicle may include at least one processor comprising circuitry and a memory. The memory may include instructions that when executed by the circuitry cause the at least one processor to receive an image representative of an environment of the host vehicle, to identify a first segment of the image, to provide the first segment to a first trained network, the first trained network being configured to generate a first output indicative of a state of a traffic light, to identify a second segment of the image, to provide the second segment to a second trained network, the second trained network being configured to generate a second output indicative of a proposed navigational action, to determine, based on both the first output and the second output a planned navigational action and to cause the host vehicle to take the planned navigational action.
1. A navigation system for a host vehicle, the system comprising:
at least one processor comprising circuitry and a memory, wherein the memory includes instructions that when executed by the circuitry cause the at least one processor to:
receive from an image capture device associated with the host vehicle a captured image representative of an environment of the host vehicle;
identify a first segment of the captured image associated with a traffic light;
provide the first segment of the captured image to a first trained network, the first trained network being configured to generate a first output indicative of a state of the traffic light;
identify a second segment of the captured image that includes contextual information associated with the traffic light;
provide the second segment to a second trained network, the second trained network being configured to generate a second output indicative of a proposed navigational action for the host vehicle relative to the traffic light;
determine, based on both the first output from the first trained network and the second output from the second trained network a planned navigational action for the host vehicle; and
cause the host vehicle to take the planned navigational action.
2. The system of claim 1 , wherein the first output indicative of a state of the traffic light includes an indication of an illumination state of one or more lamps included on the traffic light.
3. The system of claim 1 , wherein the first output indicative of a state of the traffic light includes an indication of a color associated with one or more illuminated lamps associated with the traffic light.
4. The system of claim 1 , wherein the first output indicative of a state of the traffic light includes spot detection.
5. The system of claim 4 , wherein spot detection includes at least one of illumination detection, color detection and shape detection.
6. The system of claim 1 , wherein the first output indicative of a state of the traffic light represents at least one of a stop, go, or slow signal.
7. The system of claim 1 , wherein the memory includes instructions that when executed by the circuitry cause the at least one processor to determine whether the traffic light is relevant to the host vehicle.
8. The system of claim 7 , wherein the determination of whether the traffic light is relevant to the host vehicle is determined based on traffic light relevancy information stored in a map.
9. The system of claim 8 , wherein the traffic light relevancy information is stored in the map for each of a plurality of lanes of a road segment.
10. The system of claim 8 , wherein the traffic light relevancy is further based on a determined lane of travel for the host vehicle.
11. The system of claim 7 , wherein the first segment of the captured image is provided to the first trained network based on whether the traffic light is determined to be relevant to the host vehicle.
12. The system of claim 7 , wherein the first segment of the captured image is not provided to the first trained network if the traffic light is determined as not relevant to the host vehicle.
13. The system of claim 1 , wherein the first segment of the captured image includes a representation of the traffic light that is at least partially obscured.
14. The system of claim 13 , wherein the traffic light is at least partially obscured by one or more objects.
15. The system of claim 14 , wherein the one or more objects includes at least one of a truck or a tree limb.
16. The system of claim 13 , wherein the traffic light is at least partially obscured by an environmental condition.
17. The system of claim 16 , wherein the environmental condition includes at least one of fog, rain, snow or light glare.
18. The system of claim 1 , wherein the first segment of the captured image is determined based on map information storing a location of the traffic light.
19. The system of claim 18 , wherein the first segment of the captured image corresponds to a region of the captured image where, based on the map information, a representation of the traffic light is expected to appear.
20. The system of claim 19 , wherein the representation of the traffic light is not visible or not clearly visible in the first segment, the first output indicative of a state of the traffic light is associated with a confidence level below a predetermined threshold, and in response, the second output of the second trained network is weighted more heavily than the first output of the first trained network in determining the planned navigational action for the host vehicle.
21. The system of claim 1 , wherein the second segment of the captured image includes the entire captured image.
22. The system of claim 1 , wherein the second segment of the captured image includes the first segment of the captured image.
23. The system of claim 1 , wherein the contextual information includes a state of a pedestrian crossing signal.
24. The system of claim 1 , wherein the contextual information includes a representation of a pedestrian present in an intersection associated with the traffic light.
25. The system of claim 1 , wherein the contextual information includes one or more indicators of motion of a pedestrian.
26. The system of claim 1 , wherein the contextual information includes one or more indicators of motion of one or more target vehicles.
27. The system of claim 1 , wherein an indicator of a confidence level associated with the first output indicative of the state of the traffic light is output from the first trained network.
28. The system of claim 27 , wherein the indicator of the confidence level output from the first trained network is used in determining a weight distribution between the first output of the first trained network and the second output of the second trained network, the weight distribution being used in determining the planned navigational action for the host vehicle.
29. The system of claim 1 , wherein an indicator of a confidence level associated with the second output indicative of a proposed navigational action for the host vehicle relative to the traffic light is output from the second trained network.
30. The system of claim 29 , wherein the indicator of the confidence level output from the second trained network is used in determining a weight distribution between the first output of the first trained network and the second output of the second trained network, the weight distribution being used in determining the planned navigational action for the host vehicle.
31. The system of claim 1 , wherein the navigational action indicated by the second trained network includes at least one of stop, slow, maintain speed or go.
32. A method for navigating a host vehicle, the method comprising:
receive from an image capture device associated with the host vehicle a captured image representative of an environment of the host vehicle;
identify a first segment of the captured image associated with a traffic light;
provide the first segment of the captured image to a first trained network, the first trained network being configured to generate a first output indicative of a state of the traffic light;
identify a second segment of the captured image that includes contextual information associated with the traffic light;
provide the second segment to a second trained network, the second trained network being configured to generate a second output indicative of a proposed navigational action for the host vehicle relative to the traffic light;
determine, based on both the first output from the first trained network and the second output from the second trained network a planned navigational action for the host vehicle; and
cause the host vehicle to take the planned navigational action.