Multi-modal, multi-technique vehicle signal detection
A vehicle includes one or more cameras that capture a plurality of two-dimensional images of a three-dimensional object. A light detector and/or a semantic classifier search within those images for lights of the three-dimensional object. A vehicle signal detection module fuses information from the light detector and/or the semantic classifier to produce a semantic meaning for the lights. The vehicle can be controlled based on the semantic meaning. Further, the vehicle can include a depth sensor and an object projector. The object projector can determine regions of interest within the two-dimensional images, based on the depth sensor. The light detector and/or the semantic classifier can use these regions of interest to efficiently perform the search for the lights.
1. A method, comprising:
receiving a two-dimensional image of a first three-dimensional object from a camera, wherein the first three-dimensional object is a vehicle;
receiving a first signal indicating first three-dimensional coordinates of the first three-dimensional object;
receiving a second signal indicating second three-dimensional coordinates of a second three-dimensional object;
projecting the first three-dimensional coordinates onto the two-dimensional image;
determining a portion of the first three-dimensional object is occluded in the two-dimensional image by the second three-dimensional object, based on the first three-dimensional coordinates of the first three-dimensional object, and the second three-dimensional coordinates of the second three-dimensional object;
determining a region of interest in the two-dimensional image based on the projection; and
searching for a vehicle signal light of the first three-dimensional object within the region of interest.
2. The method of claim 1 , wherein projecting the first three-dimensional coordinates onto the two-dimensional image comprises applying a matrix transformation to the first three-dimensional coordinates.
3. The method of claim 1 , further comprising:
controlling a vehicle, based on the vehicle signal light of the first three-dimensional object, wherein the controlling is accelerating, braking, or steering the vehicle.
4. The method of claim 1 , further comprising:
in response to determining the portion of the first three-dimensional object is occluded, applying a mask to the two-dimensional image to remove pixels not associated with regions where vehicle signal lights would be present.
5. The method of claim 1 , wherein projecting the first three-dimensional coordinates comprises projecting a surface of a three-dimensional polygon corresponding to the first three-dimensional object.
6. The method of claim 1 , wherein the first signal indicating the first three-dimensional coordinates of the first three-dimensional object includes three-dimensional coordinates of a plurality of surfaces of the first three-dimensional object, and the region of interest is determined based on the three-dimensional coordinates of the plurality of surfaces of the first three-dimensional object.
7. The method of claim 1 , further comprising:
cropping the two-dimensional image to the region of interest to produce a cropped image, wherein the searching is performed on the cropped image.
8. One or more non-transitory, computer-readable media encoded with instructions that, when executed by one or more processing units, perform a method comprising:
receiving a two-dimensional image of a first three-dimensional object from a camera, wherein the first three-dimensional object is a vehicle;
receiving a first signal indicating first three-dimensional coordinates of the first three-dimensional object;
receiving a second signal indicating second three-dimensional coordinates of a second three-dimensional object;
projecting the first three-dimensional coordinates onto the two-dimensional image;
determining a portion of the first three-dimensional object is occluded in the two-dimensional image by the second three-dimensional object, based on the first three-dimensional coordinates of the first three-dimensional object, and the second three-dimensional coordinates of the second three-dimensional object;
determining a region of interest in the two-dimensional image based on the projection; and
searching for a vehicle signal light of the first three-dimensional object within the region of interest.
9. The one or more non-transitory, computer-readable media of claim 8 , wherein projecting the first three-dimensional coordinates onto the two-dimensional image comprises applying a matrix transformation to the first three-dimensional coordinates.
10. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
controlling a vehicle, based on the vehicle signal light of the first three-dimensional object, wherein the controlling is accelerating, braking, or steering the vehicle.
11. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
in response to determining the portion of the first three-dimensional object is occluded, applying a mask to the two-dimensional image to remove pixels not associated with the first three-dimensional object.
12. The one or more non-transitory, computer-readable media of claim 8 , wherein projecting the first three-dimensional coordinates comprises projecting a surface of a three-dimensional polygon corresponding to the first three-dimensional object.
13. The one or more non-transitory, computer-readable media of claim 8 , wherein the first signal indicating the first three-dimensional coordinates of the first three-dimensional object includes three-dimensional coordinates of a plurality of surfaces of the first three-dimensional object, and the region of interest is determined based on the three-dimensional coordinates of the plurality of surfaces of the first three-dimensional object.
14. The one or more non-transitory, computer-readable media of claim 8 , the method further comprising:
cropping the two-dimensional image to the region of interest to produce a cropped image, wherein the searching is performed on the cropped image.
15. A system, comprising:
a memory including instructions;
a processor to execute the instructions; and
an object projector, encoded in the instructions, to:
receive a two-dimensional image of a first three-dimensional object from a camera, wherein the first three-dimensional object is a vehicle;
receive a first signal indicating first three-dimensional coordinates of the first three-dimensional object;
receive a second signal indicating second three-dimensional coordinates of a second three-dimensional object;
project the first three-dimensional coordinates onto the two-dimensional image;
determine a portion of the first three-dimensional object is occluded in the two-dimensional image by the second three-dimensional object, based on the first three-dimensional coordinates of the first three-dimensional object, and the second three-dimensional coordinates of the second three-dimensional object; and
determine a region of interest in the two-dimensional image based on the projection; and
a vehicle signal detector, encoded in the instructions, to:
search for a vehicle signal light of the first three-dimensional object within the region of interest.
16. The system of claim 15 , wherein the vehicle signal detector further is to:
skip light detection in the portion of the first three-dimensional object in the two-dimensional image occluded by the second three-dimensional object.
17. The system of claim 15 , further comprising:
a vehicle control system to accelerate, brake, or steer a vehicle, based on the vehicle signal light of the first three-dimensional object.
18. The system of claim 15 , wherein the vehicle signal detector further is to:
in response to determining the portion of the first three-dimensional object is occluded, determine any lights within the portion of the first three-dimensional object in the two-dimensional image occluded by the second three-dimensional object do not belong to the first three-dimensional object.
19. The system of claim 15 , wherein projecting the first three-dimensional coordinates comprises projecting a surface of a three-dimensional polygon corresponding to the first three-dimensional object.
20. The system of claim 15 , wherein the first signal indicating the first three-dimensional coordinates of the first three-dimensional object includes three-dimensional coordinates of a plurality of surfaces of the first three-dimensional object, and the region of interest is determined based on the three-dimensional coordinates of the plurality of surfaces of the first three-dimensional object.