Robust object detection and classification using static-based cameras and events-based cameras
Techniques are disclosed to facilitate, in autonomous vehicles, the robust detection and classification of objects in a scene using a static sensors in conjunction with event-based sensors. A trained system architecture may be implemented, and the fusion of both sensors thus allows for the consideration of scenes with overexposure, scenes with underexposure, as well as scenes in which there is no movement. In doing so, the autonomous vehicle may detect and classify objects in conditions in which each sensor, if operating separately, would not otherwise be able to classify (or classify with high uncertainty) due to the sensing environment.
1. An object detection and classification system of an autonomous vehicle (AV), the object detection and classification system comprising:
one or more processors; and
a memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to:
obtain a first set of images associated with a scene, the first set of images being encoded using static-based camera sensor data, each respective one of the first set of images being identified with a frame from among a plurality of frames;
for each respective frame from among the plurality of frames, obtain a second set of images associated with the scene, the second set of images including encoded information representing events occurring within each respective frame using event-based camera sensor data;
receive the first set of images and the second set of images via separate respective channel inputs, each one of the separate respective channel inputs being a dedicated channel input configured to provide, to a same input layer of a neural network, a different image from among the first set of images and the second set of images;
process (i) the first set of images including the static-based camera sensor data, and (ii) the encoded information included in the second set of images to determine a location and type of one or more objects included in the scene; and
provide the location and type of the one or more objects included in the scene to a control system of the AV to perform one or more autonomous navigation tasks.
2. The object detection and classification system of claim 1 , wherein the different images from among the first set of images include a cyan wavelength image, a magenta wavelength image, and a yellow wavelength image, and
wherein the different images from among the second set of images include a positive event image and a negative event image.
3. The object detection and classification system of claim 2 , wherein the one or more processors are configured to, for each respective frame from among the plurality of frames:
encode the received event-based camera sensor data into (i) the positive event image using a number of a positive events occurring within each respective frame, and (ii) the negative event image using a number of a negative events occurring within each respective frame.
4. The object detection and classification system of claim 2 , wherein the one or more processors are configured to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into the positive event image and the negative event image by:
assigning, to each pixel of the positive event image, an intensity gradient value that is based upon a number of aggregated positive events detected within each respective frame by each respective pixel of an event camera image as indicated by the event-based camera sensor data; and
assigning, to each pixel of the negative event image, an intensity gradient value that is based upon a number of aggregated negative events detected within each respective frame by each respective pixel of an event camera image as indicated by the event-based camera sensor data.
5. The object detection and classification system of claim 1 ,
wherein the one or more processors are configured to use a combination of the first set of images and the second set of images received via the channel inputs to determine the location and type of the one or more objects included in the scene when the scene is a static road scene that causes the received event-based camera sensor data to indicate no positive events and no negative events.
6. The object detection and classification system of claim 1 , wherein the one or more processors are configured to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into the positive event image and the negative event image by:
applying a kernel of m-by-m pixel dimensions centered on each respective pixel coordinate within the positive event image to encode the positive event image in accordance with a time surface encoding, with m being an integer value; and
applying a kernel of m-by-m pixel dimensions centered on each respective pixel coordinate within the negative event image to encode the negative event image in accordance with a time surface encoding, with m being an integer value.
7. An autonomous vehicle (AV), comprising:
a media control unit configured to (i) obtain a first set of images associated with a scene, the first set of images being encoded using static-based camera sensor data, each respective one of the first set of images being identified with a frame from among a plurality of frames, and (ii) for each respective frame from among the plurality of frames, obtain a second set of images associated with the scene, the second set of images including encoded information representing events occurring within each respective frame using event-based camera sensor data; and
an electronic control unit (ECU) configured to (i) receive the first set of images and the second set of images via separate respective channel inputs, each one of the separate respective channel inputs being a dedicated channel input configured to provide, to a same input layer of a neural network, a different image from among the first set of images and the second set of images, (ii) process the first set of images including the static-based camera sensor data, (iii) process the second set of images including the encoded information included in the second set of images, (iv) determine a location and type of one or more objects included in the scene, and (v) provide the location and type of the one or more objects included in the scene to a control system of the AV to perform one or more autonomous navigation tasks.
8. The AV of claim 7 , wherein the different images from among the first set of images include a cyan wavelength image, a magenta wavelength image, and a yellow wavelength image, and
wherein the different images from among the second set of images include a positive event image and a negative event image.
9. The AV of claim 8 , wherein the media control unit is configured to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into (i) the positive event image using a number of a positive events occurring within each respective frame, and (ii) the negative event image using a number of a negative events occurring within each respective frame.
10. The AV of claim 9 , wherein the media control unit is configured to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into the positive event image and the negative event image by:
assigning, to each pixel of the positive event image, an intensity gradient value based upon a number of aggregated positive events detected within each respective frame by each respective pixel of an event camera image as indicated by the event-based camera sensor data; and
assigning, to each pixel of the negative event image, an intensity gradient value based upon a number of aggregated negative events detected within each respective frame by each respective pixel of an event camera image as indicated by the event-based camera sensor data.
11. The AV of claim 7 ,
wherein the ECU is configured to use a combination of the first set of images and the second set of images received via the channel inputs to determine the location and type of the one or more objects included in the scene when the scene is a static road scene that causes the received event-based camera sensor data to indicate no positive events and no negative events.
12. The AV of claim 9 , wherein the media control unit is configured to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into the positive event image and the negative event image by:
applying a kernel of m-by-m pixel dimensions centered on each respective pixel coordinate within the positive event image to encode the positive event image in accordance with a time surface encoding, with m being an integer value; and
applying a kernel of m-by-m pixel dimensions centered on each respective pixel coordinate within the negative event image to encode the negative event image in accordance with a time surface encoding, with m being an integer value.
13. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors of a control system associated with an autonomous vehicle (AV), cause the AV to:
obtain a first set of images associated with a scene, the first set of images being encoded using static-based camera sensor data, each respective one of the first set of images being identified with a frame from among a plurality of frames;
for each respective frame from among the plurality of frames, obtain a second set of images associated with the scene, the second set of images including encoded information representing events occurring within each respective frame using event-based camera sensor data;
receive the first set of images and the second set of images via separate respective channel inputs, each one of the separate respective channel inputs being a dedicated channel input configured to provide, to a same input layer of a neural network, a different image from among the first set of images and the second set of images;
process (i) the first set of images including the static-based camera sensor data, and (ii) the second set of images including the encoded information, to determine a location and type of one or more objects included in the scene; and
provide the location and type of the one or more objects included in the scene to a control system of the AV to perform one or more autonomous navigation tasks.
14. The non-transitory computer-readable medium of claim 13 , wherein the different images from among the first set of images include a cyan wavelength image, a magenta wavelength image, and a yellow wavelength image, and
wherein the different images from among the second set of images include a positive event image and a negative event image.
15. The non-transitory computer-readable medium of claim 14 , wherein the instructions, when executed by the one or more processors of the control system associated with the AV, cause the AV to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into (i) the positive event image using a number of a positive events occurring within each respective frame, and (ii) the negative event image using a number of a negative events occurring within each respective frame.
16. The non-transitory computer-readable medium of claim 15 , wherein the instructions, when executed by the one or more processors of the control system associated with the AV, cause the AV to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into the positive event image and the negative event image by:
assigning, to each pixel of the positive event image, an intensity gradient value based upon a number of aggregated positive events detected within each respective frame by each respective pixel of an event camera image as indicated by the event-based camera sensor data; and
assigning, to each pixel of the negative event image, an intensity gradient value based upon a number of aggregated negative events detected within each respective frame by each respective pixel of an event camera image as indicated by the event-based camera sensor data.
17. The non-transitory computer-readable medium of claim 13 ,
wherein the instructions, when executed by the one or more processors of the control system associated with the AV, cause the AV to use a combination of the first set of images and the second set of images received via the channel inputs to determine the location and type of the one or more objects included in the scene when the scene is a static road scene that causes the received event-based camera sensor data to indicate no positive events and no negative events.
18. The non-transitory computer-readable medium of claim 13 , wherein the instructions, when executed by the one or more processors of the control system associated with the AV, cause the AV to, for each respective frame from among the plurality of frames, encode the received event-based camera sensor data into the positive event image and the negative event image by:
applying a kernel of m-by-m pixel dimensions centered on each respective pixel coordinate within the positive event image to encode the positive event image in accordance with a time surface encoding, with m being an integer value; and
applying a kernel of m-by-m pixel dimensions centered on each respective pixel coordinate within the negative event image to encode the negative event image in accordance with a time surface encoding, with m being an integer value.
19. The object detection and classification system of claim 1 , wherein the first set of images are encoded using the static-based camera sensor data, which is generated via a static-based camera independently of the event-based camera sensor data, and
wherein the second set of images are encoded using the event-based camera sensor data, which is generated via an event-based camera independently of the static-based camera sensor data.
20. The object detection and classification system of claim 19 , wherein the second set of images include multiple event camera images per each respective one of the plurality of frames that identify multiple events over each respective frame.
21. The object detection and classification system of claim 1 , wherein the one or more processors are configured to use a combination of the first set of images and the second set of images received via the channel inputs to determine the location and type of the one or more objects included in the scene when the first set of images do not indicate a presence of the one or more objects in the scene.