Detecting objects non-visible in color images
A computer-implemented method of detecting one or more objects in a driving environment located externally to a vehicle, and a vehicle imaging system configured to detect one or more objects. The computer-implemented method includes training a first neural network to detect objects in a color video stream, the first neural network having a plurality of mid-level color features at a plurality of scales, and training a second neural network, operatively coupled to color neural network and an infrared video stream, to match, at the plurality of scales, mid-level infrared features of the second neural network to mid-level color features of the first neural network. A pixel-level invisibility map is then generated from the color video stream and the infrared video stream by determining differences, at each of the plurality of scales, between mid-level color features at the first neural network and mid-level infrared features at the second infrared neural network, and coupling the result to a fusing function.
1. A computer-implemented method, comprising:
training a first neural network to detect objects in a color video stream, the first neural network having a plurality of mid-level color features at a plurality of scales;
training a second neural network, operatively coupled to color neural network and an infrared video stream, to match, at the plurality of scales, mid-level infrared features of the second neural network to mid-level color features of the first neural network; and
generating a pixel-level invisibility map from the color video stream and the infrared video stream by determining differences, at each of the plurality of scales, between mid-level color features at the first neural network and mid-level infrared features at the second infrared neural network, and coupling the result to a fusing function.
2. The computer-implemented method of claim 1 , wherein:
the first neural network comprises a deep neural network, and
the second neural network comprises a deep neural network.
3. The computer-implemented method of claim 2 , wherein each deep neural network comprises a You Only Look Once, version 3 (YOLO-v3) network.
4. The computer-implemented method of claim 1 , wherein the invisibility map predicts a likelihood of a presence of one or more non-visible objects for every pixel in an image of the color video stream.
5. The computer-implemented method of claim 1 , further comprising aligning one or more image pairs, each image pair comprising an image from a raw color video stream and an image from a raw infrared video stream, to form the color video stream and the infrared video stream.
6. The computer-implemented method of claim 1 , wherein determining differences further comprises determining at least one Euclidian distance between the mid-level color features at the first neural network and the mid-level infrared features at the second neural network.
7. A vehicle imaging system, comprising:
a first trained neural network, operatively coupled to a color video stream and operable to detect objects in the color video stream, the first neural network having mid-level color features at a plurality of scales;
a second trained neural network, operatively coupled to an infrared video stream and the first trained neural network, and operable to match, at the plurality of scales, mid-level infrared features of the second trained neural network to the mid-level color features of the first trained neural network coupled to the color video stream; and
a computing device, operatively coupled to the first trained neural network and the second trained neural network, the computing device having one or more processors and a non-transitory memory, operatively coupled to the one or more processors, comprising a set of instructions executable by the one or more processors to cause the one or more processors to:
generate a pixel-level invisibility map from the color video stream and the infrared video stream, by determining differences, at each scale in the plurality of scales, between the mid-level color features at the first trained neural network and the mid-level infrared features at the second trained neural network, and coupling the result to a fusing function.
8. The vehicle imaging system of claim 7 , wherein:
the first neural network comprises a deep neural network, and
the second neural network comprises a deep neural network.
9. The vehicle imaging system of claim 8 , wherein each deep neural network comprises a You Only Look Once, version 3 (YOLO-v3) network.
10. The vehicle imaging system of claim 7 , wherein the invisibility map predicts a likelihood of a presence of one or more non-visible objects for every pixel in an image of the color video stream.
11. The vehicle imaging system of claim 7 , wherein the one or more processors are to execute the set of instructions to align one or more image pairs, each image pair comprising an image from a raw color video stream and an image from a raw infrared video stream, to form the color video stream and the infrared video stream.
12. The vehicle imaging system of claim 7 , wherein determining differences further comprises determining at least one Euclidian distance between the mid-level color features at the first neural network and the mid-level infrared features at the second neural network.
13. A computer-implemented method, comprising:
dynamically detecting a driving environment located externally to a vehicle;
receiving image data of the detected external driving environment;
dynamically conducting, via a first neural network, image analysis of the image data to detect objects in the detected external driving environment, the first neural network having a plurality of mid-level color features at a plurality of scales;
generating by a second neural network, in response to the image analysis, paired images of color images and infrared images of the detected external driving environment, the second neural network configured to match, at the plurality of scales, mid-level infrared features of the second neural network to mid-level color features of the first neural network; and
generating, in response to the paired images, an invisibility map of the detected external driving environment of the vehicle.
14. The computer-implemented method of claim 13 , wherein the image data comprises image date related to a raw color video stream.
15. The computer-implemented method of claim 14 , wherein the image data comprises image date related to a raw infrared video stream.
16. The computer-implemented method of claim 15 , further comprising aligning one or more image pairs, each image pair comprising an image from the raw color video stream and an image from the raw infrared video stream, to form the color video stream and the infrared video stream.
17. The computer-implemented method of claim 13 , wherein the invisibility map comprises a pixel-level invisibility map to predict a likelihood of a presence of one or more non-visible objects for every pixel in an image of the color video stream.
18. The computer-implemented method of claim 13 , further comprising, in response to generating the invisibility map, automatically generating one or more alert signals of a presence of one or more detected objects in the detected external driving environment.
19. The computer-implemented method of claim 13 , wherein generating the invisibility map comprises determining differences, at each scale in the plurality of scales, between the mid-level color features at the first trained neural network and the mid-level infrared features at the second trained neural network, and coupling the result to a fusing function.