Multimodal detection with integrated sensors
A system for multimodal detection is provided. The system comprises a light collection and distribution device configured to perform at least one of collecting light signals from a field-of-view (FOV) and distributing the light signals to a plurality of detectors. The light signals have a plurality of wavelengths comprising at least a first wavelength and a second wavelength. The system further comprises a multimodal sensor comprising the plurality of detectors. The plurality of detectors comprises at least a light detector of a first type and a light detector of a second type. The light detector of the first type is configured to detect light signals having a first light characteristic. The light detector of the first type is configured to perform distance measuring based on light signals having the first wavelength. The light detector of the second type is configured to detect light signals having a second light characteristic.
1. A system for multimodal detection, comprising:
a light collection and distribution device configured to perform at least one of collecting light signals from a field-of-view (FOV) and distributing the light signals to a plurality of sensors, wherein the light signals have a plurality of wavelengths comprising at least a first wavelength and a second wavelength, the second wavelength being different from the first wavelength, wherein one or both of the light collection and distribution device and a signal separation device are configured to separate the light signals to obtain light signals having the first wavelength and light signals having the second wavelength, and direct the light signals having the first wavelength and light signals having the second wavelength to different types of sensors of the plurality of sensors;
a multimodal sensor comprising the plurality of sensors, wherein the plurality of sensors comprises at least a light sensor of a first type and a light sensor of a second type, wherein:
the light sensor of the first type is configured to detect light signals having a first light characteristic of a plurality of light characteristics, the light sensor of the first type being configured to perform distance measuring based on light signals having the first wavelength;
the light sensor of the second type is configured to detect light signals having a second light characteristic of the plurality of light characteristics, the first light characteristic being different from the second light characteristic; and
the light sensor of the first type and the light sensor of the second type are both mounted to a first side of a printed circuit board (PCB); and
a processor mounted to a second side of the PCB such that the processor and the light sensors of the first type and second type share the PCB, and the second side of the PCB is opposite to the first side of the PCB, the processor being configured to generate fused data based on depth data from the light sensor of the first type and image data from the light sensor of the second type, wherein the multimodal sensor is more compact in size compared to using discrete sensors each having a processor.
2. The system of claim 1 , further comprising:
a light source configured to provide light including at least one of: visible light, near infrared (NIR) light, short wavelength IR (SWIR) light, medium wavelength IR (MWIR) light, and long wavelength IR (LWIR) light; and
a transmitter configured to transmit the light toward the FOV, wherein the light signals having the first wavelength are formed based on the transmitted light.
3. The system of claim 2 , wherein the light source comprises an array of light emitting elements, and wherein the plurality of sensors are positioned corresponding to the array of light emitting elements.
4. The system of claim 1 , further comprising one or more optical or electrical scanners configured to perform at least one of a point scan or a line scan of the FOV.
5. The system of claim 4 , wherein the one or more optical or electrical scanners are controlled to scan a region of interest (ROI) with a higher resolution than scanning a non-ROI.
6. The system of claim 1 , wherein the light collection and distribution device comprises one or more of: refraction optics, diffractive optics, and reflection optics.
7. The system of claim 6 , wherein the refraction optics comprises a beam splitter configured to direct a portion of the light signals to a first sensor of the plurality of sensors and direct another portion of the light signals to a second sensor of the plurality of sensors.
8. The system of claim 6 , wherein the diffractive optics comprises grating configured to separate the light signals having at least one of different wavelengths, different intensities, or different polarizations.
9. The system of claim 6 , wherein the reflection optics comprises a Newtonian based reflection device configured to direct a portion of the light signals to a first sensor of the plurality of sensors and direct another portion of the light signals to a second sensor of the plurality of sensors.
10. The system of claim 6 , wherein the reflection optics comprises a Schmidt-Cassegrain based reflection device configured to direct a portion of the light signals to a first sensor of the plurality of sensors and direct another portion of the light signals to a second sensor of the plurality of sensors.
11. The system of claim 1 , wherein the plurality of wavelengths comprises wavelengths in at least two of: a visible light wavelength range, a near infrared (NIR) wavelength range, a short-wavelength infrared range, a mid-wavelength infrared range, and a long infrared wavelength range.
12. The system of claim 1 , wherein the signal separation device is configured to separate the light signals to form separated light signals having a plurality of different light characteristics.
13. The system of claim 12 , wherein the plurality of different light characteristics comprises one or more of a light wavelength, a light intensity, a light angular direction, and a light polarization.
14. The system of claim 12 , wherein the signal separation device comprises a spectrum separation device configured to separate the light signals to form the separated light signals having different wavelengths.
15. The system of claim 12 , wherein the signal separation device comprises a spatial separation device configured to separate the light signals to form the separated light signals corresponding to at least one of different spatial positions of the plurality of sensors or different angular directions of the light signals.
16. The system of claim 15 , wherein the spatial separation device comprises optical fibers or a micro lens array configured to separate the light signals to form the separated light signals and direct the separated light signals to respective sensors of the plurality of sensors.
17. The system of claim 12 , wherein the signal separation device comprises a polarization separation device configured to separate the light signals to form the separated light signals having different polarizations.
18. The system of claim 1 , wherein the plurality of sensors are positioned at a focal plane of the light collection and distribution device and the light signals are directed to the focal plane without a signal separation device.
19. The system of claim 1 , wherein at least two of the plurality of sensors of the multimodal sensor are mounted to two different device packages, two different modules, or two different PCBs.
20. The system of claim 1 , wherein the plurality of sensors comprises detectors forming a detector array.
21. The system of claim 20 , wherein the detector array comprises a hybrid integration of different types of detectors in a single semiconductor chip.
22. The system of claim 20 , wherein the detector array is electrically coupled to a readout circuitry in a same device package, or wherein the detector array is electrically coupled to the readout circuitry via wire bonding or flip-chip bonding.
23. The system of claim 20 , wherein the detector array is disposed in a separate module from a readout circuitry.
24. The system of claim 20 , wherein the detector array comprises different types of detectors disposed in one or more semiconductor dies or a portion of a semiconductor wafer.
25. The system of claim 20 , wherein the detector array and a readout circuitry are both disposed in a semiconductor wafer.
26. The system of claim 1 , wherein the light sensor of the first type is configured to detect light signals having NIR wavelengths, and the light sensor of the second type is configured to detect light signals having a visible light wavelength range, a near infrared (NIR) wavelength range, a short-wavelength infrared range, a mid-wavelength infrared range, or a long infrared wavelength range.
27. A method performed by a multimodal detection system comprising a light collection and distribution device and a multimodal sensor comprising a plurality of sensors having at least a light sensor of a first type and a light sensor of a second type, the method comprising:
performing, by the light collection and distribution device, at least one of collecting light signals from a field-of-view (FOV) and distributing the light signals to the plurality of sensors, wherein the light signals have a plurality of wavelengths comprising at least a first wavelength and a second wavelength, the second wavelength being different from the first wavelength;
separating the light signals to obtain light signals having the first wavelength and light signals having the second wavelength;
directing the light signals having the first wavelength and light signals having the second wavelength to different types of sensors of the plurality of sensors;
detecting, by the light sensor of the first type, light signals having a first light characteristic of a plurality of light characteristics;
detecting, by the light sensor of the second type, light signals having a second light characteristic of the plurality of light characteristics, the first light characteristic being different from the second light characteristic,
wherein the light sensor of the first type and the light sensor of the second type are both mounted to a first side of a printed circuit board (PCB); and
generating, by a processor mounted to a second side of the PCB, fused data based on depth data from the light sensor of the first type and image data from the light sensor of the second type, wherein the processor and the light sensors of the first type and second type share the PCB, and the second side of the PCB is opposite to the first side of the PCB, and wherein the multimodal sensor is more compact in size compared to using discrete sensors each having a processor.
28. An integrated sensing system comprising a system for multimodal detection, the system comprising:
a light collection and distribution device configured to perform at least one of collecting light signals from a field-of-view (FOV) and distributing the light signals to a plurality of sensors, wherein the light signals have a plurality of wavelengths comprising at least a first wavelength and a second wavelength, the second wavelength being different from the first wavelength, wherein one or both of the light collection and distribution device and a signal separation device are configured to separate the light signals to obtain light signals having the first wavelength and light signals having the second wavelength, and direct the light signals having the first wavelength and light signals having the second wavelength to different types of sensors of the plurality of sensors;
a multimodal sensor comprising the plurality of sensors, wherein the plurality of sensors comprises at least a light sensor of a first type and a light sensor of a second type, wherein:
the light sensor of the first type is configured to detect light signals having a first light characteristic of a plurality of light characteristics, the light sensor of the first type being configured to perform distance measuring based on light signals having the first wavelength;
the light sensor of the second type is configured to detect light signals having a second light characteristic of the plurality of light characteristics, the first light characteristic being different from the second light characteristic; and
the light sensor of the first type and the light sensor of the second type are both mounted to a first side of a printed circuit board (PCB); and
a processor mounted to a second side of the PCB such that the processor and the light sensors of the first type and second type share the PCB, and the second side of the PCB is opposite to the first side of the PCB, the processor being configured to generate fused data based on depth data from the light sensor of the first type and image data from the light sensor of the second type, wherein the multimodal sensor is more compact in size compared to using discrete sensors each having a processor.
29. A vehicle comprising an integrated sensing system having a system for multimodal detection, the system for multimodal detection comprising:
a light collection and distribution device configured to perform at least one of collecting light signals from a field-of-view (FOV) and distributing the light signals to a plurality of sensors, wherein the light signals have a plurality of wavelengths comprising at least a first wavelength and a second wavelength, the second wavelength being different from the first wavelength, wherein one or both of the light collection and distribution device and a signal separation device are configured to separate the light signals to obtain light signals having the first wavelength and light signals having the second wavelength, and direct the light signals having the first wavelength and light signals having the second wavelength to different types of sensors of the plurality of sensors;
a multimodal sensor comprising the plurality of sensors, wherein the plurality of sensors comprises at least a light sensor of a first type and a light sensor of a second type, wherein:
the light sensor of the first type is configured to detect light signals having a first light characteristic of a plurality of light characteristics, the light sensor of the first type being configured to perform distance measuring based on light signals having the first wavelength;
the light sensor of the second type is configured to detect light signals having a second light characteristic of the plurality of light characteristics, the first light characteristic being different from the second light characteristic; and
the light sensor of the first type and the light sensor of the second type are both mounted to a first side of a printed circuit board (PCB); and
a processor mounted to a second side of the PCB such that the processor and the light sensors of the first type and second type share the PCB, and the second side of the PCB is opposite to the first side of the PCB, the processor being configured to generate fused data based on depth data from the light sensor of the first type and image data from the light sensor of the second type, wherein the multimodal sensor is more compact in size compared to using discrete sensors each having a processor.