Multi-sensor fusion system and autonomous mobile apparatus
The present disclosure relates to a multi-sensor fusion system and an autonomous mobile apparatus. The multi-sensor fusion system includes: a trigger module including a pulse signal output, the pulse signal output being used to output a pulse signal; and a plurality of depth camera modules, at least one depth camera module including a trigger signal generation module and a trigger signal output, the trigger signal generation module being used to generate a trigger signal according to the pulse signal, and the trigger signal output being connected to the trigger signal generation module, and used to output the trigger signal, where the trigger signal is used for triggering the at least one depth camera module to perform an exposure operation, and other depth camera modules perform exposure operations according to the received trigger signal output by the trigger signal output.
1. A multi-sensor fusion system, comprising:
a trigger module comprising a pulse signal output, the pulse signal output being used to output a pulse signal; and
a plurality of depth camera modules, at least one depth camera module comprising a trigger signal generation module and a trigger signal output, the trigger signal generation module being used to generate a trigger signal according to the pulse signal, and the trigger signal output being connected to the trigger signal generation module, and used to output the trigger signal, wherein
the trigger signal is used for triggering the at least one depth camera module to perform an exposure operation, and other depth camera modules perform exposure operations according to the received trigger signal output by the trigger signal output;
wherein the trigger module comprises a global positioning system (GPS) positioning module and the GPS positioning module is used to update local time of the depth camera modules, and processors of the depth camera modules record a timestamp of obtained image information according to a updated local time; and
the multi-sensor fusion system further comprises:
a host in communication connection with each depth camera module and configured to receive the image information obtained by the depth camera modules, and fuse the image information according to the timestamp.
2. The multi-sensor fusion system according to claim 1 , wherein the plurality of depth camera modules comprise a main camera and at least one auxiliary camera; the main camera comprises the trigger signal generation module and the trigger signal output; and
each auxiliary camera comprises a trigger signal input, and the trigger signal input is used to receive the trigger signal output by the trigger signal output.
3. The multi-sensor fusion system according to claim 2 , wherein each main camera comprises a depth camera and a red-green-blue (RGB) camera; the trigger signal generation module further comprises a pulse signal input, and the pulse signal input is connected to the pulse signal output; and
the trigger signal output comprises:
a first output used to output the trigger signal to the depth camera and the RGB camera of the main camera; and
a second output used to output the trigger signal to a camera of each auxiliary camera.
4. The multi-sensor fusion system according to claim 1 , wherein the GPS positioning module further comprises a serial output port, each depth camera module comprises a processor, the processor comprises a first input and a serial input port, the serial input port is used to be connected to the serial output port of the GPS positioning module to receive a serial signal output by the serial output port, and the first input is connected to the pulse signal output to receive the pulse signal output by the pulse signal output; and
the processor is configured to record a first local time when a target edge of the pulse signal is received, obtain a universal time coordinated (UTC) time from the serial signal when the target edge is received, and record a second local time when the UTC time is obtained, so as to determine a current UTC time corresponding to the second local time according to the first local time, the second local time and the obtained UTC time, and update local time of the processor according to the current UTC time.
5. The multi-sensor fusion system according to claim 4 , wherein the processor comprises a calculation module, and the calculation module calculates the current UTC time based on a sum of the obtained UTC time and a difference between the second local time and the first local time.
6. The multi-sensor fusion system according to claim 4 , wherein the processor is configured to update the local time when a difference between the second local time and the current UTC time is greater than a preset threshold.
7. The multi-sensor fusion system according to claim 1 , further comprising:
a LIDAR device in communication connection with the GPS positioning module and the host respectively, the GPS positioning module being further used to update local time of the LIDAR device, wherein
the LIDAR device is triggered according to the received pulse signal output by the pulse signal output to obtain distance information and record a timestamp of the obtained distance information according to the updated local time; and
the host fuses the distance information and the image information according to the timestamp.
8. The multi-sensor fusion system according to claim 1 , wherein the GPS positioning module is in communication connection with the host, and the GPS positioning module is further used to position absolute positioning information of an autonomous mobile apparatus to which the GPS positioning module belongs; and
the host is configured to obtain the absolute positioning information, and obtain relative positioning information of the autonomous mobile apparatus according to the image information, the absolute positioning information and the relative positioning information being used for planning a motion path of the autonomous mobile apparatus.
9. An autonomous mobile apparatus, comprising: a multi-sensor fusion system, the multi-sensor fusion system comprising:
a trigger module comprising a pulse signal output, the pulse signal output being used to output a pulse signal; and
a plurality of depth camera modules, at least one depth camera module comprising a trigger signal generation module and a trigger signal output, the trigger signal generation module being used to generate a trigger signal according to the pulse signal, and the trigger signal output being connected to the trigger signal generation module, and used to output the trigger signal, wherein
the trigger signal is used for triggering the at least one depth camera module to perform an exposure operation, and other depth camera modules perform exposure operations according to the received trigger signal output by the trigger signal output;
wherein the trigger module comprises a global positioning system (GPS) positioning module, and the GPS positioning module is used to update local time of the depth camera modules, and processors of the depth camera modules record a timestamp of obtained image information according to a updated local time; and
the multi-sensor fusion system further comprises:
a host in communication connection with each depth camera module and configured to receive the image information obtained by the depth camera modules, and fuse the image information according to the timestamp.
10. The autonomous mobile apparatus according to claim 9 , wherein the plurality of depth camera modules comprise a main camera and at least one auxiliary camera; the main camera comprises the trigger signal generation module and the trigger signal output; and
each auxiliary camera comprises a trigger signal input, and the trigger signal input is used to receive the trigger signal output by the trigger signal output.
11. The autonomous mobile apparatus according to claim 10 , wherein each main camera comprises a depth camera and a red-green blue (RGB) camera; the trigger signal generation module further comprises a pulse signal input, and the pulse signal input is connected to the pulse signal output; and
the trigger signal output comprises:
a first output used to output the trigger signal to the depth camera and the RGB camera of the main camera; and
a second output used to output the trigger signal to a camera of each auxiliary camera.
12. The autonomous mobile apparatus according to claim 9 , wherein the GPS positioning module further comprises a serial output port, each depth camera module comprises a processor, the processor comprises a first input and a serial input port, the serial input port is used to be connected to the serial output port of the GPS positioning module to receive a serial signal output by the serial output port, and the first input is connected to the pulse signal output to receive the pulse signal output by the pulse signal output; and
the processor is configured to record first local time when a target edge of the pulse signal is received, obtain universal time coordinated (UTC) time from the received serial signal when the target edge is received, and record second local time when the UTC time is obtained, so as to determine current UTC time corresponding to the second local time according to the first local time, the second local time and the obtained UTC time, and update local time of the processor according to the current UTC time.
13. The autonomous mobile apparatus according to claim 12 , wherein the processor comprises a calculation module, and the calculation module calculates the current UTC time based on a sum of the obtained UTC time and a difference between the second local time and the first local time.
14. The autonomous mobile apparatus according to claim 12 , wherein the processor is configured to update the local time when a difference between the second local time and the current UTC time is greater than a preset threshold.
15. The autonomous mobile apparatus according to claim 9 , further comprising:
a LIDAR device in communication connection with the GPS positioning module and the host respectively, the GPS positioning module being further used to update local time of the LIDAR device, wherein
the LIDAR device is triggered according to the received pulse signal output by the pulse signal output to obtain distance information and record a timestamp of the obtained distance information according to the updated local time; and
the host fuses the distance information and the image information according to the timestamp.
16. The autonomous mobile apparatus according to claim 9 , wherein the GPS positioning module is in communication connection with the host, and the GPS positioning module is further used to position an autonomous mobile apparatus to which the GPS positioning module belongs based on absolute positioning information; and
the host is configured to obtain the absolute positioning information, and obtain relative positioning information of the autonomous mobile apparatus according to the image information, the absolute positioning information and the relative positioning information being used for planning a motion path of the autonomous mobile apparatus.