METHOD AND SYSTEM FOR RE-PROJECTING AND COMBINING SENSOR DATA FOR VISUALIZATION
There is provided a system and method of re-projecting and combining sensor data of a scene from a plurality of sensors for visualization. The method including: receiving the sensor data from the plurality of sensors; re-projecting the sensor data from each of the sensors into a new viewpoint; localizing each of the re-projected sensor data; combining the localized re-projected sensor data into a combined image; and outputting the combined image. In a particular case, the receiving and re-projecting can be performed locally at each of the sensors.
1 . A computer-implemented method of re-projecting and combining sensor data of a scene from a plurality of sensors for visualization, the method comprising:
receiving the sensor data from the plurality of sensors;
re-projecting the sensor data from each of the sensors into a new viewpoint by:
calibrating each of the sensors to determine one or more calibration values for the respective sensor; and
determining a mapping function to obtain an undistorted image representative of the new viewpoint for each sensor;
localizing each of the re-projected sensor data;
combining the localized re-projected sensor data into a combined image; and
outputting the combined image.
2 . The method of claim 1 , wherein the sensor data comprises red-green-blue-depth (RGB-D) channel values for each of a plurality of pixels.
3 . The method of claim 2 , wherein the values of the RGB-D channels are from a combination of two or more sensors.
4 . The method of claim 1 , wherein calibrating each of the sensors comprises determining intrinsic parameters for each of the sensors, the intrinsic parameters comprising distortions and camera matrix.
5 . The method of claim 1 , wherein the localization comprises performing simultaneous localization and mapping (SLAM) to position and orient the re-projected sensor data relative to each other.
6 . The method of claim 5 , wherein simultaneous localization and mapping comprises performing point-to-plane iterative closest point.
7 . The method of claim 1 , further comprising adding supporting elements to the combined image.
8 . The method of claim 7 , wherein the supporting elements comprise graphical elements associated with one or more objects located in the scene.
9 . The method of claim 1 , wherein the combined image comprises a top-down view.
10 . A non-transitory machine-readable medium storing instructions, which when executed cause a processor to:
receive sensor data representing a scene from a plurality of sensors;
re-project the sensor data from each of the sensors into a new viewpoint by:
calibrating each of the sensors to determine one or more calibration values for the respective sensor; and
determining a mapping function to obtain an undistorted image representative of the new viewpoint for each sensor;
localize each of the re-projected sensor data;
combine the localized re-projected sensor data into a combined image; and
output the combined image.
11 . The non-transitory machine-readable medium of claim 10 , wherein the sensor data comprises red-green-blue-depth (RGB-D) channel values for each of a plurality of pixels.
12 . The non-transitory machine-readable medium of claim 11 , wherein the values of the RGB-D channels are from a combination of two or more sensors.
13 . The non-transitory machine-readable medium of claim 12 , wherein re-projecting the sensor data comprises:
generating a point cloud for the respective sensor by applying the calibration values to each of the depth D channel values for each pixel; and
applying a matrix representative of the new viewpoint to the point cloud for each sensor.
14 . The non-transitory machine-readable medium of claim 10 , wherein calibrating each of the sensors comprises determining intrinsic parameters for each of the sensors, the intrinsic parameters comprising distortions and camera matrix.
15 . The non-transitory machine-readable medium of claim 10 , wherein the localization comprises performing simultaneous localization and mapping (SLAM) to position and orient the re-projected sensor data relative to each other.
16 . The non-transitory machine-readable medium of claim 15 , wherein simultaneous localization and mapping comprises performing point-to-plane iterative closest point.
17 . The non-transitory machine-readable medium of claim 10 , wherein the instructions further configure the processor to add supporting elements to the combined image.
18 . The non-transitory machine-readable medium of claim 17 , wherein the supporting elements comprise graphical elements associated with one or more objects located in the scene.
19 . The non-transitory machine-readable medium of claim 10 , wherein the combined image comprises a top-down view.