Multicamera image processing
A multicamera image processing system is disclosed. In various embodiments, image data is received from each of a plurality of sensors associated with a workspace, the image data comprising for each sensor in the plurality of sensors one or both of visual image information and depth information. Image data from the plurality of sensors is merged to generate a merged point cloud data. Segmentation is performed based on visual image data from at least a subset of the sensors in the plurality of sensors to generate a segmentation result. One or both of the merged point cloud data and the segmentation result is/are used to generate a merged three dimensional and segmented view of the workspace.
1. A system, comprising:
a communication interface configured to receive image data from each of a plurality of cameras associated with a workspace; and
a processor coupled to the communication interface and configured to:
merge image data from the plurality of cameras to generate a merged point cloud data;
perform segmentation based on visual image data from a subset of the plurality of cameras to generate a segmentation result, wherein the segmentation result includes an indication of an object boundary for one or more objects;
use the merged point cloud data and the segmentation result to generate a merged three dimensional and segmented view of the workspace, including by:
for at least a subset of the plurality of cameras, generating a point cloud for each object and computing a corresponding centroid; and
segmenting objects within the workspace based at least in part on centroids of corresponding to each object; and
provide the merged three dimensional and segmented view of the workspace as an output to a module configured to determine a strategy to grasp an object present in the workspace using a robot.
2. The system of claim 1 , wherein:
the segmentation result is obtained based on performing segmentation using RGB data from a camera;
the segmentation result comprises a plurality of RGB pixels; and
a subset of the plurality of RGB pixels is identified based at least in part on determination that the corresponding RGB pixels are associated with an object boundary.
3. The system of claim 1 , wherein merging the image data from the plurality of cameras to generate the merged point cloud data comprises:
translating to a global three dimensional coordinate framework respective positions and orientations of objects and features of the workspace as captured in separately acquired views associated with image data from the plurality of cameras.
4. The system of claim 1 , where the image data from the plurality of cameras are dynamically merged to provide a continuously updated merged point cloud data.
5. The system of claim 1 , wherein the processor is configured to:
autonomously detect that at least one of the plurality of cameras requires recalibration; and
in response to detecting that the at least one of the plurality of cameras requires recalibration, recalibrate the at least one of the plurality of cameras.
6. The system of claim 1 , wherein recalibrating the at least one of the plurality of cameras comprises:
using a camera mounted to a robotic actuator to relocate a fiducial marker in the workspace;
re-estimating camera-to-workspace transformation using fiducial markers; and
recalibrating the at least one of the plurality of cameras to a marker on the robot.
7. The system of claim 1 , wherein generating the merged three dimensional and segmented view of the workspace comprises:
applying a workspace filter in connection with removing one or more of an image and point cloud data associated with portions of the workspace, features of the workspace, or items in the workspace.
8. The system of claim 7 , wherein the workspace filter removes image or point cloud data associated with portions of the workspace, features of the workspace, or items in the workspace that are not required for determining the strategy to grasp the object present in the workspace.
9. The system of claim 7 , wherein the workspace filter removes statistical outlier data.
10. The system of claim 1 , wherein the plurality of cameras includes one or more three dimensional (3D) cameras.
11. The system of claim 1 , wherein the image data includes RGB data.
12. The system of claim 1 , wherein the processor is configured to use one or both of the merged point cloud data and the segmentation result to perform a box fit with respect to the object in the workspace.
13. The system of claim 1 , wherein the processor is further configured to implement the strategy to grasp the object using the robot.
14. The system of claim 6 , wherein the processor is configured to grasp the object in connection with a robotic operation to pick the object from an origin location and place the object in a destination location in the workspace.
15. The system of claim 1 , wherein the processor is further configured to use the merged three dimensional and segmented view of the workspace to display a visualization of the workspace.
16. The system of claim 1 , wherein generating the merged three dimensional and segmented view of the workspace comprises de-projecting into the merged point cloud a set of points comprising the segmentation result.
17. The system of claim 1 , wherein the processor is further configured to subsample the merged point cloud data.
18. The system of claim 17 , wherein the processor is further configured to perform cluster processing on the subsampled point cloud data.
19. The system of claim 18 , wherein the processor is configured to use the subsampled and clustered point cloud data and the segmentation result to generate a box fit result with respect to the object in the workspace.
20. The system of claim 1 , wherein the merged point cloud data and the segmentation result is used to determine a trajectory via which a robotic arm is to move the object to a destination location.
21. A method, comprising:
receiving image data from each of a plurality of cameras associated with a workspace;
merging image data from the plurality of cameras to generate a merged point cloud data;
performing segmentation based on visual image data from a subset of the plurality of cameras to generate a segmentation result, wherein the segmentation result includes an indication of an object boundary for one or more objects;
using one or both of the merged point cloud data and the segmentation result to generate a merged three dimensional and segmented view of the workspace, including by:
for at least a subset of the plurality of cameras, generating a point cloud for each object and computing a corresponding centroid; and
segmenting objects within the workspace based at least in part on centroids of corresponding to each object; and
providing the merged three dimensional and segmented view of the workspace as an output to a module configured to determine a strategy to grasp an object present in the workspace using a robot.
22. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving image data from each of a plurality of cameras associated with a workspace;
merging image data from the plurality of cameras to generate a merged point cloud data;
performing segmentation based on visual image data from a subset of the plurality of cameras to generate a segmentation result, wherein the segmentation result includes an indication of an object boundary for one or more objects;
using one or both of the merged point cloud data and the segmentation result to generate a merged three dimensional and segmented view of the workspace, including by:
for at least a subset of the plurality of cameras, generating a point cloud for each object and computing a corresponding centroid; and
segmenting objects within the workspace based at least in part on centroids of corresponding to each object; and
providing the merged three dimensional and segmented view of the workspace as an output to a module configured to determine a strategy to grasp an object present in the workspace using a robotic arm.