Reliable robotic manipulation in a cluttered environment
One embodiment can provide a robotic system. The robotic system can include a robotic arm comprising an end-effector, an illumination unit comprising a plurality of single-color light sources of different colors, a structured-light projector to project codified light patterns onto a scene, one or more cameras to capture pseudo-color images of the scene illuminated by the single-color light sources of different colors and images of the scene with the projected codified light patterns, a pose-determination unit to determine a pose of a component based on the pseudo-color images and the images of the scene with the projected codified light patterns, a path-planning unit to generate a motion plan for the end-effector based on the determined pose of the component and a current pose of the end-effector, and a robotic controller to control movement of the end-effector according to the motion plan to allow the end-effector to grasp the component.
1 . A robotic system comprising:
a robotic arm comprising an end-effector;
an illumination unit comprising a plurality of single-color light sources of different colors;
a structured-light projector;
one or more black-and-white cameras; and
a computer system comprising a processor and a storage device storing instructions that when executed by the processor cause the processor to perform a method, the method comprising:
estimating an initial pose of a component of interest;
controlling the robotic arm to move the end-effector based on the estimated initial pose of the component without activating the illumination unit, the structured light projector, and the back-and-white cameras;
in response to determining that the end-effector is in vicinity of the component, activating the illumination unit, the structured light projector, and the black-and-white cameras, such that the black-and-white cameras capture pseudo-color images of the scene under the illumination of the illumination unit and images of the scene with the codified light patterns projected by the structured light projector, wherein a respective pseudo-color image is captured when the scene is illuminated by a single-color light source of a corresponding color;
concatenating multiple pseudo-color images of different colors in increasing wavelength order;
inputting the concatenated multiple pseudo-color images of different colors into a neural network with multiple input channels, wherein each pseudo-color image is sent to a corresponding input channel;
determining a refined pose of the component of interest based on an output of the neural network;
generating a motion plan for the end-effector based on the determined refined pose of the component and a current pose of the end-effector; and
controlling movement of the end-effector according to the motion plan to allow the end-effector to grasp the component of interest.
2 . The robotic system of claim 1 , wherein the method further comprises compensating for errors in the movement of the end-effector.
3 . The robotic system of claim 2 , wherein compensating for errors in the movement of the end-effector comprises applying a machine-learning technique to determine a controller-desired pose corresponding to a camera-instructed pose of the end-effector such that, when the robotic controller controls the movement of the end-effector based on the controller-desired pose, the end-effector achieves, as observed by the cameras, the camera-instructed pose.
4 . The robotic system of claim 1 , wherein the single-color light sources comprise light-emitting diodes (LEDs), and wherein colors of the single-color light sources range between ultraviolet and infrared.
5 . The robotic system of claim 1 , wherein determining the pose further comprises generating a segmentation mask for an image of the scene based on the output of the neural network.
6 . The robotic system of claim 5 , wherein the method further comprises generating a three-dimensional (3D) point cloud of the component of interest by overlaying the segmentation mask on the images of the scene with the projected codified light patterns.
7 . The robotic system of claim 6 , wherein the pose of the component of interest is determined based on the 3D point cloud and a geometric model of the component.
8 . The robotic system of claim 5 , wherein the neural network comprises a Mask Region-based Convolutional Neural Network (Mask R-CNN).
9 . The robotic system of claim 1 , wherein the codified light patterns are encoded based on maximum min-SW gray codes.
10 . A computer-implemented method for controlling a robotic arm, the method comprising:
generating, by a robotic controller, an initial set of instructions to control the robotic arm to move an end-effector towards a component of interest in a work scene without activating a refinement subsystem comprising an illumination unit with a plurality of single-color light sources of different colors, a structured light projector, and one or more back-and-white cameras;
in response to determining that the end-effector is within a vicinity of the component of interest, activating the refinement subsystem by:
alternately turning on the plurality of single-color light sources of different colors to illuminate the work scene;
activating the structured-light projector to project codified light patterns onto the work scene; and
activating the one or more black-and-white cameras to capture pseudo-color images of the work scene and images of the work scene with the codified light patterns projected by the structured-light projector, wherein a respective pseudo-color image is captured when the scene is illuminated by a single-color light source of a corresponding color;
concatenating multiple pseudo-color images of different colors in increasing wavelength order;
inputting the concatenated multiple pseudo-color images of different colors into a neural network with multiple input channels, wherein each pseudo-color image is sent to a corresponding input channel;
determining a refined pose of the component of interest based on an output of the neural network;
generating a set of refined instructions based on the determined refined pose of the component and a current pose of the end-effector; and
controlling, by the robotic controller, movement of the end-effector according to the set of refined instructions to allow the end-effector to grasp the component of interest.
11 . The method of claim 10 , further comprising compensating for errors in the movement of the end-effector.
12 . The method of claim 11 , wherein compensating for errors in the movement of the end-effector comprises applying a machine-learning technique to determine a controller-desired pose corresponding to a camera-instructed pose of the end-effector such that, when the robotic controller controls the movement of the end-effector based on the controller-desired pose, the end-effector achieves, as observed by the cameras, the camera-instructed pose.
13 . The method of claim 10 , wherein the single-color light sources comprise light-emitting diodes (LEDs), and wherein colors of the single-color light sources range between ultraviolet and infrared.
14 . The method of claim 10 , wherein determining the pose further comprises generating a segmentation mask for an image of the work scene based on the output of the neural network.
15 . The method of claim 14 , further comprising generating a three-dimensional (3D) point cloud of the component of interest by overlaying the segmentation mask on the images of the scene with the projected codified light patterns.
16 . The method of claim 15 , wherein the pose of the component of interest is determined based on the 3D point cloud and a geometric model of the component.
17 . The method of claim 14 , wherein the neural network comprises a Mask Region-based Convolutional Neural Network (Mask R-CNN).
18 . The method of claim 10 , wherein the codified light patterns are encoded based on maximum min-SW gray codes.