Systems, methods, and control modules for controlling grasping by robot systems
Systems, methods, and control modules for operating robot systems are described. A touch heatmap is generated for an object, indicating at least one touch region for the object. Touch regions indicate regions of an object which are most suitable or most prone to touching when the object is grasped. At least one end effector is controlled in accordance with at least one grasp primitive, to grasp the object in accordance with at least one touch region specified in the touch heatmap.
1 . A robot system comprising:
a robot body having at least one end effector;
at least one sensor;
a robot controller including at least one processor and at least one non-transitory processor-readable storage medium storing processor-executable instructions which, when executed by the at least one processor, cause the robot system to:
capture, by the at least one sensor, sensor data including a representation of an object;
access, in a library of object models, an object model representation of the object based on the sensor data;
access a touch heatmap associated with the object model, the touch heatmap indictive of a plurality of touch regions of the object model;
access a work of the robot system;
select a subse uch regions from the plurality of touch regions based at least in part on the work objective of the robot system; and
control, by the robot controller, the at least one end effector to grasp the object based at least in part on the subset of touch regions of the object model.
2 . The robot system of claim 1 , wherein:
the at least one non-transitory processor-readable storage medium further stores a library of grasp primitives;
the processor-executable instructions further cause the at least one processor to access a first grasp primitive from the library of grasp primitives; and
the processor-executable instructions which cause the robot controller to control the at least one end effector to grasp the object further cause the robot controller to control the at least one end effector to grasp the object in accordance with the first grasp primitive.
3 . The robot system of claim 2 , wherein the touch heatmap is further indicative of at least one grasp primitive including the first grasp primitive for grasping the object model at the at least one touch region.
4 . The robot system of claim 2 , wherein the processor-executable instructions further cause the robot controller to select the first grasp primitive from the library of grasp primitives based at least in part on the touch heatmap.
5 . The robot system of claim 1 , wherein the processor-executable instructions which cause the robot controller to select the subset of touch regions cause the robot controller to select the subset of touch regions based at least in part on proximity of the subset of touch regions at the object to the robot body or to the at least one end effector.
6 . The robot system of claim 1 , wherein:
the at least one non-transitory processor-readable storage medium further stores a library of grasp primitives usable by the at least one end effector; and
the processor-executable instructions which cause the robot controller to select the subset of touch regions cause the robot controller to select the subset of touch regions based at least in part on the subset of touch regions being graspable in accordance with at least one grasp primitive in the library of grasp primitives.
7 . The robot system of claim 1 , wherein:
the at least one non-transitory processor-readable storage medium further stores a library of grasp primitives usable by the at least one end effector;
the processor-executable instructions further cause the robot controller to select a first grasp primitive of the library of grasp primitives capable of grasping the object in accordance with the subset of touch regions; and
the processor-executable instructions which cause the robot controller to control the at least one end effector to grasp the object based at least in part on the subset of touch regions further cause the robot controller to grasp the object in accordance with the first grasp primitive.
8 . The robot system of claim 1 , wherein:
the at least one end effector includes a first end effector and a second end effector;
the processor-executable instructions that cause the robot controller to select a subset of touch regions from the plurality of touch regions, further cause the robot controller to select a first subset of touch regions and a second subset of touch regions from the plurality of touch regions; and
the processor-executable instructions which cause the robot controller to control the at least one end effector to grasp the object cause the robot controller to control the first end effector to grasp the object in accordance with the first subset of touch regions and to control the second end effector to grasp the object in accordance with the second subset of touch regions.
9 . The robot system of claim 1 , wherein the touch heatmap is stored as metadata associated with the object model.
10 . A robot system comprising:
a robot body having at least one end effector;
at least one sensor;
a robot controller including at least one processor and at least one non-transitory processor-readable storage medium storing processor-executable instructions which, when executed by the at least one processor, cause the robot system to:
capture, by the at least one sensor, sensor data including a representation of an object;
access, in a library of object models, an object model representation of the object based on the sensor data;
access a touch heatmap associated with the object model, the touch heatmap indictive of a plurality of touch regions of the object model and the touch heatmap indicative of frequency of touch at each touch region in the plurality of touch regions;
select a subset of touch regions from the plurality of touch regions based at least in part on frequency of touch as indicated in the touch heatmap; and
control, by the robot controller, the at least one end effector to grasp the object based at least in part on the subset of touch regions of the object model.
11 . The robot system of claim 10 , wherein:
the at least one non-transitory processor-readable storage medium further stores a library of grasp primitives;
the processor-executable instructions further cause the at least one processor to access a first grasp primitive from the library of grasp primitives; and
the processor-executable instructions which cause the robot controller to control the at least one end effector to grasp the object further cause the robot controller to control the at least one end effector to grasp the object in accordance with the first grasp primitive.
12 . The robot system of claim 11 , wherein the touch heatmap is further indicative of at least one grasp primitive including the first grasp primitive for grasping the object model at the at least one touch region.
13 . The robot system of claim 11 , wherein the processor-executable instructions further cause the robot controller to select the first grasp primitive from the library of grasp primitives based at least in part on the touch heatmap.
14 . The robot system of claim 10 , wherein the processor-executable instructions which cause the robot controller to select the subset of touch regions cause the robot controller to select the subset of touch regions based at least in part on proximity of the subset of touch regions at the object to the robot body or to the at least one end effector.
15 . The robot system of claim 10 , wherein:
the at least one non-transitory processor-readable storage medium further stores a library of grasp primitives usable by the at least one end effector; and
the processor-executable instructions which cause the robot controller to select the subset of touch regions cause the robot controller to select the subset of touch regions based at least in part on the subset of touch regions being graspable in accordance with at least one grasp primitive in the library of grasp primitives.
16 . The robot system of claim 10 , wherein:
the at least one non-transitory processor-readable storage medium further stores a library of grasp primitives usable by the at least one end effector;
the processor-executable instructions further cause the robot controller to select a first grasp primitive of the library of grasp primitives capable of grasping the object in accordance with the subset of touch regions; and
the processor-executable instructions which cause the robot controller to control the at least one end effector to grasp the object based at least in part on the subset of touch regions further cause the robot controller to grasp the object in accordance with the first grasp primitive.
17 . The robot system of claim 10 , wherein:
the at least one end effector includes a first end effector and a second end effector;
the processor-executable instructions that cause the robot controller to select a subset of touch regions from the plurality of touch regions, further cause the robot controller to select a first subset of touch regions and a second subset of touch regions from the plurality of touch regions; and
the processor-executable instructions which cause the robot controller to control the at least one end effector to grasp the object cause the robot controller to control the first end effector to grasp the object in accordance with the first subset of touch regions and to control the second end effector to grasp the object in accordance with the second subset of touch regions.
18 . The robot system of claim 10 , wherein the touch heatmap is stored as metadata associated with the object model.