Device and method for controlling a robot
A method for controlling a robot device. The method includes acquiring an image(s) of in a workspace of the robot device; determining, by a neural network, object hierarchy information specifying stacking relations of the objects with respect to each other in the workspace of the robot device and confidence information for the object hierarchy information from the image(s); if the confidence information indicates a confidence above a confidence threshold, manipulating an object of the objects; if the confidence information indicates a confidence lower than the confidence threshold, acquiring an additional image of the objects and determining, by the neural network, additional object hierarchy information specifying stacking relations of the objects with respect to each other in the workspace of the robot device and additional confidence information for the additional object hierarchy information from the additional image and control the robot using the additional object hierarchy information.
1 . A method for controlling a robot device, comprising the following steps:
acquiring at least one image of a plurality of objects in a workspace of the robot device;
determining, by a neural network, a first adjacency matrix representing object hierarchy information specifying stacking relations of the plurality of objects with respect to each other in the workspace of the robot device, and representing confidence information for the object hierarchy information from the at least one image;
when the confidence information indicates a confidence for the object hierarchy information above a confidence threshold, manipulating an object of the plurality of objects;
when the confidence information indicates a confidence for the object hierarchy information lower than the confidence threshold, acquiring an additional image of the plurality of objects and determining, by the neural network, a second adjacency matrix representing additional object hierarchy information specifying stacking relations of the plurality of objects with respect to each other in the workspace of the robot device, and representing additional confidence information for the additional object hierarchy information from the additional image,
wherein each entry of the first adjacency matrix and the second adjacency matrix specifies a stacking relation between an object associated with a row of the entry and an object associated with a column of the entry and specifies the confidence in the stacking relation between the object associated with the row of the entry and the object associated with the column of the entry;
combining the first adjacency matrix and the second adjacency matrix; and
controlling a physical action of the robot device using the combination of the first adjacency matrix and the second adjacency matrix.
2 . The method of claim 1 , wherein the combining of the first adjacency matrix and the second adjacency matrix by an adjacency matrix combination includes, for each adjacency matrix to be combined, performing an element-wise logit transformation of the adjacency matrix, summing the transformed adjacency matrices and applying an inverse logit transformation to the sum.
3 . The method of claim 1 , further comprising, when the confidence information indicates the confidence for the object hierarchy information above the confidence threshold, determining an object of the plurality of objects to be manipulated from the object hierarchy information by determining an object on which none of the other objects is stacked and manipulating the determined object.
4 . A controller configured to control a robot device, the controller configured to:
acquire at least one image of a plurality of objects in a workspace of the robot device;
determine, by a neural network, a first adjacency matrix representing object hierarchy information specifying stacking relations of the plurality of objects with respect to each other in the workspace of the robot device, and representing confidence information for the object hierarchy information from the at least one image;
when the confidence information indicates a confidence for the object hierarchy information above a confidence threshold, manipulate an object of the plurality of objects;
when the confidence information indicates a confidence for the object hierarchy information lower than the confidence threshold, acquire an additional image of the plurality of objects and determining, by the neural network, a second adjacency matrix representing additional object hierarchy information specifying stacking relations of the plurality of objects with respect to each other in the workspace of the robot device, and representing additional confidence information for the additional object hierarchy information from the additional image,
wherein each entry of the first adjacency matrix and the second adjacency matrix specifies a stacking relation between an object associated with a row of the entry and an object associated with a column of the entry and specifies the confidence in the stacking relation between the object associated with the row of the entry and the object associated with the column of the entry;
combine the first adjacency matrix and the second adjacency matrix; and
control a physical action of the robot device using the combination of the first adjacency matrix and the second adjacency matrix.
5 . A non-transitory computer-readable medium on which are stored instructions for controlling a robot device, the instruction, when executed by a computer, causing the computer to perform the following steps:
acquiring at least one image of a plurality of objects in a workspace of the robot device;
determining, by a neural network, a first adjacency matrix representing object hierarchy information specifying stacking relations of the plurality of objects with respect to each other in the workspace of the robot device, and representing confidence information for the object hierarchy information from the at least one image;
when the confidence information indicates a confidence for the object hierarchy information above a confidence threshold, manipulating an object of the plurality of objects;
when the confidence information indicates a confidence for the object hierarchy information lower than the confidence threshold, acquiring an additional image of the plurality of objects and determining, by the neural network, a second adjacency matrix representing additional object hierarchy information specifying stacking relations of the plurality of objects with respect to each other in the workspace of the robot device, and representing additional confidence information for the additional object hierarchy information from the additional image,
wherein each entry of the first adjacency matrix and the second adjacency matrix specifies a stacking relation between an object associated with a row of the entry and an object associated with a column of the entry and specifies the confidence in the stacking relation between the object associated with the row of the entry and the object associated with the column of the entry;
combining the first adjacency matrix and the second adjacency matrix; and
controlling a physical action of the robot device using the combination of the first adjacency matrix and the second adjacency matrix.