Grasping apparatus, learning apparatus, learned model, grasping system, determination method, and learning method

1. A grasping apparatus, comprising:

image data acquisition circuitry configured to acquire image data of a stack of flexible objects being photographed, the stack of flexible objects including a plurality of flexible objects in a stacked state and each flexible object in a folded state;

a grasping device configured to grasp a flexible object that is located on top of the stack of flexible objects; and

control circuitry configured to:

prior to performance of a grasp operation on the flexible object that is located on top of the stack of flexible objects with a portion of the grasping device configured to come in contact with the flexible object that is located on top of the stack of flexible objects during the grasp operation, control a motion of the portion of the grasping device to deform ends of a top surface of the flexible object that is located on top of the stack of flexible objects by causing the portion of the grasping device to apply a pressing force from a side surface direction to the flexible object that is located on top of the stack of flexible objects;

acquire, using the image data acquisition circuitry, image data of the stack of flexible objects after the ends of the top surface of the flexible object that is located on top of the stack of flexible objects are deformed by the portion of the grasping device; and

determine, using the image data of the stack of flexible objects in which the ends of the top surface of the flexible object that is located on top of the stack of flexible objects have been deformed by the portion of the grasping device, whether a part of the stack of flexible objects in the image data is suitable for being grasped with the portion of the grasping device according to the grasp operation.

2. The grasping apparatus according to claim 1 , wherein the control circuitry is further configured to determine a contact position of the flexible object that is located on top of the stack of flexible objects in contact with the portion of the grasping device to deform the ends of the top surface of the flexible object that is located on top of the stack of flexible objects for the motion using image data of the stack of flexible objects before the ends of the top surface of the flexible object that is located on top of the stack of flexible objects are deformed according to the motion, wherein

for deforming the ends of the top surface of the flexible object that is located on top of the stack of flexible objects, the control circuitry is further configured to control the grasping device such that the grasping device displaces the flexible object that is located on top of the stack of flexible objects in the determined contact position using the portion of the grasping device.

3. A learning apparatus, comprising:

control circuitry configured to:

acquire training data including image data of a stack of flexible objects being photographed, the stack of flexible objects including a plurality of flexible objects in a stacked state and each flexible object in a folded state; and

perform calculation of machine learning using the acquired training data, wherein

the image data includes first image data and second image data of the stack of flexible objects in which ends of a top surface of a flexible object that is located on top of the stack of flexible objects have been deformed from a regular stacked pattern shape by application of a pressing force from a side surface direction by a portion of a grasping device configured to come in contact with the flexible object that is located on top of the stack of flexible objects during a grasp operation prior to the grasp operation being performed on the flexible object that is located on the top of the stack of flexible objects by the portion of the grasping device,

the first image data being image data of the ends, of the top surface of the flexible object that is located on top of the stack of flexible objects, that are combined into one end by folding the flexible object that is located on top of the stack of flexible objects,

the second image data being image data of the ends, of the top surface of the flexible object that is located on top of the stack of flexible objects, that are not combined into one end by folding the flexible object that is located on top of the stack of flexible objects, and

the control circuitry is further configured to calculate to learn a part of the flexible object that is located on top of the stack of flexible objects suitable for being grasped using the training data.

4. A non-transitory computer readable medium storing a learned model for causing a computer to execute a method comprising:

learning a weighting coefficient of a neural network constituting the learned model is learned using training data including image data of a stack of flexible objects being photographed,

the stack of flexible objects including a plurality of flexible objects in a stacked state and each flexible object in a folded state,

the image data including first image data and second image data of the stack of flexible objects in which ends of a top surface of a flexible object that is located on top of the stack of flexible objects have been deformed from a regular stacked pattern shape by application of a pressing force from a side surface direction by a portion of a grasping device configured to come in contact with the flexible object that is located on top of the stack of flexible objects during a grasp operation prior to the grasp operation being performed on the flexible object that is located on the top of the stack of flexible objects by the portion of the grasping device,

the first image data being image data of the ends, of the top surface of the flexible object that is located on top of the stack of flexible objects, that are combined into one end by folding the flexible object that is located on top of the stack of flexible objects,

the second image data being image data of the ends, of the top surface of the flexible object that is located on top of the stack of flexible objects, that are not combined into one end by folding the flexible object that is located on top of the stack of flexible objects; and

performing a calculation according to the weighting coefficient for the image data of the stack of flexible objects that is input to an input layer of the neural network and output, from an output layer of the neural network, a quantified value indicating whether a part of the stack of flexible objects in the image data is suitable for being grasped from an output layer of the neural network.

5. A grasping system, comprising:

a camera configured to photograph surrounding objects;

a grasping apparatus including:

image data acquisition circuitry configured to acquire image data of a stack of flexible objects photographed by the camera, the stack of flexible objects including a plurality of flexible objects in a stacked state and each flexible object in a folded state;

a grasping device configured to grasp a flexible object that is located on top of the stack of flexible objects;

a control circuitry unit configured to:

prior to performance of a grasp operation on the flexible object that is located on top of the stack of flexible objects with a portion of the grasping device configured to come in contact with the flexible object that is located on top of the stack of flexible objects during the grasp operation, control a motion of the portion of the grasping device to deform ends of a top surface of the flexible object that is located on top of the stack of flexible objects by causing the grasping device to apply a pressing force from a side surface direction to the flexible object that is located on top of the stack of flexible objects, and

acquire, using the image data acquisition circuitry, image data of the stack of flexible objects after the ends of the top surface of the flexible object that is located on top of the stack of flexible objects are deformed by the portion of the grasping device; and

a server configured to determine, using the image data of the stack of flexible objects in which the ends of the top surface of the flexible object that is located on top of the stack of flexible objects have been deformed by the portion of the grasping device, whether a part of the stack of flexible objects in the image data is suitable for being grasped with the portion of the grasping device according to the grasp operation.

6. A method performed in a grasping apparatus, the method comprising:

prior to performance of a grasp operation on a flexible object that is located on top of a stack of flexible objects with a portion of a grasping device configured to come in contact with the flexible object that is located on top of the stack of flexible objects during the grasp operation, controlling a motion of the portion of the grasping device to deform ends of a top surface of the flexible object that is located on top of a stack of flexible objects by causing the portion of the grasping device to apply a pressing force from a side surface direction to the flexible object that is located on top of the stack of flexible objects, the stack of flexible objects including a plurality of flexible objects in a stacked state and each flexible object in a folded state;

acquiring, using image data acquisition circuitry, image data of the stack of flexible objects after the ends of the top surface of the flexible object that is located on top of the stack of flexible objects are deformed by the portion of the grasping device; and

determining, using the image date of the stack of flexible objects in which the ends of the top surface of the flexible object that is located on top of the stack of flexible objects have been deformed by the portion of the grasping device, whether a part of the stack of flexible objects in the image data is suitable for being grasped with the portion of the grasping device according to the grasp operation.

7. A learning method, comprising:

acquiring training data including image data of a stack of flexible objects being photographed, the stack of flexible objects including a plurality of flexible objects in a stacked state and each flexible object in a folded state; and

performing, using the acquired training data, calculation of machine learning to determine a part of the stack of flexible objects suitable for being grasped, wherein

the image data includes first image data and second image data of the stack of flexible objects in which ends of a top surface of a flexible object that is located on top of the stack of flexible objects have been deformed from a regular stacked pattern shape by application of a pressing force from a side surface direction by a portion of a grasping device configured to come in contact with the flexible object that is located on top of the stack of flexible objects during a grasp operation prior to the grasp operation being performed on the flexible object that is located on the top of the stack of flexible objects by the portion of the grasping device,

the first image data being image data of the ends, of the top surface of the flexible object that is located on top of the stack of flexible objects, that are combined into one end by folding the flexible object that is located on top of the stack of flexible objects, and

the second image data being image data of the ends, of the top surface of the flexible object that is located on top of the stack of flexible objects, that are not combined into one end by folding the flexible object.