Control apparatus, control method, and computer-readable storage medium storing a control program

1. A control apparatus for controlling an operation of a manipulator, the control apparatus comprising:

a processor configured with a program to perform operations comprising:

operation as a first data obtainer configured to obtain first sensing data from a first sensor system configured to observe an endpoint of the manipulator at a current time;

operation as a first estimator configured to calculate a first estimate value of current coordinates of the endpoint at the current time in an observation space based on the obtained first sensing data using a first estimation model;

operation as a second data obtainer configured to obtain second sensing data from a second sensor system configured to observe the endpoint of the manipulator at the current time;

operation as a second estimator configured to calculate a second estimate value of current coordinates of the endpoint at the current time in the observation space based on the obtained second sensing data using a second estimation model;

operation as an adjuster configured to calculate a gradient of an error between the first estimate value and the second estimate value, and to adjust a parameter value for at least one of the first estimation model or the second estimation model to reduce the error based on the calculated gradient;

operation as a command determiner configured to determine a control command to drive the manipulator to cause actual coordinates of the endpoint to be closer to a goal value based on at least one of the first estimate value or the second estimate value in accordance with the gradient of the error; and

operation as a driver configured to provide the determined control command to the manipulator to drive the manipulator based on the determined control command.

2. The control apparatus according to claim 1 , wherein the processor is configured with the program such that operation as the adjuster further comprises:

obtaining, in response to a contact of the endpoint of the manipulator with an object, a boundary value of the coordinates of the endpoint on a boundary surface of the contact with the object:

calculating a gradient of a first error between the first estimate value estimated at the contact and the obtained boundary value, adjusts the parameter value for the first estimation model to reduce the first error based on the calculated gradient of the first error;

calculating a gradient of a second error between the second estimate value estimated at the contact and the obtained boundary value; and

adjusting the parameter value for the second estimation model to reduce the second error based on the calculated gradient of the second error.

3. The control apparatus according to claim 2 , wherein

the manipulator comprises one or more joints,

the first sensor system comprises an encoder configured to measure an angle of each of the one or more joints, and

the second sensor system comprises a camera.

4. The control apparatus according to claim 3 , wherein

the manipulator further comprises an end-effector configured to hold a workpiece,

a target point of the endpoint of the manipulator is set to the end-effector when the end-effector is not holding the workpiece,

the target point of the endpoint of the manipulator is set to the workpiece when the end-effector is holding the workpiece, and

the first sensor system further comprises a touch sensor configured to estimate a positional relationship of the workpiece held with the end-effector.

5. The control apparatus according to claim 3 , wherein the control command comprises a command for at least one of the one or more joints.

6. The control apparatus according to claim 1 , wherein

the manipulator comprises one or more joints,

the first sensor system comprises an encoder configured to measure an angle of each of the one or more joints, and

the second sensor system comprises a camera.

7. The control apparatus according to claim 6 , wherein

the manipulator further comprises an end-effector configured to hold a workpiece,

a target point of the endpoint of the manipulator is set to the end-effector when the end-effector is not holding the workpiece,

the target point of the endpoint of the manipulator is set to the workpiece when the end-effector is holding the workpiece, and

the first sensor system further comprises a touch sensor configured to estimate a positional relationship of the workpiece held with the end-effector.

8. The control apparatus according to claim 1 , wherein

the manipulator comprises one or more joints,

the first sensor system comprises an encoder configured to measure an angle of each of the one or more joints, and

the second sensor system comprises a camera.

9. The control apparatus according to claim 8 , wherein the control command comprises a command for at least one of the one or more joints.

10. The control apparatus according to claim 1 , wherein at least one of the first estimation model and the second estimation model comprises a machine learning model.

11. The control apparatus according to claim 1 , wherein at least one of the first sensing data and the second sensing data is obtained over a predetermined amount of time and averaged to reduce noise.

12. The control apparatus according to claim 1 , wherein absent noise in the first sensing data and the second sensing data, the first estimate value is the same as the second estimate value.

13. A control method for controlling an operation of a manipulator, the method being implementable by a computer, the method comprising:

obtaining a first sensing data from a first sensor system configured to observe an endpoint of the manipulator at a current time;

calculating a first estimate value of a coordinates of the endpoint at the current time in an observation space based on the obtained first sensing data using a first estimation model;

obtaining a second sensing data from a second sensor system configured to observe the endpoint of the manipulator at the current time;

calculating a second estimate value of the coordinates of the endpoint at the current time in the observation space based on the obtained second sensing data using a second estimation model;

calculating a gradient of an error between the first estimate value and the second estimate value;

adjusting a parameter value for at least one of the first estimation model or the second estimation model to reduce the error based on the calculated gradient;

determining a control command for to drive the manipulator to cause the coordinates of the endpoint to be closer to a goal value based on at least one of the first estimate value or the second estimate value in accordance with the gradient of the error; and

providing the determined control command to the manipulator to drive the manipulator.

14. The method according to claim 13 , wherein at least one of the first estimation model and the second estimation model comprises a machine learning model.

15. The method according to claim 13 , wherein at least one of the first sensing data and the second sensing data is obtained over a predetermined amount of time and averaged to reduce noise.

16. A non-transitory computer-readable storage medium storing a control program for controlling an operation of a manipulator, the program, which when read and executed, causes a computer to perform operations comprising:

obtaining a first sensing data from a first sensor system configured to observe an endpoint of the manipulator at a current time;

calculating a first estimate value of a coordinates of the endpoint at the current time in an observation space based on the obtained first sensing data using a first estimation model;

obtaining a second sensing data from a second sensor system configured to observe the endpoint of the manipulator at the current time;

calculating a second estimate value of the coordinates of the endpoint at the current time in the observation space based on the obtained second sensing data using a second estimation model;

calculating a gradient of an error between the first estimate value and the second estimate value;

adjusting a parameter value for at least one of the first estimation model or the second estimation model to reduce the error based on the calculated gradient;

determining a control command for to drive the manipulator to cause the coordinates of the endpoint to be closer to a goal value based on at least one of the first estimate value or the second estimate value in accordance with the gradient of the error; and

providing the determined control command to the manipulator to drive the manipulator.

17. The non-transitory computer-readable storage medium according to claim 16 , wherein at least one of the first estimation model and the second estimation model comprises a machine learning model.

18. The non-transitory computer-readable storage medium according to claim 16 , wherein at least one of the first sensing data and the second sensing data is obtained over a predetermined amount of time and averaged to reduce noise.