Remote operation method, storage medium, remote operation device, and remote operation system

1 . A remote operation method of a robot comprising:

a conversion step of converting an operation input with respect to an operation-side device operated by an operator to a joint angle command value of the robot;

an acquisition step of acquiring actual visual information of a space in which the robot is present;

a generation step of generating virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;

a projection step of projecting the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and projecting a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired to the screen viewed by the operator,

wherein the virtual visual information, in order to reduce an effect of transmission latencies between the operation-side device and the robot, is calculated through an inverse conversion of the three-dimensional representation for predicting parameters such as a motion and a shape from the visual information by regression with visual information stored in the operation-side device between a first time at which the actual visual information is acquired and a second time at which the actual visual information acquired at the first time arrives at the operation-side device operated by the operator as inputs and is calculated by interpolating the parameters between times using a time-series network; and

a step of transmitting the virtual visual information to the robot while reducing transmission latencies between the operation-side device and the robot based on the virtual visual information, and actuating, based on control data included in the virtual visual information, an actuator associated with the robot during remote operation of the robot.

2 . The remote operation method of a robot according to claim 1 , wherein the generation step includes generating the virtual visual information by inputting the joint angle command value, a posture of the screen viewed by the operator, and the acquired actual visual information to a trained model.

3 . The remote operation method of a robot according to claim 1 , wherein the virtual visual information is generated using posture information obtained by estimating postures of the robot and an object to be operated at a time at which the actual visual information is acquired.

4 . The remote operation method of a robot according to claim 1 , wherein the actual visual information includes meta data including time information indicating a time at which an image is captured, position information of an object to be operated recognized through a recognition process performed on the captured image, and posture information of the object to be operated, and

wherein the generation step includes generating the virtual visual information using estimation information obtained by estimating the posture of the robot and the posture of the object to be operated using the meta data.

5 . The remote operation method of a robot according to claim 1 , wherein the actual visual information includes RGB data and depth data.

6 . A remote operation method of a robot comprising:

a conversion step of converting an operation input with respect to an operation-side device operated by an operator to a joint angle command value of the robot;

an acquisition step of acquiring actual visual information of a space in which the robot is present;

a generation step of generating virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;

a projection step of projecting the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and projecting a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired to the screen viewed by the operator,

wherein the projection step, in order to reduce an effect of transmission latencies between the operation-side device and the robot, includes determining a mixture ratio of the actual visual information and the virtual visual information according to a rate of change of the actual visual information with time; and

a step of transmitting the virtual visual information to the robot while reducing the transmission latencies between the operation-side device and the robot based on the virtual visual information, and actuating, based on control data included in the virtual visual information, an actuator associated with the robot during remote operation of the robot.

7 . A non-transitory computer-readable storage medium storing a program, the program causing a remote operation computer to perform:

converting an operation input with respect to an operation-side device operated by an operator to a joint angle command value of the robot;

acquiring actual visual information of a space in which the robot is present;

generating virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;

projecting the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and projecting a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired,

wherein the virtual visual information, in order to reduce an effect of transmission latencies between the operation-side device and the robot, is calculated through an inverse conversion of the three-dimensional representation for predicting parameters such as a motion and a shape from the visual information by regression with visual information stored in the operation-side device between a first time at which the actual visual information is acquired and a second time at which the actual visual information acquired at the first time arrives at the operation-side device operated by the operator as inputs and is calculated by interpolating the parameters between times using a time-series network, and

transmitting the virtual visual information to the robot while reducing the transmission latencies between the operation-side device and the robot based on the virtual visual information, and actuating, based on control data included in the virtual visual information, an actuator associated with the robot during remote operation of the robot.

8 . A remote operation device comprising:

a conversion unit configured to convert an operation input with respect to an operation-side device operated by an operator to a joint angle command value of the robot;

an acquisition unit configured to acquire actual visual information of a space in which the robot is present;

a generation unit configured to generate virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;

a projection unit configured to project the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and to project a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired,

wherein the virtual visual information, in order to reduce an effect of transmission latencies between the operation-side device and the robot, is calculated through an inverse conversion of the three-dimensional representation for predicting parameters such as a motion and a shape from the visual information by regression with visual information stored in the operation-side device between a first time at which the actual visual information is acquired and a second time at which the actual visual information acquired at the first time arrives at the operation-side device operated by the operator as inputs and is calculated by interpolating the parameters between times using a time-series network, and

a communication unit configured to transmit the virtual visual information to the robot while reducing the transmission latencies between the operation-side device and the robot based on the virtual visual information, and actuate, based on control data included in the virtual visual information, an actuator associated with the robot during remote operation of the robot.

9 . A remote operation system comprising a remote operation device and a remote-location device,

wherein the remote-location device includes:

a robot; and

a visual sensor detecting an actual visual image of an environment in which the robot is present, and

wherein the remote operation device includes:

a conversion unit configured to convert an operation input with respect to an operation-side device operated by an operator to a joint angle command value of the robot;

an acquisition unit configured to acquire actual visual information of a space in which the robot is present;

a generation unit configured to generate virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;

a projection unit configured to project the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and to project a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired,

wherein the virtual visual information, in order to reduce an effect of transmission latencies between the operation-side device and the robot, is calculated through an inverse conversion of the three-dimensional representation for predicting parameters such as a motion and a shape from the visual information by regression with visual information stored in the operation-side device between a first time at which the actual visual information is acquired and a second time at which the actual visual information acquired at the first time arrives at the operation-side device operated by the operator as inputs and is calculated by interpolating the parameters between times using a time-series network, and

a communication unit configured to transmit the virtual visual information to the robot while reducing the transmission latencies between the operation-side device and the robot based on the virtual visual information, and actuate, based on control data included in the virtual visual information, an actuator associated with the robot during remote operation of the robot.