Teleoperation system and method thereof
A method and teleoperation system, includes provision of first information on an environment of a teleoperated device by at least one sensor module that is included in the teleoperated device and prediction of an action intended by the operator based on an input of the operator by an action predictor module that is included in the information gathering assistant module. It is determined whether first information on an environment is sufficient to execute the intended action in the environment by a reasoning agent module. Then second information on the environment is determined, which is suitable to enhance the environment information which is useful to execute the intended action in the environment by the reasoning agent module. Provision of second information is controlled by at least one information provision unit, and the first and second information on the environment is fused by an information fusion module.
1 . A teleoperation system, comprising:
a teleoperated device;
an information gathering assistant module;
at least one information provision unit,
wherein
the at least one information provision unit comprises a movable platform and/or a controllable movable platform, and/or is removably attachable to the movable platform and/or the controllable movable platform, and
the controllable movable platform is a portion of the teleoperated device;
an information fusion module; and
a user interface,
wherein
the teleoperated device comprises at least one sensor module that is configured to provide first information on an environment of the teleoperated system,
the user interface comprises an input device,
the information gathering assistant module comprises an action predictor module configured to predict an intended action based on an input of the operator and a reasoning agent module configured to determine second information on the environment, and is configured to control provision of the second information on the environment by the at least one information provision unit,
wherein in a case that the portion of the teleoperated device belongs to a portion that is not controllable without changing the outcome of the intended action of the operator,
the information gathering assistant module comprises a decision agent module that is configured to determine the at least one information provision unit that is attached to the portion of the teleoperated device as being capable of providing the second information on the environment only if the controllable movable portion is currently not controlled by the operator, and
the information fusion module is configured to fuse the first and second information.
2 . The teleoperation system according to claim 1 , wherein
the decision engine module is configured to determine the type and/or amount of the at least one information provision unit for providing the second information based on at least one of a current state of the system, operator preferences and success rates estimated by the reasoning agent.
3 . The teleoperation system according to claim 1 , wherein
the at least one information provision unit comprises an electrical storage, and
the teleoperation system and/or the teleoperated device comprises a base- or charging station to charge the at least one information provision unit.
4 . The teleoperation system according to claim 1 , wherein
the first and second information is fused into a 3D visual representation.
5 . The teleoperation system according to claim 1 , wherein
the user interface comprises a visual interface that is configured to visualize the fused information.
6 . The teleoperation system according to claim 1 , wherein
the reasoning agent module is further configured to determine viewpoints in the environment that define the location at which at least one information provision unit can be positioned in order to provide the second information on the environment and/or to determine a region of interest.
7 . The teleoperation system according to claim 6 , wherein
the region of interest is determined based on at least one of the inputs of the operator, an assistant function behavior and a determination of a low-quality region.
8 . The teleoperation system according to claim 6 , wherein
the at least one information provision unit is removably attachable to a stationary platform at a position that coincides with any of the determined viewpoints.
9 . The teleoperation system according to claim 1 , wherein
the teleoperation system further comprises a persistent memory that is configured to store the fused information on the environment and/or store a past environment information.
10 . The teleoperation system according to claim 1 , wherein
at least one of the information gathering assistant module and the information provision unit comprises an information provision unit controller that is configured to provide control signals based on which the information provision unit provides the second information on the environment.
11 . The teleoperation system according to claim 1 , wherein
the at least one information provision unit comprises at least one of a sensor and a mirror.
12 . The teleoperation system according to claim 10 , wherein
the information provision unit controller is configured to change a pose of at least one of the sensor and the mirror.
13 . The teleoperation system according to claim 1 , wherein
the information provision unit controller is further configured to calculate a trajectory for a respective controllable movable platform.
14 . The teleoperation system according to claim 13 , wherein
the calculated trajectory is updated to avoid collisions with the teleoperated device and the at least one information provision unit.
15 . A method for a teleoperation system, comprising:
providing first information on an environment of a teleoperated device by at least one sensor module that is comprised in the teleoperated device,
wherein
at least one information provision unit of the teleoperation system comprises a movable platform and/or a controllable movable platform, and/or is removably attachable to the movable platform and/or the controllable movable platform, and
the controllable movable platform is a portion of the teleoperated device;
predicting an intended action based on an input of the operator by an action predictor module that is comprised in the information gathering assistant module;
determining if the first information on an environment is sufficient to execute the intended action in the environment by a reasoning agent module;
determining second information on the environment suitable to enhance the environment information which is useful to execute the intended action in the environment by the reasoning agent module;
controlling provision of second information by the at least one information provision unit by the information gathering assistant module,
wherein the at least one information provision unit that is attached to the portion of the teleoperated device as being capable of providing second information on the environment is determined by a decision agent module that is comprised in the information gathering assistant module only if the controllable movable portion is currently not controlled by the operator in a case that the portion of the teleoperated device belongs to a portion that is not controllable without changing the outcome of the intended action of the operator; and
fusing the first and second information on the environment by an information fusion module.