Camera and end-effector planning for visual servoing
Various aspects of techniques, systems, and use cases may be used for camera and end-effector planning for visual servoing for example in redundant manipulators. A technique may include generating a set of paths of an end effector of a robotic arm, and ranking the set of paths using an objective function that results in an improvement to a distance between the end effector and a target while maintaining a collision mitigation path for the end effector and minimizing occlusion of a camera affixed to a joint of the robotic arm. The technique may include converting a path of the set of paths into a trajectory based on the ranking, and outputting the trajectory for controlling the robotic arm to move the end effector.
1 . At least one non-transitory machine-readable medium, including instructions, which when executed by processing circuitry, cause the processing circuitry to perform operations to:
generate a set of paths of an end effector of a robotic arm by attempting to satisfy a visual-servoing task to provide a set of velocities in an end-effector task-space and satisfying a visibility-aware planning task to provide a velocity in a camera task-space, respective paths of the set of paths each including at least one respective trajectory to traverse the respective paths from a first position to a respective end position;
determine a ranking of the set of paths using an objective function that results in a decrease in a distance between the end effector and a target while maintaining a collision mitigation path for the end effector and minimizing occlusion of a camera affixed to a joint of the robotic arm;
convert a path of the set of paths into a trajectory based on the ranking, the trajectory being one of the at least one respective trajectory; and
output the trajectory to cause control of the robotic arm to move the end effector.
2 . The at least one machine-readable medium of claim 1 , wherein the operations further cause the processing circuitry to:
capture a first image using a second camera affixed to the end effector; and
identify an occlusion in the first image.
3 . The at least one machine-readable medium of claim 2 , wherein to output the trajectory, the operations include outputting a control signal to control the robotic arm based on a second image received from the camera affixed to the joint of the robotic arm.
4 . The at least one machine-readable medium of claim 2 , further comprising operations to project a second image received from the camera affixed to the joint of the robotic arm into a perspective corresponding to the second camera affixed to the end effector.
5 . The at least one machine-readable medium of claim 1 , wherein to rank the set of paths using the objective function, the operations include using optimization constraints including at least one of minimizing self-occlusion, minimizing scene occlusion, maintaining line of sight, or minimizing camera dynamics.
6 . The at least one machine-readable medium of claim 5 , wherein the optimization constraints include at least one of maintaining a field of view of a second camera affixed to the end effector or a field of view of the camera affixed to the joint of the robotic arm occlusion mitigation.
7 . The at least one machine-readable medium of claim 1 , wherein the operations further cause the processing circuitry to:
receive a trajectory for the end effector of the robotic arm; and
identify a next position for the end effector from the trajectory; and
wherein to generate the set of paths of the end effector of the robotic arm, the operations include using an image captured by the camera on the joint of the robotic arm.
8 . The at least one machine-readable medium of claim 1 , wherein to output the trajectory, the operations include outputting control signals for manipulating a set of motors of the robotic arm to apply at least one velocity to a link of the robotic arm that is independent of a velocity of the end effector.
9 . The at least one machine-readable medium of claim 1 , wherein to satisfy the visibility-aware planning task, the operations include determining a plurality of velocities, including the velocity, that are independent of a trajectory of the end effector.
10 . The at least one machine-readable medium of claim 1 , wherein when at least one of the tasks cannot be satisfied, the operations include determining the set of paths according to an optimization that satisfies the visual-servoing task while minimizing a deviation from the velocity in the camera task-space.
11 . A system comprising:
processing circuitry; and
memory including instructions, which when executed by the processing circuitry, cause the processing circuitry to:
generate a set of paths of an end effector of a robotic arm by attempting to satisfy a visual-servoing task to provide a set of velocities in an end-effector task-space and satisfying a visibility-aware planning task to provide a velocity in a camera task-space, respective paths of the set of paths each including at least one respective trajectory to traverse the respective paths from a first position to a respective end position;
determine a ranking of the set of paths using an objective function that results in a decrease in a distance between the end effector and a target while maintaining a collision mitigation path for the end effector and minimizing occlusion of a camera affixed to the end effector;
convert a path of the set of paths into a trajectory based on the ranking, the trajectory being one of the at least one respective trajectory; and
output the trajectory for to cause control of the robotic arm to move the end effector.
12 . The system of claim 11 , wherein the instructions further cause the processing circuitry to:
capture a first image using a second camera affixed to a joint of the robotic arm;
identify an occlusion in the first image; and
receive a second image from the second camera affixed to the joint of the robotic arm.
13 . The system of claim 12 , wherein to output the trajectory, the instructions further cause the processing circuitry to output a control signal to control the robotic arm based on the second image.
14 . The system of claim 12 , wherein the instructions further cause the processing circuitry to project the second image into a perspective corresponding to the second camera affixed to the end effector.
15 . The system of claim 11 , wherein to rank the set of paths using the objective function, the instructions further cause the processing circuitry to use optimization constraints including at least one of minimizing self-occlusion, minimizing scene occlusion, maintaining line of sight, or minimizing camera dynamics.
16 . The system of claim 15 , wherein the optimization constraints include at least one of maintaining a field of view of a second camera affixed to the end effector or a field of view of the camera affixed to the joint of the robotic arm occlusion mitigation.
17 . The system of claim 11 , wherein the instructions further cause the processing circuitry to:
receive a trajectory for the end effector of the robotic arm; and
identify a next position for the end effector from the trajectory; and
wherein to generate the set of paths of the end effector of the robotic arm, the operations include using an image captured by the camera on the joint of the robotic arm.
18 . An apparatus comprising:
means for generating a set of paths of an end effector of a robotic arm by attempting to satisfy a visual-servoing task to provide a set of velocities in an end-effector task-space and satisfying a visibility-aware planning task to provide a velocity in a camera task-space, respective paths of the set of paths each including at least one respective trajectory to traverse the respective paths from a first position to a respective end position;
means for ranking the set of paths using an objective function that results in an a decrease in a distance between the end effector and a target while maintaining a collision mitigation path for the end effector and minimizing occlusion of a camera affixed to a joint of the robotic arm;
means for converting a path of the set of paths into a trajectory based on the ranking, the trajectory being one of the at least one respective trajectory; and
means for outputting the trajectory to cause control of the robotic arm to move the end effector.
19 . The apparatus of claim 18 , wherein the means for outputting the trajectory outputs control signals for manipulating a set of motors of the robotic arm to apply a set of control commands to a set of robot links of the robotic arm that are independent of a velocity of the end effector.