User interface and related flow for controlling a robotic arm

1 . A method, comprising:

displaying, via a user interface (UI), a first control input for generating a first path for a robot to traverse over a surface of a target object, wherein the target object comprises a set of real-world targets, wherein the first path comprises a set of virtual points, and wherein each virtual point of the set of virtual points comprises a normal of a set of normals, each normal corresponding to a pose for a robot tool to access a real-world target of the set of real-world targets;

displaying, via a first pane of the UI, a real-time scanning of the target object by the robot, the real-time scanning displaying a first visual marker overlaid on the target object indicating that the robot has identified a real-world target in real time;

generating the first path based on a first selection of the first control input;

detecting, via the UIL, a modification of a virtual point of the first path, wherein the modification comprises adding the virtual point to the virtual points, moving the virtual point, or deleting the virtual point from the set of virtual points;

displaying, via a second pane of the UI, the modification of the virtual point based on detecting the modification of the virtual point, the second pane distinct from the first pane; and

generating a second path based on a second selection of a second control input to generate the second path for the robot to traverse over the surface of the target object, wherein the second path comprises a set of robot tool poses based on the set of normals.

2 . The method of claim 1 , wherein the method further comprises displaying the first path, wherein the first path further comprises respective directional marker between pairs of virtual points.

3 . The method of claim 1 , wherein the method further comprises:

receiving a third selection of a third control input for identifying an object in a point cloud representation of a target object; and

causing the object to be identified based on receiving the third selection of the third control input for identifying the object in a point cloud representation of a target object.

4 . The method of claim 1 , wherein the method further comprises:

receiving a third selection of a third control input for generating a trajectory for the robot; and

causing the generation of the trajectory for the robot based on receiving the third selection of a third control input for generating the trajectory for the robot.

5 . The method of claim 1 , wherein the method further comprises:

receiving a third selection of a third control input for determining a class of a real-world target using a machine learning model; and

causing a determination of the class of the real-world target based on receiving the third selection of the third control input for determining the class of the real-world target using the machine learning model.

6 . The method of claim 1 , wherein the method further comprises:

receiving a third selection of a third control input for registering a working frame with a reference frame; and

causing the working frame to be registered with the reference frame based on receiving the third selection of the third control input for registering the working frame with the reference frame.

7 . The method of claim 1 , wherein the UI is in operable communication with an application executing on a computing device, and wherein the application is in operable communication with the robot.

8 . The method of claim 1 , wherein the method further comprises:

displaying a first field for receiving a first input parameter for guiding generation of the first path.

9 . A system comprising:

a user interface (UI);

one or more processors; and

a computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

displaying, via the UI, a first control input for generating a first path for a robot to traverse over a surface of a target object, wherein the target object comprises a set of real-world targets, wherein the first path comprises a set of virtual points, and wherein each virtual point of the set of virtual points comprises a normal of a set of normals, each normal, corresponding to pose for a robot tool to access a real-world target of the set of real-world targets;

displaying, via a first pane of the UI, a real-time scanning of the target object by the robot, the real-time scanning displaying a first visual marker indicating that the robot has identified a real-world target in real time;

generating the first path based on a first selection of the first control input;

detecting, via the UI, a modification of a virtual point of the first path, wherein the modification comprises adding the virtual point to the virtual points, moving the virtual point, or deleting the virtual point from the set of virtual points;

displaying, via a second pane of the UI, the modification of the virtual point based on detecting the modification of the virtual point, the second pane distinct from the first pane; and

generating a second path based on a second selection of a second control input to generate the second path for the robot to traverse over the surface of the target object, wherein the second path comprises a set of robot tool poses based on the set of normals.

10 . The system of claim 9 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising displaying the first path, wherein the first path further comprises respective directional marker between pairs of virtual points.

11 . The system of claim 9 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

receiving a third selection of a third control input for identifying an object in a point cloud representation of a target object; and

causing the object to be identified based on receiving the third selection of the third control input for identifying the object in a point cloud representation of a target object.

12 . The system of claim 9 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

receiving a third selection of a third control input for generating a trajectory for the robot; and

causing the generation of the trajectory for the robot based on receiving the third selection of a third control input for generating the trajectory for the robot.

13 . The system of claim 9 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

receiving a third selection of a third control input for determining a class of a real-world target using a machine learning model; and

causing a determination of the class of the real-world target based on receiving the third selection of the third control input for determining the class of the real-world target using the machine learning model.

14 . The system of claim 9 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

receiving a third selection of a third control input for registering a working frame with a reference frame; and

causing the working frame to be registered with the reference frame based on receiving the third selection of the third control input for registering the working frame with the reference frame.

15 . One or more non-transitory computer-readable media including one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

displaying, via a user interface (UI), a first control input for generating a first path for a robot to traverse over a surface of a target object, object, wherein the target object comprises a set of real-world targets, wherein the first path comprises a set of virtual points, and wherein each virtual point of the set of virtual points comprises a normal of a set of normals, each normal, corresponding to pose for a robot tool to access a real-world target of the set of real-world targets;

displaying, via a first pane of the UI, a real-time scanning of the target object by the robot, the real-time scanning displaying a first visual marker overlaid on the target object indicating that the robot has identified a real-world target in real time;

generating the first path based on a first selection of the first control input;

detecting, via the UIL, a modification of a virtual point of the first path, wherein the modification comprises adding the virtual point to the virtual points, moving the virtual point, or deleting the virtual point from the set of virtual points;

displaying, via a second pane of the UI, the modification of the virtual point based on detecting the modification of the virtual point, the second pane distinct from the first pane; and

generating a second path based on a second selection of a second control input to generate the second path for the robot to traverse over the surface of the target object, wherein the second path comprises a set of robot tool poses based on the set of normals.

16 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising displaying the first path, wherein the first path further comprises respective directional marker between pairs of virtual points.

17 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

receiving a third selection of a third control input for identifying an object in a point cloud representation of a target object; and

causing the object to be identified based on receiving the third selection of the third control input for identifying the object in a point cloud representation of a target object.

18 . The one or more non-transitory computer-readable media of claim 15 , wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to perform operations comprising:

receiving a third selection of a third control input for generating a trajectory for the robot; and

causing the generation of the trajectory for the robot based on receiving the third selection of a third control input for generating the trajectory for the robot.