Systems, apparatuses, and methods for robotic learning and execution of skills including navigation and manipulation functions
Systems, apparatus, and methods are described for robotic learning and execution of skills. A robotic apparatus can include a memory, a processor, sensors, and one or more movable components (e.g., a manipulating element and/or a transport element). The processor can be operatively coupled to the memory, the movable elements, and the sensors, and configured to obtain information of an environment, including one or more objects located within the environment. In some embodiments, the processor can be configured to learn skills through demonstration, exploration, user inputs, etc. In some embodiments, the processor can be configured to execute skills and/or arbitrate between different behaviors and/or actions. In some embodiments, the processor can be configured to execute skills and/or behaviors using cached trajectories or plans. In some embodiments, the processor can be configured to execute skills requiring navigation and manipulation behaviors.
1. A robotic device, comprising:
a base supported on a transport element;
a manipulating element coupled to the base and including an end effector;
a set of sensors; and
a memory storing a plan cache including a plurality of plans for executing a skill, each plan from the plurality of plans being associated with a starting pose of the base and the end effector relative to a physical object in an environment near the robotic device;
a processor operatively coupled to the base, the manipulating element, the set of sensors, and the memory, the processor configured to:
scan, using at least one sensor from the set of sensors, the environment near the robotic device;
identify, based on the scan of the environment near the robotic device, a location of the physical object in the environment near the robotic device;
determine a current pose of the base and the end effector relative to the location of the physical object;
select a first plan from the plurality of plans having a first starting pose of the base and the end effector that is substantially the same as the current pose of the base and the end effector;
execute the first plan;
in response to determining that the skill has not been successfully executed,
select a second plan from the plan cache based on the current pose of the base and the end effector, the second plan having a second starting pose of the base and the end effector that is different from the first starting pose; and
execute the second plan.
2. The robotic device of claim 1 , wherein the processor is further configured to:
move the base and the end effector to a different pose relative to the location of the physical object; and
determine the current pose of the base and the end effector when the base and the end effector is in the different pose,
the processor configured to select the second plan after determining the current pose of the base and the end effector when the base and the end effector is in the different pose.
3. The robotic device of claim 1 , wherein the processor is further configured to:
detect a collision with an obstacle during the execution of the first plan; and
in response to detecting the collision, terminate the execution of the first plan and move the end effector back into a predefined safe position.
4. The robotic device of claim 3 , wherein the processor is configured to select the second plan from the plan cache for triggering executing the skill that avoids the collision with the obstacle.
5. The robotic device of claim 4 , wherein the processor is further configured to generate the second plan based on parameters associated with the obstacle.
6. The robotic device of claim 4 , wherein the processor is further configured to select a second plan from the plurality of plans that avoids the collision with the obstacle.
7. The robotic device of claim 3 , wherein the processor is configured to obtain information of the obstacle by retrieving information of objects near at least one of a marker associated with the physical object or the physical object from a map of the environment stored in the memory.
8. The robotic device of claim 1 , wherein the processor is further configured to:
detect a collision with an obstacle during the execution of the second plan;
in response to detecting the collision, terminate the execution of the second plan and move the end effector back into a predefined safe position; and
generate a modified plan for triggering executing the skill that avoids the collision with the obstacle.
9. The robotic device of claim 1 , wherein the physical object includes at least one of surfaces, walls, doors, equipment, supplies, instruments, tools, furniture, or humans.
10. A robotic device, comprising:
a base supported on a transport element;
a manipulating element coupled to the base and including an end effector;
a set of sensors; and
a memory storing a plan cache including a plurality of plans for executing a skill, each plan from the plurality of plans being associated with a starting pose of the base and the end effector relative to a physical object in an environment near the robotic device;
a processor operatively coupled to the base, the manipulating element, the set of sensors, and the memory, the processor configured to:
scan, using at least one sensor from the set of sensors, the environment near the robotic device;
identify, based on the scan of the environment near the robotic device, a location of the physical object in the environment near the robotic device;
determine a current pose of the base and the end effector relative to the location of the physical object;
select a first plan from the plurality of plans having a starting pose of the base and the end effector that is substantially the same as the current pose of the base and the end effector;
execute the first plan;
in response to detecting a collision with an obstacle during the execution of the first plan, terminate the execution of the first plan and move the end effector into a predefined safe position; and
execute a second plan that avoids the collision with the obstacle.
11. The robotic device of claim 10 , wherein the processor is configured to select the second plan from the plan cache for triggering executing the skill that avoids the collision with the obstacle.
12. The robotic device of claim 10 , wherein the processor is further configured to generate the second plan based on parameters associated with the obstacle.
13. The robotic device of claim 10 , wherein the processor is further configured to:
move the base and the end effector to a different pose relative to the location of the physical object; and
determine the current pose of the base and the end effector when the base and the end effector is in the different pose,
the processor configured to select the second plan after determining the current pose of the base and the end effector when the base and the end effector is in the different pose.
14. The robotic device of claim 10 , wherein the processor is configured to obtain information of the obstacle by retrieving information of objects near at least one of a marker associated with the physical object or the physical object from a map of the environment stored in the memory.
15. The robotic device of claim 10 , wherein the physical object includes at least one of surfaces, walls, doors, equipment, supplies, instruments, tools, furniture, or humans.
16. A robotic device, comprising:
a memory;
a base supported on a transport element;
a manipulating element coupled to the base and including an end effector;
a set of one or more sensors; and
a processor operatively coupled to the memory, the base, the manipulating element, and the set of sensors, the processor configured to:
in response to receiving a first user input indicating a starting pose for executing a skill, register the starting pose of the base and the end effector relative to a physical object in an environment;
obtain, using the set of sensors and during a demonstration of a manipulation required to execute the skill, sensor data at a plurality of keyframes of the demonstration;
in response to receiving a second user input indicating a target pose for interacting with the physical object to execute the skill, register the target pose of the end effector relative to the physical object;
generate a plan for executing the skill based at least in part on the starting pose, the sensor data collected at the plurality of keyframes, and the target pose;
identify a plurality of additional starting poses for executing the skill;
generate a plurality of additional plans for executing the skill based at least in part on the plurality of additional starting poses, the sensor data collected at the plurality of keyframes, and the target pose; and
store the plan and the plurality of additional plans for executing the skill as a plan cache in the memory.
17. The robotic device of claim 16 , wherein the physical object is a first physical object, the processor is further configured to:
detect, using at least one sensor from the set of sensors, a second physical object;
determine to execute the skill; and
select, based on a pose of the base and the end effector relative to the second physical object, a specific plan from the plan cache to execute to interact with the second physical object to execute the skill.
18. The robotic device of claim 17 , wherein the processor is further configured to:
scan, using at least one sensor of the set of sensors, the environment near the robotic device; and
identify, based on the scan of the environment, a location of the physical object,
the processor being configured to register the starting pose of the base and the end effector relative to the physical object based at least in part on a position of the physical object relative to the base.
19. The robotic device of claim 17 , wherein the processor is further configured to:
obtain information of an obstacle in an environment near the second physical object;
determine whether executing the specific plan would cause a collision between the robotic device and the obstacle; and
in response to determining that executing the specific plan would cause a collision, generate a modified plan for executing the skill.
20. The robotic device of claim 19 , wherein the processor is configured to obtain information of the obstacle by retrieving information of objects near at least one of a marker associated with the second physical object or the second physical object from a map of the environment stored in the memory.
21. The robotic device of claim 17 , wherein the processor is further configured to:
determine whether executing the specific plan corresponding to the second physical object would result in successful execution of the skill;
in response to determining that the specific plan would result in successful execution of the skill, execute the specific plan.
22. The robotic device of claim 21 , wherein the processor is further configured to:
in response to determining that the specific plan would not result in successful execution of the skill, generate a modified plan for executing the skill.
23. The robotic device of claim 16 , wherein the processor is further configured to:
in response to determining that the execution of the specific plan corresponding to the second physical object did not result in successful execution of the skill, select a different plan from the plan cache; and
execute the different plan.
24. The robotic device of claim 16 , wherein the physical object includes at least one of surfaces, walls, doors, equipment, supplies, instruments, tools, furniture, or humans.