Robotic control using value distributions

1. A method implemented by one or more processors of a robot to perform a robotic task, the method comprising:

generating state data associated with the robot in an environment, wherein the state data is generated based at least in part on vision data captured by a vision component of the robot;

sampling a plurality of candidate actions that are available for performance by the robot given the state data;

processing, utilizing a machine learning model that has been trained to learn a value function, a plurality of state-action pairs to generate a corresponding plurality of value distributions, wherein each state-action pair includes the state data and one of the plurality of sampled candidate actions;

selecting, from the plurality of state-action pairs, the state-action pair corresponding to the value distribution that satisfies one or more criteria; and

controlling the robot to implement the sampled candidate action of the selected state-action pair.

2. The method of claim 1 , further comprising calculating, for each value distribution of the plurality of value distributions, a summary value, wherein the selecting includes selecting the state-action pair corresponding to the summary value that satisfies one or more of the criteria.

3. The method of claim 2 , wherein the summary value comprises a mean of the value distribution.

4. The method of claim 2 , further comprising distorting the value distribution based on a desired measure of risk seeking, wherein the summary value is calculated based on the distorted value distribution.

5. The method of claim 4 , wherein the desired measure of risk seeking is determined from one or more signals.

6. The method of claim 5 , wherein one or more of the signals includes a measure of entropy associated with the environment.

7. The method of claim 5 , wherein one or more of the signals includes a measure of uncertainty associated with the environment.

8. The method of claim 5 , wherein one or more of the signals includes one or more attributes of the robotic task.

9. The method of claim 8 , wherein the one or more attributes include a measure of skill of the robot in performing the robotic task.

10. The method of claim 8 , wherein the one or more attributes include a measure of difficulty of the robotic task.

11. The method of claim 5 , wherein one or more of the signals includes a user preference.

12. The method of claim 5 , further comprising performing object recognition processing on the vision data to identify a class of an object present in the environment;

wherein one or more of the signals includes the class of the object.

13. The method of claim 1 , wherein the plurality of candidate actions are sampled from a continuous action space.

14. A robot, comprising:

a vision sensor viewing an environment;

a trained critic neural network model stored in one or more non-transitory computer readable media, the trained critic neural network model representing a learned value function;

at least one processor configured to:

generate state data associated with the robot in an environment, wherein the state data is generated based at least in part on vision data captured by the vision sensor;

sample a plurality of candidate actions that are available for performance by the robot given the state data;

process, based on the trained critic neural network learning model, a plurality of state-action pairs to generate a corresponding plurality of value distributions, wherein each state-action pair includes the state data and one of the plurality of sampled candidate actions;

select, from the plurality of state-action pairs, the state-action pair corresponding to the value distribution that satisfies one or more criteria; and

control the robot to implement the sampled candidate action of the selected state-action pair.

15. The robot of claim 14 , wherein the at least one processor is further to calculate, for each value distribution of the plurality of value distributions, a summary value, wherein the selecting includes selecting the state-action pair corresponding to the summary value that satisfies one or more of the criteria.

16. The robot of claim 15 , wherein the summary value comprises a mean of the value distribution.

17. The robot of claim 15 , further comprising distorting the value distribution based on a desired measure of risk seeking, wherein the summary value is calculated based on the distorted value distribution.

18. The robot of claim 17 , wherein the desired measure of risk seeking is determined from one or more signals.

19. The robot of claim 18 , wherein one or more of the signals includes one or more of:

a user preference;

a measure of entropy associated with the environment;

a measure of uncertainty associated with the environment;

one or more attributes of the robotic task;

a measure of skill of the robot in performing the robotic task; or

a measure of difficulty of the robotic task.

20. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors of a robot, cause the one or more processors to perform the following operations:

generating state data associated with the robot in an environment, wherein the state data is generated based at least in part on vision data captured by a vision component of the robot;

sampling a plurality of candidate actions that are available for performance by the robot given the state data;

processing, utilizing a machine learning model that has been trained to learn a value function, a plurality of state-action pairs to generate a corresponding plurality of value distributions, wherein each state-action pair includes the state data and one of the plurality of sampled candidate actions;

selecting, from the plurality of state-action pairs, the state-action pair corresponding to the value distribution that satisfies one or more criteria; and

controlling the robot to implement the sampled candidate action of the selected state-action pair.