Robot Natural Language Term Disambiguation and Entity Labeling

1 . A robot comprising:

a body and one or more physically moveable components;

one or more processors; and

one or more storage devices storing instructions that are operable, when executed by the one or more processors, to cause the robot to perform operations comprising:

receiving a natural language command from a user;

identifying an ambiguous term in the command, wherein the ambiguous term references a location or an entity in an environment of the robot;

identifying a user location indicator from one or more sensor inputs;

computing a location within the environment of the robot using the location indicator identified from the one or more sensor inputs;

computing resolution data using the computed location, wherein the resolution data resolves the reference of the ambiguous term;

generating one or more actions using the natural language command and the resolved reference of the ambiguous term; and

executing the one or more actions.

2 . The robot of claim 1 , wherein the one or more actions comprise assigning a label to an entity in the environment of the robot based on the computed resolution data.

3 . The robot of claim 2 , wherein assigning a label to an entity in the environment comprises assigning a name of a room to a physical location within the environment.

4 . The robot of claim 3 , wherein the operations further comprise:

receiving a second natural language command that references the room name; and

performing a behavior relating to the particular room in response to receiving the second natural language command.

5 . The robot of claim 4 , wherein performing the behavior relating to the particular room comprises navigating to the particular room.

6 . The robot of claim 2 , wherein the name assigned to the physical entity is a name of a user in the environment.

7 . The robot of claim 2 , wherein the operations further comprise:

receiving a second natural language command from a user, the second natural language command having an ambiguous term;

determining that the ambiguous term is the name of a label assigned to an entity in the robot's environment; and

generating a modified command that replaces the ambiguous term with the name of the label assigned to the entity in the robot's environment.

8 . The robot of claim 1 , wherein the operations further comprise:

determining that the ambiguous term includes a possessive pronoun;

identifying a user who issued the command; and

associating the entity with the identified user who issued the command.

9 . The robot of claim 2 , wherein the operations further comprise:

obtaining an image of the robot's environment containing the physical entity;

labeling the image with the name assigned to the physical entity; and

providing the labeled image as a training example to a computer vision system.

10 . The robot of claim 9 , wherein the operations further comprise:

training, by the computer vision system, a model using the image labeled with the name assigned to the physical entity in the environment of the robot.

11 . The robot of claim 10 , wherein the model is a robot-specific model for use by only the robot or the model is shared model for use by one or more other robots.

12 . The robot of claim 1 , wherein identifying a user location indicator captured from an image of the user comprises:

determining that a user location indicator is not within a field of view of an integrated camera of the robot; and

in response, performing one or more movement behaviors until a user location indicator is within the field of view of the integrated camera.

13 . The robot of claim 12 , wherein performing the one or more movement behaviors comprises:

determining a direction from which a stream of captured audio originated; and

turning toward the direction, navigating toward the direction, or both.

14 . The robot of claim 13 , wherein the operations further comprise:

determining that an obstacle blocks a view of a user; and

in response, performing one or more movement behaviors to navigate around the obstacle.

15 . The robot of claim 1 , wherein computing a location within the environment of the robot using the location indicator captured from the image of the user comprises:

identifying a gesture or a gaze by the user toward a particular location in the environment of the robot;

generating a vector based on the gesture made by the user; and

computing a location as an intersection point of the vector when extended.

16 . The robot of claim 15 , wherein the intersection point is a point on a surface of the environment.

17 . The robot of claim 15 , wherein the gesture made by the user comprises moving an arm or pointing a finger.

18 . The robot of claim 15 , wherein identifying a gesture or a gaze by the user toward a particular location in the environment of the robot comprises evaluating criteria in the following order:

determining whether a large gesture is detected;

determining whether a small gesture is detected; and

determining whether a gaze is detected.

19 . The robot of claim 1 , wherein computing resolution data using the computed location comprises:

determining that the computed location is not within a field of view of an integrated camera;

in response, performing one or more movement behaviors until the computed location is within the field of view of the integrated camera.

20 . The robot of claim 1 , wherein computing resolution data using the computed location comprises computing a name of a physical entity at the computed location in the environment of the robot.

21 . The robot of claim 1 , wherein computing the resolution data using the computed location comprises:

determining that the ambiguous term is a location ambiguity; and

in response, generating a navigation action based on the computed location.

22 . The robot of claim 1 , wherein computing the resolution data using the computed location comprises:

determining that the ambiguous term is an entity ambiguity;

in response, determining a name of an entity at the computed location; and

generating an action based on the name of the entity at the computed location

23 . An apparatus comprising:

one or more physically moveable components;

one or more processors; and

one or more storage devices storing instructions that are operable, when executed by the one or more processors, to cause the apparatus to perform operations comprising:

receiving a natural language command from a user;

identifying an ambiguous term in the command, wherein the ambiguous term references a location or an entity in an environment of the apparatus;

identifying a user location indicator from one or more sensor inputs;

computing a location within the environment of the apparatus using the location indicator identified from the one or more sensor inputs;

computing resolution data using the computed location, wherein the resolution data resolves the reference of the ambiguous term;

generating one or more actions using the natural language command and the resolved reference of the ambiguous term; and

executing the one or more actions.

24 . One or more non-transitory computer storage media encoded with computer program instructions that when executed by a robot having one or more processors cause the robot to perform operations comprising:

receiving a natural language command from a user;

identifying an ambiguous term in the command, wherein the ambiguous term references a location or an entity in an environment of the robot;

identifying a user location indicator from one or more sensor inputs;

computing a location within the environment of the robot using the location indicator identified from the one or more sensor inputs;

computing resolution data using the computed location, wherein the resolution data resolves the reference of the ambiguous term;

generating one or more actions using the natural language command and the resolved reference of the ambiguous term; and

executing the one or more actions.