Object hand-over between robot and actor
An example robotic device may include an arm having a palm and fingers, a depth sensor disposed within the palm, and a control system. The control system may be configured to detect an indication to receive an object from an actor, and in response, cause the robotic device to enter a receive-object mode. When the robotic device is in the receive-object mode, the control system is further configured to: receive, from the depth sensor, depth data indicating a distance between the palm and the object; when the distance is greater than a first threshold and less than a second threshold, cause the arm to move towards the object; when the distance exceeds the second threshold, maintain the arm in a fixed position; and when the distance drops below the first threshold, cause the two or more fingers to close to grasp the object.
1. A robotic device comprising:
an arm comprising a palm and two or more fingers;
a first sensor;
a depth sensor disposed within the palm; and
a control system configured to:
receive, from the first sensor, first data associated with an actor holding an object;
detect an indication to receive the object from the actor based on the first data; and
in response to detecting the indication to receive the object from the actor, cause the robotic device to enter a receive-object mode, wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
receive, from the depth sensor, depth data indicating a distance between the palm and the object;
when the distance between the palm and the object is greater than a first threshold and less than a second threshold, cause the arm to move towards the object;
when the distance between the palm and the object exceeds the second threshold, maintain the arm in a fixed position; and
when the distance between the palm and the object drops below the first threshold, cause the two or more fingers to close to grasp the object.
2. The robotic device of claim 1 , wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
move the arm into a first position to wait for hand-over of the object from the actor.
3. The robotic device of claim 2 , wherein the control system is further configured to move the arm into the first position by:
moving the arm away from the object to a distance outside of the second threshold to gesture that the robotic device is in the receive-object mode and is ready to receive the object from the actor.
4. The robotic device of claim 2 , wherein the control system is further configured to move the arm into the first position by:
moving the arm along a predetermined trajectory to gesture that the robotic device is in the receive-object mode and is ready to receive the object from the actor.
5. The robotic device of claim 1 , wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
determine the distance between the palm and the object based on a perpendicular component of a distance represented by the depth data, wherein the first threshold and the second threshold each represent respective distances along an axis perpendicular to the palm.
6. The robotic device of claim 1 , wherein the robotic device further comprises an image sensor, and wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
receive, from the image sensor, image data indicating a position of the object relative to the palm; and
based on the image data, adjust a lateral position of the arm to place the object within a field of view of the depth sensor.
7. The robotic device of claim 1 , wherein the first sensor comprises an image sensor, wherein the first data is image data, and wherein the control system is configured to detect the indication to receive the object from the actor by:
identifying, based on the image data, (i) the object and (ii) a virtual skeleton representing the actor; and
detecting the indication to receive the object from the actor based on a pose of the virtual skeleton and a position of the object relative to the virtual skeleton reaching a predetermined configuration.
8. The robotic device of claim 1 , wherein the first sensor comprises a microphone communicatively connected to the control system, and wherein the control system is configured to detect the indication to receive the object from the actor by:
detecting, by way of the microphone, an utterance by the actor; and
determining that the utterance indicates to receive the object from the actor.
9. The robotic device of claim 1 , wherein the first sensor is an image sensor, wherein the first data is image data, wherein the robotic device further comprises an audio speaker and a microphone, and wherein the control system is configured to detect the indication to receive the object from the actor by:
identifying, based on the image data, (i) the object and (ii) a virtual skeleton representing the actor;
in response to a pose of the virtual skeleton and a position of the object relative to the virtual skeleton reaching a predetermined configuration, causing the audio speaker to generate an utterance requesting hand-over of the object from the actor to the arm; and
detecting, by way of the microphone, an utterance by the actor indicating to receive the object from the actor.
10. The robotic device of claim 1 , wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
cause the two or more fingers to open to expose the depth sensor disposed within the palm before receiving the depth data.
11. The robotic device of claim 1 , wherein the first sensor comprises an image sensor, and wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
receive, from the image sensor, image data representing the actor holding the object; and
based on the image data, verify that the depth data received from the depth sensor indicates the distance between the palm and the object.
12. The robotic device of claim 1 , further comprising an image sensor disposed within the palm, wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
receive, from the image sensor, image data representing the actor holding the object; and
based on the image data, verify that the depth data received from the depth sensor indicates the distance between the palm and the object.
13. The robotic device of claim 1 , wherein, when the robotic device is in the receive-object mode, the control system is further configured to:
determine a speed with which to move the arm towards the object based on the distance between the palm and the object.
14. The robotic device of claim 1 , wherein the depth sensor disposed within the palm comprises a first time-of-flight sensor having a first range and a second time-of-flight sensor having a second range greater than the first range, and wherein the control system is further configured to receive the depth data by:
receiving first depth data from the first time-of-flight sensor and second depth data from the second time-of-flight sensor; and
determining the distance between the palm and the object based on at least one of the first depth data and the second depth data.
15. A method comprising:
receiving, from a first sensor on a robotic device and by a control system of the robotic device, first data associated with an actor holding an object;
based on the first data, detecting, by the control system, an indication to receive the object from the actor by an arm of the robotic device, wherein the arm comprises a palm and two or more fingers; and
in response to detecting the indication to receive the object from the actor, operating the robotic device in a receive-object mode, wherein operating in the receive-object mode comprises:
receiving, from a depth sensor disposed within the palm and by the control system, depth data indicating a distance between the palm and the object;
when the distance between the palm and the object is greater than a first threshold and less than a second threshold, causing, by the control system, the arm to move towards the object;
when the distance between the palm and the object exceeds the second threshold, causing, by the control system, the arm to be maintained in a fixed position; and
when the distance between the palm and the object drops below the first threshold, causing, by the control system, the two or more fingers to close to grasp the object.
16. The method of claim 15 , wherein operating in the receive-object mode further comprises:
moving the arm into a first position to wait for hand-over of the object from the actor.
17. The method of claim 16 , wherein moving the arm into the first position comprises:
moving the arm away from the object to a distance outside of the second threshold to gesture that the robotic device is in the receive-object mode and is ready to receive the object from the actor.
18. The method of claim 15 , wherein operating in the receive-object mode further comprises:
determining the distance between the palm and the object based on a perpendicular component of a distance represented by the depth data, wherein the first threshold and the second threshold each represent respective distances along an axis perpendicular to the palm.
19. The method of claim 15 , wherein operating in the receive-object mode further comprises:
receiving, from an image sensor on the robotic device, image data indicating a position of the object relative to the palm; and
based on the image data, adjusting a lateral position of the arm to place the object within a field of view of the depth sensor.
20. A non-transitory computer readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
receiving, from a first sensor on a robotic device, first data associated with an actor holding an object;
based on the first data, detecting an indication to receive the object from the actor by an arm of the robotic device, wherein the arm comprises a palm and two or more fingers; and
in response to detecting the indication to receive the object from the actor, operating the robotic device in a receive-object mode, wherein operating in the receive-object mode comprises:
receiving, from a depth sensor disposed within the palm, depth data indicating a distance between the palm and the object;
when the distance between the palm and the object is greater than a first threshold and less than a second threshold, causing the arm to move towards the object;
when the distance between the palm and the object exceeds the second threshold, causing the arm to be maintained in a fixed position; and
when the distance between the palm and the object drops below the first threshold, causing the two or more fingers to close to grasp the object.