Device and method to identify functional parts of tools for robotic manipulation

1. A system for identifying functional elements of an object for robotic manipulation, the system comprising one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of:

causing a robot to repeatedly manipulate an object, wherein the robot comprises an audio sensor and a plurality of touch sensors;

recording a three-dimensional (3D) location of each audio event that produces a response as measured by the audio sensor during manipulation of the object;

recording a 3D location of each tactile event that produces a response as detected by the plurality of touch sensors during manipulation of the object;

determining a 3D location of each audio and tactile event in 3D space;

generating a 3D audio point cloud denoting the 3D locations of the audio events;

generating a 3D tactile point cloud denoting the 3D locations of the tactile events;

registering the 3D audio point cloud and the 3D tactile point cloud with a 3D model point cloud of the object; and

annotating the 3D model point cloud of the object with the audio events to generate an annotated 3D model point cloud of the object, which encodes the location of a functional element of the object.

2. The system as set forth in claim 1 , wherein the one or more processors further perform operations of:

labeling a point in the 3D audio point cloud as positive if an audio event was detected; and

labeling a point in the 3D audio point cloud as negative if an audio event was not detected.

3. The system as set forth in claim 2 , wherein the one or more processors further perform operations of:

merging the 3D audio point cloud with the 3D tactile point cloud to generate a 3D merged point cloud; and

aligning the 3D merged point cloud with the 3D model point cloud of the object.

4. The system as set forth in claim 3 , wherein the one or more processors further perform an operation of labeling each point in the 3D model point cloud of the object with an estimated probability that an audio event will be detected if the robot manipulates the object at that point.

5. The system as set forth in claim 4 , wherein one or more processors further perform an operation of determining the estimated probability as follows:

defining N(p i ) as a set of points in the 3D audio point cloud which are within a predetermined distance of a point p i ;

defining m audio as a number of points in N(p i ) for which an audio event was detected; and

outputting m audio /|N(p i )|.

6. A computer-implemented method for identifying functional elements of an object for robotic manipulation, comprising an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of:

causing a robot to repeatedly manipulate an object, wherein the robot comprises an audio sensor and a plurality of touch sensors;

recording a three-dimensional (3D) location of each audio event that produces a response as measured by the audio sensor during manipulation of the object;

recording a 3D location of each tactile event that produces a response as detected by the plurality of touch sensors during manipulation of the object;

determining a 3D location of each audio and tactile event in 3D space;

generating a 3D audio point cloud denoting the 3D locations of the audio events;

generating a 3D tactile point cloud denoting the 3D locations of the tactile events;

registering the 3D audio point cloud and the 3D tactile point cloud with a 3D model point cloud of the object; and

annotating the 3D model point cloud of the object with the audio events to generate an annotated 3D model point cloud of the object, which encodes the location of a functional element of the object.

7. The method as set forth in claim 6 , wherein the data processor further performs operations of:

merging the 3D audio point cloud with the 3D tactile point cloud to generate a 3D merged point cloud; and

aligning the 3D merged point cloud with the 3D model point cloud of the object.

8. The method as set forth in claim 7 , wherein the data processor further performs operations of:

merging the 3D audio point cloud with the 3D tactile point cloud to generate a 3D merged point cloud; and

aligning the 3D merged point cloud with the 3D model point cloud of the object.

9. The method as set forth in claim 8 , wherein the data processor further performs an operation of labeling each point in the 3D model point cloud of the object with an estimated probability that an audio event will be detected if the robot manipulates the object at that point.

10. The method as set forth in claim 9 , wherein the data processor further performs an operation of determining the estimated probability as follows:

defining N(p i ) as a set of points in the 3D audio point cloud which are within a predetermined distance of a point p i ;

defining m audio as a number of points in N(p i ) for which an audio event was detected; and

outputting m audio / |N(p i )|.

11. A computer program product for identifying functional elements of an object for robotic manipulation, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

causing a robot to repeatedly manipulate an object, wherein the robot comprises an audio sensor and a plurality of touch sensors;

recording a three-dimensional (3D) location of each audio event that produces a response as measured by the audio sensor during manipulation of the object;

recording a 3D location of each tactile event that produces a response as detected by the plurality of touch sensors during manipulation of the object;

determining a 3D location of each audio and tactile event in 3D space;

generating a 3D audio point cloud denoting the 3D locations of the audio events;

generating a 3D tactile point cloud denoting the 3D locations of the tactile events;

registering the 3D audio point cloud and the 3D tactile point cloud with a 3D model point cloud of the object; and

annotating the 3D model point cloud of the object with the audio events to generate an annotated 3D model point cloud of the object, which encodes the location of a functional element of the object.

12. The computer program product as set forth in claim 11 , further comprising instructions for causing the processor to perform operations of:

labeling a point in the 3D audio point cloud as positive if an audio event was detected; and

labeling a point in the 3D audio point cloud as negative if an audio event was not detected.

13. The computer program product as set forth in claim 12 , further comprising instructions for causing the processor to perform operations of:

merging the 3D audio point cloud with the 3D tactile point cloud to generate a 3D merged point cloud; and

aligning the 3D merged point cloud with the 3D model point cloud of the object.

14. The computer program product as set forth in claim 13 , further comprising instructions for causing the processor to perform an operation of labeling each point in the 3D model point cloud of the object with an estimated probability that an audio event will be detected if the robot manipulates the object at that point.

15. The computer program product as set forth in claim 14 , further comprising instructions for causing the processor to perform an operation of determining the estimated probability as follows:

defining N(p i ) as a set of points in the 3D audio point cloud which are within a predetermined distance of a point p i ;

defining m audio as a number of points in N(p i ) for which an audio event was detected; and

outputting m audio /|N(p i )|.