Multimodal sensor array for robotic systems
A multimodal sensing architecture utilizes an array of single sensor or multi-sensor groups (superpixels) to facilitate advanced object-manipulation and recognition tasks performed by mechanical end effectors in robotic systems. The single-sensors/superpixels are spatially arrayed over contact surfaces of the end effector fingers and include, e.g., pressure sensors and vibration sensors that facilitate the simultaneous detection of both static and dynamic events occurring on the end effector, and optionally include proximity sensors and/or temperature sensors. A readout circuit receives the sensor data from the superpixels and transmits the sensor data onto a shared sensor data bus. An optional multimodal control generator receives and processes the sensor data and generates multimodal control signals that cause the robot system's control circuit to adjust control operations performed by the end effector or other portions of the robot mechanism and when the sensor data indicates non-standard operating conditions.
1. A method for controlling a robotic system including a robot mechanism having an end effector including a plurality of spatially arrayed superpixels, each said superpixel including an associated pressure sensor and an associated vibration sensor disposed over a corresponding contact surface portion of the end effector, the end effector being configured to grasp a target object, and the robot mechanism being configured to move the target object while being grasped by the end effector, the method comprising:
controlling the pressure sensors of said plurality of spatially arrayed superpixels such that each said pressure sensor generates associated time-based localized static event data values in response to static forces applied to said corresponding contact surface portion on the end effector while the target object is grasped by the end effector;
controlling the vibration sensors of said plurality of spatially arrayed superpixels such that said each said vibration sensor generates associated time-based localized dynamic event data values in response to vibrational forces applied to said corresponding contact surface portion on the end effector while the robot mechanism is moving the target object;
while the robot mechanism is moving the target object, identifying a location of a non-standard operating condition using changes in at least one of said time-based localized static event data values and said time-based localized dynamic event data values; and
adjusting the end effector in response to said identified non-standard operating condition location.
2. The method of claim 1 , further comprising:
using both said static event data and said dynamic event data to identify slipping of said target object relative to said end actuator, and
conrolling the end effector to increase a grip force applied to the target object in response to said identified slipping.
3. The method of claim 1 ,
wherein identifying said non-standard operating condition location includes using both said static event data and said dynamic event data to identify an undesirable contact location at which undesirable contact between said target object and a secondary object occurs, and
wherein adjusting the end effector comprises controlling the robot mechanism to move the target object to a location that relative to the secondary object such that the undesirable contact is avoided.
4. The method of claim 1 , wherein controlling the robotic system comprises performing an assembly process including the target object and a secondary object.
5. The method of claim 4 , wherein the target object comprises a hollow structure having an open bottom end, and wherein performing the assembly process comprises mounting the target object over the secondary object such that at least a portion of the secondary object extends through the open bottom end into the hollow structure when the assembly process is successfully completed.
6. The method of claim 4 , wherein identifying the location of the non-standard operating condition comprises detecting a misalignment between said target object and said secondary object by detecting changes in one or more time-based localized dynamic event data values generated by associated vibration sensors located adjacent to an impact location between said target object and said secondary object during said assembly process.
7. The method of claim 4 , wherein identifying the location of the non-standard operating condition comprises detecting a misalignment between said target object and said secondary object by detecting changes in one or more time-based localized dynamic event data values generated by associated vibration sensors located adjacent to a scraping-type contact location between said target object and said secondary object during said assembly process.
8. The method of claim 4 , further comprising using both said static event data and said dynamic event data to detect an expected final contact event that confirms a successful completion of said assembly process.