IP Library Granted Patent US 10,229,317
Granted Patent B2
US 10,229,317 · App. 15/230,412 · Granted Mar 12, 2019

Selectively downloading targeted object recognition modules

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,229,317
App. No.
15/230,412
Granted
Mar 12, 2019
Kind
B2
Abstract

Methods, apparatus, systems, and computer-readable media are provided for downloading targeted object recognition modules that are selected from a library of candidate targeted object recognition modules based on various signals. In some implementations, an object recognition client may be operated to facilitate object recognition for a robot. It may download targeted object recognition module(s). Each targeted object recognition module may facilitate inference of an object type or pose of an observed object. The targeted object module(s) may be selected from a library of targeted object recognition modules based on various signals, such as a task to be performed by the robot. The object recognition client may obtain vision data capturing at least a portion of an environment in which the robot operates. The object recognition client may determine, based on the vision data and the downloaded object recognition module(s), information about an observed object in the environment.

Claims (49)

1. A computer-implemented method comprising:

operating, by one or more processors integral with a robot, an object recognition client to facilitate object recognition for the robot;

determining an assigned task to be performed by the robot;

determining a range of a wireless access point through which the robot is able to engage in wireless network communication with a remote computing system;

determining that the assigned task requires the robot to travel to an area that is outside of the range of the wireless access point, such that the robot will be unable to engage in wireless network communication with the remote computing system while in the area;

based on the assigned task, and while the robot is within the range of the wireless access point, downloading, to the robot by the object recognition client, from the remote computing system, one or more targeted object recognition modules, wherein each targeted object recognition module is usable by the object recognition client to calculate, at the robot, an object type or pose of an observed object, and wherein the one or more targeted object recognition modules are selected from a library of targeted object recognition modules that is remote from the robot based on the assigned task;

determining that the downloading is complete;

after determining that the downloading is complete, operating the robot to travel outside of the range of the wireless access point into the area;

obtaining, by the object recognition client, from one or more vision sensors integral with the robot, vision data capturing, from a perspective of the robot, at least a portion of the second area;

determining, at the robot by the object recognition client, based on the vision data and the one or more downloaded object recognition modules, information about an object in the second area that is detected in the vision data captured by the one or more vision sensors integral with the robot; and

operating the robot to perform the assigned task.

2. The computer-implemented method of claim 1 , wherein the one or more targeted object modules are selected from the library based at least in part on one or more expected objects or object types associated with the assigned task assigned to the robot.

3. The computer-implemented method of claim 2 , wherein the expected objects or object types are determined based on objects encountered by one or more robots historically when performing tasks sharing one or more attributes with the assigned task.

4. The computer-implemented method of claim 1 , wherein the one or more targeted object recognition modules are selected from the library based at least in part on available resources of the robot.

5. The computer-implemented method of claim 1 , wherein the one or more targeted object recognition modules are selected from the library based at least in part on the vision data.

6. The computer-implemented method of claim 1 , further comprising selecting, by the object recognition client, the one or more targeted object recognition modules from the library.

7. The computer-implemented method of claim 1 , wherein the one or more targeted object recognition modules are selected from the library by the remote computing system based on one or more attributes of the assigned task provided to the remote computing system by the object recognition client.

8. The computer-implemented method of claim 1 , wherein the one or more vision sensors includes a camera configured to obtain depth data, and the vision data comprises a point cloud associated with the environment.

9. The computer-implemented method of claim 1 , further comprising selecting, by the object recognition client, from multiple competing inferences of object types or poses provided by multiple downloaded targeted object recognition modules, a given inference of an object type or pose based on a measure of confidence associated with the given inference.

10. A system comprising one or more processors integral with the robot and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:

operate an object recognition client at a robot to perform object recognition for the robot, wherein the object recognition client is configured to:

determine an assigned task to be performed by the robot;

determine a range of a wireless access point through which the robot is able to engage in wireless network communication with a remote computing system;

determine that the assigned task requires the robot to travel to an area that is outside of the range of the wireless access point, such that the robot will be unable to engage in wireless network communication with the remote computing system while in the area;

based on the assigned task, and while the robot is within the range of the wireless access point, download, to the robot from the remote computing system, one or more targeted object recognition modules, wherein each targeted object recognition module is usable by the object recognition client to calculate, at the robot, an object type or pose of an observed object, and wherein the one or more targeted object modules are selected from a library of targeted object recognition modules that is remote from the robot based on the assigned task;

determine that the downloading is complete;

after determining that the downloading is complete, operate the robot to travel outside of the range of the wireless access point into the area;

obtain, from one or more vision sensors integral with the robot, vision data capturing, from a perspective of the robot, at least a portion of the second area;

determine, at the robot based on the vision data and the one or more downloaded object recognition modules, information about an object in the second area that is detected in the vision data captured by the one or more vision sensors integral with the robot; and

operate the robot to perform the assigned task.

11. The system of claim 10 , wherein the one or more targeted object modules are selected from the library based at least in part on one or more expected objects or object types associated with the assigned task.

12. The system of claim 10 , wherein the one or more targeted object recognition modules are selected from the library based at least in part on available resources of the robot.

13. The system of claim 10 , wherein the one or more targeted object recognition modules are selected from the library based at least in part on the vision data.

14. The system of claim 10 , wherein the one or more targeted object recognition modules are selected from the library by the object recognition client.

15. The system of claim 10 , wherein the one or more targeted object recognition modules are selected from the library by the remote computing system based on one or more attributes of the assigned task provided to the remote computing system by the object recognition client.

16. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors integral with a robot, cause the one or more processors to perform the following operations:

determining an assigned task to be performed by the robot;

determining a range of a wireless access point through which the robot is able to engage in wireless network communication with a remote computing system;

determining that the assigned task requires the robot to travel to an area that is outside of the range of the wireless access point, such that the robot will be unable to engage in wireless network communication with the remote computing system while in the area;

based on the assigned task, and while the robot is within the range of the wireless access point, downloading, to the robot from the remote computing system, one or more targeted object recognition modules, wherein each targeted object recognition module is usable by the object recognition client to calculate, at the robot, an object type or pose of an observed object, and wherein the one or more targeted object modules are selected from a library of targeted object recognition modules that is remote from the robot based on the assigned task;

determining that the downloading is complete;

after determining that the downloading is complete, operating the robot to travel outside of the range of the wireless access point into the area;

obtaining, from one or more vision sensors integral with the robot, vision data capturing, from a perspective of the robot, at least a portion of the second area;

determining, at the robot based on the vision data and the one or more downloaded object recognition modules, information about an object in the second area that is detected in the vision data captured by the one or more vision sensors integral with the robot; and

operating the robot to perform the assigned task.

17. The method of claim 1 , further comprising:

operating the robot to travel towards the area outside of the range of the wireless access point during the downloading;

determining that the robot is proximate an edge of the range of the wireless access point before determining that the downloading is complete; and

operating the robot to cease moving until determining that the downloading is complete.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 29, 2025
From: GOOGLE LLC
To: GDM HOLDING LLC
Reel/Frame 071109/0342 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2023
From: X DEVELOPMENT LLC
To: GOOGLE LLC
Reel/Frame 063992/0371 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2016
From: GOOGLE INC.
To: X DEVELOPMENT LLC
Reel/Frame 039900/0610 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 8, 2016
From: RAJKUMAR, NARESHKUMAR; HINTERSTOISSER, STEFAN; BAJRACHARYA, MAX
To: GOOGLE INC.
Reel/Frame 039360/0603 →