IP Library Granted Patent US 9,399,290
Granted Patent B2
US 9,399,290 · App. 14/493,676 · Granted Jul 26, 2016

Enhancing sensor data by coordinating and/or correlating data attributes

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,399,290
App. No.
14/493,676
Granted
Jul 26, 2016
Kind
B2
Abstract

A method includes receiving first sensor data acquired by a first sensor in communication with a cloud computing system. The first sensor data has a first set of associated attributes including a time and a location at which the first sensor data was acquired. The method also includes receiving second sensor data acquired by a second sensor in communication with the cloud computing system. The second data has a second set of associated attributes including a time and a location at which the second sensor data was acquire. Further, the method includes generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes and instructing a robot in communication with the cloud computing system to perform a task based at least in part on the data processing result.

Claims (33)

1. A method comprising:

receiving first sensor data acquired by a first sensor that is in communication with a cloud computing system, wherein the first sensor data has a first set of associated attributes including a location at which the first sensor data was acquired, wherein the first sensor data is associated with at least one object in an environment in which one or more robots reside, and wherein the one or more robots are in communication with the cloud computing system;

receiving second sensor data acquired by a second sensor that is in communication with the cloud computing system, wherein the second data has a second set of associated attributes including a location at which the second sensor data was acquired, and wherein the second sensor data is associated with the at least one object in the environment;

generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes, wherein the data processing result includes a localization of the first sensor data and the second sensor data; and

in response to the data processing result being indicative of a threshold high level of certainty of an identification of the at least one object, wherein the data processing result being indicative of the threshold high level of certainty of the identification is based at least in part on the localization being conclusive, providing instructions to the one or more robots to perform a task associated with the identified at least one object.

2. The method of claim 1 , wherein the first sensor data includes one or more of a first image of the at least one object and a first audio signal generated by the at least one object, wherein the second sensor data includes one or more of a second image of the at least one object and a second audio signal generated by the at least one object, wherein the first image and the second image include images of the at least one object from different perspectives, and wherein the first audio signal and the second audio signal are different audio signals generated by the at least one object.

3. The method of claim 1 , wherein the first sensor data includes a first acoustic signal that represents an audible command, wherein the second sensor data includes a second acoustic signal that represents the audible command, and wherein the instructions are related to executing the audible command.

4. The method of claim 1 , further comprising:

removing noise from at least one of the first sensor data and the second sensor data based at least in part on one or more of the first set of associated attributes and the second set of associated attributes.

5. The method of claim 1 , wherein at least one of the one or more robots includes a mechanical actuator.

6. The method of claim 1 , wherein one or more of the first sensor and the second sensor are one and only one sensor.

7. The method of claim 1 , wherein the one or more robots include at least a first robot and a second robot, wherein the first sensor is associated with the first robot, and wherein the second sensor is associated with the second robot.

8. The method of claim 7 , wherein at least one of the first sensor and the second sensor is in wireless communication with the first and second robots, respectively.

9. The method of claim 1 , wherein the localization is based on an identification of a location of the first sensor relative to the at least one object and an identification of a location of the second sensor relative to the at least one object.

10. The method of claim 1 , wherein the first set of associated attributes further includes a time at which the first sensor data was acquired, wherein the second set of associated attributes further includes a time at which the second sensor data was acquired, wherein the data processing result further includes a synchronization of the first sensor data and the second sensor data, and wherein the synchronization is based at least in part on (i) an identification of one or more data points from the first sensor data and the second sensor data and (ii) an extrapolation of the identified data points to a common time.

11. The method of claim 1 , wherein providing instructions to the one or more robots to perform the task associated with the identified at least one object comprises providing instructions to the one or more robots to interact with the identified at least one object.

12. A cloud computing system, comprising:

a computing device configured to receive a request to perform a task, wherein the computing device is configured to communicate with the cloud computing system to:

receive first sensor data associated with first attributes and acquired by a first sensor, wherein the first attributes include a location at which the first sensor data was acquired, and wherein the first sensor data is associated with at least one object in an environment,

receive second sensor data associated with second attributes and acquired by a second sensor, wherein the second attributes include a location at which the second sensor data was acquired, and wherein the second sensor data is associated with the at least one object in the environment,

process the first attributes and the second attributes to localize the first sensor data and the second sensor data and to generate a data processing result representative of the localized first sensor data and second sensor data, and

in response to the data processing result being indicative of a threshold high level of certainty of an identification of the at least one object, wherein the data processing result being indicative of the threshold high level of certainty of the identification is based at least in part on the localization being conclusive, provide instructions to perform the task, wherein the task is associated with the identified at least one object.

13. The system of claim 12 , wherein the first sensor data and second sensor data include acoustic signals that represent an audible command, and wherein the computing device is configured to communicate with the cloud computing system to provide instructions to execute the audible command.

14. The system of claim 12 , wherein the computing device is configured to communicate with the cloud computing system to process at least one of the first attributes and the second attributes to remove noise from at least one of the first sensor data and the second sensor data.

15. The system of claim 12 , wherein the computing device is a server device and the instructions are for a client device to perform the task, and wherein the client device includes a mechanical actuator.

16. The system of claim 12 , wherein the first sensor data is received from a first client device that is in communication with the cloud computing system, wherein the first sensor is coupled to the first client device, wherein the second sensor data is received from a second client device that is in communication with the cloud computing system, wherein the second sensor is coupled to the second client device.

17. The system of claim 12 , wherein the first attributes further include a time at which the first sensor data was acquired, wherein the second attributes further include a time at which the second sensor data was acquired different from the time at which the first sensor data was acquired, wherein processing the first attributes and the second attributes further comprises processing the first attributes and the second attributes to synchronize the first sensor data and the second sensor data, wherein the data processing result is further representative of the synchronized first sensor data and the second sensor data, and wherein synchronization of the first sensor data and the second sensor data is based at least in part on (i) an identification of one or more data points from the first sensor data and the second sensor data and (ii) an extrapolation of the identified data points to a common time.

18. A non-transitory computer readable medium having stored therein instructions executable by at least one processor to cause the at least one processor to perform functions comprising:

accessing first sensor data acquired by a first sensor that is in communication with a cloud computing system, wherein the first sensor data has a first set of associated attributes including a time at which the first sensor data was acquired, wherein the first sensor data is associated with at least one object in an environment in which one or more robots reside, and wherein the one or more robots are in communication with the cloud computing system;

accessing second sensor data acquired by a second sensor that is in communication with the cloud computing system, wherein the second data has a second set of associated attributes including a time at which the second sensor data was acquired, and wherein the second sensor data is associated with the at least one object in the environment;

generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes, wherein the data processing result includes a synchronization of the first sensor data and the second sensor data, and wherein the synchronization is based at least in part on (i) an identification of one or more data points from the first sensor data and the second sensor data and (ii) an extrapolation of the identified data points to a common time; and

in response to the data processing result being indicative of a threshold high level of certainty of an identification of the at least one object, wherein the data processing result being indicative of the threshold high level of certainty of the identification is based at least in part on the synchronization being conclusive, providing instructions to the one or more robots to perform a task associated with the identified at least one object.

19. The non-transitory computer readable medium of claim 18 , wherein the first set of associated attributes further includes a location at which the first sensor data was acquired, wherein the second set of associated attributes further includes a location at which the second sensor data was acquired, wherein the data processing result further includes a localization of the first sensor data and the second sensor data, and wherein the data processing result being indicative of the threshold high level of certainty of the identification of the at least one object is further based on the localization being conclusive.

Assignments (6)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 29, 2025
From: GOOGLE LLC
To: GDM HOLDING LLC
Reel/Frame 071109/0342 →
CORRECTIVE ASSIGNMENT TO CORRECT THE THE REMOVAL OF THE INCORRECTLY RECORDED APPLICATION NUMBERS 14/149802 AND 15/419313 PREVIOUSLY RECORDED AT REEL: 44144 FRAME: 1. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME. Recorded Mar 4, 2024
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 068092/0502 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 21, 2023
From: X DEVELOPMENT LLC
To: GOOGLE LLC
Reel/Frame 064658/0001 →
CHANGE OF NAME Recorded Oct 6, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044144/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2016
From: GOOGLE INC.
To: X DEVELOPMENT LLC
Reel/Frame 039900/0610 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 23, 2014
From: FRANCIS, ANTHONY G., JR.; HICKMAN, RYAN; POURSOHI, ARSHAN
To: GOOGLE INC.
Reel/Frame 033797/0585 →