IP Library Granted Patent US 8,874,266
Granted Patent B1
US 8,874,266 · App. 13/564,313 · Granted Oct 28, 2014

Enhancing sensor data by coordinating and/or correlating data attributes

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 8,874,266
App. No.
13/564,313
Granted
Oct 28, 2014
Kind
B1
Abstract

A method includes receiving first sensor data acquired by a first sensor in communication with a cloud computing system. The first sensor data has a first set of associated attributes including a time and a location at which the first sensor data was acquired. The method also includes receiving second sensor data acquired by a second sensor in communication with the cloud computing system. The second data has a second set of associated attributes including a time and a location at which the second sensor data was acquire. Further, the method includes generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes and instructing a robot in communication with the cloud computing system to perform a task based at least in part on the data processing result.

Claims (37)

1. A method, comprising:

receiving first sensor data acquired by a first sensor that is in communication with the cloud computing system, wherein the first sensor data has a first set of associated attributes including a time and a location at which the first sensor data was acquired, and wherein the first sensor data includes one or more of a first image of at least one object and a first audio signal generated by the at least one object;

receiving second sensor data acquired by a second sensor that is in communication with the cloud computing system, wherein the second data has a second set of associated attributes including a time and a location at which the second sensor data was acquired, wherein the second sensor data includes one or more of a second image of the at least one object and a second audio signal generated by the at least one object, and wherein the first image and the second image are different images and the first audio signal and the second audio signal are different audio signals;

generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes, wherein the data processing result includes a synchronization of the first sensor data and the second sensor data and further includes a localization of the first sensor data and the second sensor data, wherein the synchronization is based at least in part on an identification of one or more data points from the first sensor data and the second sensor data and an extrapolation of the identified data points to a common time;

in response to the data processing result being inconclusive of an identification of the at least one object, wherein the data processing result being inconclusive is based at least in part on the localization being inconclusive, receiving third sensor data by at least one third sensor that is in communication with the cloud computing system, wherein the third sensor data has a third set of associated attributes including a time and a location at which the third sensor data is acquired, and wherein the third sensor data includes one or more of a third image of the at least one object and a third audio signal generated by the at least one object;

generating an updated data processing result based at least in part on the first sensor data and associated attributes, the second sensor data and associated attributes, and the third sensor data and associated attributes; and

in response to the updated data processing result being conclusive of the identification of the at least one object, providing instructions to one or more robots that are in communication with the cloud computing system to perform a task associated with the identified at least one object.

2. The method of claim 1 , wherein the first image, the second image, and the third image include images of the at least one object from different perspectives.

3. The method of claim 1 , wherein the task includes interacting with the identified at least one object.

4. The method of claim 1 , wherein one or more of the first audio signal, the second audio signal, and the third audio signal include acoustic signals that represent an audible command, and wherein the instructions are related to executing the audible command.

5. The method of claim 1 , further comprising removing noise from at least one of the first sensor data, the second sensor data, and the third sensor data based at least in part on one or more of the first set of associated attributes, the second set of associated attributes, and the third set of associated attributes.

6. The method of claim 1 , wherein the one or more robots include a mechanical actuator.

7. The method of claim 1 , wherein one or more of the first sensor, the second sensor, and the at least one third sensor are one and only one sensor.

8. The method of claim 1 , wherein the one or more robots include at least a first robot and a second robot, wherein the first sensor is associated with the first robot, and wherein the second sensor is associated with the second robot.

9. The method of claim 8 , wherein at least one of the first sensor and the second sensor is in wireless communication with the first and second robots, respectively.

10. The method of claim 1 , wherein receiving the third sensor data based on the data processing result being inconclusive of the identification of the at least one object comprises receiving the third sensor data based on the data processing result being indicative of a threshold low level of certainty of the identification.

11. The method of claim 1 , wherein the localization is based on an identification of a location of the first sensor relative to the at least one object and an identification of a location of the second sensor relative to the at least one object.

12. A cloud computing system, comprising:

a computing device configured to receive a request to perform a task, wherein the computing device is configured to communicate with a cloud computing system to:

receive first sensor data associated with first attributes and acquired by a first sensor, wherein the first attributes include a time and a location at which the first sensor data was acquired, and wherein the first sensor data includes one or more of a first image of at least one object and a first audio signal generated by the at least one object,

receive second sensor data associated with second attributes and acquired by a second sensor, wherein the second attributes include a time and a location at which the second sensor data was acquired, wherein the second sensor data includes one or more of a second image of the at least one object and a second audio signal generated by the at least one object, and wherein the first image and the second image are different images and the first audio signal and the second audio signal are different audio signals,

process the first attributes and the second attributes to localize and synchronize the first sensor data and the second sensor data and to generate a data processing result representative of the localized and synchronized first sensor data and second sensor data, wherein synchronizing the first sensor data and the second sensor data is based at least in part on an identification of one or more data points from the first sensor data and the second sensor data and an extrapolation of the identified data points to a common time,

in response to the data processing result being inconclusive of an identification of the at least one object, wherein the data processing result being inconclusive is based at least in part on the localized first sensor data and second sensor data being inconclusive, receive third sensor data associated with third attributes and acquired by at least one third sensor, wherein the third attributes include a time and a location at which the third sensor data was acquired, and wherein the third sensor data includes one or more of a third image of the at least one object and a third audio signal generated by the at least one object,

process the first attributes, the second attributes, and the third attributes to localize and synchronize the first sensor data, the second sensor data, and the third sensor data and to generate an updated data processing result, and

in response to the updated data processing result being conclusive of the identification of the at least one object, responsively provide instructions to perform the task, wherein the task is associated with the identified at least one object.

13. The system of claim 12 , wherein the first sensor data, the second sensor data, and the third sensor data include images of the at least one object from different perspectives.

14. The system of claim 12 , wherein the first and second sensor data include acoustic signals that represent an audible command, and wherein the computing device is configured to communicate with the cloud computing system to provide instructions to execute the audible command.

15. The system of claim 12 , wherein the computing device is configured to communicate with the cloud computing system to process at least one of the first attributes, the second attributes, and the third attributes to remove noise from at least one of the first sensor data, the second sensor data, and the third sensor data.

16. The system of claim 12 , wherein the computing device is a server device and the instructions are for a client device to perform the task.

17. The system of claim 16 , wherein the client device includes a mechanical actuator.

18. A non-transitory computer readable medium having stored therein instructions executable by at least one processor to cause the at least one processor to perform functions comprising:

accessing first sensor data acquired by a first sensor that is in communication with a cloud computing system, wherein the first sensor data has a first set of associated attributes including a time and a location at which the first sensor data was acquired, and wherein the first sensor data includes one or more of a first image of at least one object and a first audio signal generated by the at least one object;

accessing second sensor data acquired by a second sensor that is in communication with the cloud computing system, wherein the second data has a second set of associated attributes including a time and a location at which the second sensor data was acquired, wherein the second sensor data includes one or more of a second image of the at least one object and a second audio signal generated by the at least one object, and wherein the first image and the second image are different images and the first audio signal and the second audio signal are different audio signals;

generating a data processing result based at least in part on the first sensor data, the first set of associated attributes, the second sensor data, and the second set of associated attributes, wherein the data processing result includes a synchronization of the first sensor data and the second sensor data and further includes a localization of the first sensor data and the second sensor data, wherein the synchronization is based at least in part on an identification of one or more data points from the first sensor data and the second sensor data and an extrapolation of the identified data points to a common time;

in response to the data processing result being inconclusive of an identification of the at least one object, wherein the data processing result being inconclusive is based at least in part on the localization being inconclusive, accessing third sensor data acquired by at least one third sensor that is in communication with the cloud computing system, wherein the third sensor data has a third set of associated attributes including a time and a location at which the third sensor data is acquired, and wherein the third sensor data includes one or more of a third image of the at least one object and a third audio signal generated by the at least one object;

generating an updated data processing result based at least in part on the first sensor data and associated attributes, the second sensor data and associated attributes, and the third sensor data and associated attributes; and

in response to the updated data processing result being conclusive of the identification of the at least one object, providing instructions to one or more robots that are in communication with the cloud computing system to perform a task associated with the identified at least one object.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 29, 2025
From: GOOGLE LLC
To: GDM HOLDING LLC
Reel/Frame 071109/0342 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 21, 2023
From: X DEVELOPMENT LLC
To: GOOGLE LLC
Reel/Frame 064658/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2016
From: GOOGLE INC.
To: X DEVELOPMENT LLC
Reel/Frame 039900/0610 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 14, 2012
From: FRANCIS JR., ANTHONY G.; HICKMAN, RYAN; POURSOHI, ARSHAN
To: GOOGLE INC.
Reel/Frame 028785/0368 →