IP Library Granted Patent US 9,485,366
Granted Patent B2
US 9,485,366 · App. 15/066,105 · Granted Nov 1, 2016

Methods and systems for communicating sensor data on a mobile device

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,485,366
App. No.
15/066,105
Granted
Nov 1, 2016
Kind
B2
Abstract

Methods and systems for communicating sensor data on a mobile device are described. An example method involves receiving, by a processor and from an inertial measurement unit (IMU), sensor data corresponding to a first timeframe, and storing the sensor data using a data buffer. The processor may also receive image data and sensor data corresponding to a second timeframe. The processor may then generate a digital image that includes at least the image data corresponding to the second timeframe and the sensor data corresponding to the first timeframe and the second timeframe. The processor may embed the stored sensor data corresponding to the first timeframe and the second timeframe in pixels of the digital image. And the processor may provide the digital image to an application processor of the mobile device.

Claims (64)

1. A method comprising:

receiving, at a first processor of a device, a first digital image comprising first sensor data corresponding to a first timeframe, second sensor data corresponding to a second timeframe subsequent to the first timeframe, and image data corresponding to the second timeframe, wherein the first sensor data and second sensor data are embedded in pixels of the first digital image; and

responsive to determining that the first sensor data was not correctly received by the first processor via a source other than the first digital image, extracting the first sensor data from the first digital image for use at the first processor.

2. The method of claim 1 , further comprising:

providing a second digital image from a second processor for receipt by the first processor, the second digital image comprising the first sensor data embedded in pixels of the second digital image; and

wherein determining that the first sensor data was not correctly received by the first processor comprises determining, at the first processor, at least one of: the second digital image was corrupted during transmission from the second processor to the first processor; and the second digital image was dropped by the first processor.

3. The method of claim 1 , wherein:

the first sensor data comprises sensor data obtained from an inertial measurement unit (IMU) during the first timeframe;

the second sensor data comprises sensor data obtained from the IMU during the second timeframe; and

the image data comprises image data obtained from at least one camera of the device during the second timeframe.

4. The method of claim 1 , wherein:

the first digital image includes a first timestamp associated with the first sensor data and a second timestamp associated with the second sensor data; and

the method further comprises:

determining a validity of the second sensor data based on a comparison of a time span between the first and second timestamps to a specified threshold.

5. The method of claim 1 , wherein:

the first digital image further comprises depth information associated with the image data; and

receiving the first digital image comprises extracting, at the first processor, the depth information from the first digital image.

6. The method of claim 1 , wherein:

the first digital image further comprises computer-vision information associated with the image data, wherein the computer-vision information comprises one or more image features of the first digital image; and

receiving the first digital image comprises extracting, at the first processor, the computer-vision information from the first digital image.

7. The method of claim 1 , wherein:

extracting the first sensor data from the first digital image for use at the first processor comprises using the extracted first sensor data for estimating at least one of a current location or current orientation of the device.

8. A device comprising:

a first processor, the first processor configured to:

receive a first digital image comprising first sensor data corresponding to a first timeframe, second sensor data corresponding to a second timeframe subsequent to the first timeframe, and image data corresponding to the second timeframe, wherein the first sensor data and second sensor data are embedded in pixels of the first digital image; and

responsive to determining that the first sensor data was not correctly received by the first processor via a source other than the first digital image, extract the first sensor data from the first digital image for use at the first processor.

9. The device of claim 8 , further comprising:

a second processor, the second processor to provide the first digital image for receipt by the first processor and to provide a second digital image for receipt by the first processor, the second digital image comprising the first sensor data embedded in pixels of the second digital image; and

wherein the first processor is to determine that the first sensor data was not correctly received by the first processor by determining at least one of: the second digital image was corrupted during transmission from the second processor to the first processor; and the second digital image was dropped by the first processor.

10. The device of claim 9 , further comprising:

an inertial measurement unit (IMU) to provide the first sensor data during the first timeframe and to provide the second sensor data during the second timeframe; and

at least one camera to provide the image data during the second timeframe.

11. The device of claim 8 , wherein:

the first digital image includes a first timestamp associated with the first sensor data and a second timestamp associated with the second sensor data; and

the first processor further is to determine a validity of the second sensor data based on a comparison of a time span between the first and second timestamps to a specified threshold.

12. The device of claim 8 , wherein:

the first digital image further comprises depth information associated with the image data; and

the first processor further is to extract the depth information from the first digital image.

13. The device of claim 8 , wherein:

the first digital image further comprises computer-vision information associated with the image data, wherein the computer-vision information comprises one or more image features of the first digital image; and

the first processor further is to extract the computer-vision information from the first digital image.

14. The device of claim 8 , wherein:

the first processor is to use the extracted first sensor data for estimating at least one of a current location or current orientation of the device.

15. The device of claim 8 , wherein:

the first processor and second processor are coupled via a camera bus interface.

16. The device of claim 8 , wherein:

the device comprises a mobile device.

17. A method comprising:

receiving, by a processor of a device and from an inertial measurement unit (IMU) of the device, sensor data corresponding to a first timeframe;

storing, by the processor, the sensor data corresponding to the first timeframe using a data buffer;

receiving, by the processor and from at least one camera of the device, image data corresponding to a second timeframe, wherein the second timeframe is subsequent to the first timeframe;

receiving, by the processor and from the IMU, sensor data corresponding to the second timeframe;

generating, by the processor, a digital image that includes at least the image data corresponding to the second timeframe, the stored sensor data corresponding to the first timeframe, and the sensor data corresponding to the second timeframe, wherein the stored sensor data corresponding to the first timeframe and the sensor data corresponding to the second timeframe are embedded in pixels of the digital image; and

providing, by the processor, the digital image to another component of the device.

18. The method of claim 17 , further comprising:

determining time stamps associated with the first timeframe and the second timeframe respectively;

wherein the digital image further includes the determined time stamps; and

wherein the determined time stamps are embedded in additional pixels of the digital image.

19. The method of claim 17 , further comprising:

receiving, by the processor and from a depth processor of the device, depth data corresponding to the second timeframe, wherein the digital image further comprises the depth data.

20. The method of claim 17 , further comprising:

determining computer-vision data using the image data corresponding to the first timeframe, wherein the computer-vision data comprises one or more image features of the digital image;

storing the computer-vision data corresponding to the first timeframe using the data buffer; and

determining computer-vision data using the image data corresponding to the second timeframe, wherein the digital image further comprises the stored computer-vision data corresponding to the first timeframe and the computer-vision data corresponding to the second timeframe.

Assignments (2)
CHANGE OF NAME Recorded Dec 5, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044695/0115 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 16, 2017
From: FUNG, JAMES; HESCH, JOEL; LEE, JOHNNY
To: GOOGLE INC.
Reel/Frame 043873/0937 →