IP Library Granted Patent US 11,233,956
Granted Patent B2
US 11,233,956 · App. 16/836,764 · Granted Jan 25, 2022

Sensor system with low power sensor devices and high power sensor devices

Inventors: Shaomin Xiong (Fremont, CA); Toshiki Hirano (San Jose, CA); Haoyu Wu (Sunnyvale, CA)
Assignee: Western Digital Technologies, Inc.
H04N5/355G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,233,956
App. No.
16/836,764
Granted
Jan 25, 2022
Kind
B2
Abstract

In one embodiment, a sensor device includes a sensor configured to obtain sensor data and a communication interface configured to communicate with a set of secondary sensor devices. The set of secondary sensor devices comprise a set of secondary sensors and a set of secondary processing devices. The set of secondary sensors obtain additional sensor data. The set of secondary processing devices process the additional sensor data based on a set of secondary machine learning models, and generate activation data based on the additional sensor data. The sensor device also includes a processing device. The processing device is also configured to receive the activation data from the set of secondary sensor devices and cause the sensor to obtain first sensor data based on the activation data. The processing device is further configured to generate one or more inferences based on the first sensor data and a machine learning model.

Claims (71)

1. A sensor device, comprising:

a sensor configured to obtain sensor data, the sensor data comprises at least an image;

a communication interface configured to communicate with a set of secondary sensor devices, wherein:

the set of secondary sensor devices comprise a set of secondary sensors and a set of secondary processing devices;

the set of secondary sensors are configured to obtain additional sensor data; and

the set of secondary processing devices are configured to process the additional sensor data based on a set of secondary machine learning models, and to generate activation data based on processing the additional sensor data through the set of secondary machine learning models; and

a processing device coupled to the sensor, the processing device configured to:

operate the sensor device in a first power state;

receive the activation data from the set of secondary sensor devices;

transition the sensor device to a second power state that consumes more power than the first power state;

cause the sensor to obtain first sensor data based on the activation data; and

generate one or more inferences based on the first sensor data and a machine learning model, wherein:

the activation data indicates a portion of the image to be analyzed using the machine learning model; and

the sensor device at the second power state consumes more power than each secondary sensor device.

2. The sensor device of claim 1 , wherein the one or more inferences are generated further based on the activation data.

3. The sensor device of claim 2 , wherein the activation data indicates a portion of the sensor data to be analyzed using the machine learning model.

4. The sensor device of claim 3 , wherein:

the sensor data further comprises a video;

the activation data indicates a portion of the video to be analyzed using the machine learning model.

5. The sensor device of claim 2 , wherein the activation data comprises intermediate layer outputs of one or more secondary machine learning models of the set of secondary machine learning models.

6. The sensor device of claim 5 , wherein the processing device is further configured to:

bypassing one or more layers of the machine learning model by providing the intermediate layer outputs to an intermediate layer of the machine learning model.

7. The sensor device of claim 1 , wherein the processing device is further configured to:

adjust one or more parameters of the sensor based on the activation data.

8. The sensor device of claim 7 , wherein the activation data comprises a status of an object detected by at least one of the set of secondary machine learning models.

9. The sensor device of claim 7 , wherein the one or more parameters comprises one or more of a shutter speed, an exposure time, an aperture, a frame rate, and an ISO setting.

10. The sensor device of claim 1 , the one or more inferences are generated further based on the additional sensor data.

11. The sensor device of claim 1 , wherein the processing device is further configured to:

transmit adjustment data to the set of secondary sensor devices, wherein the set of secondary sensor devices adjust the generation of the one or more inferences based on the adjustment data.

12. The sensor device of claim 1 , wherein:

the sensor device and one or more of the set of secondary sensor devices are positioned at different locations;

the first sensor data and the additional sensor data indicate different views of an object; and

the first sensor data and the additional sensor data improves an accuracy of inferences generated by the machine learning model.

13. The sensor device of claim 1 , wherein the set of secondary sensor devices are configured to continually obtain the additional sensor data.

14. A system comprising the sensor device and the set of secondary sensor devices of claim 1 .

15. A sensor device, comprising:

a sensor configured to obtain sensor data, the sensor data comprises at least an image;

a communication interface configured to communicate with a primary sensor device, wherein:

the primary sensor device comprises a primary sensor and a primary processing device;

the primary sensor is configured to obtain primary sensor data; and

the primary processing device is configured to process the primary sensor data through a primary machine learning model based on activation data; and

a processing device coupled to the sensor, the processing device configured to:

receive the sensor data from the sensor;

generate one or more inferences based on the sensor data and a machine learning model, wherein the activation data indicates a portion of the image to be analyzed using the machine learning model; and

transmit activation data to the primary sensor device based on the one or more inferences, wherein:

the sensor device consumes less power than the primary sensor device; and

the activation data causes the primary sensor device to obtain additional sensor data.

16. The sensor device of claim 15 , wherein:

the sensor data further comprises a video;

the activation data indicates a portion of additional sensor data to be analyzed using the primary machine learning model; and

the activation data indicates a portion of the video to be analyzed using the machine learning model.

17. The sensor device of claim 15 , wherein the activation data comprises intermediate layer outputs one or more intermediate layers of the machine learning model.

18. The sensor device of claim 15 , wherein the sensor device and the primary sensor device are located at different locations.

19. The sensor device of claim 15 , wherein the processing device is further configured to:

receive adjustment data from the primary sensor device; and

adjust the generation of the one or more inferences based on the adjustment data.

20. The sensor device of claim 15 , wherein the processing device is further configured to:

receive adjustment data from the primary sensor device; and

adjust one or more parameters of the sensor based on the adjustment data.

21. The sensor device of claim 15 , wherein the activation data comprises a status of an object detected by the machine learning model.

22. A system comprising the sensor device and the primary sensor device of claim 15 .

23. A method, comprising:

operating a sensor device in a first power state;

receiving, by the sensor device, activation data from a set of secondary sensor devices, wherein:

the set of secondary sensor devices comprise a set of secondary sensors and a set of secondary processing devices;

the set of secondary sensors are configured to obtain additional sensor data; and

the set of secondary processing devices are configured to process the additional sensor data based on a set of secondary machine learning models, and to generate the activation data based on processing the additional sensor data through the set of secondary machine learning models; and

causing a sensor of the sensor device to obtain sensor data and first sensor data based on the activation data; and

generating one or more inferences based on the first sensor data and a machine learning model, wherein:

the activation data indicates a portion of the sensor data to be analyzed using the machine learning model, wherein the sensor data comprises an image; and

the sensor device consumes more power than each of the set of secondary sensor devices.

Assignments (5)
PATENT COLLATERAL AGREEMENT - A&R LOAN AGREEMENT Recorded Aug 21, 2023
From: WESTERN DIGITAL TECHNOLOGIES, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 064715/0001 →
PATENT COLLATERAL AGREEMENT - DDTL LOAN AGREEMENT Recorded Aug 21, 2023
From: WESTERN DIGITAL TECHNOLOGIES, INC.
To: JPMORGAN CHASE BANK, N.A.
Reel/Frame 067045/0156 →
RELEASE OF SECURITY INTEREST AT REEL 053482 FRAME 0453 Recorded Feb 8, 2022
From: JPMORGAN CHASE BANK, N.A.
To: WESTERN DIGITAL TECHNOLOGIES, INC.
Reel/Frame 058966/0279 →
SECURITY INTEREST Recorded May 14, 2020
From: WESTERN DIGITAL TECHNOLOGIES, INC.
To: JPMORGAN CHASE BANK, N.A., AS AGENT
Reel/Frame 053482/0453 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 1, 2020
From: XIONG, SHAOMIN; HIRANO, TOSHIKI; WU, HAOYU
To: WESTERN DIGITAL TECHNOLOGIES, INC.
Reel/Frame 052548/0070 →