IP Library Granted Patent US 11,610,078
Granted Patent B2
US 11,610,078 · App. 16/706,608 · Granted Mar 21, 2023

Low variance region detection for improved high variance region detection using machine learning

Inventors: Kratarth Goel (Albany, CA); James William Vaisey Philbin (Palo Alto, CA); Sarah Tariq (Palo Alto, CA)
Assignee: Zoox, Inc.
G06K9/623G06K9/6221G06N3/0454G06N20/00G06V20/58G06V40/103G06V40/161
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,610,078
App. No.
16/706,608
Granted
Mar 21, 2023
Kind
B2
Abstract

Using detection of low variance regions for improving detection is described. In an example, sensor data can be received from a sensor associated with a vehicle. The sensor data can represent an environment. An indication of a low variance region associated with the sensor data can be determined and an indication of a high variance region associated with the sensor data can be determined based at least in part on the indication of the low variance region. The vehicle can be controlled based on at least one of the sensor data or the indication of the high variance region.

Claims (67)

1. A method comprising:

receiving, from an image capturing device associated with a vehicle, image data representing an environment;

inputting at least a portion of the image data into a machine learned model;

determining, by the machine learned model, an indication of a position of a low variance region associated with the image data;

determining, based at least partly on the indication of the position of the low variance region, an indication of a high variance region in the image data represented by a bounding box, the low variance region being a sub-region of the high variance region;

determining, based at least in part on the bounding box, a trajectory; and

controlling the vehicle based at least partly on the trajectory.

2. The method as claim 1 recites, wherein determining the indication of the high variance region comprises:

performing a statistical analysis of the indication of the low variance region and an associated portion of the image data that is proximate to the low variance region; and

determining, based at least partly on the statistical analysis and the associated portion of the image data that is proximate to the low variance region, a location and extents of the high variance region.

3. The method as claim 1 recites, wherein the machine learned model comprises a first machine learned model and the indication of the position of the low variance region comprises a first output, and

wherein determining the indication of the high variance region comprises:

inputting the first output into a second machine learned model; and

receiving, from the second machine learned model, the bounding box,

wherein the second machine learned model is trained to detect high variance regions in image data.

4. The method as claim 1 recites, wherein the machine learned model comprises a first machine learned model and the indication of the position of the low variance region comprises a first output, and

wherein determining the indication of the high variance region comprises:

determining, based at least partly on analyzing the image data using a second machine learned model, a second output indicating at least one feature of the high variance region; and

inputting the first output and the second output into a third machine learned model, wherein the third machine learned model is trained to determine high variance regions in image data.

5. The method as claim 1 recites, further comprising:

receiving, based at least partly on the image data, a plurality of classifications of an object identified in the image data and a plurality of confidence scores, an individual confidence score corresponding to an individual classification;

reducing, based at least partly on the indication of the position of the low variance region, a threshold associated with a classification of the plurality of classifications, wherein the classification corresponds to a high variation region;

determining that a confidence score associated with the classification meets or exceeds the threshold; and

determining the indication of the high variance region based at least partly on determining that the confidence score meets or exceeds the threshold.

6. A system comprising:

one or more processors; and

computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving sensor data representing an environment;

determining an indication of a position of a low variance region associated with the sensor data;

determining an indication of a high variance region associated with the sensor data based at least in part on the indication of the position of the low variance region, the low variance region being a sub-region of the high variance region; and

controlling the system based on at least one of the sensor data or the indication of the high variance region.

7. The system as claim 6 recites, the operations further comprising:

determining an additional indication of a high variance region associated with the sensor data; and

controlling the system further based at least in part on the additional indication of the high variance region.

8. The system as claim 7 recites, the operations further comprising:

determining, based at least in part on the indication of the high variance region and the additional indication of the high variance region, a combined indication of the high variance region; and

controlling the system further based at least in part on the combined indication of the high variance region.

9. The system as claim 6 recites, wherein the indication of the position of the low variance region is one or more of represented in the sensor data or derived from the sensor data.

10. The system as claim 6 recites, wherein:

the determining the indication of the position of the low variance region is based at least in part on analyzing at least a portion of the sensor data using a neural network,

the low variance region comprises a representation of a face in the sensor data, and

the high variance region comprises a representation of a pedestrian in the sensor data.

11. The system as claim 10 recites, wherein the neural network is trained based at least in part on utilizing another neural network to project features associated with input data into an image space to generate reconstructed input data and enforcing consistency between the input data and the reconstructed input data using a loss function.

12. The system as claim 6 recites, wherein determining the indication of the high variance region comprises:

inputting the indication of the position of a low variance region into a portion of a machine learned model trained to detect high variance regions in sensor data; and

analyzing the indication of the position of the low variance region by the portion of the machine learned model.

13. The system as claim 6 recites, the operations further comprising:

receiving a plurality of classifications of an object identified in the sensor data and a plurality of confidence scores, an individual confidence score corresponding to an individual classification;

reducing a threshold associated with a classification of the plurality of classifications associated with the high variance region;

determining that a confidence score associated with the classification meets or exceeds the threshold; and

determining the indication of the high variance region based at least partly on determining that the confidence score meets or exceeds the threshold.

14. The system as claim 13 recites, wherein reducing the threshold associated with the classification is based at least in part on determining the indication of the position of the low variance region.

15. The system as claim 6 recites, wherein the indication of the high variance region comprises a bounding box.

16. One or more computer-readable media that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving, from a sensor associated with a vehicle, sensor data representing an environment within which the vehicle is located;

inputting, into a machine learned model, at least a portion of the sensor data;

determining, by the machine learned model, an indication of a low variance region associated with the sensor data, wherein the low variance region indicates at least a portion of an object that varies from object to object, within a same object classification, at or below a threshold variance; and

determining an indication of a high variance region based at least in part on a portion of the sensor data associated with the low variance region, the low variance region being a sub-region of the high variance region.

17. The one or more computer-readable media as claim 16 recites, the operations further comprising determining the indication of the high variance region based at least in part on the indication of the low variance region associated with the portion of the sensor data.

18. The one or more computer-readable media as claim 16 recites, the operations further comprising determining the indication of the high variance region based at least in part on analyzing at least the portion of the sensor data using at least one of another machine learned model or another portion of the machine learned model.

19. The one or more computer-readable media as claim 16 recites, the operations further comprising:

determining, based at least in part on the indication of the high variance region, a trajectory along which the vehicle is to travel; and

controlling the vehicle based at least partly on the trajectory.

20. The one or more computer-readable media as claim 16 recites, wherein:

the sensor data is image data;

the low variance region is associated with at least one of a front view of a face, a side view of a face, or a back of a face in the image data; and

the high variance region is associated with a pedestrian.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 7, 2020
From: GOEL, KRATARTH; PHILBIN, JAMES WILLIAM VAISEY; TARIQ, SARAH
To: ZOOX, INC.
Reel/Frame 052598/0462 →
Continuity (3)
Continuation In Part 16457524 · Jun 28, 2019
Provisional Application 62925698 · Oct 24, 2019
Related Publication 20200410281A1 · Dec 31, 2020
Cited By (1)
US 12,272,155