Granted Patent B2

US 11,494,921 · App. 16/395,948 · Granted Nov 8, 2022

Inventors: Saleh ElHattab (San Francisco, CA); Justin Joel Delegard (West Chester, OH); Bodecker John DellaMaria (San Francisco, CA); Brian Tuan (Cupertino, CA); Jennifer Winnie Leung (Berkeley, CA); Sylvie Lee (San Francisco, CA); Jesse Michael Chen (San Francisco, CA); Kirti Varun Munjeti (Frisco, TX); Frances Peijin Guo (San Jose, CA)

Assignee: Samsara Networks Inc.

G06T7/248G06N20/00G06T7/90G06V20/46G06T2207/10024

Abstract

Claims (79)

1. A system comprising:

at least one sensor device to generate sensor data comprising a plurality of data streams;

a memory; and

at least one hardware processor to perform operations comprising:

receiving sensor data at a sensor device that includes a dashcam mounted at a vehicle, the sensor data comprising monocular image data;

applying a stereoscopic inference model to the monocular image data, the stereoscopic inference model trained to generate a 3-dimensional (3D) depth model based on the monocular image data;

constructing the #D depth model based on the monocular image data generated by the dashcam and the stereoscopic inference model;

detecting an event based on the 3D depth model;

accessing at least a portion of the plurality of data streams in response to the detecting the event; and

causing display of a presentation of the portion of the plurality of data streams at a client device, the presentation of the portion of the plurality of data streams including an identifier associated with the sensor device.

2. The system of claim 1 , wherein the detecting the event based on the sensor data includes:

performing a comparison of the 3D depth model against one or more threshold values; and

detecting the event based on the comparison.

3. The system of claim 1 , wherein the sensor data includes video data, and the detecting the event includes:

extracting a set of features from the video data;

applying the set of features from the video data to a machine learned model; and

detecting the event based on an output of the machine learned model.

4. The system of claim 1 , wherein the sensor data includes image data that comprises a set of image features, and the detecting the event based on the sensor data includes:

determining a point of gaze based on the image features; and

detecting the event based on the point of gaze.

5. The system of claim 1 , wherein the at least one hardware processor performs operations further comprising:

applying a first portion of the plurality of data streams to a machine learned model at the sensor device;

detecting a precursor to the event at the sensor device based on an output of the machine learned model;

accessing a second portion of the plurality of data streams in response to the detecting the precursor to the event at the sensor device; and

wherein the detecting the event includes detecting the event based on the second portion of the plurality of data streams.

6. The system of claim 1 , wherein the at least one hardware processor performs operations further comprising:

applying the sensor data from a first portion of the plurality of data streams to a first machine learned model;

detecting a precursor to the event based on a first output of the first machine learned model;

applying the sensor data from a second portion of the plurality of data streams to a second machine learned model in response to the detecting the precursor to the event; and

wherein the detecting the event includes detecting the event based on a second output of the second machine learned model.

7. A method comprising:

receiving sensor data at a sensor device that includes a dashcam mounted at a vehicle, the sensor data monocular image data from one or more of a plurality of data streams;

applying a stereoscopic inference model to the monocular image data, the stereoscopic inference model trained to generate a 3-dimensional (3D) depth model based on the monocular image data;

constructing the #D depth model based on the monocular image data generated by the dashcam and the stereoscopic inference model;

detecting an event based on the 3D depth model;

accessing at least a portion of the plurality of data streams in response to the detecting the event; and

causing display of a presentation of the portion of the plurality of data streams at a client device, the presentation of the portion of the plurality of data streams including an identifier associated with the sensor device.

8. The method of claim 7 , wherein the detecting the event based on the sensor data includes:

performing a comparison of the 3D depth model against one or more threshold values; and

detecting the event based on the comparison.

9. The method of claim 7 , wherein the sensor data includes video data, and the detecting the event includes:

extracting a set of features from the video data;

applying the set of features from the video data to a machine learned model; and

detecting the event based on an output of the machine learned model.

10. The method of claim 7 , wherein the sensor data includes image data that comprises a set of image features, and the detecting the event based on the sensor data includes:

determining a point of gaze based on the image features; and

detecting the event based on the point of gaze.

11. The method of claim 7 , wherein the method further comprises:

applying a first portion of the plurality of data streams to a machine learned model at the sensor device;

detecting a precursor to the event at the sensor device based on an output of the machine learned model;

accessing a second portion of the plurality of data streams in response to the detecting the precursor to the event at the sensor device; and

wherein the detecting the event includes detecting the event based on the second portion of the plurality of data streams.

12. The method of claim 7 , wherein the method further comprises:

applying the sensor data from a first portion of the plurality of data streams to a first machine learned model;

detecting a precursor to the event based on a first output of the first machine learned model;

applying the sensor data from a second portion of the plurality of data streams to a second machine learned model in response to the detecting the precursor to the event; and

wherein the detecting the event includes detecting the event based on a second output of the second machine learned model.

13. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving sensor data at a sensor device that includes a dashcam mounted at a vehicle, the sensor data monocular image data from one or more of a plurality of data streams;

applying a stereoscopic inference model to the monocular image data, the stereoscopic inference model trained to generate a 3-dimensional (3D) depth model based on the monocular image data;

to constructing the #D depth model on the monocular image data generated by the dashcam and the stereoscopic inference model;

detecting an event based on the 3D depth model;

accessing at least a portion of the plurality of data streams in response to the detecting the event; and

causing display of a presentation of the portion of the plurality of data streams at a client device, the presentation of the portion of the plurality of data streams including an identifier associated with the sensor device.

14. The non-transitory machine-readable storage medium of claim 13 , wherein the detecting the event based on the sensor data includes:

performing a comparison of the 3D depth model against one or more threshold values; and

detecting the event based on the comparison.

15. The non-transitory machine-readable storage medium of claim 13 , wherein the sensor data includes video data, and the detecting the event includes:

extracting a set of features from the video data;

applying the set of features from the video data to a machine learned model; and

detecting the event based on an output of the machine learned model.

16. The non-transitory machine-readable storage medium of claim 13 , wherein the sensor data includes image data that comprises a set of image features, and the detecting the event based on the sensor data includes:

determining a point of gaze based on the image features; and

detecting the event based on the point of gaze.

17. The non-transitory machine-readable storage medium of claim 13 , wherein the instructions cause the machine to perform operations further comprising:

applying a first portion of the plurality of data streams to a machine learned model at the sensor device;

detecting a precursor to the event at the sensor device based on an output of the machine learned model;

accessing a second portion of the plurality of data streams in response to the detecting the precursor to the event at the sensor device; and

wherein the detecting the event includes detecting the event based on the second portion of the plurality of data streams.

Assignments (2)

Continuity (1)

Cited By (41)

US 12,197,610 US 12,213,090 US 12,228,944 US 12,253,617 US 12,256,021 US 12,260,616 US 12,269,498 US 12,284,461 US 12,289,181 US 12,306,010 US 12,327,445 US 12,328,639 US 12,344,168 US 12,346,712 US 12,367,718 US 12,375,876 US 12,391,256 US 12,397,785 US 12,426,007 US 12,438,947 US 12,444,306 US 12,445,285 US 12,450,329 US 12,464,045 US 12,479,446 US 12,501,178 US 12,511,947 US 12,524,314 US 12,534,097 US 12,561,624 US 12,565,143 US 12,626,200 US 12,630,050 US 12,646,402 US 12,651,529 US 12,662,152 US 12,665,989 US 12,671,464 US 12,675,419 US 12,682,753 US 12,701,004