Systems and methods for asynchronously processing image data
Systems and methods for asynchronously processing image data are provided. The systems and methods involve: at least one camera operable to capture images; and at least one processor operable to: apply at least one first machine learning model to a first subset of the images to detect an event and determine a first predicted probability in substantially real-time; determine that the first predicted probability satisfies a first confidence criterion; in response, apply at least one second machine learning model to a second subset of the images to determine a second predicted probability asynchronously and with a longer cumulative execution time; determine that the second predicted probability of the event satisfies a second confidence criterion; and in response, transmit an indication of the event, whereby a user remotely located from the imaging device can be notified of the event.
1 . An onboard imaging device for monitoring a vehicle comprising:
at least one camera operable to capture images depicting an area in front of the vehicle and/or an interior of the vehicle; and
at least one processor comprising at least one primary processor and at least one coprocessor, the at least one processor operable to:
apply, by the at least one coprocessor, at least one first machine learning model to a first subset of the images to detect an unsafe driving event and determine a first predicted probability of the unsafe driving event, the at least one first machine learning model being executed in substantially real-time with respect to the first subset of the images being captured;
determine whether the first predicted probability of the unsafe driving event satisfies a first confidence criterion;
in response to determining that the first predicted probability of the unsafe driving event satisfies a first confidence criterion:
apply by the at least one primary processor, at least one second machine learning model to a second subset of the images to determine a second predicted probability of the unsafe driving event, the at least one second machine learning model being executed asynchronously with respect to the at least one first machine learning model, the application of the at least one of second machine learning model having a longer cumulative execution time than the application of the at least one first machine learning model, and the second subset of the images comprising more image data than the first subset of the images;
determine whether the second predicted probability of the unsafe driving event satisfies a second confidence criterion; and
in response to determining that the second predicted probability of the unsafe driving event satisfies the second confidence criterion, transmit an indication of the unsafe driving event, whereby a user remotely located from the imaging device can be notified of the unsafe driving event.
2 . The imaging device of claim 1 , wherein the first subset of the images has a lower frame rate than the second subset of the images.
3 . The imaging device of claim 1 , wherein the at least one coprocessor comprises at least one graphical processing unit (GPU) and/or at least one digital signal processor (DSP) and the at least one primary processor comprises at least one central processing unit (CPU).
4 . The imaging device of claim 1 , wherein:
applying the at least one second machine learning model to the second subset of the images comprises applying the at least one second machine learning model to the second subset of the images to detect at least one other unsafe driving event that was not detected by the at least one first machine learning model; and
an indication of the at least one other unsafe driving event is also transmitted with the indication of the unsafe driving event.
5 . The imaging device of claim 1 , wherein:
applying the at least one second machine learning model to the second subset of the images comprises applying at least one second machine learning model to the second subset of the images to identify contextual data associated with the unsafe driving event; and
the contextual data is also transmitted with the indication of the first unsafe driving event.
6 . The imaging device of claim 1 , wherein the at least one processor is further operable to:
determine whether the first predicted probability of the unsafe driving event satisfies a third confidence criterion; and
in response to determining that the first predicted probability of the unsafe driving event satisfies the third confidence criterion, generate an audio alert at the imaging device.
7 . The imaging device of claim 1 , wherein the second confidence criterion is stricter than the first confidence criterion.
8 . The imaging device of claim 1 , further comprising:
at least one accelerometer operable to generate acceleration data; and
at least one GPS receiver operable to generate location data; wherein the at least one processor is further operable to:
in response to determining that the predicted probability of the unsafe driving event does not satisfy the first confidence criterion, determine whether at least one of the acceleration data or the location data satisfies a fourth confidence criterion; and
in response to determining that the at least one of the acceleration data or the location data satisfies a fourth confidence criterion, apply the at least one second machine learning model to the second subset of the images to determine the second predicted probability of the event.
9 . The imaging device of claim 1 , wherein the unsafe driving event comprises:
the vehicle tailgating another vehicle,
the vehicle nearly colliding or colliding with another vehicle and/or other object,
the vehicle straddling two lanes,
the vehicle performing a rolling stop,
the vehicle crossing a solid lane,
a driver of the vehicle drinking and/or eating,
the driver not wearing a seatbelt,
the driver using a phone,
the driver smoking,
the driver not viewing the road, and/or
the driver yawning.
10 . The imaging device of claim 1 , wherein the at least one first machine model and the at least one second machine learning model comprise at least one common model.
11 . A method for asynchronously processing image data, the method comprising operating at least one processor, comprising at least one primary processor and at least one coprocessor, to:
capture, using at least one camera, images of an area in front of a vehicle and/or an interior of the vehicle;
apply, by the at least one coprocessor, at least one first machine learning model to a first subset of the images to detect an unsafe driving event and determine a first predicted probability of the unsafe driving event, the at least one first machine learning model being executed in substantially real-time with respect to the first subset of the images being captured;
determine whether the first predicted probability of the unsafe driving event satisfies a first confidence criterion;
in response to determining that the first predicted probability of the unsafe driving event satisfies a first confidence criterion:
apply, by the at least one primary processor, at least one second machine learning model to a second subset of the images to determine a second predicted probability of the unsafe driving event, the at least one second machine learning model being executed asynchronously with respect to the at least one first machine learning model, the application of the at least one of second machine learning model having a longer cumulative execution time than the application of the at least one first machine learning model, and the second subset of the images comprising more image data than the first subset of the images;
determine whether the second predicted probability of the unsafe driving event satisfies a second confidence criterion; and
in response to determining that the second predicted probability of the unsafe driving event satisfies the second confidence criterion, transmit an indication of the unsafe driving event, whereby a user remotely located from the imaging device can be notified of the unsafe driving event.
12 . The method of claim 11 , wherein the first subset of the images has a lower frame rate than the second subset of the images.
13 . The method of claim 11 , wherein the at least one coprocessor comprises at least one graphical processing unit (GPU) and/or at least one digital signal processor (DSP) and the at least one primary processor comprises at least one central processing unit (CPU).
14 . The method of claim 11 , wherein:
applying the at least one second machine learning model to the second subset of the images comprises applying the at least one second machine learning model to the second subset of the images to detect at least one other unsafe driving event that was not detected by the at least one first machine learning model; and
an indication of the at least one other unsafe driving event is also transmitted with the indication of the unsafe driving event.
15 . The method of claim 11 , wherein:
applying the at least one second machine learning model to the second subset of the images comprises applying at least one second machine learning model to the second subset of the images to identify contextual data associated with the unsafe driving event; and
the contextual data is also transmitted with the indication of the first unsafe driving event.
16 . The method of claim 11 , further comprising operating the at least one processor to:
determine whether the first predicted probability of the unsafe driving event satisfies a third confidence criterion; and
in response to determining that the first predicted probability of the unsafe driving event satisfies the third confidence criterion, generate an audio alert at the imaging device.
17 . The method of claim 11 , wherein the second confidence criterion is stricter than the first confidence criterion.
18 . The method of claim 11 , further comprising operating the at least one processor to:
in response to determining that the predicted probability of the unsafe driving event does not satisfy the first confidence criterion, determine whether at least one of acceleration data or location data satisfies a fourth confidence criterion;
in response to determining that the at least one of the acceleration data or the location data satisfies a fourth confidence criterion, apply the at least one second machine learning model to the second subset of the images to determine the second predicted probability of the event.
19 . The method of claim 11 , wherein the unsafe driving event comprises:
the vehicle tailgating another vehicle,
the vehicle nearly colliding or colliding with another vehicle and/or other object,
the vehicle straddling two lanes,
the vehicle performing a rolling stop,
the vehicle crossing a solid lane,
a driver of the vehicle drinking and/or eating,
the driver not wearing a seatbelt,
the driver using a phone,
the driver smoking,
the driver not viewing the road, and/or
the driver yawning.
20 . The method of claim 11 , wherein the at least one first machine model and the at least one second machine learning model comprise at least one common model.
21 . A non-transitory computer readable medium having instructions stored thereon executable by at least one processor to implement a method for asynchronously processing image data, the at least one processor comprising at least one primary processor and at least one coprocessor, the method comprising operating at least one processor to:
capture, using at least one camera, images of an area in front of a vehicle and/or an interior of the vehicle;
apply, by the at least one coprocessor, at least one first machine learning model to a first subset of the images to detect an unsafe driving event and determine a first predicted probability of the unsafe driving event, the at least one first machine learning model being executed in substantially real-time with respect to the first subset of the images being captured;
determine whether the first predicted probability of the unsafe driving event satisfies a first confidence criterion;
in response to determining that the first predicted probability of the unsafe driving event satisfies a first confidence criterion:
apply, by the at least one primary processor, at least one second machine learning model to a second subset of the images to determine a second predicted probability of the unsafe driving event, the at least one second machine learning model being executed asynchronously with respect to the at least one first machine learning model, the application of the at least one of second machine learning model having a longer cumulative execution time than the application of the at least one first machine learning model, and the second subset of the images comprising more image data than the first subset of the images;
determine whether the second predicted probability of the unsafe driving event satisfies a second confidence criterion; and
in response to determining that the second predicted probability of the unsafe driving event satisfies the second confidence criterion, transmit the second subset of the images with an indication of the unsafe driving event, whereby a user remotely located from the imaging device can be notified of the unsafe driving event.