Inward/outward vehicle monitoring for remote reporting and in-cab warning enhancements
Systems and methods are provided for intelligent driving monitoring systems, advanced driver assistance systems and autonomous driving systems, and providing alerts to the driver of a vehicle, based on anomalies detected between driver behavior and environment captured by the outward facing camera. Various aspects of the driver, which may include his direction of sight, point of focus, posture, gaze, is determined by image processing of the upper visible body of the driver, by a driver facing camera in the vehicle. Other aspects of environment around the vehicle captured by the multitude of cameras in the vehicle are used to correlate driver behavior and actions with what is happening outside to detect and warn on anomalies, prevent accidents, provide feedback to the driver, and in general provide a safer driver experience.
1. A method comprising:
capturing, by at least one processor of a computing device with an outward facing camera, first visual data outside of a vehicle;
determining, by the at least one processor based on the first visual data, a potentially unsafe driving condition outside of the vehicle;
capturing, by the at least one processor with a driver facing camera, second visual data of a driver of the vehicle;
determining, by the at least one processor based on the second visual data, whether the driver is looking toward the potentially unsafe driving condition or away from the potentially unsafe driving condition;
activating, by the at least one processor, an in-vehicle alert in response to determining the potentially unsafe driving condition, that the driver is looking away from the potentially unsafe driving condition, and that an amount of time in which the vehicle will encounter the potentially unsafe driving condition satisfies a first threshold of time; and
transmitting, by the at least one processor to a remote server located remotely from the vehicle, a remote alert in response to the determination that the driver is not looking toward the potentially unsafe driving condition and after the in-vehicle alert is activated.
2. The method of claim 1 , wherein determining whether the driver is looking toward the potentially unsafe driving condition comprises monitoring, by the at least one processor based on the second visual data, at least one of a pose of the driver or a gaze of the driver.
3. The method of claim 2 , wherein monitoring at least one of the pose of the driver or the gaze of the driver comprises:
locating, by the at least one processor, a plurality of keypoints associated with the driver in the second visual data; and
determining, by the at least one processor, a distance between a first keypoint and a second keypoint of the plurality of keypoints at a first time before the determination of the potentially unsafe driving condition.
4. The method of claim 3 , wherein the first keypoint corresponds to a first location in the visual data corresponding to a first shoulder of the driver and the second keypoint corresponds to a second location in the second visual data corresponding to a second shoulder of the driver.
5. The method of claim 3 , wherein determining the distance between the first keypoint and the second keypoint comprises:
determining, over a predetermined amount of time, a plurality of distances between the first keypoint and the second keypoint; and
determining a median of the plurality of determined distances.
6. The method of claim 3 , wherein the monitoring at least one of the pose of the driver or the gaze of the driver further comprises:
determining, by the at least one processor, a second distance between the first keypoint and the second keypoint at a second time; and
determining, by the at least one processor, a scaling factor based on the second distance and the distance.
7. The method of claim 6 , wherein the monitoring at least one of the pose of the driver or the gaze of the driver further comprises:
determining, by the at least one processor, a third distance between a third keypoint and a fourth keypoint; and
determining, by the at least one processor, a deviation from a driver pose based at least in part on the third distance and the scaling factor.
8. The method of claim 7 , wherein the third keypoint corresponds to a third location in the second visual data corresponding to a first eye of the driver and the fourth keypoint corresponds to a fourth location in the second visual data corresponding to a second eye of the driver.
9. The method of claim 8 , further comprising:
determining, by the at least one processor, a distance between the third keypoint and the fourth keypoint; and
determining the gaze of the driver based on the third distance, the distance between the third keypoint and the fourth keypoint, and the scaling factor.
10. The method of claim 9 , wherein the gaze is further determined based on a determined angle or distance between an ear of the driver and a nose of the driver.
11. The method of claim 1 , further comprising:
receiving, by the at least one processor from a remote driver monitoring device, a request for live video captured by at least one of the outward facing camera or the driver facing camera, wherein the request is received after transmission of the remote alert; and
transmitting, by the at least one processor in response to the request, the live video captured by at least one of the outward facing camera or the driver facing camera.
12. The method of claim 1 , wherein the in-vehicle alert is an audio alert played through a speaker in the vehicle, a visual alert on a display in the vehicle, or a light in the vehicle.
13. The method of claim 1 , further comprising transmitting, by the at least one processor to the remote server, the first visual data, the second visual data, and information about the in-vehicle alert and whether the driver is looking toward the potentially unsafe driving condition.
14. The method of claim 1 , wherein the amount of time in which the vehicle will encounter the potentially unsafe driving condition is determined based on at least one of a speed of the vehicle, a distance from the vehicle to an object associated with the potentially unsafe driving condition, or a speed of the object associated with the potentially unsafe driving condition.
15. The method of claim 14 , wherein the object associated with the potentially unsafe driving condition is one of a traffic light, a stop sign, an intersection, a railroad crossing, a lane or road boundary, or a second vehicle.
16. The method of claim 1 , wherein the determination of whether the driver is looking toward the potentially unsafe driving condition or away from the potentially unsafe driving condition comprises monitoring, by the at least one processor based on the second visual data, at least one of a pose of the driver or a gaze of the driver.
17. A method comprising:
capturing, by at least one processor of a computing device with an outward facing camera, first visual data outside of a vehicle;
determining, by the at least one processor based on the first visual data, a potentially unsafe driving condition outside of the vehicle and an amount of time in which the vehicle will encounter the potentially unsafe driving condition;
capturing, by the at least one processor with a driver facing camera, second visual data of a driver of the vehicle;
determining, by the at least one processor based on the second visual data, whether the driver has looked in a direction of the potentially unsafe driving condition within a predetermined threshold of time of the determination of unsafe driving condition, wherein:
an in-vehicle alert is suppressed when the driver has looked in the direction of the potentially unsafe driving condition within the predetermined threshold of time and
an in-vehicle alert is activated when the driver has not looked in the direction of the potentially unsafe driving condition within the predetermined threshold of time; and
transmitting, by the at least one processor to a remote server located remotely from the vehicle, the first visual data, the second visual data, information about the in-vehicle alert suppression or activation, and information about whether the driver looked in the direction of the potentially unsafe driving condition.
18. The method of claim 17 , wherein the in-vehicle alert is an audio alert played through a speaker in the vehicle or a visual alert on a display or a light in the vehicle.
19. The method of claim 17 , wherein the amount of time in which the vehicle will encounter the potentially unsafe driving condition is determined based on at least one of a lateral velocity of the vehicle, a speed of the vehicle, a distance from the vehicle to an object associated with the potentially unsafe driving condition, or a speed of the object associated with the potentially unsafe driving condition.
20. The method of claim 19 , wherein the object associated with the potentially unsafe driving condition is a lane or road boundary or a second vehicle.
21. The method of claim 17 , wherein the determination of whether the driver has looked in the direction of the potentially unsafe driving condition comprises monitoring, by the at least one processor based on the second visual data, at least one of a pose of the driver or a gaze of the driver.