Methods and devices for triggering vehicular actions based on passenger actions
Autonomous driving system methods and devices which trigger vehicular actions based on the monitoring of one or more occupants of a vehicle are presented. The methods, and corresponding devices, may include identifying a plurality of features in a plurality of subsets of image data detailing the one or more occupants; tracking changes over time of the plurality of features over the plurality of subsets of image data; determining a state, from a plurality of states, of the one or more occupants based on the tracked changes; and triggering the vehicular action based on the determined state.
1. A system for triggering a vehicular action based on monitoring of one or more occupants of a vehicle, the system comprising:
at least one processor comprising circuitry and a memory, wherein the memory includes instructions that when executed by the circuitry cause the at least one processor to:
receive, from an image acquisition device, a plurality of subsets of image data detailing the one or more occupants;
identify a plurality of features of the one or more occupants in the plurality of subsets of image data;
identify, based on the plurality of subsets of image data, at least one change in the plurality of features over time, wherein identifying the at least one change includes inputting at least some of the plurality of subsets of image data into a spatio-temporal model;
determine a state of the one or more occupants from a plurality of states based on the at least one change, wherein the spatio-temporal model includes an output layer providing a score for each of the plurality of states, and wherein the state that is associated with the one or more occupants is based on the score;
determine, based on the determined state of the one or more occupants and an assumed minimum braking rate of the vehicle, a theoretical safe distance between the vehicle and at least one object; and
trigger the vehicular action based on the determined theoretical safe distance.
2. The system of claim 1 , wherein each of the plurality of states is associated with a corresponding time value and wherein the theoretical safe distance is determined based on a time value associated with the determined state of the one or more occupants.
3. The system of claim 2 , wherein the time value represents an estimated reaction time of the driver.
4. The system of claim 2 , wherein the time value is determined using a look-up table correlating the plurality of states to the corresponding time values.
5. The system of claim 1 , wherein at least one of the plurality of states corresponds to a classification of an action being performed by the one or more occupants while driving.
6. The system of claim 1 , wherein determining the state of the one or more occupants includes selecting a state from the plurality of states having a highest score.
7. The system of claim 6 , wherein triggering the vehicular action based on the determined theoretical safe distance includes comparing the theoretical safe distance to the real physical distance.
8. The system of claim 1 , wherein the memory incudes instructions that when executed by the circuitry further cause the at least one processor to determine a real physical distance to the at least one object.
9. The system of claim 8 , wherein triggering the vehicular action includes causing a braking response of the vehicle until the physical distance exceeds the theoretical safe distance.
10. The system of claim 1 , wherein the theoretical safe distance between the vehicle and at least one object is further determined based on an assumed maximum acceleration value of the vehicle.
11. The system of claim 1 , wherein triggering the vehicular action includes causing a notification to be presented to the at least one occupant.
12. The system of claim 11 , wherein the notification is presented via a user interface of the vehicle.
13. The system of claim 1 , wherein the vehicular action includes a braking response of the vehicle.
14. The system of claim 1 , wherein identifying the plurality of features includes applying a Feature-Level Temporal Filtering based model.
15. The system of claim 1 , wherein identifying the plurality of features includes inputting the plurality of subsets of image data into a neural network.
16. A method for triggering a vehicular action based on monitoring of one or more occupants of a vehicle, the method comprising:
receiving, from an image acquisition device, a plurality of subsets of image data detailing the one or more occupants;
identifying a plurality of features of the one or more occupants in the plurality of subsets of image data;
identifying, based on the plurality of subsets of image data, at least one change in the plurality of features over time, wherein identifying the at least one change includes inputting at least some of the plurality of subsets of image data into a spatio-temporal model;
determining a state of the one or more occupants from a plurality of states based on the at least one change, wherein the spatio-temporal model includes an output layer providing a score for each of the plurality of states, and wherein the state that is associated with the one or more occupants is based on the score;
determining, based on the determined state of the one or more occupants and an assumed minimum braking rate of the vehicle, a theoretical safe distance between the vehicle and at least one object; and
triggering the vehicular action based on the determined theoretical safe distance.
17. The method of claim 16 , wherein each of the plurality of states is associated with a corresponding time value and wherein the theoretical safe distance is determined based on a time value associated with the determined state of the one or more occupants.
18. The method of claim 17 , wherein the time value represents an estimated reaction time of the driver.
19. The method of claim 16 , wherein the method further includes determining a real physical distance to the at least one object and wherein triggering the vehicular action based on the determined theoretical safe distance includes comparing the theoretical safe distance to the real physical distance.
20. A non-transitory computer readable medium containing instructions that when executed by at least one processor, cause the at least one processor to perform a method for triggering a vehicular action based on monitoring of one or more occupants of a vehicle, the method comprising:
receiving, from an image acquisition device, a plurality of subsets of image data detailing the one or more occupants;
identifying a plurality of features of the one or more occupants in the plurality of subsets of image data;
identifying, based on the plurality of subsets of image data, at least one change in the plurality of features over time, wherein identifying the at least one change includes inputting at least some of the plurality of subsets of image data into a spatio-temporal model;
determining a state of the one or more occupants from a plurality of states based on the at least one change, wherein the spatio-temporal model includes an output layer providing a score for each of the plurality of states, and wherein the state that is associated with the one or more occupants is based on the score;
determining, based on the determined state of the one or more occupants and an assumed minimum braking rate of the vehicle, a theoretical safe distance between the vehicle and at least one object; and
triggering the vehicular action based on the determined theoretical safe distance.
21. The non-transitory computer readable medium of claim 20 , wherein at least one of the plurality of states corresponds to a classification of an action being performed by the one or more occupants while driving.
22. The non-transitory computer readable medium of claim 20 , wherein the theoretical safe distance between the vehicle and at least one object is further determined based on an assumed maximum acceleration value of the vehicle.
23. The non-transitory computer readable medium of claim 20 , wherein triggering the vehicular action includes causing a notification to be presented to the at least one occupant.