Appearance and movement based model for determining risk of micro mobility users
The systems and methods disclosed herein provide a risk prediction system that uses trained machine learning models to make predictions that a VRU will take a particular action. The system first receives, in a video stream, an image depicting a VRU operating a micro-mobility vehicle and extract the depictions from the image. The extraction process may be determined by bounding box classifiers trained to identify various VRUs and micro-mobility vehicles. The system feeds the extracted depictions to machine learning models and receives, as an output, risk profiles for the VRU and the micro-mobility vehicle. The risk profile may include data associated with the VRU/micro-mobility vehicle determined based on classifications of the VRU and the micro-mobility vehicles. The system may then generate a prediction that the VRU operating the micro-mobility vehicle will take a particular action based on the risk profile.
1. A method comprising:
receiving, in a video stream, an image depicting a human operating a micro-mobility vehicle;
extracting one or more depictions associated with the human and the micro-mobility vehicle from the image;
inputting the one or more depictions into one or more machine learning models;
receiving as output, from the one or more machine learning models, a set of classifications representative of at least an appearance of the human, the appearance of the human predicted from a plurality of candidate appearances of the human, each of the plurality of candidate appearances of the human representative of a different object worn by the human;
generating a risk profile based on the set of classifications; and
generating a prediction that the human will take a particular action while operating the micro-mobility vehicle based on the risk profile.
2. The method of claim 1 , wherein the prediction comprises a confidence score corresponding to a likelihood that a particular risk is posed by the human.
3. The method of claim 1 , wherein the risk profile is determined by:
classifying a type of the micro-mobility vehicle; and
determining, based on the type of micro-mobility vehicle, one or more vehicle control parameters associated with the micro-mobility vehicle.
4. The method of claim 3 , wherein the one or more vehicle control parameters include one or more of a range of movement, speed capabilities, braking capabilities, and acceleration capabilities.
5. The method of claim 1 , wherein the prediction that the human will take the particular action is predicted by a machine learned model based on the risk profile.
6. The method of claim 1 , further comprising:
determining, based on the risk profile, a set of instructions for transmission to an autonomous, semi-autonomous vehicle, a vehicle with advanced driver-assistance systems (ADAS), or an intelligent infrastructure system.
7. The method of claim 1 , wherein determining the risk profile comprises:
determining a movement associated with the human operating the micro-mobility vehicle.
8. The method of claim 1 , wherein extracting the one or more depictions associated with the human and the micro-mobility vehicle from the image is based on one or more bounding polygon classifiers.
9. The method of claim 1 , wherein the prediction is further based on contextual information including a time of a day, or location, wherein the contextual information is extracted from the image.
10. The method of claim 1 , wherein the output further comprises a first set of distributions representative of the set of classifications, and wherein generating the risk profile is further based on the first set of distributions.
11. A non-transitory computer-readable medium comprising memory with instructions encoded thereon, the instructions causing one or more processors to perform operations when executed, the instructions comprising instructions to:
receive, in a video stream, an image depicting a human operating a micro-mobility vehicle;
extract one or more depictions associated with the human and the micro-mobility vehicle from the image;
input the one or more depictions into one or more machine learning models;
receive as output, from the one or more machine learning models, a set of classifications representative of at least an appearance of the human, the appearance of the human predicted from a plurality of candidate appearances of the human, each of the plurality of candidate appearances of the human representative of a different object worn by the human;
generate a risk profile based on the set of classifications; and
generate a prediction that the human will take a particular action while operating the micro-mobility vehicle based on the risk profile.
12. The non-transitory computer-readable medium of claim 11 , wherein the prediction comprises a confidence score corresponding to a likelihood that a particular risk is posed by the human.
13. The non-transitory computer-readable medium of claim 11 , wherein the risk profile is determined by:
classifying a type of the micro-mobility vehicle; and
determining, based on the type of micro-mobility vehicle, one or more vehicle control parameters associated with the micro-mobility vehicle.
14. The non-transitory computer-readable medium of claim 13 , wherein the one or more vehicle control parameters include one or more of a range of movement, speed capabilities, braking capabilities, and acceleration capabilities.
15. The non-transitory computer-readable medium of claim 11 , wherein the prediction that the human will take the particular action is predicted by a machine learned model based on the risk profile.
16. The non-transitory computer-readable medium of claim 11 , the instructions further comprising instructions to:
determining, based on the risk profile, a set of instructions for transmission to an autonomous, semi-autonomous vehicle, a vehicle with advanced driver-assistance systems (ADAS), or an intelligent infrastructure system.
17. The non-transitory computer-readable medium of claim 11 , wherein the instructions to determine the risk profile comprise instructions to:
determine a movement associated with the human operating the micro-mobility vehicle.
18. The non-transitory computer-readable medium of claim 11 , wherein extracting the one or more depictions associated with the human and the micro-mobility vehicle from the image is based on one or more bounding polygon classifiers.
19. The non-transitory computer-readable medium of claim 11 , wherein the prediction is further based on contextual information including a time of a day, or location, wherein the contextual information is extracted from the image.
20. The non-transitory computer-readable medium of claim 11 , wherein the output further comprises a first set of distributions representative of the set of classifications, and wherein generating the risk profile is further based on the first set of distributions.