Detection of loss-of-control vulnerable road users in automotive environments
The disclosed systems and techniques are directed to identifying and responding to presence of vulnerable road users (VRUs) in driving environments that are at risk of loss of control of their driving trajectories. The techniques include collecting, using a sensing system of an autonomous vehicle, sensing data for an environment of the autonomous vehicle and processing the sensing data by one or more machine learning models to identify a plurality of reference points associated with a VRU in the environment. The techniques further include identifying one or more height differentials for the plurality of reference points, determining that the VRU is at risk of loss of control, based at least on a change of the one or more height differentials, and causing a control system of the autonomous vehicle to perform an avoidance action.
1 . A system comprising:
a sensing system of an autonomous vehicle, the sensing system to collect sensing data for an environment of the autonomous vehicle; and
a perception system of the autonomous vehicle, the perception system to:
process the sensing data by one or more machine learning models to identify a plurality of reference points associated with a vulnerable road user (VRU) in the environment;
identify one or more height differentials for the plurality of reference points;
determine that the VRU is at risk of loss of control, based at least on a change of the one or more height differentials; and
cause a control system of the autonomous vehicle to perform an avoidance action.
2 . The system of claim 1 , wherein the plurality of reference points comprise at least one reference point associated with an upper body of the VRU and at least one reference point associated with a lower body of the VRU.
3 . The system of claim 2 , wherein to determine that the VRU is at risk of loss of control, the perception system is to detect that the change of the one or more height differentials satisfies a threshold condition.
4 . The system of claim 3 , wherein the threshold condition comprises at least one of:
the one or more height differentials experiencing at least a threshold change, or
the one or more height differentials experiencing at least a threshold rate of change.
5 . The system of claim 1 , wherein the VRU is determined to be at risk of loss of control in further view of an acceleration of the VRU.
6 . The system of claim 1 , wherein the one or more machine learning models are further to determine a prone state score for the VRU, wherein the prone state score characterizes a likelihood that the VRU is in a prone position or transitioning to the prone position, and wherein the VRU is determined to be at risk of loss of control in further view of the prone state score.
7 . The system of claim 6 , wherein the VRU is determined to be at risk of loss of control based at least on one of:
the change of the one or more height differentials satisfying a first threshold condition and the prone state score satisfying a second threshold condition;
a combination of the one or more height differentials and the prone state score satisfying a third threshold condition; or
an acceleration of the VRU satisfying a fourth threshold condition and at least one of:
the one or more height differentials satisfying a fifth threshold condition, or
the prone state score satisfying a sixth threshold condition.
8 . The system of claim 6 , wherein the one or more machine learning models comprise:
a backbone trained to process the sensing data and generate one or more intermediate features;
a first classifier trained to process the one or more intermediate features and output the plurality of reference points; and
a second classifier trained to process the one or more intermediate features and output the prone state score.
9 . The system of claim 1 , wherein the sensing data comprises camera data and at least one of lidar data or radar data, and wherein the sensing system comprises:
a camera to collect the camera data; and
at least one of a lidar sensor to collect the lidar data, or a radar sensor to collect the radar data; and
wherein to identify the plurality of reference points, the one or more machine learning models are trained to process:
a portion of the camera data associated with the VRU, and
at least a portion of the lidar data associated with the VRU or a portion of the radar data associated with the VRU.
10 . The system of claim 1 , wherein the VRU comprises at least one of:
a pedestrian,
a bicyclist,
a motorcyclist,
a scooter rider,
a skateboard rider, or
a wheelchair rider.
11 . A system comprising:
a sensing system of an autonomous vehicle, the sensing system to collect sensing data for an environment of the autonomous vehicle; and
a perception system of the autonomous vehicle, the perception system to:
process the sensing data by one or more machine learning models to:
identify a plurality of reference points associated with a vulnerable road user (VRU) in the environment;
determine a prone state score for the VRU, wherein the prone state score characterizes a likelihood that the VRU is in a prone position or transitioning to the prone position; and
identify one or more height differentials for the plurality of reference points;
determine that the VRU is at risk of loss of control, based on an acceleration of the VRU and at least one of:
a change of the one or more height differentials; or
the prone state score; and
cause a control system of the autonomous vehicle to perform an avoidance action.
12 . The system of claim 11 , wherein the VRU is determined to be at risk of loss of control based at least on one of:
the acceleration of the VRU satisfying a first threshold condition and the one or more height differentials satisfying a second threshold condition, or
a combination of the acceleration of the VRU and the one or more height differentials satisfying a third threshold condition.
13 . The system of claim 11 , wherein the VRU is determined to be at risk of loss of control based at least on one of:
the acceleration of the VRU satisfying a first threshold condition and the prone state score satisfying a second threshold condition, or
a combination of the acceleration of the VRU and the prone state score satisfying a third threshold condition.
14 . A method comprising:
collecting, using a sensing system of an autonomous vehicle, sensing data for an environment of the autonomous vehicle;
processing the sensing data by one or more machine learning models to identify a plurality of reference points associated with a vulnerable road user (VRU) in the environment;
identifying one or more height differentials for the plurality of reference points;
determining that the VRU is at risk of loss of control, based at least on a change of the one or more height differentials; and
causing a control system of the autonomous vehicle to perform an avoidance action.
15 . The method of claim 14 , wherein determining that the VRU is at risk of loss of control comprises:
detecting that the change of the one or more height differentials satisfies a threshold condition, wherein the threshold condition comprises at least one of:
the one or more height differentials experiencing at least a threshold change, or
the one or more height differentials experiencing at least a threshold rate of change.
16 . The method of claim 14 , wherein determining that the VRU is at risk of loss of control is further based on an acceleration of the VRU.
17 . The method of claim 14 , wherein the one or more machine learning models are further to determine a prone state score for the VRU, wherein the prone state score characterizes a likelihood that the VRU is in a prone position or transitioning to the prone position, and wherein determining that the VRU is at risk of loss of control is further based on the prone state score.
18 . The method of claim 17 , wherein the determining that the VRU is at risk of loss of control is based at least on one of:
the change of the one or more height differentials satisfying a first threshold condition and the prone state score satisfying a second threshold condition;
a combination of the one or more height differentials and the prone state score satisfying a third threshold condition; or
an acceleration of the VRU satisfying a fourth threshold condition and at least one of:
the one or more height differentials satisfying a fifth threshold condition, or
the prone state score satisfying a sixth threshold condition.
19 . The method of claim 17 , wherein processing the sensing data by one or more machine learning models comprises:
processing, using a backbone, the sensing data to generate one or more intermediate features;
processing, using a first classifier, the one or more intermediate features to output the plurality of reference points; and
processing, using a second classifier, the one or more intermediate features to output the prone state score.
20 . The method of claim 14 , wherein the sensing data comprises camera data and at least one of lidar data or radar data, and wherein the sensing system comprises:
a camera to collect the camera data; and
at least one of a lidar sensor to collect the lidar data, or a radar sensor to collect the radar data; and
wherein to identify the plurality of reference points, the one or more machine learning models are trained to process:
a portion of the camera data associated with the VRU, and
at least a portion of the lidar data associated with the VRU or a portion of the radar data associated with the VRU.