Granted Patent B2

US 12,187,276 · App. 17/908,918 · Granted Jan 7, 2025

Inventors: Dongchun Ren (Beijing, CN); Mingyu Fan (Beijing, CN); Huaxia Xia (Beijing, CN); Yanliang Zhu (Beijing, CN); Deheng Qian (Beijing, CN); Xin Li (Beijing, CN)

Assignee: Beijing Sankuai Online Technology Co., Ltd.

B60W30/0956B60W40/04B60W50/0097G06N3/049G06T7/246G06V20/58B60W2050/0028B60W2420/403B60W2554/4041B60W2554/4044B60W2556/10G06T2207/20081G06T2207/20084G06T2207/30241G06T2207/30261

Abstract

Claims (90)

1. An obstacle trajectory prediction method, implemented by a vehicle, comprising:

monitoring one or more obstacles around the vehicle;

obtaining, for each of the one or more obstacles, historical status information and current status information of the obstacle;

determining a current interaction feature under current joint action of the vehicle and the one or more obstacles according to historical status information and current status information of the vehicle and the historical status information and the current status information of each of the one or more obstacles;

obtaining a future motion trajectory of the vehicle planned by the vehicle, and determining a future motion trajectory feature of the vehicle according to the future motion trajectory of the vehicle;

determining a global interaction feature under the joint action of the vehicle and the one or more obstacles according to the determined current interaction feature and the future motion trajectory feature of the vehicle;

determining, for a to-be-predicted obstacle in the one or more obstacles, an individual interaction feature of the to-be-predicted obstacle under the joint action according to current status information of the to-be-predicted obstacle and the global interaction feature; and

inputting the individual interaction feature of the to-be-predicted obstacle and information about an environment around the vehicle into a pre-trained trajectory prediction model, to cause the trajectory prediction model to output a future motion trajectory of the to-be-predicted obstacle, wherein the future motion trajectory of the to-be-predicted obstacle is used by the vehicle to avoid the to-be-predicted obstacle.

2. The method according to claim 1 , wherein determining the current interaction feature under current joint action of the vehicle and the one or more obstacles according to the historical status information and current status information of the vehicle and the historical status information and the current status information of each of the one or more obstacles comprises:

determining position features of the vehicle and each of the one or more obstacles according to the current status information of the vehicle and the current status information of each of the one or more obstacles;

obtaining hidden variables respectively corresponding to the vehicle and each of the one or more obstacles according to the historical status information and the current status information of the vehicle and the historical status information and the current status information of each of the one or more obstacles, and determining tracking features of the vehicle and each of the one or more obstacles according to the hidden variables respectively corresponding to the vehicle and each of the one or more obstacles, wherein the hidden variables are used for representing a status difference of the vehicle or each of the one or more obstacles from a historical status to a current status;

determining a motion feature of the vehicle according to the historical status information and the current status information of the vehicle; and

determining the current interaction feature under the current joint action of the vehicle and the one or more obstacles according to the position features, the tracking features, and the motion feature of the vehicle.

3. The method according to claim 2 , wherein the pre-trained trajectory prediction model comprises a long short-term memory (LSTM) model comprising an encoder and a decoder; and

inputting the individual interaction feature of the to-be-predicted obstacle and information about the environment around the vehicle into the pre-trained trajectory prediction model, to cause the trajectory prediction model to output the future motion trajectory of the to-be-predicted obstacle comprises:

determining a status difference of the to-be-predicted obstacle from a historical status to a current status according to historical status information and current status information of the to-be-predicted obstacle;

inputting the individual interaction feature of the to-be-predicted obstacle, the information about the environment around the vehicle, and the status difference of the to-be-predicted obstacle into the encoder, to cause the encoder to output a hidden variable corresponding to the to-be-predicted obstacle; and

inputting the hidden variable corresponding to the to-be-predicted obstacle, the individual interaction feature of the to-be-predicted obstacle, the information about the environment around the vehicle, and the status difference of the to-be-predicted obstacle into the decoder, to cause the decoder to output the future motion trajectory of the to-be-predicted obstacle.

4. The method according to claim 3 , wherein obtaining the hidden variable corresponding to the vehicle according to the historical status information and the current status information of the vehicle comprises:

determining a status difference of the vehicle from the historical status to the current status according to the historical status information and the current status information of the vehicle; and

inputting an individual interaction feature of the vehicle, the information about the environment around the vehicle, and the status difference of the vehicle into the encoder, to cause the encoder to output the hidden variable corresponding to the vehicle, wherein

the individual interaction feature of the vehicle is obtained according to the current status information of the vehicle and the global interaction feature; and

obtaining the hidden variables respectively corresponding to each of the one or more obstacles according to the historical status information and the current status information of each of the one or more obstacles comprises:

for each of the one or more obstacles, determining, a status difference of the obstacle from the historical status to the current status according to the historical status information and the current status information of the obstacle; and

inputting an individual interaction feature of the obstacle, the information about the environment around the vehicle, and the status difference of the obstacle from the historical status to the current status into the encoder, to cause the encoder to output the hidden variable corresponding to the obstacle.

5. The method according to claim 1 , wherein determining the individual interaction feature of the to-be-predicted obstacle under the joint action according to the current status information of the to-be-predicted obstacle and the global interaction feature comprises:

determining a feature vector corresponding to the current status information of the to-be-predicted obstacle as a current status vector of the to-be-predicted obstacle;

determining a feature vector corresponding to the global interaction feature as a global interaction vector; and

determining the individual interaction feature of the to-be-predicted obstacle under the joint action according to a vector dot product of the current status vector of the to-be-predicted obstacle and the global interaction vector.

6. The method according to claim 1 , wherein inputting the information about the environment around the vehicle into the pre-trained trajectory prediction model comprises:

acquiring an actual image of a current environment around the vehicle;

determining a global environment feature according to the actual image;

determining, according to a position of the to-be-predicted obstacle in the actual image, a position of a local environment feature corresponding to the to-be-predicted obstacle in the global environment feature as a reference position;

determining an environment feature corresponding to the reference position in the global environment feature as the local environment feature corresponding to the to-be-predicted obstacle; and

inputting the determined local environment feature corresponding to the to-be-predicted obstacle into the pre-trained trajectory prediction model.

7. The method according to claim 6 , wherein determining the global environment feature according to the actual image comprises:

recognizing key elements comprised in the actual image;

determining positions of the key elements in the actual image;

generating an abstract image corresponding to the actual image according to the positions of the key elements in the actual image and predetermined models respectively matching the key elements; and

determining the global environment feature according to the abstract image.

8. A non-transitory computer-readable storage medium, storing a computer program, the computer program, when executed by a controller, causing the controller to implement operations comprising:

monitoring one or more obstacles around a vehicle;

obtaining, for each of the one or more obstacles, historical status information and current status information of the obstacle;

determining a current interaction feature under current joint action of the vehicle and the one or more obstacles according to historical status information and current status information of the vehicle and the historical status information and the current status information of each of the one or more obstacles;

obtaining a future motion trajectory of the vehicle planned by the vehicle, and determining a future motion trajectory feature of the vehicle according to the future motion trajectory of the vehicle;

determining a global interaction feature under the joint action of the vehicle and the one or more obstacles according to the determined current interaction feature and the future motion trajectory feature of the vehicle;

determining, for a to-be-predicted obstacle in the one or more obstacles, an individual interaction feature of the to-be-predicted obstacle under the joint action according to current status information of the to-be-predicted obstacle and the global interaction feature; and

inputting the individual interaction feature of the to-be-predicted obstacle and information about an environment around the vehicle into a pre-trained trajectory prediction model, to cause the trajectory prediction model to output a future motion trajectory of the to-be-predicted obstacle, wherein the future motion trajectory of the to-be-predicted obstacle is used by the vehicle to avoid the to-be-predicted obstacle.

9. An unmanned driving device, comprising:

a processor; and

a memory configured to store instructions executable by the processor, wherein the instructions, when executed, causes the processor to implement operations comprising:

monitoring one or more obstacles around a vehicle;

obtaining, for each of the one or more obstacles, historical status information and current status information of the obstacle;

determining a current interaction feature under current joint action of the vehicle and the one or more obstacles according to historical status information and current status information of the vehicle and the historical status information and the current status information of each of the one or more obstacles;

obtaining a future motion trajectory of the vehicle planned by the vehicle, and determining a future motion trajectory feature of the vehicle according to the future motion trajectory of the vehicle;

determining a global interaction feature under the joint action of the vehicle and the one or more obstacles according to the determined current interaction feature and the future motion trajectory feature of the vehicle;

determining, for a to-be-predicted obstacle in the one or more obstacles, an individual interaction feature of the to-be-predicted obstacle under the joint action according to current status information of the to-be-predicted obstacle and the global interaction feature; and

inputting the individual interaction feature of the to-be-predicted obstacle and information about an environment around the vehicle into a pre-trained trajectory prediction model, to cause the trajectory prediction model to output a future motion trajectory of the to-be-predicted obstacle, wherein the future motion trajectory of the to-be-predicted obstacle is used by the vehicle to avoid the to-be-predicted obstacle.

10. The device according to claim 9 , wherein determining the current interaction feature under current joint action of the vehicle and the one or more obstacles according to historical status information and current status information of the vehicle and the historical status information and the current status information of each of the one or more obstacles comprises:

determining position features of the vehicle and each of the one or more obstacles according to the current status information of the vehicle and the current status information of each of the one or more obstacles;

obtaining hidden variables respectively corresponding to the vehicle and each of the one or more obstacles according to the historical status information and the current status information of the vehicle and the historical status information and the current status information of each of the one or more obstacles, and determining tracking features of the vehicle and each of the one or more obstacles according to the hidden variables respectively corresponding to the vehicle and each of the one or more obstacles, wherein the hidden variables are used for representing a status difference of the vehicle or each of the one or more obstacles from a historical status to a current status;

determining a motion feature of the vehicle according to the historical status information and the current status information of the vehicle; and

determining the current interaction feature under the current joint action of the vehicle and the one or more obstacles according to the position features, the tracking features, and the motion feature of the vehicle.

11. The device according to claim 10 , wherein the pre-trained trajectory prediction model comprises a long short-term memory (LSTM) model comprising an encoder and a decoder; and

inputting the individual interaction feature of the to-be-predicted obstacle and information about the environment around the vehicle into the pre-trained trajectory prediction model, to cause the trajectory prediction model to output the future motion trajectory of the to-be-predicted obstacle comprises:

determining a status difference of the to-be-predicted obstacle from a historical status to a current status according to historical status information and current status information of the to-be-predicted obstacle;

inputting the individual interaction feature of the to-be-predicted obstacle, the information about the environment around the vehicle, and the status difference of the to-be-predicted obstacle into the encoder, to cause the encoder to output a hidden variable corresponding to the to-be-predicted obstacle; and

inputting the hidden variable corresponding to the to-be-predicted obstacle, the individual interaction feature of the to-be-predicted obstacle, the information about the environment around the vehicle, and the status difference of the to-be-predicted obstacle into the decoder, to cause the decoder to output the future motion trajectory of the to-be-predicted obstacle.

12. The device according to claim 11 , wherein obtaining the hidden variable corresponding to the vehicle according to the historical status information and the current status information of the vehicle comprises:

determining a status difference of the vehicle from the historical status to the current status according to the historical status information and the current status information of the vehicle; and

inputting an individual interaction feature of the vehicle, the information about the environment around the vehicle, and the status difference of the vehicle into the encoder, to cause the encoder to output the hidden variable corresponding to the vehicle, wherein

the individual interaction feature of the vehicle is obtained according to the current status information of the vehicle and the global interaction feature; and

obtaining the hidden variables respectively corresponding to each of the one or more obstacles according to the historical status information and the current status information of each of the one or more obstacles comprises:

for each of the one or more obstacles, determining, a status difference of the obstacle from the historical status to the current status according to the historical status information and the current status information of the obstacle; and

inputting an individual interaction feature of the obstacle, the information about the environment around the vehicle, and the status difference of the obstacle from the historical status to the current status into the encoder, to cause the encoder to output the hidden variable corresponding to the obstacle.

13. The device according to claim 9 , wherein determining the individual interaction feature of the to-be-predicted obstacle under the joint action according to current status information of the to-be-predicted obstacle and the global interaction feature comprises:

determining a feature vector corresponding to the current status information of the to-be-predicted obstacle as a current status vector of the to-be-predicted obstacle;

determining a feature vector corresponding to the global interaction feature as a global interaction vector; and

determining the individual interaction feature of the to-be-predicted obstacle under the joint action according to a vector dot product of the current status vector of the to-be-predicted obstacle and the global interaction vector.

14. The device according to claim 9 , wherein inputting information about the environment around the vehicle into the pre-trained trajectory prediction model comprises:

acquiring an actual image of a current environment around the vehicle;

determining a global environment feature according to the actual image;

determining, according to a position of the to-be-predicted obstacle in the actual image, a position of a local environment feature corresponding to the to-be-predicted obstacle in the global environment feature as a reference position;

determining an environment feature corresponding to the reference position in the global environment feature as the local environment feature corresponding to the to-be-predicted obstacle; and

inputting the determined local environment feature corresponding to the to-be-predicted obstacle into the pre-trained trajectory prediction model.

15. The device according to claim 14 , wherein determining the global environment feature according to the actual image comprises:

recognizing key elements comprised in the actual image;

determining positions of the key elements in the actual image;

generating an abstract image corresponding to the actual image according to the positions of the key elements in the actual image and predetermined models respectively matching the key elements; and

determining the global environment feature according to the abstract image.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (14)