IP Library Granted Patent US 12,240,501
Granted Patent B2
US 12,240,501 · App. 16/910,395 · Granted Mar 4, 2025

Convolution operator selection

Inventors: Stephen G. McGill (Broomall, PA); Paul Drews (Watertown, MA); Guy Rosman (Newton, MA)
Assignee: Toyota Research Institute, Inc.
B60W60/00272B60W60/00274G06N3/08B60W2552/10B60W2554/4046B60W2554/4049
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,240,501
App. No.
16/910,395
Granted
Mar 4, 2025
Kind
B2
Abstract

A vehicle system includes one or more sensors configured to capture aspects of an environment and a computing device. The computing device is configured to receive information about the environment captured by the one or more sensors, determine one or more structures within the environment based on the received information, select a kernel that is parameterized for predicting a vehicle trajectory based on the one or more structures determined within the environment, and perform a convolution of the selected kernel and an array defining the environment, wherein the convolution predicts a future trajectory of a vehicle within the environment.

Claims (43)

1. A vehicle system comprising:

one or more sensors configured to capture aspects of an environment; and

a computing device configured to:

receive information about the environment captured by the one or more sensors,

determine one or more structures within the environment based on the received information,

select a kernel from a plurality of kernels that correspond to various environments, the selected kernel is parameterized for predicting a vehicle trajectory based on the one or more structures determined within the environment,

perform a convolution of the selected kernel and an array defining the environment, wherein the convolution predicts a future trajectory of a vehicle within the environment, wherein the future trajectory of the vehicle defines one or more predicted future locations of the vehicle in the environment,

obtain a sensed vehicle trajectory of the vehicle,

compare the sensed vehicle trajectory of the vehicle with the predicted future trajectory of the vehicle, and

select, in response to the comparison indicating an inaccurate predicted future trajectory of the vehicle, a second kernel that is parameterized for predicting a vehicle trajectory, wherein the second kernel is based on the one or more structures determined within the environment and the comparison of the sensed vehicle trajectory of the vehicle with the predicted future trajectory of the vehicle.

2. The vehicle system of claim 1 , wherein the computing device is further configured to generate one or more kernels comprising parameters that are learned based on the one or more structures of the environment and vehicle behaviors corresponding to the environment.

3. The vehicle system of claim 2 , wherein the vehicle behaviors comprise at least one of expected vehicle trajectories within the environment, a vehicle speed, or a rule of a road that applies to the environment.

4. The vehicle system of claim 1 , wherein the computing device is further configured to define a plurality of discretized locations within the environment and generate an occupancy map based on the predicted future trajectory of the vehicle, wherein the occupancy map is defined by the plurality of discretized locations within the environment defining a probability distribution of a likelihood that the vehicle will be located at the plurality of discretized locations within the environment.

5. The vehicle system of claim 4 , wherein the computing device is further configured to provide the probability distribution to an advanced driver assistance system for determining an action to take in response to the probability distribution.

6. The vehicle system of claim 1 , wherein the one or more structures comprises identification of at least one of a width of a road, a number of lanes of the road, a weather condition, a time of day, a speed limit, a traffic density, or an urban or rural environment.

7. The vehicle system of claim 1 , wherein the one or more structures comprises the vehicle.

8. The vehicle system of claim 1 , wherein the kernel is based on a heat equation.

9. A computing device configured to:

receive information about an environment;

determine one or more structures within the environment based on the received information;

select a kernel from a plurality of kernels that correspond to various environments, the selected kernel is parameterized for predicting a vehicle trajectory based on the one or more structures determined within the environment;

perform a convolution of the selected kernel and an array defining the environment, wherein the convolution predicts a future trajectory of a vehicle within the environment, wherein the future trajectory of the vehicle defines one or more predicted future locations of the vehicle in the environment;

obtain a sensed vehicle trajectory of the vehicle;

compare the sensed vehicle trajectory of the vehicle with the predicted future trajectory of the vehicle; and

select, in response to the comparison indicating an inaccurate predicted future trajectory of the vehicle, a second kernel that is parameterized for predicting a vehicle trajectory, wherein the second kernel is based on the one or more structures determined within the environment and the comparison of the sensed vehicle trajectory of the vehicle with the predicted future trajectory of the vehicle.

10. The computing device of claim 9 , wherein the computing device is further configured to generate one or more kernels comprising parameters that are learned based on the one or more structures of the environment and vehicle behaviors corresponding to the environment.

11. The computing device of claim 10 , wherein the vehicle behaviors comprise at least one of expected vehicle trajectories within the environment, a vehicle speed, or a rule of a road that applies to the environment.

12. The computing device of claim 9 , wherein the computing device is further configured to define a plurality of discretized locations within the environment and generate an occupancy map based on the predicted future trajectory of the vehicle, wherein the occupancy map is defined by the plurality of discretized locations within the environment defining a probability distribution of a likelihood that the vehicle will be located at the plurality of discretized locations within the environment.

13. The computing device of claim 12 , wherein the computing device is further configured to provide the probability distribution to an advanced driver assistance system for determining an action to take in response to the probability distribution.

14. The computing device of claim 9 , wherein the one or more structures comprises identification of at least one of a width of a road, a number of lanes of the road, a weather condition, a time of day, a speed limit, a traffic density, or an urban or rural environment.

15. A method for predicting future events, the method comprising:

receiving information about an environment;

determining one or more structures within the environment based on the received information;

selecting a kernel from a plurality of kernels that correspond to various environments, the selected kernel is parameterized for predicting a vehicle trajectory based on the one or more structures determined within the environment;

performing a convolution of the selected kernel and an array defining the environment, wherein the convolution predicts a future trajectory of a vehicle within the environment, wherein the future trajectory of the vehicle defines one or more predicted future locations of the vehicle in the environment;

obtaining a sensed vehicle trajectory of the vehicle;

comparing the sensed vehicle trajectory of the vehicle with the predicted future trajectory of the vehicle; and

selecting, in response to the comparison indicating an inaccurate predicted future trajectory of the vehicle, a second kernel that is parameterized for predicting a vehicle trajectory, wherein the second kernel is based on the one or more structures determined within the environment and the comparison of the sensed vehicle trajectory of the vehicle with the predicted future trajectory of the vehicle.

16. The method of claim 15 , further comprising:

defining a plurality of discretized locations within the environment; and

generating an occupancy map based on the predicted future trajectory of the vehicle, wherein the occupancy map is defined by the plurality of discretized locations within the environment defining a probability distribution of a likelihood that the vehicle will be located at the plurality of discretized locations within the environment.

17. The method of claim 16 , further comprising:

providing the probability distribution to an advanced driver assistance system for determining an action to take in response to the probability distribution.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 5, 2025
From: TOYOTA RESEARCH INSTITUTE, INC.
To: TOYOTA JIDOSHA KABUSHIKI KAISHA
Reel/Frame 071024/0731 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 24, 2020
From: MCGILL, STEPHEN G.; DREWS, PAUL; ROSMAN, GUY
To: TOYOTA RESEARCH INSTITUTE, INC.
Reel/Frame 053025/0730 →
Continuity (1)
Related Publication 20210403056A1 · Dec 30, 2021
References Cited (22)
US 9472019B2 · Jenkins · 2016 [cited by applicant]
US 9760806B1 · Ning et al. · 2017 [cited by applicant]
US 20080126274A1 · Jannarone · 2008 [cited by examiner]
US 20110246156A1 · Zecha et al. · 2011 [cited by applicant]
US 20180012085A1 · Blayvas et al. · 2018 [cited by applicant]
US 20180284785A1 · Berntorp · 2018 [cited by examiner]
US 20190147335A1 · Wang · 2019 [cited by examiner]
US 20190294164A1 · Wagenmaker · 2019 [cited by examiner]
US 20190346844A1 · Chen et al. · 2019 [cited by applicant]
US 20200202144A1 · Porter · 2020 [cited by examiner]
US 20200225321A1 · Kruglick · 2020 [cited by examiner]
US 20200239029A1 · Kim · 2020 [cited by examiner]
US 20210101606A1 · Zhang · 2021 [cited by examiner]
US 20220136853A1 · Ferencz · 2022 [cited by examiner]
EP 3219564A1 · 2017 [cited by applicant]
Ganesh, Prakhar, “Types of Convolution Kernels: Simplified,” https://web.archive.org/web/20191019064344/https://towardsdatascience.com/types-of-convolution-kernels-simplified-f040cb307c37#:˜: text=Convolution%20is%20usi… [cited by examiner]
Brain4Cars: Car that knows before you do via sensory-fusion deep learning architecture https://arxiv.org/pdf/1601.00740.pdf Jan. 5, 2016. [cited by applicant]
How future cars will predict your driving maneuvers before you make them https://www.technologyreview.com/s/545506/how-future-cars-will-predict-your-driving-maneuvers-before-you-make-them/ Jan. 15, 2016. [cited by applicant]
AI can predict the future location of vehicles https://news.developer.nvidia.com/ai-can-predict-the-future-location-of-vehicles/ Sep. 27, 2018. [cited by applicant]
Learning to predict other vehicles' trajectones for sale autonomous driving https://medium.com/@hello_isee/learning-to-predict-other-vehicles-trajectories-for-safe-autonomous-driving-cbfb1b8053a4 Jun. 17, 2019. [cited by applicant]
Multi-Agent Tensor Fusion for Contextual Trajectory Prediction Zhao et al., CVPR 2019 URL: https://arxiv.org/pdf/1904.04776.pdf Jul. 28, 2019. [cited by applicant]
A dual learning model for vehicle trajectory prediction https://www.researchgate.net/publication/338754579 A_Dual_Learning_Model_for_Vehicle_Trajectory_Prediction Jan. 2020. [cited by applicant]