IP Library › Granted Patent US 12,450,768
Granted Patent B2
US 12,450,768 · App. 17/328,566 · Granted Oct 21, 2025

End-to-end tracking of objects

Inventors: Davi Eugenio Nascimento Frossard (Toronto, CA); Raquel Urtasun (Toronto, CA)
Assignee: Aurora Operations, Inc.
G06T7/70B60K31/0008G01S17/89G06N3/045G06N3/08G06T7/20G06T7/248G06T7/90G06V20/58B60K2031/0016G06T2207/10024G06T2207/20084G06T2207/30241G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,450,768
App. No.
17/328,566
Granted
Oct 21, 2025
Kind
B2
Abstract

Systems and methods for detecting and tracking objects are provided. In one example, a computer-implemented method includes receiving sensor data from one or more sensors. The method includes inputting the sensor data to one or more machine-learned models including one or more first neural networks configured to detect one or more objects based at least in part on the sensor data and one or more second neural networks configured to track the one or more objects over a sequence of sensor data. The method includes generating, as an output of the one or more first neural networks, a 3D bounding box and detection score for a plurality of object detections. The method includes generating, as an output of the one or more second neural networks, a matching score associated with pairs of object detections. The method includes determining a trajectory for each object detection.

Claims (61)

1. An autonomous vehicle computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the one or more processors to perform operations, the operations comprising:

receiving, through one or more sensors of an autonomous vehicle, sensor data associated with an environment, the sensor data comprising RGB data and one or more LIDAR point clouds;

determining an object detection associated with an object within the environment based at least in part on one or more first models and the sensor data;

determining a detection score for the object detection, wherein determining a detection score comprises:

fusing the one or more LIDAR point clouds with the RGB data by extracting one or more features from the RGB data; and

encoding one or more binary parameters associated with the object detection, wherein at least one binary parameter of the one or more binary parameters is indicative of whether the object detection is associated with a trajectory beginning or a trajectory ending;

the one or more first models comprising one or more first machine-learned models configured to detect the object in the environment based at least in part on the sensor data being input into the one or more first models and configured to determine the detection score for the object detection based on the one or more binary parameters;

tracking the object detection over a sequence of sensor data inputs based at least in part on one or more second models,

the one or more second models comprising one or more second machine-learned models configured to track the object over the sequence of sensor data inputs, the one or more second models being different from the one or more first models; and

generating a trajectory for the object based at least in part on one or more linear constraints configured to link the object detection over the sequence of sensor data inputs.

2. The autonomous vehicle computing system of claim 1 , wherein determining the object detection associated with the object within the environment comprises:

generating, as an output of the one or more first machine-learned models, a bounding box for the object detection.

3. The autonomous vehicle computing system of claim 2 , wherein the bounding box is a three-dimensional bounding box.

4. The autonomous vehicle computing system of claim 2 , wherein the one or more sensors comprise a LIDAR sensor, and wherein generating the bounding box for the object detection comprises:

generating the bounding box for the object detection based at least in part on the one or more LIDAR point clouds acquired by the LIDAR sensor.

5. The autonomous vehicle computing system of claim 1 , wherein the operations further comprise:

generating one or more vehicle control signals for the autonomous vehicle based at least in part on the trajectory for the object detection.

6. The autonomous vehicle computing system of claim 1 , wherein the one or more first machine-learned models comprises one or more first neural networks and wherein the one or more second machine-learned models comprise one or more second neural networks.

7. The autonomous vehicle computing system of claim 1 , wherein the sequence of sensor data inputs includes first image data and second image data, the first image data representing an earlier point in time than the second image data.

8. A computer-implemented method, comprising:

receiving sensor data associated with an environment, the sensor data comprising RGB data and one or more LIDAR point clouds;

determining an object detection associated with an object within the environment based at least in part on one or more first models and the sensor data;

determining a detection score for the object detection, wherein determining a detection score comprises:

fusing the one or more LIDAR point clouds with the RGB data by extracting one or more features from the RGB data; and

encoding one or more binary parameters associated with the object detection, wherein at least one binary parameter of the one or more binary parameters is indicative of whether the object detection is associated with a trajectory beginning or a trajectory ending;

the one or more first models comprising one or more first machine-learned models configured to detect the object in the environment based at least in part on the sensor data being input into the one or more first models and configured to determine the detection score for the object detection based on the one or more binary parameters;

tracking the object detection over a sequence of sensor data inputs based at least in part on one or more second models,

the one or more second models comprising one or more second machine-learned models configured to track the object over the sequence of sensor data inputs, the one or more second models being different from the one or more first models; and

generating a trajectory for the object based at least in part on one or more linear constraints configured to link the object detection over the sequence of sensor data inputs.

9. The computer-implemented method of claim 8 , wherein determining the object detection associated with the object within the environment comprises:

generating, as an output of the one or more first machine-learned models, a bounding box for the object detection.

10. The computer-implemented method of claim 9 , wherein the bounding box is a three-dimensional bounding box.

11. The computer-implemented method of claim 9 , wherein generating the bounding box for the object detection comprises:

generating the bounding box for the object detection based at least in part on the one or more LIDAR point clouds.

12. The computer-implemented method of claim 8 , wherein tracking the object detection over the sequence of sensor data inputs comprises:

generating, as an output of the one or more second machine-learned models, a matching score associated with the object detection over the sequence of sensor data inputs.

13. The computer-implemented method of claim 12 , wherein generating the trajectory for the object comprises:

determining the trajectory for the object detection based at least in part on the matching score associated with the object detection over the sequence of sensor data inputs.

14. The computer-implemented method of claim 8 , wherein the one or more first machine-learned models comprises one or more first convolutional neural networks and wherein the one or more second machine-learned models comprise one or more second convolutional neural networks.

15. The computer-implemented method of claim 8 , wherein the sensor data is received through one or more sensors located onboard an autonomous vehicle, and wherein the method further comprises:

generating one or more vehicle control signals for the autonomous vehicle based at least in part on the trajectory for the object detection.

16. An autonomous vehicle, comprising:

a sensor system configured to generate sensor data of an environment external to the autonomous vehicle; and

a vehicle computing system, comprising:

one or more processors; and

one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the vehicle computing system to perform operations, the operations comprising:

receiving the sensor data, the sensor data comprising RGB data and one or more LIDAR point clouds;

inputting the sensor data to one or more machine-learned models comprising one or more first models configured to detect one or more objects in the environment based at least in part on the sensor data and one or more second models configured to track the one or more objects over a sequence of sensor data inputs, the one or more second models being different from the one or more first models;

determining, based at least in part on the one or more first models, an object detection associated with an object within the environment;

determining a detection score for the object detection, wherein determining a detection score comprises:

fusing the one or more LIDAR point clouds with the RGB data by extracting one or more features from the RGB data; and

encoding one or more binary parameters associated with the object detection, wherein:

 the detection score for the object detection is based on the one or more binary parameters, and

 at least one binary parameter of the one or more binary parameters is indicative of whether the object detection is associated with a trajectory beginning or a trajectory ending;

tracking, based at least in part on the one or more second models, the object detection over the sequence of sensor data inputs; and

generating a trajectory for the object based at least in part on one or more linear constraints configured to link the object detection over the sequence of sensor data inputs.

17. The autonomous vehicle of claim 16 , wherein the operations further comprise:

generating one or more vehicle control signals for the autonomous vehicle based at least in part on the trajectory for the object detection.

18. The autonomous vehicle of claim 17 , wherein the vehicle control signals are based at least in part on a motion plan indicative of a future motion of the autonomous vehicle, wherein the motion plan is based at least in part on the trajectory for the object detection.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 14, 2024
From: UATC, LLC
To: AURORA OPERATIONS, INC.
Reel/Frame 067733/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 25, 2021
From: FROSSARD, DAVI EUGENIO NASCIMENTO; URTASUN, RAQUEL
To: UBER TECHNOLOGIES, INC.
Reel/Frame 057902/0901 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 25, 2021
From: UBER TECHNOLOGIES, INC.
To: UATC, LLC
Reel/Frame 057902/0960 →
Continuity (3)
Continuation 16122203 · Sep 5, 2018
Provisional Application 62586700 · Nov 15, 2017
Related Publication 20210362596A1 · Nov 25, 2021
References Cited (88)
US 8818702B2 · Zeng · 2014 [cited by examiner]
US 9471851B1 · Cao et al. · 2016 [cited by applicant]
US 10037471B2 · Satzoda · 2018 [cited by examiner]
US 10073456B2 · Mudalige · 2018 [cited by examiner]
US 10459444B1 · Kentley-Klay · 2019 [cited by examiner]
US 11468285B1 · Tang · 2022 [cited by examiner]
US 20120197856A1 · Banka et al. · 2012 [cited by applicant]
US 20170193390A1 · Weston et al. · 2017 [cited by applicant]
US 20180157939A1 · Butt et al. · 2018 [cited by applicant]
US 20180348346A1 · Vallespi-Gonzalez et al. · 2018 [cited by applicant]
US 20180348374A1 · Laddha et al. · 2018 [cited by applicant]
US 20190049242A1 · Adams et al. · 2019 [cited by applicant]
US 20190096086A1 · Xu · 2019 [cited by examiner]
US 20190130569A1 · Liu et al. · 2019 [cited by applicant]
US 20190137979A1 · Akella et al. · 2019 [cited by applicant]
US 20190188866A1 · Mehrseresht · 2019 [cited by applicant]
CN 105910827 · 2016 [cited by applicant]
WO WO2010042068 · 2010 [cited by applicant]
Jiang, H., Fels, S., & Little, J. J. (Jun. 2007). A linear programming approach for multiple object tracking. In 2007 IEEE Conference on Computer Vision and Pattern Recognition (pp. 1-8). IEEE. (Year: 2007). [cited by examiner]
U.S. Appl. No. 16/124,966, filed Sep. 7, 2018, 68 pages. [cited by applicant]
Bottger et al., “Measuring the Accuracy of Object Detectors and Trackers,” In German Conference on Pattern Recognition (pp. 415-426). Springer, Cham. (Year: 2017). [cited by applicant]
Karpathy et al., “Large-Scale Video Classification with Convolutional Neural Networks,” In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 1725-1732). (Year: 2014). [cited by applicant]
Alahi, et al., “Social LSTM: Human Trajectory Prediction in Crowded Spaces.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-971, 2016. [cited by applicant]
Ahuja et al., “Network Flows: Theory, Algorithms, and Applications”, Prentice Hall, Upper Saddle River, New Jersey, 1993, 863 pages. [cited by applicant]
Benfold et al., “Stable Multi-Target Tracking in Real-Time Surveillance Video” Conference on Computer Vision and Pattern Recognition, Colorado Springs, Colorado, Jun. 21-23, 2011, 8 pages. [cited by applicant]
Berclaz et al., “Multiple Object Tracking Using K-Shortest Paths Optimization”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, Issue 9, Sep. 2011, 16 pages. [cited by applicant]
Bernardin et al., “Evaluating Multiple Object Tracking Performance: The Clear Mot Metrics”, Journal on Image and Video Processing, vol. 2008, Article ID 246309, Dec. 2008, 10 pages. [cited by applicant]
Bertsekas et al., “Data Networks”, Prentice Hall, Englewood Cliffs, New Jersey, 113 pages. [cited by applicant]
Birchfield et al., “Multiway Cut for Stereo and Motion with Slanted Surfaces”, International Conference on Computer Vision, Kerkyra, Greece, Sep. 20-27, 1999, 7 pages. [cited by applicant]
Chen, et al., “3D object proposals using stereo imagery for accurate object class detection.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. [cited by applicant]
Chen et al., “Multi-View 3D Object Detection Network for Autonomous Driving”, Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, Jul. 21-26, 2017, 9 pages. [cited by applicant]
Choi et al., “A General Framework for Tracking Multiple People from a Moving Camera”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, Issue 7, Jul. 2013, 16 pages. [cited by applicant]
Choi et al., “Near-Online Multi-Target Tracking with Aggregated Local Flow Descriptor”, International Conference on Computer Vision, Las Condes, Chile, Dec. 11-18, 2015, 9 pages. [cited by applicant]
Collins et al., “Hybrid Stochastic/Deterministic Optimization for Tracking Sports Players and Pedestrians”, European Conference on Computer Vision, Zurich, Switzerland, Sep. 6-12, 2014, 16 pages. [cited by applicant]
Dai, et al., “R-FCN: Object Detection Via Region-Based Fully Convolutional Networks”, In Advances in neural information processing systems, pp. 379-387, 2016. [cited by applicant]
Feichtenhofer et al., “Detect to Track and Track to Detect”, International Conference on Computer Vision, Venice Italy, Oct. 22-29, 2017, 9 pages. [cited by applicant]
Gaidon et al., “Online Domain Adaptation for Multi-Object Tracking”, arXiv:1508-00776v1, Aug. 4, 2015, 13 pages. [cited by applicant]
Geiger et al., “3D Traffic Scene Understanding from Movable Platforms”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, Issue 5, May 2014, 14 pages. [cited by applicant]
Geiger et al., “Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite”, Conference on Computer Vision and Pattern Recognition, Providence, Rhode Island, Jun. 16-21, 2012, 8 pages. [cited by applicant]
Gong et al., “Multi-Hypothesis Motion Planning for Visual Object Tracking”, International Conference on Computer Vision, Barcelona, Spain, Nov. 6-13, 2011, pp. 619-626. [cited by applicant]
Gurobi Optimization Inc., “Gurobi Optimizer Reference Manual”, 2016, 773 pages. [cited by applicant]
Haeusler et al., “Ensemble Learning for Confidence Measure in Stereo Vision”, Conference on Computer Vision and Pattern Recognition, Portland, Oregan, Jun. 23-28, 2013, 8 pages. [cited by applicant]
He et al., “Mask R-CNN”, arXiv:1703.06870v3, Jan. 24, 2018, 12 pages. [cited by applicant]
Held et al., “Learning to Track at 100 FPS with Deep Regression Networks”, arXiv:1604.01802v2, Aug. 16, 2016, 26 pages. [cited by applicant]
Howard et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications”, arXiv:1704.04861v1, Apr. 17, 2017, 9 pages. [cited by applicant]
Huang et al., “Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors”, Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, Jul. 21-26, 2017, 10 pages. [cited by applicant]
Iandola et al., “Squeezenet: AlexNet-Level Accuracy with 50X Fewer Parameters and <0.5MB Model Size”, arXiv:1602.07360v4, Nov. 4, 2016, 13 pages. [cited by applicant]
Karlsruhe Institute of Technology, “The KITTI Vision Benchmark Suite”, http://www.cvlibs.net/datasets/kitti/eval_tracking.php, retrieved on Sep. 18, 2018, 6 pages. [cited by applicant]
Khan et al., “Efficient Particle Filter-Based Tracking of Multiple Interacting Targets using an MRF-Based Motion Model”, International Conference on Intelligent Robots and Systems, Las Vegas, Nevada, Oct. 27-31, 2003, 6… [cited by applicant]
Kingma et al., “ADAM: A Method for Stochastic Optimization”, arXiv:1412.6980v9, Jan. 30, 2017, 15 pages. [cited by applicant]
Kuo et al., “Multi-Target Tracking by On-Line Learned Discriminative Appearance Models” Conference on Computer Vision and Pattern Recognition, San Francisco, California, Jun. 13-18, 2010, 8 pages. [cited by applicant]
Leal-Taixé et al., “Learning by Tracking: Siamese CNN for Robust Target Association”, Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, Nevada, Jun. 26-Jul. 1, 2016, 8 pages. [cited by applicant]
Lee et al., “Distant Future Prediction in Dynamic Scenes with Interacting Agents”, Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, Jul. 21-26, 2017, 10 pages. [cited by applicant]
Lee et al., “Multi-Class Multi-Object Tracking Using Changing Point Detection”, arXiv:1608.08434v1, Aug. 30, 2016, 16 pages. [cited by applicant]
Lenz et al., “FollowMe: Efficient Online Min-Cost Flow Tracking with Bounded Memory and Computation”, Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, Jun. 8-10, 2015, 9 pages. [cited by applicant]
Li, “3D Fully Convolutional Network for Vehicle Detection in Point Cloud”,2016, arXiv:1611.08069v2, Jan. 16, 2017, 5 pages. [cited by applicant]
Li, et al., “Learning to Associate: HybridBoosted Multi-Target Tracker for Crowded Scene”, Conference on Computer Vision and Pattern Recognition, Miami Beach, Florida, Jun. 20-25, 2009, 8 pages. [cited by applicant]
Lin et al., “Focal Loss for Dense Object Detection”, arXiv:1708.02002v2, Feb. 7, 2018, 10 pages. [cited by applicant]
Liu et al., “SSD: Single Shot Multibox detector”, arXiv:1512.02325v5, Dec. 29, 2016, 17 pages. [cited by applicant]
Luo et al., “Efficient Deep Learning for Stereo Matching”, Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, Jun. 26-Jul. 1, 2016, 9 pages. [cited by applicant]
Ma et al., “Forecasting Interactive Dynamics of Pedestrians with Fictitious Play”, arXiv:1604.01431v3, Mar. 28, 2017, 9 pages. [cited by applicant]
Ma et al., “Hierarchical Convolutional Features for Visual Tracking”, International Conference on Computer Vision, Las Condes, Chile, Dec. 11-18, 2015, pp. 3074-3082. [cited by applicant]
Mathieu et al., “Deep Multi-Scale Video Prediction Beyond Mean Square Error”, arXiv:1511.05440v6, Feb. 26, 2016, 14 pages. [cited by applicant]
Milan et al., “Continuous Energy Minimization for Multitarget Tracking”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, No. 1, Jan. 2014, pp. 58-72. [cited by applicant]
Milan et al., Detection- and Trajectory-Level Exclusion in Multiple Object Tracking, Conference on Computer Vision and Pattern Recognition, Columbus, Ohio, Jun. 25-27, 2013, 8 pages. [cited by applicant]
Milan et al., “Online Multi-Target Tracking Using Recurrent Neural Networks”, Association for the Advancement of Artificial Intelligence, San Francisco, California, Feb. 4-9, 2017, 8 pages. [cited by applicant]
Nam et al., “Learning Multi-Domain Convolutional Neural Networks for Visual Tracking”, Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, Jun. 26-Jul. 1, 2016, pp. 4293-4302. [cited by applicant]
Osep et al., “Combined Image-and World-Space Tracking in Traffic Scenes”, International Conference on Robotics and Automation, Singapore, China, May 29-Jun. 3, 2017, 8 pages. [cited by applicant]
Pellegrini et al., “You'll Never Walk Alone: Modeling Social Behavior for Multi-Target Tracking”, International Conference on Computer Vision, Kyoto, Japan, Sep. 29-Oct. 2, 2009, pp. 261-268. [cited by applicant]
Redmon et al., “Yolo9000: Better, Faster, Stronger”, Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, Jul. 21-26, 2017, 9 pages. [cited by applicant]
Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, Neural Information Processing Systems, Montreal, Canada, Dec. 7-12, 2015, 9 pages. [cited by applicant]
Riahi et al., “Multiple Object Tracking Based on Sparse Generative Appearance Modeling” International Conference on Image Processing, Quebec City, Canada, Sep. 27-30, 2015, 5 pages. [cited by applicant]
Shitrit et al., “Multi-Commodity Network Flow for Tracking Multiple People”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, Issue 8, Aug. 2014, 14 pages. [cited by applicant]
Simonyan et al., “Very Deep Convolutional Networks for Large-Scale Image Recognition”, arXiv:1409.1556v6, Apr. 10, 2015, 14 pages. [cited by applicant]
Spyropoulos et al., “Learning to Detect Ground Control Points for Improving the Accuracy of Stereo Matching”, Conference on Computer Vision and Pattern Recognition, Columbus, Ohio, Jun. 24-27, 2014, 8 pages. [cited by applicant]
Srivastava et al., “Unsupervised Learning of Video Representations Using LSTMs”, International Conference on Machine Learning, Lille, France, Jul. 6-11, 2015, pp. 843-852. [cited by applicant]
Tao et al., “Siamese Instance Search for Tracking”, Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, Jun. 26-Jul. 1, 2016, pp. 1420-1429. [cited by applicant]
Walker et al., “An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders”, arXiv:1606.07873v1, Jun. 25, 2016, 17 pages. [cited by applicant]
Wang et al., “Learning A Deep Compact Image Representation for Visual Tracking”, Advances in Neural Information Processing Systems, Lake Tahoe, Nevada, Dec. 5-10, 2013, pp. 809-817. [cited by applicant]
Wang et al., “Learning Optimal Parameters for Multi-Target Tracking with Contextual Interactions”, International Journal of Computer Vision, vol. 122, No. 3, 2017, pp. 484-501. [cited by applicant]
Wang et al., “Visual Tracking with Fully Convolutional Networks”, International Conference on Computer Vision, Las Condes, Chile, Dec. 11-18, 2015, pp. 3119-3127. [cited by applicant]
Wu et al., “SqueezeDet: Unified, Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Driving”, Honolulu, Hawaii, Jul. 21-26, 2017, 9 pages. [cited by applicant]
Xiang et al., “Learning to Track: Online Multi-Object Tracking by Decision Making”, International Conference on Computer Vision, Las Condes, Chile, Dec. 11-18, 2015, 9 pages. [cited by applicant]
Yang et al., “An Online Learned CRF Model for Multi-Target Tracking”, Conference on Computer Vision and Pattern Recognition, Providence, Rhode Island, Jun. 16-21, 2012, 8 pages. [cited by applicant]
Yoon et al., “Bayesian Multi-Object Tracking Using Motion Context from Multiple Objects”, Winter Conference on Applications of Computer Vision, Waikoloa Beach, Hawaii, Jan. 6-9, 2015, 8 pages. [cited by applicant]
Yoon et al., “Online Multi-Object Tracking via Structural Constraint Event Aggregation”, Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, Jun. 26-Jul. 1, 2016, 9 pages. [cited by applicant]
Zbontar et al., “Computing the Stereo Matching Cost with a Convolutional Neural Network”, Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, Jun. 8-10, 2015, 8 pages. [cited by applicant]
Zhang et al., “Global Data Association for Multi-Object Tracking Using Network Flows” Conference on Computer Vision and Pattern Recognition, Anchorage, Alaska, Jun. 23-28, 2008, 8 pages. [cited by applicant]