IP Library › Granted Patent US 12,496,714
Granted Patent B2
US 12,496,714 · App. 18/200,347 · Granted Dec 16, 2025

Collision-free motion generation

Inventors: Balakumar Sundaralingam (Seattle, WA); Siva Kumar Sastry Hari (Sunnyvale, CA); Adam Harper Fishman (Seattle, WA); Caelan Reed Garrett (Seattle, WA); Alexander James Millane (Zurich, CH); Elena Oleynikova (Zurich, CH); Ankur Handa (Seattle, WA); Fabio Tozeto Ramos (Seattle, WA); Nathan Donald Ratliff (Seattle, WA); Karl Van Wyk (Issaquah, WA); Dieter Fox (Seattle, WA)
Assignee: NVIDIA CORPORATION
B25J9/1664
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,496,714
App. No.
18/200,347
Granted
Dec 16, 2025
Kind
B2
Abstract

Apparatuses, systems, and techniques to perform collision-free motion generation (e.g., to operate a real-world or virtual robot). In at least one embodiment, at least a portion of the collision-free motion generation is performed in parallel.

Claims (69)

1 . A method comprising:

using at least one first parallel processing unit (“PPU”) to generate a plurality of seed trajectories in parallel based at least in part on an initial joint configuration and a plurality of seed joint configurations that would position at least a portion of a first manipulator in a goal pose;

using at least one second PPU to modify the plurality of seed trajectories in parallel to obtain a plurality of modified trajectories;

selecting one of the plurality of modified trajectories based at least in part on cost values calculated for the plurality of modified trajectories; and

providing a motion plan based at least in part on the selected modified trajectory to a second manipulator to cause the second manipulator to move in accordance with the motion plan.

2 . The method of claim 1 , wherein using the at least one second PPU to modify the plurality of seed trajectories comprises, for a particular one of the plurality of seed trajectories, performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising performing at least one optimization on the particular seed trajectory.

3 . The method of claim 2 , wherein the at least one optimization comprises a gradient-free optimization, and a gradient-based optimization.

4 . The method of claim 3 , wherein the gradient-free optimization comprises a particle-based optimization.

5 . The method of claim 3 , wherein the gradient-based optimization is based at least in part on a Limited-memory Broyden-Fletcher-Goldfarb-Shanno (“L-BFGS”) algorithm.

6 . The method of claim 2 , wherein each of the one or more iterations comprises calculating a task cost, and

the at least one stopping condition is satisfied based at least in part on a comparison between the task cost and a threshold value.

7 . The method of claim 2 , wherein the at least one stopping condition is satisfied when at least a desired number of iterations have been performed.

8 . The method of claim 1 , wherein using the at least one first PPU to generate the plurality of seed trajectories in parallel comprises:

using inverse kinematics to obtain the plurality of seed joint configurations;

using at least one third PPU to modify the plurality of seed joint configurations to obtain a plurality of modified joint configurations; and

interpolating between the initial joint configurations and the plurality of modified joint configurations to obtain at least a portion of the plurality of seed trajectories.

9 . The method of claim 8 , wherein using the at least one third PPU to modify the plurality of seed joint configurations comprises, for a particular one of the plurality of seed joint configurations, performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising performing at least one optimization on the particular seed joint configuration.

10 . The method of claim 1 , wherein using the at least one first PPU to generate the plurality of seed trajectories in parallel comprises:

using geometric planning to obtain at least a portion of the plurality of seed trajectories.

11 . The method of claim 1 , wherein the second manipulator is the first manipulator.

12 . The method of claim 1 , wherein using the at least one second PPU to optimize the plurality of seed trajectories comprises, for a particular one of the plurality of seed trajectories, performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising:

performing at least one optimization on the particular seed trajectory to obtain optimization results comprising a number of steps having a previous step size,

obtaining a new step size determined based at least in part on at least one limitation of a device to perform the selected optimized trajectory, the selected optimized trajectory; and

scaling the selected optimized trajectory to comprise the number of steps having the new step size.

13 . The method of claim 12 , wherein the at least one limitation comprises at least one velocity constraint, at least one acceleration constraint, or at least one jerk constraint.

14 . The method of claim 12 , wherein each of the one or more iterations comprises:

calculating a task cost based at least in part on one or more constituent cost values; and

modifying at least a portion of the one or more constituent cost values based at least in part on a new time step.

15 . A system comprising:

one or more circuits to:

generate a plurality of seed trajectories in parallel based at least in part on an initial joint configuration and a plurality of seed joint configurations that would position at least a portion of a first manipulator in a goal pose,

calculate cost values for a plurality of modified trajectories obtained by modifying the plurality of seed trajectories in parallel, and

select a modified trajectory of the plurality of modified trajectories based at least in part on the cost values; and

a second manipulator to move in accordance with the selected modified trajectory.

16 . The system of claim 15 , wherein the second manipulator is different from the first manipulator.

17 . The system of claim 15 , wherein the second manipulator is at least a portion of an autonomous device or a semi-autonomous device.

18 . The system of claim 15 , wherein the second manipulator is at least a portion of an autonomous vehicle, an aerial drone, a cleaning device, a legged robot, or a walking robot.

19 . The system of claim 15 , wherein the one or more circuits are to modify a particular one of the plurality of seed trajectories by performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising performing a gradient-free optimization on the particular seed trajectory to obtain first optimization results, and performing a gradient-based optimization on the first optimization results to obtain second optimization results.

20 . The system of claim 15 , wherein the one or more circuits are to generate the plurality of seed trajectories in parallel by:

using inverse kinematics to obtain the plurality of seed joint configurations;

modifying the plurality of seed joint configurations to obtain a plurality of modified joint configurations; and

interpolating between the initial joint configurations and the plurality of modified joint configurations to obtain at least a portion of the plurality of seed trajectories.

21 . The system of claim 15 , wherein the one or more circuits are to generate at least a portion of the plurality of seed trajectories in parallel using geometric planning.

22 . The system of claim 21 , wherein the geometric planning comprises:

identifying at least one collision fee path segment by searching along directions between at least one vertex of a graph and a plurality of target points in the graph in parallel for at least one endpoint that does not collide with any obstacles;

adding to the graph at least one new vertex corresponding to the at least one endpoint; and

adding to the graph at least one new edge between the at least one vertex and the at least one new vertex.

23 . The system of claim 15 , wherein the one or more circuits are to generate a motion plan based at least in part on the selected modified trajectory and the second manipulator that is to move in accordance with the motion plan.

24 . The system of claim 15 , wherein the one or more circuits are to modify a particular one of the plurality of seed trajectories by performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising performing at least one optimization on the particular seed trajectory, the at least one optimization comprising calculating a collision cost value associated with one or more collisions encountered along the particular seed trajectory.

25 . The system of claim 24 , wherein calculating the collision cost value comprises:

determining a new position for a virtual object positioned at a third position between first and second positions, the new position being based at least in part on the third position and a first distance between the third position and a selected first virtual obstacle when the virtual object is not in a collision at the third position; and

until the virtual object would be positioned at a desired position with respect to at least one of the first position and the second position when the virtual object is positioned at the new position:

if the first distance is not greater than a second distance based at least in part on the new position and the desired position, calculating a collision cost if the virtual object would collide with at least one second virtual obstacle when the virtual object is at the new position, the new position being a previous position,

redetermining the first distance between the previous position and a selected second virtual obstacle, and

redetermining the new position based on the previous position and the first distance.

26 . The system of claim 15 , wherein the one or more circuits are to modify a particular one of the plurality of seed trajectories by performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising performing at least one optimization on the particular seed trajectory, the at least one optimization comprising calculating a collision cost value associated with one or more collisions encountered between components of a device moving in accordance with the particular seed trajectory.

27 . The system of claim 26 , wherein calculating the collision cost value comprises:

identifying as a set of movable objects any pairs of smaller objects representing the device that include first and second objects that are movable with respect to one another;

calculating distances between any of the pairs of the smaller objects included in the set of movable objects in parallel; and

using the distances to identify as a set of colliding objects any of the pairs of the smaller objects that are to collide with one another.

28 . The system of claim 15 , wherein the one or more circuits are to modify a particular one of the plurality of seed trajectories by performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising performing at least one optimization on the particular seed trajectory, the at least one optimization comprising:

generating a plurality of data structures representing links, at least a portion of the links being movable with respect to one another; and

performing a plurality of kinematic processes in parallel, the plurality of kinematic processes comprising at least one of a plurality of forward kinematic processes or a plurality of backward kinematic processes, the plurality of forward kinematic processes to use the plurality of data structures to obtain locations of virtual objects representing the links, the plurality of backward kinematic processes to use the plurality of data structures to obtain gradients for the virtual objects.

29 . At least one parallel processing unit (“PPU”) comprising:

one or more circuits to perform instructions that when performed by the at least one PPU cause the at least one PPU to:

generate a plurality of seed trajectories in parallel based at least in part on an initial joint configuration and a plurality of seed joint configurations that would position at least a portion of a first manipulator in a goal pose,

calculate cost values for a plurality of modified trajectories obtained by modifying the plurality of seed trajectories in parallel, and

provide a motion plan based at least in part on the cost values to a second manipulator to cause the second manipulator to move in accordance with the motion plan.

30 . The at least one PPU of claim 29 , wherein the instructions when performed by the at least one PPU cause the at least one PPU to modify a particular one of the plurality of seed trajectories by performing one or more iterations until at least one stopping condition is satisfied, each of the one or more iterations comprising performing at least one optimization on the particular seed trajectory.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 23, 2023
From: SUNDARALINGAM, BALAKUMAR; HARI, SIVA KUMAR SASTRY; FISHMAN, ADAM HARPER; GARRETT, CAELAN REED; MILLANE, ALEXANDER JAMES; OLEYNIKOVA, ELENA; HANDA, ANKUR; TOZETO RAMOS, FABIO; RATLIFF, NATHAN DONALD; VAN WYK, KARL; FOX, DIETER
To: NVIDIA CORPORATION
Reel/Frame 063728/0977 →
Continuity (2)
Provisional Application 63411495 · Sep 29, 2022
Related Publication 20240131706A1 · Apr 25, 2024
References Cited (71)
US 9895803B1 · Oslund · 2018 [cited by examiner]
US 11623346B2 · Colasanto · 2023 [cited by examiner]
US 12380418B2 · Cella · 2025 [cited by examiner]
US 20040122557A1 · Chandhoke · 2004 [cited by examiner]
US 20100138810A1 · Komatsu · 2010 [cited by examiner]
US 20120165982A1 · Kim · 2012 [cited by examiner]
US 20190202056A1 · Xiong · 2019 [cited by examiner]
US 20210146532A1 · Rodriguez Garcia · 2021 [cited by examiner]
US 20210220994A1 · Colasanto · 2021 [cited by examiner]
US 20210253128A1 · Nister · 2021 [cited by examiner]
US 20210331754A1 · Whitman · 2021 [cited by examiner]
US 20220163969A1 · Li · 2022 [cited by examiner]
US 20230048578A1 · Campos Macias · 2023 [cited by examiner]
US 20230226695A1 · Ames · 2023 [cited by examiner]
US 20240001961A1 · Myung · 2024 [cited by examiner]
US 20240017738A1 · Vozar · 2024 [cited by examiner]
US 20240075617A1 · Wu · 2024 [cited by examiner]
US 20240092357A1 · Kobilarov · 2024 [cited by examiner]
US 20240261969A1 · Yamaguchi · 2024 [cited by examiner]
US 20250065499A1 · Wahrburg · 2025 [cited by examiner]
US 20250249586A1 · Murray · 2025 [cited by examiner]
Dai et al., Improving Trajectory Optimization using a Roadmap Framework, Oct. 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 8674-8681 (Year: 2018). [cited by examiner]
Merkt et al., Leveraging Precomputation with Problem Encoding for Warm-Starting Trajectory Optimization in Complex Environments, Oct. 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5… [cited by examiner]
Henrich et al., Parallel Processing Approaches in Robotics, Jul. 7-11, 1997, IEEE International Symposium on Industrial Electronics , pp. 1-7 (Year: 1997). [cited by examiner]
Apgar et al., “Fast Online Trajectory Optimization for the Bipedal Robot Cassie,” Robotics: Science and Systems, vol. 101, 2018, 8 pages. [cited by applicant]
Beeson et al., “TRAC-IK: An Open-Source Library for Improved Solving of Generic Inverse Kinematics,” IEEE-RAS International Conference on Humanoid Robots, 2015, 8 pages. [cited by applicant]
Bertsekas et al., “Reinforcement Learning and Optimal Control,” Athena Scientific, 2018, 268 pages. [cited by applicant]
Bhardwaj et al., “STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation,” Proceedings of the Conference on Robot Learning, Nov. 2021, 10 pages. [cited by applicant]
Carpentier et al., “The Pinocchio C++ library—A fast and Flexible Implementation of Rigid Body Dynamics Algorithms and Their Analytical Derivatives,” IEEE International Symposium on System Integrations, 2019, 6 pages. [cited by applicant]
Chamzas et al., “Motionbenchmaker: A tool to generate and benchmark motion planning datasets,” IEEE Robotics and Automation Letters, 7(2): 2021, 8 pages. [cited by applicant]
Coumans et al., “:PyBullet: A Python Module for Physics Simulation for Games,” Robotics and Machine Learning, retrieved http://pybullet.org, 2016, 10 pages. [cited by applicant]
Dellaert, “Factor Graphs: Exploiting Structure in Robotics,” Annual Review of Control; Robotics; and Autonomous Systems, 2021, 28 pages. [cited by applicant]
Gammell et al., “Batch Informed Trees (BIT*): Sampling-based Optimal Planning via the Heuristically Guided Search of Implicit Random Geometric Graphs,” IEEE international conference on robotics and automation, 2015, 8 p… [cited by applicant]
Garrett et al., “PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning,” Proceedings of the International Conference on Automated Planning and Scheduling, vol. 30, 2020, 12 pag… [cited by applicant]
GitHub, “Tesseract,” retrieved from https://github.com/tesseract-robotics/tesseract, Sep. 5, 2022, 6 pages. [cited by applicant]
Hansen, “The CMA Evolution Strategy: A Tutorial,” 2016, 39 pages. [cited by applicant]
Ichnowski et al., “GOMP: Grasp-Optimized Motion Planning for Bin Picking,” IEEE International Conference on Robotics and Automation, 2020, 8 pages. [cited by applicant]
IEEE, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages. [cited by applicant]
Kalakrishnan et al., “STOMP: Stochastic Trajectory Optimization for Motion Planning,” IEEE International Conference on Robotics and Automation, 2011, 6 pages. [cited by applicant]
Kuffner et al., “RRT-Connect: An Efficient Approach to Single-Query Path Planning,” ICRA, vol. 2, 2000, 7 pages. [cited by applicant]
Kunz et al., “Probabilistically Complete Kinodynamic Planning for Robot Manipulators with Acceleration Limits,” IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, 7 pages. [cited by applicant]
Kunz et al., “Time-Optimal Trajectory Generation for Path Following with Bounded Acceleration and Velocity,” Robotics: Science and Systems VIII, 2012, 8 pages. [cited by applicant]
Lavalle et al., “Rapidly-Exploring Random Trees: Progress and Prospects,” Algorithmic and Computational Robotics: New Directions, 2001, 16 pages. [cited by applicant]
Lavalle, “Planning Algorithms,” Cambridge, U.K.: Cambridge University Press, 2006, 512 pages. [cited by applicant]
Ma et al., “A Complete Recipe for Stochastic Gradient MCMC,” Advances in Neural Information Processing Systems, 2015, 16 pages. [cited by applicant]
Medeiros et al., “Trajectory Optimization for Wheeled-Legged Quadrupedal Robots Driving in Challenging Terrain,” IEEE Robotics and Automation Letter, 5(3): 2020, 8 pages. [cited by applicant]
Meier et al., “Differentiable and Learnable Robot Models,” 2022, 6 pages. [cited by applicant]
Mukadam et al., “Continuous-Time Gaussian Process Motion Planning via Probabilistic Inference,” The International Journal of Robotics Research, 37(11): 2018, 24 pages. [cited by applicant]
Murray et al., “Robot Motion Planning on a Chip,” 2016, 9 pages. [cited by applicant]
Nocedal et al., “Numerical Optimization,” Springer 1999, 683 pages. [cited by applicant]
Nvidia, “Programming Guide :: CUDA Toolkit Documentation,” retrieved from https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html, 2022, 10 pages. [cited by applicant]
Oleynikova et al., “GitHub—Nvidia-Isaac/Nvblox: A GPU-accelerated TSDF and ESDF Library for Robots Equipped with RGB-D Cameras,” retrieved from https://github.com/nvidia-isaac/nvblox, 2022, 12 pages. [cited by applicant]
Oleynikova et al., “Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning,” IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017, 8 pages. [cited by applicant]
Pineda et al., “Theseus: A Library for Differentiable Nonlinear Optimization,” 2022, 26 pages. [cited by applicant]
Posa et al., “Direct Trajectory Optimization of Rigid Body Dynamical Systems Through Contact,” Algorithmic Foundations of Robotics X, 2013, 22 pages. [cited by applicant]
Ratliff et al., “CHOMP: Gradient Optimization Techniques for Efficient Motion Planning,” IEEE International Conference on Robotics and Automation, 2009, 8 pages. [cited by applicant]
Ratliff et al., “Understanding the Geometry of Workspace Obstacles in Motion Optimization,” 2015, 8 pages. [cited by applicant]
Schmidt et al., “DART: Dense Articulated Real-time Tracking with Consumer Depth Cameras,” Autonomous Robots, 39(3): 2015, 20 pages. [cited by applicant]
Schulman et al., “Motion Planning with Sequential Convex Optimization and Convex Collision Checking,” The International Journal of Robotics Research, 33(9): 2014, 17 pages. [cited by applicant]
Srinivasa et al., “A System for Multi-Step Mobile Manipulation: Architecture, Algorithms, and Experiments,” International Symposium on Experimental Robotics, 2016, 12 pages. [cited by applicant]
Starke et al., “Memetic Evolution for Generic Full-Body Inverse Kinematics in Robotics and Animation,” IEEE Transactions on Evolutionary Computation, 23(3): 2018, 15 pages. [cited by applicant]
Strub et al., “Adaptively Informed Trees (AIT*): Fast Asymptotically Optimal Path Planning through Adaptive Heuristics,” IEEE International Conference on Robotics and Automation, 2020, 8 pages. [cited by applicant]
Sucan et al., “The Open Motion Planning Library,” IEEE Robotics & Automation Magazing, Dec. 2012, 10 Pages. [cited by applicant]
Sundaralingam et al., “Relaxed-Rigidity Constraints: Kinematic Trajectory Optimization and Collision Avoidance for In-Grasp Manipulation,” Autonomous Robots, 43(2): 2019, 15 pages. [cited by applicant]
Todorov et al., “A Generalized Iterative LQG Method for Locally-Optimal Feedback Control of Constrained Nonlinear Stochastic Systems,” Proceedings of the American Control Conference, vol. 1, 2005, 7 pages. [cited by applicant]
Toussaint et al., “KOMO: Newton Methods for Korder Markov Constrained Motion Problems,” 2014, 6 pages. [cited by applicant]
Toussaint, “Logic-Geometric Programming: An Optimization-Based Approach to Combined Task and Motion Planning,” Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015, 7 pages. [cited by applicant]
Toussaint, “Robot Trajectory Optimization using Approximate Inference,” Proceedings of the 26th Annual International Conference on Machine Learning, 2009, 8 pages. [cited by applicant]
Wagener et al., “An Online Learning Approach to Model Predictive Control,” Oct. 9, 2019, 18 pages. [cited by applicant]
Welling et al., “Bayesian Learning via Stochastic Gradient Langevin Dynamics,” Proceedings of the International Conference on Machine Learning, 2011, 8 pages. [cited by applicant]
Zucker et al., “CHOMP: Covariant Hamiltonian Optimization for Motion Planning,” The International Journal of Robotics Research, 32(9-10): 2013, 45 pages. [cited by applicant]