Granted Patent B2

US 12,229,982 · App. 17/190,312 · Granted Feb 18, 2025

Inventors: Filippo Brizzi (London, GB); Luca del Pero (London, GB); Tayyab Naseer (London, GB); Lorenzo Peppoloni (London, GB)

Assignee: Lyft, Inc.

G06T7/70G06F18/22H04N13/204B60W10/18B60W10/20B60W30/00B60W60/001B60W2420/403G06T2207/10016G06T2207/30241G06T2207/30252

Abstract

Claims (70)

1. A computer-implemented method comprising:

receiving, at a computing system, sensor data captured by a camera-based sensor system associated with a vehicle in an environment;

processing, by the computing system, the received sensor data and thereby determining a sensor-based track for an agent that is identified within the sensor data, wherein the sensor-based track comprises a time-sequence of sensor-based positions of the agent;

generating, by the computing system, a first updated time-sequence of sensor-based positions of the agent by performing a first optimization operation on the time-sequence of sensor-based positions, wherein the first optimization operation includes processing at least two or more sensor-based positions of the time-sequence of sensor-based positions by beginning with a sensor-based position having a highest confidence in the time-sequence of sensor-based position observations and proceeding in a first direction;

after generating the first updated time-sequence of sensor-based positions, generating, by the computing system, a second updated time-sequence of sensor-based positions of the agent by performing a second optimization operation on the first updated time-sequence of sensor-based positions, wherein the second optimization operation includes processing at least two or more sensor-based positions of the first updated time-sequence of sensor-based positions in a second direction opposite the first direction;

deriving, by the computing system, a trajectory for the agent in the environment based on the second updated time-sequence of sensor-based positions of the agent; and

storing the derived trajectory, wherein the derived trajectory is thereafter utilized to perform a task that benefits from increased-accuracy trajectories.

2. The computer-implemented method of claim 1 , wherein determining the sensor-based track for the agent comprises:

tracking observations of the agent identified across a series of image frames within the sensor data;

assigning a common agent identifier (ID) to each of the tracked observations of the agent; and

based on the tracked observations comprising the common agent ID, determining the time-sequence of sensor-based positions.

3. The computer-implemented method of claim 2 , wherein tracking the observations of the agent identified across the series of image frames within the sensor data comprises:

based on (i) one or more respective prior positions associated with the one or more previous observations of the agent and (ii) a motion model for the agent, determining a predicted position of the agent; and

determining a match between the predicted position of the agent and a future observation of the agent based on:

an intersection of the predicted position of the agent and a future sensor-based position associated with the future observation of the agent; and

a visual matching analysis of the predicted position of the agent with the future sensor-based position associated with the future observation of the agent.

4. The computer-implemented method of claim 1 , further comprising:

determining a respective measurement error associated with each sensor-based position in the time-sequence of sensor-based positions that is based on a measured distance of the sensor-based positions from the vehicle; and

incorporating the respective measurement error associated with each sensor-based position into at least the first optimization operation or the second optimization operation.

5. The computer-implemented method of claim 4 , wherein the respective measurement error increases quadratically as the measured distance from the vehicle increases.

6. The computer-implemented method of claim 1 , further comprising:

augmenting the second updated time-sequence of sensor-based positions to include one or more additional intermediate observations sensor-based positions between each pair of sensor-based positions in the second updated time-sequence of sensor-based positions.

7. The computer-implemented method of claim 6 , wherein the sensor data is captured by a stereo camera having a frame rate of 5 frames per second, and wherein augmenting the second updated time-sequence of sensor-based positions comprises increasing a frame rate of the second updated time-sequence of sensor-based positions from 5 frames per second to 30 frames per second.

8. The computer-implemented method of claim 1 , wherein beginning with the sensor-based position having the highest confidence in the time-sequence of sensor-based positions and proceeding in the first direction comprises:

determining that one of (i) an earliest position in the time-sequence of sensor-based positions or (ii) a latest position in the time-sequence of sensor-based positions is the sensor-based position of the agent having the highest confidence in the time-sequence of sensor-based positions based on a measured distance of each respective sensor-based position from the vehicle; and

proceeding (i) chronologically forward or (ii) chronologically backward, respectively.

9. The computer-implemented method of claim 1 , wherein generating the first updated time-sequence of sensor-based positions of the agent comprises processing at least two or more sensor-based positions of the time-sequence of sensor-based positions by beginning with a position associated with an earliest observation in the time-sequence of sensor-based positions and proceeding chronologically forward; and

wherein generating the second updated time-sequence of sensor-based positions of the agent comprises processing at least two or more sensor-based positions of the first updated time-sequence of sensor-based positions by beginning with a latest sensor-based position and proceeding chronologically backward.

10. The computer-implemented method of claim 1 , wherein performing the first optimization operation on the time-sequence of sensor-based positions comprises:

based on (i) an initial sensor-based position in the time-sequence of sensor-based positions and (ii) a motion model for the agent, determining a next predicted sensor-based position for the agent;

based on (i) a next sensor-based position for the agent and (ii) the next predicted sensor-based position for the agent, determining an updated next sensor-based position for the agent having a higher confidence than the next sensor-based position for the agent; and

using the updated next sensor-based position for the agent as a next initial sensor-based position in the time-sequence of sensor-based positions to thereby determine the first updated time-sequence of sensor-based positions.

11. A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:

receive, at the computing system, sensor data captured by a camera-based sensor system associated with a vehicle in an environment;

process, by the computing system, the received sensor data and thereby determine a sensor-based track for an agent that is identified within the sensor data, wherein the sensor-based track comprises a time-sequence of sensor-based positions of the agent;

generate, by the computing system, a first updated time-sequence of sensor-based positions of the agent by performing a first optimization operation on the time-sequence of sensor-based positions, wherein the first optimization operation includes processing at least two or more sensor-based positions of the time-sequence of sensor-based positions by beginning with a sensor-based position having a highest confidence in the time-sequence of sensor-based position observations and proceeding in a first direction;

after generating the first updated time-sequence of sensor-based positions, generate, by the computing system, a second updated time-sequence of sensor-based positions of the agent by performing a second optimization operation on the first updated time-sequence of sensor-based positions, wherein the second optimization operation includes processing at least two or more sensor-based positions of the first updated time-sequence of sensor-based positions in a second direction opposite the first direction;

derive, by the computing system, a trajectory for the agent in the environment based on the second updated time-sequence of sensor-based positions of the agent; and

store the derived trajectory, wherein the derived trajectory is thereafter utilized to perform a task that benefits from increased-accuracy trajectories.

12. The computer-readable medium of claim 11 , wherein determining the sensor-based track for the agent comprises:

tracking observations of the agent identified across a series of image frames within the sensor data;

assigning a common agent identifier (ID) to each of the tracked observations of the agent; and

based on the tracked observations comprising the common agent ID, determining the time-sequence of sensor-based positions.

13. The computer-readable medium of claim 11 , wherein tracking the observations of the agent identified across the series of image frames within the sensor data comprises:

based on (i) one or more respective prior positions associated with the one or more previous observations of the agent and (ii) a motion model for the agent, determining a predicted position of the agent; and

determining a match between the predicted position of the agent and a future observation of the agent based on:

an intersection of the predicted position of the agent and a future sensor-based position associated with the future observation of the agent; and

a visual matching analysis of the predicted position of the agent with the future sensor-based position associated with the future observation of the agent.

14. The computer-readable medium of claim 11 , wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:

determine a respective measurement error associated with each sensor-based position in the time-sequence of sensor-based positions that is based on a measured distance of the sensor-based positions from the vehicle; and

incorporate the respective measurement error associated with each sensor-based position into at least the first optimization operation or the second optimization operation.

15. The computer-readable medium of claim 14 , wherein the respective measurement error increases quadratically as the measured distance from the vehicle increases.

16. The computer-readable medium of claim 11 , wherein the computer-readable medium further comprises program instructions stored thereon that are executable to cause the computing system to:

augment the second updated time-sequence of sensor-based positions to include one or more additional intermediate sensor-based positions between each pair of sensor-based positions in the second updated time-sequence of sensor-based positions.

17. The computer-readable medium of claim 16 , wherein the sensor data is captured by a stereo camera having a frame rate of 5 frames per second, and wherein augmenting the second updated time-sequence of sensor-based positions comprises increasing a frame rate of the second updated time-sequence of sensor-based positions from 5 frames per second to 30 frames per second.

18. The computer-readable medium of claim 11 , wherein beginning with the sensor-based position having the highest confidence in the time-sequence of sensor-based positions and proceeding in the first direction comprises:

determining that one of (i) an earliest position in the time-sequence of sensor-based positions or (ii) a latest position in the time-sequence of sensor-based positions is the sensor-based position of the agent having the highest confidence in the time-sequence of sensor-based positions based on a measured distance of each respective sensor-based position from the vehicle; and

proceeding (i) chronologically forward or (ii) chronologically backward, respectively.

19. The computer-readable medium of claim 11 , wherein generating the first updated time-sequence of sensor-based positions of the agent comprises processing at least two or more sensor-based positions of the time-sequence of sensor-based positions by beginning with a position associated with an earliest observation in the time-sequence of sensor-based positions and proceeding chronologically forward; and

wherein generating the second updated time-sequence of sensor-based positions of the agent comprises processing at least two or more sensor-based positions of the first updated time-sequence of sensor-based positions by beginning with a latest sensor-based position and proceeding chronologically backward.

20. A computing system comprising:

at least one processor;

a non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is capable of:

receiving, at a computing system, sensor data captured by a camera-based sensor system associated with a vehicle in an environment;

processing, by the computing system, the received sensor data and thereby determining a sensor-based track for an agent that is identified within the sensor data, wherein the sensor-based track comprises a time-sequence of sensor-based positions of the agent;

generating, by the computing system, a first updated time-sequence of sensor-based positions of the agent by performing a first optimization operation on the time-sequence of sensor-based positions, wherein the first optimization operation includes processing at least two or more sensor-based positions of the time-sequence of sensor-based positions by beginning with a sensor-based position having a highest confidence in the time-sequence of sensor-based position observations and proceeding in a first direction;

after generating the first updated time-sequence of sensor-based positions, generating, by the computing system, a second updated time-sequence of sensor-based positions of the agent by performing a second optimization operation on the first updated time-sequence of sensor-based positions, wherein the second optimization operation includes processing at least two or more sensor-based positions of the first updated time-sequence of sensor-based positions in a second direction opposite the first direction;

deriving, by the computing system, a trajectory for the agent in the environment based on the second updated time-sequence of sensor-based positions of the agent; and

storing the derived trajectory, wherein the derived trajectory is thereafter utilized to perform a task that benefits from increased-accuracy trajectories.

Assignments (2)

Continuity (1)

References Cited (11)