Granted Patent B2

US 10,943,355 · App. 16/299,527 · Granted Mar 9, 2021

Inventors: Ashton James Fagg (Pittsburgh, PA); Narek Melik-Barkhudarov (Pittsburgh, PA)

Assignee: UATC, LLC

G06T7/248G05D1/0246G06K9/00791G06K9/629G06T2207/10028G06T2207/30252

Abstract

Claims (77)

1. A computer-implemented method for movement detection, the method comprising:

obtaining, by a computing system including one or more computing devices, image data and range data representing a scene external to an autonomous vehicle, the image data including at least a first image and a second image;

determining, by the computing system, a first time associated with the first image and a second time associated with the second image, based at least in part on the image data;

identifying, by the computing system, one or more first range values in the range data that are associated with the first time, and one or more second range values in the range data that are associated with the second time;

associating, by the computing system, the one or more first range values with the first image, and the one or more second range values with the second image to synchronize the image data and the range data;

identifying, by the computing system, a set of corresponding image features from the image data, the set of corresponding image features including a first feature in the first image having a correspondence with a second feature in the second image;

determining, by the computing system, a respective distance for each of the first feature in the first image and the second feature in the second image based at least in part on the range data; and

determining, by the computing system, a velocity associated with a portion of a scene represented by the set of corresponding image features based at least in part on the respective distance for the first feature and the respective distance for the second feature.

2. The computer-implemented method of claim 1 , wherein identifying the set of corresponding image features from the image data comprises:

identifying, by the computing system, one or more keypoints based at least in part on the image data; and

generating, by the computing system, the first feature and the second feature from the image data based at least in part on the one or more keypoints.

3. The computer-implemented method of claim 2 , wherein identifying the one or more keypoints comprises:

identifying, by the computing system, a plurality of points in the first image and the second image that satisfy one or more appearance threshold criteria; and

determining, by the computing system, the one or more keypoints based at least in part on the plurality of points and the range data.

4. The computer-implemented method of claim 1 , wherein identifying the set of corresponding image features from the image data comprises:

determining, by the computing system, that the first feature and the second feature satisfy one or more matching criteria.

5. The computer-implemented method of claim 1 , wherein the first feature corresponds to one or more pixels in the first image, the second feature corresponds to one or more pixels in the second image, and determining the respective distance for each of the first feature in the first image and the second feature in the second image comprises:

generating, by the computing system, a set of two-dimensional matched points based at least in part on the one or more pixels in the first image and the one or more pixels in the second image; and

determining, by the computing system, the respective distance for the first feature in the first image and the respective distance for the second feature in the second image based at least in part on the set of two-dimensional matched points.

6. The computer-implemented method of claim 1 , the method further comprising:

generating, by the computing system, data indicative of one or more movement detections based at least in part on the velocity associated with the portion of the scene represented by the set of corresponding image features.

7. The computer-implemented method of claim 6 , wherein generating the data indicative of one or more movement detections comprises:

determining, by the computing system, a position of the first feature with respect to an image sensor used to capture the first image, and a position of the second feature with respect to an image sensor used to capture the second image.

8. The computer-implemented method of claim 6 , wherein the image data includes a geometry of one or more image sensors used to capture the first image and the second image, and generating the data indicative of one or more movement detections comprises:

determining, by the computing system, a continuous coordinate space corresponding to the first image and the second image, based at least in part on the geometry of the one or more image sensors; and

determining, by the computing system, a position of the first feature in the continuous coordinate space at a first time, and a position of the second feature in the continuous coordinate space at a second time.

9. The computer-implemented method of claim 6 , wherein generating the data indicative of one or more movement detections comprises:

performing, by the computing system, a verification of the velocity associated with the position of the scene represented by the set of corresponding image features based at least in part on one or more verification criteria; and

providing, by the computing system, the velocity associated with the position of the scene represented by the set of corresponding image features to a tracking system based at least in part on the verification.

10. The computer-implemented method of claim 1 , wherein the first image and the second image are consecutive in time.

11. The computer-implemented method of claim 1 , wherein the range data is obtained from one or more range sensors and includes a plurality of range values indicative of a distance to one or more points in the scene at one or more times.

12. A computing system, comprising:

one or more processors; and

a computer-readable medium having instructions stored thereon that, when executed by the one or more processors, cause the computing system to perform one or more operations, the operations comprising:

obtaining image data and range data representing a scene, the image data including at least a first image and a second image that depict one or more portions of the scene;

determining a first time associated with the first image and a second time associated with the second image, based at least in part on the image data;

identifying one or more first range values in the range data that are associated with the first time and one or more second range values in the range data that are associated with the second time;

associating the one or more first range values with the first image, and the one or more second range values with the second image to synchronize the image data and the range data;

determining state data based at least in part on the image data and the range data, the state data including at least a first state corresponding to the first image and a second state corresponding to the second image;

determining flow data indicative of a flow between the first state and the second state; and

determining position data for the one or more portions based at least in part on the state data and the flow data, the position data including a first position and a second position of the one or more portions.

13. The computing system of claim 12 , wherein determining the state data based at least in part on the image data and the range data comprises:

determining a first range map and a second range map based at least in part on the range data, the first range map corresponding to the first image and the second range map corresponding to the second image;

identifying a first set of range values from the range data and a second set of range values from the range data based at least in part on an interest-level threshold criteria;

determining one or more first keypoints and one or more second keypoints based at least in part on the first set of range values and the second set of range values, respectively, and the image data, the one or more first keypoints corresponding to the first image and the one or more second keypoints corresponding to the second image;

determining the first state based at least in part on the first range map, the first set of range values, and the one or more first keypoints; and

determining the second state based at least in part on the second range map, the second set of range values, and the one or more second keypoints.

14. The computing system of claim 12 , wherein determining the state data based at least in part on the image data and the range data comprises:

obtaining previously determined data indicative of the first state, the previously determined data being previously determined based at least in part on the first image and the range data;

determining a second range map based at least in part on the range data, the second range map corresponding to the second image;

identifying a second set of range values from the range data based at least in part on an interest-level threshold criteria;

determining one or more second keypoints based at least in part on the second set of range values and the image data, the one or more second keypoints corresponding to the second image; and

determining the second state based at least in part on the second range map, the second set of range values, and the one or more second keypoints.

15. The computing system of claim 12 , wherein determining the flow data indicative of the flow between the first state and the second state comprises:

extracting one or more first features from the first image, and one or more second features from the second image;

identifying one or more matching features among the one or more first features and the one or more second features; and

determining a range value associated with each of the one or more matching features.

16. The computing system of claim 12 , wherein determining the position data for the one or more portions, the position data including the first position and the second position of the one or more portions comprises:

determining the first position with respect to an image sensor used to capture the first image, the first position including a range value and an azimuth value relative to a center of the image sensor; and

determining the second position with respect to an image sensor used to capture the second image, the second position including range value and an azimuth value relative to a center of the image sensor.

17. The computing system of claim 12 , wherein determining the position data for the one or more portions, the position data including the first position and the second position of the one or more portions comprises:

determining a continuous coordinate space corresponding to the first image and the second image, based at least in part on a geometry of one or more image sensors used to capture the first image and the second image; and

determining the first position in the continuous coordinate space at a first time, and the second position in the continuous coordinate space at a second time.

18. The computing system of claim 12 , wherein the operations further comprise:

generating log data associated with determining the position data for the one or more portions;

determining an existence of one or more error conditions based at least in part on the log data; and

generating a control signal in response to the existence of the one or more error conditions to clear a processing pipeline associated with detecting a movement associated with the one or more portions.

19. An autonomous vehicle, comprising:

one or more processors; and

a computer-readable medium having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform one or more operations, the operations comprising:

obtaining image data and range data representing a scene external to an autonomous vehicle, the image data including at least a first image and a second image that depict the scene;

determining a first time associated with the first image and a second time associated with the second image, based at least in part on the image data;

identifying one or more first range values in the range data that are associated with the first time and one or more second range values in the range data that are associated with the second time;

associating the one or more first range values with the first image, and the one or more second range values with the second image to synchronize the image data and the range data;

identifying a set of corresponding image features from the image data, the set of corresponding image features including a first feature in the first image having a correspondence with a second feature in the second image;

determining a respective distance for each of the first feature in the first image and the second feature in the second image based at least in part on the range data; and

determining a velocity associated with a portion of a scene represented by the set of corresponding image features based at least in part on the respective distance for the first feature and the respective distance for the second feature.

Assignments (8)