Crowd-sourced 3D points and point cloud alignment
Systems and methods are provided for vehicle navigation. In one implementation, a host vehicle-based sparse map feature harvester system may include at least one processor programmed to receive a plurality of images captured by a camera onboard the host vehicle as the host vehicle travels along a road segment in a first direction, wherein the plurality of images are representative of an environment of the host vehicle; detect one or more semantic features represented in one or more of the plurality of images, the one or more semantic features each being associated with a predetermined object type classification; identify at least one position descriptor associated with each of the detected one or more semantic features; identify three-dimensional feature points associated with one or more detected objects represented in at least one of the plurality of images; receive position information, for each of the plurality of images, wherein the position information is indicative of a position of the camera when each of the plurality of images was captured; and cause transmission of drive information for the road segment to an entity remotely-located relative to the host vehicle, wherein the drive information includes the identified at least one position descriptor associated with each of the detected one or more semantic features, the identified three-dimensional feature points, and the position information.
1 . A host vehicle-based sparse map feature harvester system, comprising:
at least one processor comprising circuitry and a memory, wherein the memory includes instructions that when executed by the circuitry cause the at least one processor to:
receive a plurality of images captured by a camera onboard the host vehicle as the host vehicle travels along a road segment in a first direction, wherein the plurality of images are representative of an environment of the host vehicle;
detect one or more semantic features represented in one or more of the plurality of images, the one or more semantic features each being associated with a predetermined object type classification;
assign a predetermined object classification to each of the one or more semantic features;
receive position information, for each of the plurality of images, wherein the position information is indicative of a position of the camera when each of the plurality of images was captured; and
cause transmission of drive information for the road segment to an entity remotely-located relative to the host vehicle, wherein the drive information includes the predetermined object type classification assigned to each of the one or more semantic features, the identified at least one position descriptor associated with each of the detected one or more semantic features, a trajectory traveled by the host vehicle along the road segment in the first direction, and the position information, wherein the remotely-located entity includes one or more processors configured to:
receive, in addition to the drive information transmitted by the host vehicle, drive information for the road segment from each of a plurality of other vehicles traveling in a direction opposite to the host vehicle, and
analyze one or more aspects of the drive information from the host vehicle and the drive information from the plurality of other vehicles to determine a transformation for aligning three-dimensional feature points associated with one or more detected objects represented in at least one of the plurality of images with corresponding three-dimensional feature points associated with objects represented in images included in the drive information for the road segment from each of the plurality of other vehicles traveling in the direction opposite to the host vehicle.
2 . The system of claim 1 , wherein each of the three-dimensional feature points is determined based on analysis of representations of the one or more detected objects across two or more of the plurality of images and based on an ego motion of the host vehicle, the ego motion being determined based on at least one of the plurality of images or an output of at least one ego motion sensor, the ego motion sensor including at least one of a speedometer, an accelerometer, or a GPS receiver.
3 . The system of claim 1 , wherein at least some of the one or more detected objects are associated with the one or more detected semantic features.
4 . The system of claim 1 , wherein the at least one position descriptor associated with each of the detected one or more semantic features includes at least one of a two dimensional image position or a three-dimensional point location.
5 . The system of claim 4 , wherein the at least one position descriptor associated with each of the detected one or more semantic features includes the three-dimensional point location and wherein the three-dimensional point location is determined based on analysis of representations of the one or more detected objects across two or more of the plurality of images and based on an ego motion of the host vehicle, the ego motion being determined based on at least one of the plurality of images or an output of at least one ego motion sensor, the ego motion sensor including at least one of a speedometer, an accelerometer, or a GPS receiver.
6 . The system of claim 1 , wherein the one or more processors of the remotely-located entity are further configured to generate a sparse map based on the aligned drive information from the host vehicle and the drive information from the plurality of other vehicles.
7 . The system of claim 1 , wherein for the road segment, a number of identified position descriptors associated with detected semantic features is greater than a number of identified three-dimensional points.
8 . The system of claim 1 , wherein for the road segment, the memory further includes instructions that when executed by the circuitry cause the at least one processor to identify and transmit to the remotely-located entity no more than between 1 and 40 three-dimensional feature points per meter of the road segment.
9 . The system of claim 1 , wherein the three-dimensional feature points identified by the at least one processor are associated with one or more of edges or corners of at least one surface associated the at least some of the one or more detected objects.
10 . The system of claim 1 , wherein each of the three-dimensional feature points includes at least one of: an X-Y position relative to at least one of the plurality of images along with an indicator of range relative to the camera, or an X-Y-Z location relative to a real world origin.
11 . The system of claim 10 , wherein the indicator of range is determined based on: tracking across two or more of the plurality of images of an image position associated with an identified object; and an ego motion signal of the host vehicle.
12 . The system of claim 1 , wherein the position information includes at least one indicator of position determined based on: an output of a GPS sensor associated with the host vehicle, an ego motion of the host vehicle, or a combination of the output of the GPS sensor associated with the host vehicle and the one ego motion of the host vehicle.
13 . A method for harvesting sparse map features by a host vehicle, the method comprising:
receiving a plurality of images captured by a camera onboard the host vehicle as the host vehicle travels along a road segment in a first direction, wherein the plurality of images are representative of an environment of the host vehicle;
detecting one or more semantic features represented in one or more of the plurality of images, the one or more semantic features each being associated with a predetermined object type classification;
assigning a predetermined object classification to each of the one or more semantic features;
receiving position information, for each of the plurality of images, wherein the position information is indicative of a position of the camera when each of the plurality of images was captured; and
causing transmission of drive information for the road segment to an entity remotely-located relative to the host vehicle, wherein the drive information includes the predetermined object type classification assigned to each of the one or more semantic features, the identified at least one position descriptor associated with each of the detected one or more semantic features, a trajectory traveled by the host vehicle along the road segment in the first direction, and the position information, wherein the remotely-located entity includes one or more processors configured to:
receive, in addition to the drive information transmitted by the host vehicle, drive information for the road segment from each of a plurality of other vehicles traveling in a direction opposite to the host vehicle, and
analyze one or more aspects of the drive information from the host vehicle and the drive information from the plurality of other vehicles to determine a transformation for aligning three-dimensional feature points associated with one or more detected objects represented in at least one of the plurality of images with corresponding three-dimensional feature points associated with objects represented in images included in the drive information for the road segment from each of the plurality of other vehicles traveling in the direction opposite to the host vehicle.
14 . The method of claim 13 , wherein each of the three-dimensional feature points is determined based on analysis of representations of the one or more detected objects across two or more of the plurality of images and based on an ego motion of the host vehicle, the ego motion being determined based on at least one of the plurality of images or an output of at least one ego motion sensor, the ego motion sensor including at least one of a speedometer, an accelerometer, or a GPS receiver.
15 . The method of claim 13 , wherein at least some of the one or more detected objects are associated with the one or more detected semantic features.
16 . A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, are configured to cause at least one processor to perform a method for harvesting sparse map features by a host vehicle, the method comprising:
receiving a plurality of images captured by a camera onboard the host vehicle as the host vehicle travels along a road segment in a first direction, wherein the plurality of images are representative of an environment of the host vehicle;
detecting one or more semantic features represented in one or more of the plurality of images, the one or more semantic features each being associated with a predetermined object type classification;
assigning a predetermined object classification to each of the one or more semantic features;
receiving position information, for each of the plurality of images, wherein the position information is indicative of a position of the camera when each of the plurality of images was captured; and
causing transmission of drive information for the road segment to an entity remotely-located relative to the host vehicle, wherein the drive information includes the predetermined object type classification assigned to each of the one or more semantic features, the identified at least one position descriptor associated with each of the detected one or more semantic features, a trajectory traveled by the host vehicle along the road segment in the first direction, and the position information, wherein the remotely-located entity includes one or more processors configured to:
receive, in addition to the drive information transmitted by the host vehicle, drive information for the road segment from each of a plurality of other vehicles traveling in a direction opposite to the host vehicle, and
analyze one or more aspects of the drive information from the host vehicle and the drive information from the plurality of other vehicles to determine a transformation for aligning three-dimensional feature points associated with one or more detected objects represented in at least one of the plurality of images with corresponding three-dimensional feature points associated with objects represented in images included in the drive information for the road segment from each of the plurality of other vehicles traveling in the direction opposite to the host vehicle.
17 . The non-transitory computer-readable medium of claim 16 , wherein the three-dimensional feature points are associated with one or more of edges or corners of at least one surface associated the at least some of the one or more detected objects.
18 . The non-transitory computer-readable medium of claim 16 , wherein each of the three-dimensional feature points includes at least one of: an X-Y position relative to at least one of the plurality of images along with an indicator of range relative to the camera, or an X-Y-Z location relative to a real world origin.