Methods and systems for three dimensional object detection and localization

1. A method comprising:

causing, by a computing system, a radar unit to transmit radar signals into an environment of a vehicle;

receiving, at the computing system, radar reflections that represent reflections of the radar signals;

based on the radar reflections, determining a heading and a range for an object positioned in the environment, wherein the range represents a distance between the object and the vehicle;

receiving, at the computing system and from a vehicle camera, an image depicting a portion of the environment that includes the object;

based on the heading and the range for the object, removing peripheral areas of the image to generate an image patch that focuses upon the object;

providing the image patch and the heading and the range for the object as inputs into a neural network such that the neural network provides output parameters corresponding to the object, wherein the output parameters indicate a classification for the object, a center of the object, and a set of dimensions for the object from a perspective of the vehicle; and

controlling the vehicle based on the output parameters corresponding to the object.

2. The method of claim 1 , wherein causing the radar unit to transmit the radar signals into the environment of the vehicle comprises:

causing the radar unit to transmit radar signals having pulse-Doppler stretch linear frequency modulated (LFM) waveforms, wherein the radar unit is configured to use an antenna array to transmit the radar signals; and

wherein determining the heading and the range for the object positioned in the environment comprises:

processing the radar reflections using a mixing process based on the pulse-Doppler stretch LFM waveforms to determine the heading and the range for the object positioned in the environment.

3. The method of claim 1 , wherein causing the radar unit to transmit the radar signals into the environment of the vehicle comprises:

causing the radar unit to transmit the radar signals at a first time; and

wherein receiving the image depicting the portion of the environment that includes the object comprises:

receiving the image based on the vehicle camera capturing the image at a second time, wherein the second time is a threshold duration subsequent to the first time.

4. The method of claim 1 , wherein removing peripheral areas of the image to generate the image patch that focuses upon the object comprises:

identifying the object in the image based on the heading and the range for the object;

based on identifying the object, determining an outline of the object; and

removing peripheral areas of the image that represent portions of the environment outside the outline of the object.

5. The method of claim 1 , wherein controlling the vehicle based on the parameters corresponding to the object comprises:

based on the classification for the object, the center of the object, and the set of dimensions for the object from the perspective of the vehicle, determining a control strategy that enables the vehicle to avoid the object during subsequent navigation, and wherein the control strategy involves a buffer associated with avoiding the object that depends at least on the classification for the object; and

controlling the vehicle based on the control strategy.

6. The method of claim 1 , further comprising:

based on the classification for the object, determining that the object is a potential passenger;

responsive to determining that the object is the potential passenger, determining an orientation and a position of the potential passenger relative to the vehicle based on the center of the object and the set of dimensions for the object from the perspective of the vehicle; and

wherein controlling the vehicle based on the parameters corresponding to the object further comprises:

causing the vehicle to navigate at a particular speed toward the potential passenger based on the orientation the position of the potential passenger relative to the vehicle; and

stopping the vehicle proximate the potential passenger to enable the potential passenger to enter into the vehicle.

7. The method of claim 1 , wherein providing the image patch and the heading and the range for the object as inputs into the neural network such that the neural network provides output parameters corresponding to the object comprises:

based on pixels within the image patch, determining a first set of channels having a red pixel channel, a green pixel channel, and a blue pixel channel;

based on the heading and the range for the object, determining a second set of channels having a heading channel and a range channel; and

providing the first set of channels and the second set of channels as inputs into the neural network.

8. The method of claim 1 , wherein providing the image patch and the heading and the range for the object as inputs into the neural network such that the neural network provides output parameters corresponding to the object comprises:

providing the image patch and the heading and the range for the object positioned in the environment as inputs into a convolution neural network.

9. A system comprising:

a vehicle;

a radar unit coupled to the vehicle;

a vehicle camera;

a computing device configured to:

cause the radar unit to transmit radar signals into an environment of the vehicle;

receive radar reflections that represent reflections of the radar signals;

based on the radar reflections, determine a heading and a range for an object positioned in the environment, wherein the range represents a distance between the object and the vehicle;

receive, from the vehicle camera, an image depicting a portion of the environment that includes the object;

based on the heading and the range for the object, remove peripheral areas of the image to generate an image patch that focuses upon the object;

provide the image patch and the heading and the range for the object as inputs into a neural network such that the neural network provides output parameters corresponding to the object, wherein the output parameters indicate a classification for the object, a center of the object, and a set of dimensions for the object from a perspective of the vehicle; and

control the vehicle based on the output parameters corresponding to the object.

10. The system of claim 9 , wherein the computing device is further configured to:

cause the radar unit to transmit the radar signals at a first time; and

receive the image based on the vehicle camera capturing the image at a second time, wherein the second time is a threshold duration subsequent to the first time.

11. The system of claim 9 , wherein the computing device is further configured to:

identify the object in the image based on the heading and the range for the object;

based on identifying the object, determine an outline of the object; and

remove peripheral areas of the image that represent portions of the environment outside the outline of the object.

12. The system of claim 9 , wherein the computing device is further configured to:

based on the classification for the object, the center of the object, and the set of dimensions for the object from the perspective of the vehicle, determine a control strategy that enables the vehicle to avoid the object during subsequent navigation, and wherein the control strategy involves a buffer associated with avoiding the object that depends at least on the classification for the object; and

control the vehicle based on the control strategy.

13. The system of claim 12 , wherein the computing device is further configured to:

based on the classification for the object, determine that the object is a potential passenger;

responsive to determining that the object is the potential passenger, determine an orientation and a position of the potential passenger relative to the vehicle based on the center for the object and the set of dimensions for the object from the perspective of the vehicle;

cause the vehicle to navigate at a particular speed toward the potential passenger based on the orientation the position of the potential passenger relative to the vehicle; and

stop the vehicle proximate the potential passenger to enable the potential passenger to enter into the vehicle.

14. The system of claim 9 , wherein the neural network is a convolution neural network.

15. A non-transitory computer-readable medium configured to store instructions, that when executed by a computing system comprising one or more processors, causes the computing system to perform operations comprising:

causing a radar unit to transmit radar signals into an environment of a vehicle;

receiving radar reflections that represent reflections of the radar signals;

based on the radar reflections, determining a heading and a range for an object positioned in the environment, wherein the range represents a distance between the object and the vehicle;

receiving, from a vehicle camera, an image depicting a portion of the environment that includes the object;

based on the heading and the range for the object, removing peripheral areas of the image to generate an image patch that focuses upon the object;

providing the image patch and the heading and the range for the object as inputs into a neural network such that the neural network provides output parameters corresponding to the object, wherein the output parameters indicate a classification for the object, a center of the object, and a set of dimensions for the object from a perspective of the vehicle; and

providing control instructions for the vehicle based on the output parameters corresponding to the object.