Granted Patent B2

US 12,365,347 · App. 18/663,046 · Granted Jul 22, 2025

Inventors: Rama Venkata Surya Kumar Varma Bhupatiraju (Fremont, CA); Deepak Rajasekhar Karishetti (Dublin, CA); Benjamin M. Gatten (Pleasanton, CA); Sanket Goyal (Pleasanton, CA)

Assignee: Zimeno Inc.

B60W40/10A01B69/001A01B69/004A01B69/008B62D15/025G06V20/56B60W2420/403B60W2520/14B60W2552/15B60W2552/53G05D1/0088G05D1/227G06T2207/10028

Abstract

Claims (52)

1. A vehicle row follow system comprising:

a vehicle comprising at least one sensor configured to output signals serving as a basis for a three-dimensional (3D) point cloud and to output signals corresponding to a two-dimensional (2D) image of a region forward the vehicle;

a processor; and

a non-transitory computer-readable medium containing instructions to direct the processor to:

output control signals to the at least one sensor to capture a particular 2D image of a region forward the vehicle;

output control signals to the at least one sensor to output the signals serving as a basis for a particular 3D point cloud of the region forward the vehicle;

determine first estimated plant row lines in the 2D image using feature extraction and texture;

determine second estimated plant row lines in the 2D image using a fully convolutional neural network (FCN) and training images and the particular 3D point cloud;

determine a first yaw estimate of the vehicle based upon a slope of a heading line relative to a centerline between the first estimated plant row lines in the 2D image;

determine a second yaw estimate of the vehicle based upon a slope of a heading line relative to a centerline between the second estimated plant row lines in the 2D image;

identify consecutive plant rows in the particular 3D point cloud;

determine a first lateral offset of the vehicle from a centerline between the consecutive plant rows based upon the identity of the consecutive plant rows in the 3D point cloud;

determine a second lateral offset of the vehicle from a centerline between the consecutive plant rows based upon the identity of the consecutive plant rows in the 3D point cloud using the FCN;

obtain wheel odometry and inertial measurement unit signals;

determine a third yaw estimate of the vehicle and a third lateral offset of the vehicle based upon the wheel odometry and the inertial measurement unit signals;

determine a final yaw estimate of the vehicle of the vehicle and the final lateral offset by fusing of the first yaw estimate, the second yaw estimate and the third yaw estimate and by fusing of the first lateral offset, the second lateral offset and the third lateral offset using a Kalman filter; and

output steering control signals based upon the final yaw estimate and final lateral offset of the vehicle.

2. The vehicle row follow system of claim 1 , wherein the at least one sensor comprises a stereo camera.

3. The vehicle row follow system of claim 1 , wherein the instructions are to direct the processor to use a histogram and a vanishing point to determine the plant row lines in the 2D image.

4. The vehicle row follow system of claim 1 , wherein the instructions are to direct the processor to identify the consecutive plant rows in the particular 3D point cloud based upon the determined first estimated plant row lines in the 2D image or based upon the second estimated plant row lines in the 2D image.

5. The vehicle row follow system of claim 1 , wherein the fusing is with an extended Kalman filter.

6. A vehicle row following method comprising:

acquiring a two-dimensional (2D) image of a region forward a vehicle, the image depicting consecutive plant rows;

acquiring a three-dimensional (3D) point cloud of the region;

determining first estimated plant row lines corresponding to consecutive plant rows in the 2D image;

determining a first yaw estimate of the vehicle based upon a slope of a heading line relative to a centerline between the first estimated plant row lines in the 2D image;

determining second estimated plant row lines in the 2D image using a fully convolutional neural network (FCN) and training images and the particular 3D point cloud;

determine a second yaw estimate of the vehicle based upon a slope of a heading line relative to a centerline between the second estimated plant row lines in the 2D image;

identifying the consecutive plant rows in the 3D point cloud;

determine a first lateral offset of the vehicle from a centerline between the consecutive plant rows based upon the identity of the consecutive plant rows in the 3D point cloud;

determine a second lateral offset of the vehicle from a centerline between the consecutive plant rows based upon the identity of the consecutive plant rows in the 3D point cloud using the FCN;

obtaining wheel odometry and inertial measurement unit signals;

determining a third yaw estimate of the vehicle and a third lateral offset of the vehicle based upon the wheel odometry and the inertial measurement unit signals;

determining a final yaw estimate of the vehicle of the vehicle and the final lateral offset by fusing of the first yaw estimate, the second yaw estimate and the third yaw estimate and by fusing of the first lateral offset, the second lateral offset and the third lateral offset using a Kalman filter; and

outputting steering control signals based upon the final yaw estimate and final lateral offset of the vehicle;

determining a first lateral offset of the vehicle from the centerline between the consecutive plant rows based upon the identity of the consecutive plant rows in the 3D point cloud; and

outputting steering control signals based upon the determined yaw and lateral offset of the vehicle.

7. The vehicle row follow method of claim 6 , wherein the 2D image and the 3D image are acquired with a stereo camera.

8. The vehicle row follow method of claim 6 , wherein a histogram and a vanishing point are used to determine the plant row lines in the 2D image.

9. A non-transitory computer-readable medium containing instructions to direct a processor, the instructions comprising:

2D image acquisition instructions for acquiring a two-dimensional image of a region forward a vehicle, the image depicting consecutive plant rows;

3D point cloud acquisition instructions for acquiring a three-dimensional (3D) point cloud of the region;

row line instructions for determining a slope of the consecutive plant rows in the 2D image using a fully convolutional neural network and training images;

vehicle yaw instructions for determining a yaw of the vehicle based upon a slope of a heading line relative to a centerline between the consecutive plant rows in the 2D image;

point cloud row identifying instructions for identifying the consecutive plant rows in the 3D point cloud;

lateral offset instructions for determining a lateral offset of the vehicle from the centerline between the consecutive plant rows based upon the identity of the consecutive plant rows in the 3D point cloud; and

steering response instructions for outputting steering control signals based upon the determined yaw and lateral offset of the vehicle.

10. The medium of claim 9 further comprising adjustment instructions for:

obtaining wheel odometry and inertial measurement unit signals;

determining a second yaw of the vehicle with respect to the centerline and a second lateral offset of the vehicle based upon the wheel odometry and the inertial measurement unit signals;

adjusting the determined yaw of the vehicle and the determined lateral offset of the vehicle based upon the second yaw of the vehicle and the second lateral offset.

11. The medium of claim 9 , wherein the row line instructions to direct the processor to identify the consecutive plant rows in the 3D point cloud based upon the determined slope of the consecutive plant rows in the 2D image.

Assignments (1)

Continuity (2)

References Cited (60)