IP Library Granted Patent US 12,158,528
Granted Patent B2
US 12,158,528 · App. 17/490,176 · Granted Dec 3, 2024

Plant and/or vehicle locating

Inventors: Sanket Goyal (Pleasanton, CA); Christopher Travis Whitney (Palo Alto, CA)
Assignee: Zimeno Inc.
G01S19/14G06V20/56G06V30/194A01G7/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,158,528
App. No.
17/490,176
Granted
Dec 3, 2024
Kind
B2
Abstract

A plant locating system may include a vehicle supporting a Global Positioning System (GPS) antenna and a monocular camera. The system may further include a plant locating unit comprising a processing unit and a non-transitory computer-readable medium containing instructions to direct the processing unit to: acquire a sample image of a plant of interest captured at a time with the monocular camera at an unknown distance from the plant of interest (POI); determine a geographic location estimate of the GPS antenna at the time; identify a selected portion of the sample image comprising the POI; determine a distance between the POI and the monocular camera based upon the selected portion; and determine a geographic location estimate of the POI based on the geographic location estimate of the GPS antenna at the time and the determined distance between the monocular camera and the POI.

Claims (84)

1. A plant locating system comprising:

a vehicle supporting a Global Positioning System (GPS) antenna and a monocular camera facing in a sideways direction from the vehicle;

a plant locating unit comprising:

a processing unit; and

a non-transitory computer-readable medium containing instructions to direct the processing unit to:

acquire a sample image of a plant of interest captured at a time with the monocular camera at an unknown distance from the plant of interest;

determine a geographic location estimate of the GPS antenna at the time, the selected portion;

identify a selected portion of the sample image comprising the plant of interest;

determine a distance between the plant of interest and the monocular camera based upon a size of the selected portion and a number of pixels within the selected portion; and

determine a geographic location estimate of the plant of interest based on the geographic location estimate of the GPS antenna at the time and the determined distance between the monocular camera and the plant of interest.

2. The plant locating system of claim 1 further comprising a set of training images comprising images captured by the monocular camera, each of the training images comprising a training portion, wherein the processing unit identifies the selected portion of the sample image based upon the training portion of each of the training images.

3. The plant locating system of claim 2 , wherein the training portion of each of the training images and the selected portion of the sample image each comprise a window containing and larger than the plant of interest.

4. The plant locating system of claim 3 , wherein the training portion of each of the training images is human defined.

5. The plant locating system of claim 1 , wherein the selected portion comprises a window that is larger than and contains the plant of interest in the sample image and wherein the distance between the plant of interest in the monocular camera is based upon a size of the window and a number of image pixels within the window.

6. The plant locating system of claim 5 , wherein the window extends from a basal end of a stem of the plant of interest to where the plant of interest branches outwardly from the stem of the plant of interest.

7. The plant locating system of claim 1 , wherein the processing unit and the non-transitory computer-readable medium form a neural network that learns how to identify the selected portion of the sample image comprising the plant of interest from a corresponding training portion in each image of a training set of images of plants captured by the monocular camera.

8. The plant locating system of claim 1 , wherein the instructions are to direct the processing unit to determine a second geographic location estimate of a second plant of interest and to form a map comprising the geographic location estimate of the plant of interest and the second geographic location estimate of the second plant of interest.

9. The plant locating system of claim 1 , wherein the instructions are to direct the processing unit to:

acquire a second sample image of the plant of interest captured at a second time with the monocular camera at an unknown distance from the plant of interest;

determine a second geographic location estimate of the GPS antenna at the second time;

identify a selected portion of the second sample image comprising the plant of interest;

determine a second distance between the plant of interest and the monocular camera based upon the second selected portion; and

determine a second geographic location estimate of the plant of interest based on the geographic location estimate of the GPS antenna at the second time and the determined distance between the monocular camera and the plant of interest; and

determine a third geographic location estimate of the plant of interest based upon a combination of the geographic location estimate of the plant of interest and the second geographic location estimate of the plant of interest.

10. The plant locating system of claim 1 , wherein the monocular camera is vertically spaced above lowermost ground engaging portions of the vehicle by distance of no greater than 110 cm.

11. The plant locating system of claim 1 , wherein the instructions are to direct the processing unit to:

acquire a second sample image of a second plant of interest captured at a second time with the monocular camera at an unknown distance from the plant of interest, wherein the plant of interest of the second plant of interest are part of a row of plants;

determine a second geographic location estimate of the GPS antenna at the second time;

identify a second selected portion of a second sample image comprising the second plant of interest;

determine a distance between the second plant of interest and the monocular camera based upon the second selected portion;

determine a second geographic location estimate of the second plant of interest based on the geographic location estimate of the GPS antenna at the second time and the determined distance between the monocular camera and the second plant of interest; and

determine existence of a plant omission between the plant of interest and of the second plant of interest based on the geographic location estimate of the plant of interest and the second geographic location estimate of the second plant of interest.

12. The plant locating system of claim 11 , wherein the instructions are to direct the processing unit to form a map comprising the geographic location estimate for the plant of interest, the second geographic location estimate for the second plant of interest and a third geographic location for the plant omission.

13. The plant locating system of claim 1 , wherein the instructions are to direct the processing unit to output control signals to position the vehicle relative to the plant of interest based upon the geographic location estimate for the plant of interest.

14. The plant locating system of claim 1 , wherein the instructions are to direct the processing unit to output control signals to carry out an operation on the plant of interest based upon the geographic location estimate for the plant of interest.

15. The plant locating system of claim 1 , wherein the instructions are to direct the processing unit to form an evaluation of the plant of interest and to associate the evaluation with the geographic location estimate for the plant of interest.

16. A plant locating system comprising:

a vehicle supporting a Global Positioning System (GPS) antenna and a monocular camera facing in a sideways direction from the vehicle;

a plant locating unit comprising:

a processing unit; and

a non-transitory computer-readable medium containing instructions to direct the processing unit to:

acquire a sample image of a plant of interest captured at a time with the monocular camera at an unknown distance from the plant of interest;

determine a geographic location estimate of the GPS antenna at the time, the selected portion;

identify a selected portion of the sample image comprising the plant of interest;

determine a distance between the plant of interest and the monocular camera based the selected portion; and

determine a geographic location estimate of the plant of interest based on the geographic location estimate of the GPS antenna at the time and the determined distance between the monocular camera and the plant of interest, wherein the plant of interest is in a first row of plants, wherein sample image comprises depicts the plant of interest and a second plant of interest in a second row of plants and wherein the instructions are to direct the processing unit to:

identify a second selected portion of the sample image comprising the second plant of interest;

determine a distance between the second plant of interest and the monocular camera based upon the second selected portion; and

determine a geographic location estimate of the second plant of interest based on the geographic location estimate of the GPS antenna at the time and the determined distance between the monocular camera and the second plant of interest.

17. The plant locating system of claim 16 , wherein the selected portion comprises a first window having a first size and wherein the second selected portion comprises a second window having a second size different than the first size, wherein differences in size between the window and the second window are a function of different distances from the monocular camera.

18. The plant locating system of claim 17 , wherein the distance between the plant of interest and the monocular camera is based upon the first size of the first window and a number of image pixels within the first window and wherein the distance between the second plant of interest and the monocular camera is based upon the second size of the second window and a number of image pixels within the second window.

19. A plant locating system comprising:

a vehicle supporting a Global Positioning System (GPS) antenna and a monocular camera facing in a sideways direction from the vehicle;

a plant locating unit comprising:

a processing unit; and

a non-transitory computer-readable medium containing instructions to direct the processing unit to:

acquire a sample image of a plant of interest captured at a time with the monocular camera at an unknown distance from the plant of interest;

determine a geographic location estimate of the GPS antenna at the time, the selected portion;

identify a selected portion of the sample image comprising the plant of interest;

determine a distance between the plant of interest and the monocular camera based the selected portion;

determine a geographic location estimate of the plant of interest based on the geographic location estimate of the GPS antenna at the time and the determined distance between the monocular camera and the plant of interest,

acquire a second sample image of the plant of interest captured at a second time with the monocular camera at an unknown distance from the plant of interest;

determine a second geographic location estimate of the GPS antenna at the second time;

identify a selected portion of the second sample image comprising the plant of interest;

determine a second distance between the plant of interest and the monocular camera based upon the second selected portion; and

determine a second geographic location estimate of the plant of interest based on the geographic location estimate of the GPS antenna at the second time and the determined distance between the monocular camera and the plant of interest; and

determine a third geographic location estimate of the plant of interest based upon a combination of the geographic location estimate of the plant of interest and the second geographic location estimate of the plant of interest.

20. A plant locating system comprising:

a vehicle supporting a Global Positioning System (GPS) antenna and a monocular camera facing in a sideways direction from the vehicle;

a plant locating unit comprising:

a processing unit; and

a non-transitory computer-readable medium containing instructions to direct the processing unit to:

acquire a sample image of a plant of interest captured at a time with the monocular camera at an unknown distance from the plant of interest;

determine a geographic location estimate of the GPS antenna at the time, the selected portion;

identify a selected portion of the sample image comprising the plant of interest;

determine a distance between the plant of interest and the monocular camera based the selected portion;

determine a geographic location estimate of the plant of interest based on the geographic location estimate of the GPS antenna at the time and the determined distance between the monocular camera and the plant of interest,

acquire a second sample image of a second plant of interest captured at a second time with the monocular camera at an unknown distance from the plant of interest, wherein the plant of interest of the second plant of interest are part of a row of plants;

determine a second geographic location estimate of the GPS antenna at the second time;

identify a second selected portion of a second sample image comprising the second plant of interest;

determine a distance between the second plant of interest and the monocular camera based upon the second selected portion;

determine a second geographic location estimate of the second plant of interest based on the geographic location estimate of the GPS antenna at the second time and the determined distance between the monocular camera and the second plant of interest; and

determine existence of a plant omission between the plant of interest and of the second plant of interest based on the geographic location estimate of the plant of interest and the second geographic location estimate of the second plant of interest.

21. The plant locating system of claim 20 , wherein the instructions are to direct the processing unit to form a map comprising the geographic location estimate for the plant of interest, the second geographic location estimate for the second plant of interest and a third geographic location for the plant omission.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 20, 2026
From: ZIMENO INC.
To: CATERPILLAR INC.
Reel/Frame 074144/0732 →
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME AND DOCUMENTATION PREVIOUSLY RECORDED ON REEL 57654 FRAME 615. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Oct 21, 2024
From: GOYAL, SANKET; WHITNEY, CHRISTOPHER TRAVIS
To: ZIMENO INC.
Reel/Frame 069210/0078 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 30, 2021
From: GOYAL, SANKET; WHITNEY, CHRISTOPHER TRAVIS
To: ZIMENO, INC. DBA MONARCH TRACTOR
Reel/Frame 057654/0615 →
Continuity (1)
Related Publication 20230095661A1 · Mar 30, 2023