Granted Patent B2

US 10,845,810 · App. 16/539,390 · Granted Nov 24, 2020

Inventors: Arthur Flajolet (San Francisco, CA); Eric Stahl-David (San Francisco, CA); Sébastien Boyer (San Francisco, CA); Thomas Palomares (San Francisco, CA)

Assignee: FarmWise Labs, Inc.

G05D1/0094A01B69/004A01B79/005G05D1/0219G05D1/0246G06T7/579G05D2201/0201G06T2207/10028

Abstract

Claims (104)

1. A method for detecting real lateral locations of target plants relative to an autonomous machine comprises, at the autonomous machine:

at a first time, recording a first image of a first ground area at a ground-facing camera arranged on the autonomous machine;

detecting a first target plant in the first image;

accessing a lateral pixel location of the first target plant in the first image;

at a second time, for each tool module in a set of tool modules in contact with a plant bed surface and arranged behind the ground-facing camera relative to a direction of forward motion of the autonomous machine:

recording a first extension distance of the tool module; and

recording a first lateral position of the tool module relative to the ground-facing camera;

for each tool module in the set of tool modules, defining a contact point of the tool module at an intersection of the first extension distance of the tool module and the first lateral position of the tool module to generate a set of contact points;

interpolating between the set of contact points to generate the first depth profile of the plant bed surface;

estimating a real lateral location of the first target plant relative to the autonomous machine based on the lateral pixel location of the first target plant and the first depth profile of the plant bed surface; and

driving a first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the first target plant.

2. The method of claim 1 , further comprising:

autonomously navigating along a first crop row comprising the first ground area; and

in response to lateral alignment of the lateral position of the first tool module and the real lateral location of the first target plant and in response to longitudinal alignment of a longitudinal position of the first tool module and a real longitudinal location of the first target plant, triggering the first tool module to execute an agricultural operation on the first target plant.

3. The method of claim 2 , further comprising:

at a third time, recording a second image of a second ground area at the ground-facing camera, the second ground area located within the first crop row;

detecting a second target plant in the second image;

accessing a lateral pixel location of the second target plant in the second image;

at a fourth time, for each tool module in the set of tool modules:

recording a second extension distance of the tool module; and

recording a second lateral position of the tool module relative to the ground-facing camera;

estimating a second depth profile of the plant bed surface proximal the target plant based on the second extension distance and the second lateral position of each tool module in the set of tool modules;

calculating an average depth profile based on the first depth profile and the second depth profile, the average depth profile associated with the first crop row;

estimating a real lateral location of the second target plant relative to the autonomous machine based on the lateral pixel location of the second target plant and the average depth profile of the plant bed surface;

driving the first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the second target plant; and

in response to lateral alignment of the lateral position of the first tool module and the real lateral location of the second target plant and in response to longitudinal alignment of a longitudinal position of the tool module and a real longitudinal location of the second target plant, triggering the first tool module to execute an agricultural operation on the second target plant.

4. The method of claim 1 , further comprising:

detecting a second target plant in the first image;

accessing a lateral pixel location of the second target plant in the first image;

estimating a real lateral location of the second target plant relative to the autonomous machine based on the lateral pixel location of the second target plant and the first depth profile; and

driving a second tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the second target plant.

5. A method for detecting real lateral locations of target plants relative to an autonomous machine comprises, at the autonomous machine:

at a first time, recording a first image of a first ground area at a ground-facing camera arranged on the autonomous machine;

detecting a first target plant in the first image;

accessing a lateral pixel location of the first target plant in the first image;

at a second time concurrent with the first time, for each tool module in a set of tool modules in contact with a plant bed surface and arranged behind the ground-facing camera relative to a direction of forward motion of the autonomous machine:

recording a first extension distance of the tool module; and

recording a first lateral position of the tool module relative to the ground-facing camera;

estimating an intermediate depth profile of the plant bed surface at the longitudinal location of the set of tool modules based on the first extension distance and the first lateral position of each tool module in the set of tool modules;

projecting the intermediate depth profile in the direction of forward motion of the autonomous machine to estimate a first depth profile;

estimating a real lateral location of the first target plant relative to the autonomous machine based on the lateral pixel location of the first target plant and the first depth profile of the plant bed surface; and

driving a first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the first target plant.

6. The method of claim 5 , further comprising:

autonomously navigating along a first crop row comprising the first ground area; and

in response to lateral alignment of the lateral position of the first tool module and the real lateral location of the first target plant and in response to longitudinal alignment of a longitudinal position of the first tool module and a real longitudinal location of the first target plant, triggering the first tool module to execute an agricultural operation on the first target plant.

7. The method of claim 6 , further comprising:

at a third time, recording a second image of a second ground area at the ground-facing camera, the second ground area located within the first crop row;

detecting a second target plant in the second image;

accessing a lateral pixel location of the second target plant in the second image;

at a fourth time, for each tool module in the set of tool modules:

recording a second extension distance of the tool module; and

recording a second lateral position of the tool module relative to the ground-facing camera;

estimating a second depth profile of the plant bed surface based on the second extension distance and the second lateral position of each tool module in the set of tool modules;

calculating an average depth profile based on the first depth profile and the second depth profile, the average depth profile associated with the first crop row;

estimating a real lateral location of the second target plant relative to the autonomous machine based on the lateral pixel location of the second target plant and the average depth profile of the plant bed surface;

driving the first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the second target plant; and

in response to lateral alignment of the lateral position of the first tool module and the real lateral location of the second target plant, and in response to longitudinal alignment of a longitudinal position of the tool module and a real longitudinal location of the second target plant, triggering the first tool module to execute an agricultural operation on the second target plant.

8. The method of claim 5 , further comprising:

detecting a second target plant in the first image;

accessing a lateral pixel location of the second target plant in the first image;

estimating a real lateral location of the second target plant relative to the autonomous machine based on the lateral pixel location of the second target plant and the first depth profile; and

driving a second tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the second target plant.

9. The method of claim 5 , wherein estimating the intermediate depth profile of the plant bed surface based on the first extension distance of each tool module in the set of tool modules and the first lateral position of each tool module in the set of tool modules further comprises:

for each tool module in the set of tool modules, defining a contact point of the tool module at an intersection of the first extension distance of the tool module and the first lateral position of the tool module to generate a set of contact points; and

interpolating between the set of contact points to generate the intermediate depth profile of the plant bed surface.

10. The method of claim 5 :

wherein, for each tool module in the set of tool modules, recording the first extension distance of the tool module, and recording the first lateral position of the tool module relative to the ground-facing camera further comprises, in response to detecting a longitudinal location of the first target plant at a threshold distance in front of a longitudinal location of the set of tool modules:

recording the first extension distance of the tool module; and

recording the first lateral position of the tool module; and

wherein projecting the intermediate depth profile in the direction of forward motion of the autonomous machine to estimate the first depth profile further comprises projecting the intermediate depth profile to the longitudinal location of the first target plant to estimate the first depth profile.

11. A method for detecting real lateral locations of target plants relative to an autonomous machine comprises, at the autonomous machine:

at a first time, recording a first image of a first ground area at a ground-facing camera arranged on the autonomous machine;

detecting a first target plant in the first image;

accessing a lateral pixel location of the first target plant in the first image;

in response to detecting a longitudinal location of the first target plant at a threshold distance in front of a longitudinal location of the set of tool modules, for each tool module in a set of tool modules in contact with a plant bed surface and arranged behind the ground-facing camera relative to a direction of forward motion of the autonomous machine:

recording a first extension distance of the tool module; and

recording a first lateral position of the tool module relative to the ground-facing camera;

estimating an intermediate depth profile of the plant bed surface at the longitudinal location of the set of tool modules based on the first extension distance and the first lateral position of each tool module in the set of tool modules;

projecting the intermediate depth profile to the longitudinal location of the first target plant to estimate a first depth profile;

estimating a real lateral location of the first target plant relative to the autonomous machine based on the lateral pixel location of the first target plant and the first depth profile of the plant bed surface; and

driving a first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the first target plant.

12. The method of claim 11 , further comprising:

autonomously navigating along a first crop row comprising the first ground area; and

in response to lateral alignment of the lateral position of the first tool module and the real lateral location of the first target plant and in response to longitudinal alignment of a longitudinal position of the first tool module and a real longitudinal location of the first target plant, triggering the first tool module to execute an agricultural operation on the first target plant.

13. The method of claim 12 , further comprising:

at a third time, recording a second image of a second ground area at the ground-facing camera, the second ground area located within the first crop row;

detecting a second target plant in the second image;

accessing a lateral pixel location of the second target plant in the second image;

at a fourth time, for each tool module in the set of tool modules:

recording a second extension distance of the tool module; and

recording a second lateral position of the tool module relative to the ground-facing camera;

estimating a second depth profile of the plant bed surface based on the second extension distance and the second lateral position of each tool module in the set of tool modules;

calculating an average depth profile based on the first depth profile and the second depth profile, the average depth profile associated with the first crop row;

estimating a real lateral location of the second target plant relative to the autonomous machine based on the lateral pixel location of the second target plant and the average depth profile of the plant bed surface;

driving the first tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the second target plant; and

in response to lateral alignment of the lateral position of the first tool module and the real lateral location of the second target plant and in response to longitudinal alignment of a longitudinal position of the tool module and a real longitudinal location of the second target plant, triggering the first tool module to execute an agricultural operation on the second target plant.

14. The method of claim 11 , further comprising:

detecting a second target plant in the first image;

accessing a lateral pixel location of the second target plant in the first image;

estimating a real lateral location of the second target plant relative to the autonomous machine based on the lateral pixel location of the second target plant and the first depth profile; and

driving a second tool module in the set of tool modules to a lateral position laterally aligned with the real lateral location of the second target plant.

15. The method of claim 11 , wherein estimating the intermediate depth profile of the plant bed surface based on the first extension distance of each tool module in the set of tool modules and the first lateral position of each tool module in the set of tool modules further comprises:

for each tool module in the set of tool modules, defining a contact point of the tool module at an intersection of the first extension distance of the tool module and the first lateral position of the tool module to generate a set of contact points; and

interpolating between the set of contact points to generate the intermediate depth profile of the plant bed surface.

Assignments (1)

Continuity (2)

Cited By (10)

US 12,201,044 US 12,247,966 US 12,260,635 US 12,302,894 US 12,353,210 US 12,403,950 US 12,583,509 US 12,619,236 US 12,653,085 US 12,714,023