Granted Patent B2

US 12,361,593 · App. 18/609,224 · Granted Jul 15, 2025

Inventors: James Patrick Ostrowski (Mountain View, CA); John William Peake (Mountain View, CA); Jacqueline Gao Song (San Francisco, CA)

Assignee: Deere & Company

G06T7/85A01B69/001G06T7/73G06V20/188G06V20/56G06T2207/30188G06T2207/30244

Abstract

Claims (69)

1. A method for automatic calibration of a plurality of cameras on a farming machine, the method comprising:

for each camera of the plurality of cameras fixed on the farming machine, receiving an image comprising visual information representing a view of a portion of an area surrounding the farming machine;

for a first pair of cameras comprising a first camera and a second camera of the plurality of cameras, determining a first relative pose between the first pair of cameras based on visual information in both a first image received from the first camera and a second image received from the second camera;

for a second pair of cameras comprising the second camera and a third camera of the plurality of cameras, determining a second relative pose between the second pair of cameras based on visual information in both the second image received and a third image received from the third camera;

determining a calibration error for each pair of cameras of the plurality of cameras based on the first relative pose and the second relative pose, wherein determining the calibration error for the second pair of cameras includes determining a calibration error for the first pair of cameras and propagating the calibration error determined for the first pair of cameras;

generating instructions to adjust a pose of any of the plurality of cameras based on the calibration errors; and

adjusting the plurality of cameras based on the generated instructions to remedy the calibration errors.

2. The method of claim 1 , further comprising:

for a third pair of cameras comprising the third camera and a fourth camera of the plurality of cameras, determining a third relative pose between the third pair of cameras based on visual information in both the third image and a fourth image received from the fourth camera; and

wherein determining the calibration error for the third pair of cameras based on the second relative pose and the third relative pose comprises determining the calibration error for the second pair of cameras and propagating the calibration error determined for the second pair of cameras.

3. The method of claim 1 , wherein determining the calibration error comprises:

identifying an error for the first pair of cameras based on a comparison of the first relative pose with a first expected pose between the first pair of cameras, wherein the first expected pose is described in a virtual representation of the farming machine; and

identifying an error for the second pair of cameras based on a comparison of the second relative pose with a second expected pose between the second pair of cameras, wherein the second expected pose is described in the virtual representation of the farming machine.

4. The method of claim 3 , wherein determining the error for the second pair of cameras comprises propagating the error of the first pair of cameras to the second pair of cameras.

5. The method of claim 1 , wherein generating the instructions to adjust the pose of any of the plurality of cameras based on the calibration errors comprises:

determining the generated instructions to remedy the calibration error for the first pair of cameras or the second pair of cameras that minimizes adjustments to the first pair of cameras and the second pair of cameras to remedy the calibration errors.

6. The method of claim 1 , wherein determining the first relative pose comprises:

extracting position characteristics and orientation characteristics from a fiducial marker in the visual information of the first image and the second image of the first pair of cameras, wherein the first relative pose is based on the position characteristics and the orientation characteristics.

7. The method of claim 1 , further comprising:

determining the second camera is adjacent to the third camera forming the second pair of cameras.

8. The method of claim 7 , wherein determining the second camera is adjacent to the third camera comprises:

comparing visual information of the second image to visual information of a fourth image captured by a fourth camera of the plurality of cameras;

responsive to determining that the visual information of the second image does not overlap with the visual information of the fourth image, comparing the visual information of the second image to visual information of images captured by other cameras of the plurality of cameras;

identifying, from the images captured by the other cameras, the third camera that captured the third image with visual information overlapping with the visual information of the second image; and

determining that the second camera is adjacent to the third camera.

9. The method of claim 1 , wherein determining the first relative pose comprises:

measuring, by a sensor mounted on the farming machine, a first height of the first camera relative to a ground surface and a second height of the second camera relative to the ground surface;

determining a scale factor based on the first height and the second height; and

determining the first relative pose based on the scale factor.

10. The method of claim 9 , wherein determining the first relative pose comprises:

determining a first angle of the first camera relative to a ground surface and a second angle of the second camera relative to the ground surface; and

determining the first relative pose based on the first angle and the second angle.

11. A method for validating a calibration of a plurality of cameras on a farming machine having a mounting mechanism, the method comprising:

for each camera of the plurality of cameras fixed on the farming machine, receiving an image comprising visual information representing a view of a portion of an area surrounding the farming machine;

for a first pair of cameras comprising a first camera and a second camera of the plurality of cameras, determining a first relative pose between the first pair of cameras based on visual information in both a first image received from the first camera and a second image received from the second camera;

for a second pair of cameras comprising the second camera and a third camera of the plurality of cameras, determining a second relative pose between the second pair of cameras based on visual information in both the second image received and a third image received from the third camera;

determining a calibration error for each pair of cameras of the plurality of cameras based on the first relative pose and the second relative pose, wherein determining the calibration error for the second pair of cameras includes determining a calibration error for the first pair of cameras and propagating the calibration error determined for the first pair of cameras;

generating instructions to adjust a pose of any of the plurality of cameras based on the calibration errors; and

transmitting, to an operator of the farming machine, a notification comprising the calibration error and the instructions to adjust the pose of any of the plurality of cameras based on the calibration errors.

12. The method of claim 11 , further comprising:

for a third pair of cameras comprising the third camera and a fourth camera of the plurality of cameras, determining a third relative pose between the third pair of cameras based on visual information in both the third image and a fourth image received from the fourth camera; and

wherein determining the calibration error for the third pair of cameras based on the second relative pose and the third relative pose comprises determining the calibration error for the second pair of cameras and propagating the calibration error determined for the second pair of cameras.

13. The method of claim 11 , wherein determining the calibration error comprises:

identifying an error for the first pair of cameras based on a comparison of the first relative pose with a first expected pose between the first pair of cameras, wherein the first expected pose is described in a virtual representation of the farming machine; and

identifying an error for the second pair of cameras based on a comparison of the second relative pose with a second expected pose between the second pair of cameras, wherein the second expected pose is described in the virtual representation of the farming machine.

14. The method of claim 13 , wherein determining the error for the second pair of cameras comprises propagating the error of the first pair of cameras to the second pair of cameras.

15. The method of claim 11 , wherein generating the instructions to adjust the pose of any of the plurality of cameras based on the calibration errors, further comprises:

determining the generated instructions to remedy the calibration error for the first pair of cameras or the second pair of cameras that minimizes adjustments to the first pair of cameras and the second pair of cameras to remedy the calibration errors.

16. The method of claim 11 , wherein determining the first relative pose comprises:

extracting position characteristics and orientation characteristics from a fiducial marker in the visual information of the first image and the second image of the first pair of cameras, wherein the first relative pose is based on the position characteristics and the orientation characteristics.

17. The method of claim 11 , further comprising:

determining the second camera is adjacent to the third camera forming the second pair of cameras.

18. The method of claim 17 , wherein determining the second camera is adjacent to the third camera comprises:

comparing visual information of the second image to visual information of a fourth image captured by a fourth camera of the plurality of cameras;

responsive to determining that the visual information of the second image does not overlap with the visual information of the fourth image, comparing the visual information of the second image to visual information of images captured by other cameras of the plurality of cameras;

identifying, from the images captured by the other cameras, the third camera that captured the third image with visual information overlapping with the visual information of the second image; and

determining that the second camera is adjacent to the third camera.

19. The method of claim 11 , wherein determining the first relative pose comprises:

measuring, by a sensor mounted on the farming machine, a first height of the first camera relative to a ground surface and a second height of the second camera relative to the ground surface;

determining a first angle of the first camera relative to a ground surface and a second angle of the second camera relative to the ground surface;

determining a scale factor based on the first height, the second height, the first angle, and the second angle; and

determining the first relative pose based on the scale factor.

20. A non-transitory computer-readable storage medium storing instructions for automatic calibration of a plurality of cameras on a farming machine, the instructions that, when executed by a computer processor, cause the computer processor to perform operations comprising:

for each camera of the plurality of cameras fixed on the farming machine, receiving an image comprising visual information representing a view of a portion of an area surrounding the farming machine;

for a first pair of cameras comprising a first camera and a second camera of the plurality of cameras, determining a first relative pose between the first pair of cameras based on visual information in both a first image received from the first camera and a second image received from the second camera;

for a second pair of cameras comprising the second camera and a third camera of the plurality of cameras, determining a second relative pose between the second pair of cameras based on visual information in both the second image received and a third image received from the third camera;

determining a calibration error for each pair of cameras of the plurality of cameras based on the first relative pose and the second relative pose, wherein determining the calibration error for the second pair of cameras includes determining a calibration error for the first pair of cameras and propagating the calibration error determined for the first pair of cameras;

generating instructions to adjust a pose of any of the plurality of cameras based on the calibration errors; and

adjusting the plurality of cameras based on the generated instructions to remedy the calibration errors.

Assignments (2)

Continuity (2)

References Cited (11)