Granted Patent B2

US 12,266,054 · App. 17/860,761 · Granted Apr 1, 2025

Inventors: Jonathan James Millin (Santa Clara, CA); Nicholas Pilkington (Santa Clara, CA); Devin Lane (Santa Clara, CA); Christopher Sullivan (Santa Clara, CA); Michael Winn (Santa Clara, CA)

Assignee: Drone Deploy, Inc.

G06T17/05G01C11/34G05D1/0094G05D1/689G06V20/13G08G5/0013G08G5/0021G08G5/0034G08G5/006G08G5/0069G08G5/0086B64U2101/32

Abstract

Claims (69)

1. A computer-implemented method comprising:

receiving a first frame of a region of interest captured by a camera of an unmanned aerial vehicle (UAV) in association with a first camera pose;

receiving a second frame of the region of interest captured by the camera of the UAV in association with a second camera pose;

matching a feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame, based on the first camera pose and the second camera pose;

generating an orthomosaic using the first frame and the second frame, based on matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame; and

performing a bundle adjustment operation to reduce a reprojection error associated with a physical feature position using only frames that are within a temporal window corresponding to a limited number of frames that are most recently captured or processed.

2. The computer-implemented method of claim 1 , further comprising:

determining the first camera pose corresponding to the first frame;

determining the second camera pose corresponding to the second frame;

comparing the first camera pose and the second camera pose; and

determining motion information indicating motion between the first camera pose and the second camera pose, based on comparing the first camera pose and the second camera pose,

wherein the matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame comprises matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame based on the motion information.

3. The computer-implemented method of claim 2 , wherein the determining the first camera pose comprises determining the first camera pose based on one or more of: first global positioning system (GPS) information, first pitch information, first roll information, or first yaw information of the camera corresponding to the first frame,

wherein the determining the second camera pose comprises determining the second camera pose based on one or more of: second GPS information, second pitch information, second roll information, or second yaw information of the camera corresponding to the second frame.

4. The computer-implemented method of claim 1 , further comprising:

determining a first photo subset of the first frame that depicts the feature; and

determining a second photo subset of the second frame that depicts the feature,

wherein the matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame comprises matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame, based on determining the first photo subset and determining the second photo subset.

5. The computer-implemented method of claim 1 , further comprising:

performing a triangulation operation to determine a physical feature position of the feature in a three-dimensional space.

6. The computer-implemented method of claim 1 , further comprising:

providing the orthomosaic to an electronic device of a user to render the region of interest in near-real time at a display of the electronic device.

7. A device comprising:

a memory configured to store instructions; and

a processor configured to execute the instructions to perform operations comprising:

receiving a first frame of a region of interest captured by a camera of an unmanned aerial vehicle (UAV) in association with a first camera pose;

receiving a second frame of the region of interest captured by the camera of the UAV in association with a second camera pose;

matching a feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame, based on the first camera pose and the second camera pose;

generating an orthomosaic using the first frame and the second frame, based on matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame; and

performing a bundle adjustment operation to reduce a reprojection error associated with a physical feature position using only frames that are within a temporal window corresponding to a limited number of frames that are most recently captured or processed.

8. The device of claim 7 , wherein the operations further comprise:

determining the first camera pose corresponding to the first frame;

determining the second camera pose corresponding to the second frame;

comparing the first camera pose and the second camera pose; and

determining motion information indicating motion between the first camera pose and the second camera pose, based on comparing the first camera pose and the second camera pose,

wherein the matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame comprises matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame based on the motion information.

9. The device of claim 8 , wherein the determining the first camera pose comprises determining the first camera pose based on one or more of: first global positioning system (GPS) information, first pitch information, first roll information, or first yaw information of the camera corresponding to the first frame, and

wherein the determining the second camera pose comprises determining the second camera pose based on one or more of: second GPS information, second pitch information, second roll information, or second yaw information of the camera corresponding to the second frame.

10. The device of claim 7 , wherein the operations further comprise:

determining a first photo subset of the first frame that depicts the feature; and

determining a second photo subset of the second frame that depicts the feature,

wherein the matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame comprises matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame, based on determining the first photo subset and determining the second photo subset.

11. The device of claim 7 , wherein the operations further comprise:

performing a triangulation operation to determine a physical feature position of the feature in a three-dimensional space.

12. The device of claim 7 , wherein the operations further comprise:

providing the orthomosaic to an electronic device of a user to render the region of interest in near-real time at a display of the electronic device.

13. A non-transitory computer-readable medium configured to store instructions that, when executed by a processor, perform operations comprising:

receiving a first frame of a region of interest captured by a camera of an unmanned aerial vehicle (UAV) in association with a first camera pose;

receiving a second frame of the region of interest captured by the camera of the UAV in association with a second camera pose;

matching a feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame, based on the first camera pose and the second camera pose;

generating an orthomosaic using the first frame and the second frame, based on matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame; and

performing a bundle adjustment operation to reduce a reprojection error associated with a physical feature position using only frames that are within a temporal window corresponding to a limited number of frames that are most recently captured or processed.

14. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:

determining the first camera pose corresponding to the first frame;

determining the second camera pose corresponding to the second frame;

comparing the first camera pose and the second camera pose; and

determining motion information indicating motion between the first camera pose and the second camera pose, based on comparing the first camera pose and the second camera pose,

wherein the matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame comprises matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame based on the motion information.

15. The non-transitory computer-readable medium of claim 14 , wherein the determining the first camera pose comprises determining the first camera pose based on one or more of: first global positioning system (GPS) information, first pitch information, first roll information, or first yaw information of the camera corresponding to the first frame, and

wherein the determining the second camera pose comprises determining the second camera pose based on one or more of: second GPS information, second pitch information, second roll information, or second yaw information of the camera corresponding to the second frame.

16. The non-transitory computer-readable medium of 13 , wherein the operations further comprise:

determining a first photo subset of the first frame that depicts the feature; and

determining a second photo subset of the second frame that depicts the feature,

wherein the matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame comprises matching the feature of the region of interest included in the first frame with the feature of the region of interest included in the second frame, based on determining the first photo subset and determining the second photo subset.

17. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:

performing a triangulation operation to determine a physical feature position of the feature in a three-dimensional space; and

performing a bundle adjustment operation to reduce a reprojection error associated with the physical feature position.

18. The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:

providing the orthomosaic to an electronic device of a user to render the region of interest in near-real time at a display of the electronic device.

Assignments (4)

Continuity (5)

References Cited (74)