IP Library Granted Patent US 11,703,864
Granted Patent B2
US 11,703,864 · App. 17/174,583 · Granted Jul 18, 2023

Scanning structures via unmanned aerial vehicles

Inventors: Peter Henry (San Francisco, CA); Jack Zhu (San Francisco, CA); Brian Richman (San Francisco, CA); Harrison Zheng (Palo Alto, CA); Hayk Martirosyan (San Francisco, CA); Matthew Donahoe (Redwood City, CA); Abraham Bachrach (Redwood City, CA); Adam Bry (Redwood City, CA); Ryan David Kennedy (Redwood City, CA); Himel Mondal (Windsor, CA); Quentin Allen Wah Yen Delepine (Cupertino, CA)
Assignee: SKYDIO, INC.
G05D1/0094B64C39/024B64D31/06B64D47/08G05B13/0265G05B17/02G05D1/0088G05D1/101G06T7/55G06T7/74G06T17/00G06T19/20G06V20/13G06V20/64H04N23/64H04N23/695H04N23/90B64U10/13B64U2101/30G06T2207/10032G06T2207/20221G06T2219/2004
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,703,864
App. No.
17/174,583
Granted
Jul 18, 2023
Kind
B2
Abstract

In some examples, an unmanned aerial vehicle (UAV) may determine, based on a three-dimensional (3D) model including a plurality of points corresponding to a scan target, a scan plan for scanning at least a portion of the scan target. For instance, the scan plan may include a plurality of poses for the UAV to assume to capture images of the scan target. The UAV may capture with one or more image sensors, one or more images of the scan target from one or more poses of the plurality of poses. Further, the UAV may determine an update to the 3D model based at least in part on the one or more images. Additionally, the UAV may update the scan plan based at least in part on the update to the 3D model.

Claims (73)

1. An unmanned aerial vehicle (UAV) comprising:

one or more image sensors;

a propulsion mechanism; and

one or more processors configured by executable instructions to perform operations comprising:

receiving, by the one or more processors, from a computing device, an indication of a scan target;

controlling, by the one or more processors, the propulsion mechanism of the UAV to cause the UAV to assume a plurality of positions for directing a field of view of the one or more image sensors based on the indication of the scan target;

capturing, using the one or more image sensors, a plurality of images from different respective positions of the plurality of positions;

determining a three-dimensional (3D) model based on distances to one or more surfaces of the scan target determined based on the plurality of images, the 3D model including a plurality of points corresponding to one or more surfaces of the scan target;

accessing the 3D model including the plurality of points;

determining, based on the 3D model, a scan plan for scanning at least a portion of the scan target, the scan plan including a plurality of poses for the UAV to assume to capture, using one or more image sensors, images of the scan target, wherein a respective pose of the scan plan is determined based on a respective point of the 3D model;

capturing, with the one or more image sensors, while the UAV is in flight, one or more images of the scan target from one or more poses of the plurality of poses;

determining, while the UAV is in flight, an update to the 3D model based at least in part on the one or more images; and

based at least in part on the update to the 3D model, updating the scan plan while the UAV is in flight.

2. The UAV as recited in claim 1 , the operations further comprising updating, by the one or more processors of the UAV, the scan plan in real time.

3. The UAV as recited in claim 1 , the operations further comprising receiving, as the indication of the scan target, at least one of:

an indication of a 2D shape forming a boundary within which at least a portion the scan target is located;

an indication of a shape corresponding to a surface of the scan target; or

an indication of a volume within which at least a portion of the scan target is located.

4. The UAV as recited in claim 1 , wherein the respective pose is determined to be along a respective normal for the respective point at a distance from the respective point corresponding to a scanning distance parameter specified for the scan plan.

5. The UAV as recited in claim 1 , the operation of determining the scan plan further comprising:

partitioning the points of the 3D model into a subset of covered points and a subset of uncovered points;

determining a next pose for the scan plan by selecting a center point for the next pose from the subset of uncovered points;

determining points of the 3D model covered by a field of view of the next pose; and

updating the subset of covered points and the subset of uncovered points based on determining the points of the 3D model covered by the field of view of the next pose.

6. The UAV as recited in claim 1 , the operations further comprising:

detecting an obstruction to capturing an image for at least one pose of the plurality of poses; and

capturing an image of a surface of the scan target corresponding to the at least one pose by performing oblique image capture.

7. A method comprising:

accessing, by one or more processors of an unmanned aerial vehicle (UAV), a three-dimensional (3D) model including a plurality of points corresponding to one or more surfaces of a scan target;

determining, by the one or more processors, based on the 3D model, a scan plan for scanning at least a portion of the scan target, the scan plan including a plurality of poses for the UAV to assume to capture, using one or more image sensors, images of the scan target, wherein a respective pose of the scan plan is determined based on a respective point of the 3D model;

capturing, by the one or more processors, with the one or more image sensors, while the UAV is in flight, one or more images of the scan target from one or more poses of the plurality of poses;

determining, by the one or more processors, while the UAV is in flight, an update to the 3D model based at least in part on the one or more images; and

based at least in part on the update to the 3D model, updating the scan plan, by the one or more processors, while the UAV is in flight.

8. The method as recited in claim 7 , wherein at least one of determining the update to the 3D model or updating the scan plan is performed by the one or more processors onboard the UAV in real time.

9. The method as recited in claim 7 , further comprising receiving, by the one or more processors, from a computing device, an indication of the scan target, the indication of the scan target comprising at least one of:

an indication of a 2D shape forming a boundary within which at least a portion the scan target is located;

an indication of a shape corresponding to a surface of the scan target; or

an indication of a volume within which at least a portion of the scan target is located.

10. The method as recited in claim 7 , further comprising:

receiving, by the one or more processors, from a computing device, an indication of the scan target;

controlling, by the one or more processors, the propulsion mechanism of the UAV to cause the UAV to assume one or more positions for directing a field of view of the one or more image sensors based on the indication of the scan target, wherein the one or more image sensors include a plurality of lower resolution image sensors and at least one higher resolution image sensor;

capturing, using the plurality of lower-resolution image sensors, at least one image from the one or more positions; and

determining the 3D model based on distances to one or more surfaces of the scan target determined based on the at least one image captured using the lower-resolution image sensors, the 3D model initially being a lower-resolution 3D model including the plurality of points corresponding to the one or more surfaces of the scan target.

11. The method as recited in claim 10 , wherein determining the 3D model based on distances to one or more surfaces of the scan target determined based on the at least one image comprises performing Truncated Signed Distance Function (TSDF) fusion for at least two images captured from the one or more positions.

12. The method as recited in claim 7 , wherein the updated scan plan includes at least one of an additional pose or updated pose, the method further comprising controlling, by the one or more processors, a propulsion mechanism of the UAV to cause the UAV to fly to the at least one additional pose or updated pose.

13. The method as recited in claim 7 , the determining the scan plan further comprising:

partitioning the points of the 3D model into a subset of covered points and a subset of uncovered points;

determining a next pose for the scan plan by selecting a center point for the next pose from the subset of uncovered points;

determining points of the 3D model covered by a field of view of the next pose; and

updating the subset of covered points and the subset of uncovered points based on determining the points of the 3D model covered by the field of view of the next pose.

14. The method as recited in claim 7 , further comprising:

detecting an obstruction to capturing an image for at least one pose of the plurality of poses; and

capturing an image of a surface of the scan target corresponding to the at least one pose by performing an oblique image capture.

15. An unmanned aerial vehicle (UAV) comprising

one or more image sensors;

a propulsion mechanism; and

one or more processors configured by executable instructions to perform operations comprising:

determining, by the one or more processors, based on a 3D model including a plurality of points corresponding to a scan target, a scan plan for scanning at least a portion of the scan target, the scan plan including a plurality of poses for the UAV to assume to capture images of the scan target;

capturing, by the one or more processors, with the one or more image sensors, one or more images of the scan target from one or more poses of the plurality of poses;

determining, by the one or more processors, an update to the 3D model based at least in part on the one or more images; and

updating the scan plan, by the one or more processors, based at least in part on the update to the 3D model.

16. The UAV as recited in claim 15 , the operations further comprising updating the scan plan, by the one or more processors, based at least in part on the update to the 3D model, while the UAV is in flight.

17. The UAV as recited in claim 16 , the operations further comprising updating, by the one or more processors of the UAV, the scan plan in real time.

18. The UAV as recited in claim 15 , the operations further comprising determining, by the one or more processors, the update to the 3D model while the UAV is in flight.

19. The UAV as recited in claim 15 , the operations further comprising receiving, by the one or more processors, from a computing device, an indication of the scan target, the indication of the scan target comprising at least one of:

an indication of a 2D shape forming a boundary within which at least a portion the scan target is located;

an indication of a shape corresponding to a surface of the scan target; or

an indication of a volume within which at least a portion of the scan target is located.

20. The UAV as recited in claim 15 , the operations further comprising:

receiving, by the one or more processors, from a computing device, an indication of the scan target;

controlling, by the one or more processors, the propulsion mechanism of the UAV to cause the UAV to assume one or more positions for directing a field of view of the one or more image sensors based on the indication of the scan target, wherein the one or more image sensors include a plurality of lower resolution image sensors and at least one higher resolution image sensor;

capturing, using the plurality of lower-resolution image sensors, at least one image from the one or more positions; and

determining the 3D model based on distances to one or more surfaces of the scan target determined based on the at least one image captured using the lower-resolution image sensors, the 3D model initially being a lower-resolution 3D model including the plurality of points corresponding to the one or more surfaces of the scan target.

Assignments (6)
RELEASE OF SECURITY INTEREST Recorded Jul 21, 2025
From: SILICON VALLEY BANK, A DIVISION OF FIRST-CITIZENS BANK & TRUST COMPANY
To: SKYDIO, INC.
Reel/Frame 072107/0066 →
RELEASE OF SECURITY INTEREST Recorded Jul 21, 2025
From: HERCULES CAPITAL, INC.
To: SKYDIO, INC.
Reel/Frame 072128/0698 →
SECURITY INTEREST Recorded Dec 5, 2024
From: SKYDIO, INC.
To: ACQUIOM AGENCY SERVICES LLC
Reel/Frame 069516/0452 →
INTELLECTUAL PROPERTY SECURITY AGREEMENT Recorded Nov 9, 2021
From: SKYDIO, INC.
To: HERCULES CAPITAL, INC., AS COLLATERAL AND ADMINISTRATIVE AGENT
Reel/Frame 058081/0677 →
SECURITY INTEREST Recorded Nov 8, 2021
From: SKYDIO, INC.
To: SILICON VALLEY BANK
Reel/Frame 058053/0768 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 23, 2021
From: HENRY, PETER; ZHU, JACK; RICHMAN, BRIAN; ZHENG, HARRISON; MARTIROSYAN, HAYK; DONAHOE, MATTHEW; BACHRACH, ABRAHAM; BRY, ADAM; KENNEDY, RYAN DAVID; MONDAL, HIMEL; DELEPINE, QUENTIN ALLEN WAH YEN
To: SKYDIO, INC.
Reel/Frame 055378/0564 →
Continuity (3)
Provisional Application 63050860 · Jul 12, 2020
Provisional Application 62976231 · Feb 13, 2020
Related Publication 20210263488A1 · Aug 26, 2021