Visual and tactile confirmation of package presence for UAV aerial deliveries
A technique for validating a presence of a package carried by an unmanned aerial vehicle (UAV) includes: capturing an image of a scene below the UAV with a camera mounted to the UAV and oriented to face down from the UAV; analyzing the image to identify whether the package is present in the image; and determining whether the package is attached to the UAV, via a tether extending from an underside of the UAV, based at least on the analyzing of the image.
1. An unmanned aerial vehicle (UAV) adapted for package deliveries, the UAV comprising:
an airframe adapted for carrying a package;
a camera mounted to the airframe and oriented to capture an image of a scene below the airframe;
a tether mounted to the airframe at a first end and adapted to attach the package to the UAV via a second end opposite the first end; and
a control system coupled to the camera and including logic, that when executed by the control system, causes the UAV to perform operations including:
capturing the image of the scene below the airframe;
analyzing the image to classify pixels in the image deemed to satisfy a package classification; and
determining whether the package is attached to the UAV based at least on whether a threshold number of contiguous pixels in the image are classified into the package classification.
2. The UAV of claim 1 , wherein analyzing the image comprises:
performing an image segmentation on the image; and
tagging the pixels in the image deemed to satisfy the package classification.
3. The UAV of claim 1 , wherein determining whether the package is attached to the UAV further comprises:
determining whether the threshold number of contiguous pixels classified into the package classification are positioned at an expected location within the image.
4. The UAV of claim 2 , wherein determining whether the package is attached to the UAV is further based on at least one of a geometric arrangement of the contiguous pixels in the image or a position of the contiguous pixels in the image.
5. An unmanned aerial vehicle (UAV) adapted for package deliveries, the UAV comprising:
an airframe adapted for carrying a package;
a camera mounted to the airframe and oriented to capture an image of a scene below the airframe;
a tether mounted to the airframe at a first end and adapted to attach the package to the UAV via a second end opposite the first end;
a winch mounted to the airframe and adapted for raising or lowering the package to or from the UAV via the tether; and
a control system coupled to the camera and the winch, the control system including logic, that when executed by the control system, causes the UAV to perform operations including:
capturing the image of the scene below the airframe;
tracking a length of the tether dispensed by the winch;
analyzing the image; and
determining whether the package is attached to the UAV based at least on whether the analyzing indicates that a representation of the package in the image exceeds a threshold, wherein the threshold is variable based on the length of the tether dispensed when the image is acquired.
6. The UAV of claim 5 , wherein determining whether the package is attached to the UAV is further based on at least one of an acceleration or a velocity of dispensing the tether.
7. The UAV of claim 5 , wherein determining whether the package is attached to the UAV is further based on a line tension on the tether.
8. The UAV of claim 7 , wherein determining whether the package is attached to the UAV includes determining whether a correct package is attached to the UAV,
wherein the control system includes additional logic, that when executed by the control system, causes the UAV to perform additional operations including:
determining whether the line tension falls within an expected range associated with the correct package.
9. The UAV of claim 5 , wherein the control system includes additional logic, that when executed by the control system, causes the UAV to perform additional operations including:
making a first flight mission decision based upon determining the package is attached to the UAV and the package is expected to be attached to the UAV; and
making a second flight mission decision, different than the first flight mission decision, based upon determining the package is attached to the UAV and the package is not expected to be attached to the UAV.
10. The UAV of claim 9 , wherein the second flight mission decision comprises releasing the tether from the UAV.
11. The UAV of claim 5 , wherein analyzing the image to identify whether the package is present in the image comprises:
performing an image segmentation on the image;
tagging pixels in the image deemed to satisfy a package classification; and
determining the package is viewable in the image of the scene below the airframe based at least on whether a threshold number of contiguous pixels in the image are tagged as satisfying the package classification,
wherein the threshold number of contiguous pixels is a variable threshold dependent upon the length of the tether dispensed when the image is acquired.
12. The UAV of claim 5 , wherein the control system includes additional logic, that when executed by the control system, causes the UAV to perform additional operations including:
analyzing the image to estimate how far the package is from the UAV; and
validating correct operation of the winch or winch sensors that monitor operation of the winch based at least on the image and the estimate.
13. The UAV of claim 1 , wherein the control system includes additional logic, that when executed by the control system, causes the UAV to perform additional operations including:
generating navigation decisions for the UAV based at least upon the image of the scene below the airframe acquired by the camera,
wherein visual navigation and package presence decisions are both based upon images acquired by the camera.
14. A method of validating a presence of a package carried by an unmanned aerial vehicle (UAV), the method comprising:
capturing an image of a scene below the UAV with a camera mounted to the UAV and oriented to face down from the UAV;
analyzing the image to classify pixels in the image deemed to satisfy a package classification; and
determining whether the package is attached to the UAV via a tether extending from an underside of the UAV, based at least on whether a threshold number of contiguous pixels in the image are classified into the package classification.
15. The method of claim 14 , wherein determining whether the package is attached to the UAV includes determining whether a corner of the package extends from a side of the image while a majority of the package is out-of-frame of the image when the UAV is operating in a cruise segment of an aerial delivery mission.
16. The method of claim 14 , wherein analyzing the image comprises:
performing an image segmentation on the image; and
tagging the pixels in the image deemed to satisfy the package classification.
17. The method of claim 14 , wherein determining whether the package is attached to the UAV further comprises:
determining whether the threshold number of contiguous pixels classified into the package classification are positioned at an expected location within the image, the expected location changing dependent upon a mission segment of an aerial delivery mission flown by the UAV.
18. The method of claim 14 , further comprising:
raising or lowering the package via a winch coupled to the tether; and
tracking a length of the tether dispensed by the winch,
wherein determining whether the package is attached to the UAV is based at least on the analyzing of the image and the length of the tether dispensed when the image is acquired.
19. The method of claim 18 , wherein determining whether the package is attached to the UAV is further based on at least one of an acceleration of dispensing the tether, a velocity of dispensing the tether, or a line tension on the tether.
20. The method of claim 14 , wherein determining whether the package is attached to the UAV includes determining whether a correct package is attached to the UAV, the method further comprising:
determining whether a line tension of the tether falls within an expected range associated with the correct package.
21. A method of validating a presence of a package carried by an unmanned aerial vehicle (UAV), the method comprising:
capturing an image of a scene below the UAV with a camera mounted to the UAV and oriented to face down from the UAV;
analyzing the image to identify whether the package is present in the image;
determining whether the package is attached to the UAV, via a tether extending from an underside of the UAV, based at least on the analyzing of the image; and
releasing the tether when determining that an incorrect package or unexpected package has been attached to the UAV after delivering the package to a destination.