IP Library Granted Patent US 10,258,534
Granted Patent B1
US 10,258,534 · App. 15/432,416 · Granted Apr 16, 2019

Methods and systems for providing feedback based on information received from an aerial vehicle

Inventors: Maxwell Andrew Sills (San Francisco, CA); Robert Samuel Gordon (San Bruno, CA); Ian Wetherbee (San Jose, CA)
Assignee: Wing Aviation LLC
A61H3/06B64C39/024G05D1/0011G05D1/101G08G5/0034B64C2201/126G06F3/016G06F3/167G06T11/60
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,258,534
App. No.
15/432,416
Granted
Apr 16, 2019
Kind
B1
Abstract

Described herein is a control system that facilitates assistance mode(s). In particular, the control system may determine a particular assistance mode associated with an account. This particular assistance mode may specify (i) operations for an aerial vehicle to carry out in order to obtain sensor data providing environment information corresponding to a location associated with the account and (ii) feedback processes to provide feedback, via a feedback system associated with the account, that corresponds to respective environment information. The control system may transmit to the aerial vehicle an indication of the particular operations corresponding to the particular assistance mode and may then receive environment information for the location associated with the account. Based on the received environment information, the control system may apply the specified feedback processes to initiate feedback in accordance with the particular assistance mode via the associated feedback system.

Claims (59)

1. A method comprising:

determining, by a control system as part of an assistance mode being provided with respect to an account, a first location corresponding to a feedback system associated with the account, wherein the assistance mode specifies one or more feedback processes to initiate feedback via the feedback system;

receiving, by the control system from an aerial vehicle that is physically separate from the feedback system, environment information related to the first location, wherein the assistance mode further specifies one or more operations for the aerial vehicle to carry out in order to obtain sensor data providing the environment information;

based on the environment information, determining, by the control system, a second location of an obstacle relative to the first location corresponding to the feedback system; and

applying, by the control system, a particular feedback process in accordance with the assistance mode to initiate, via the feedback system, particular feedback that indicates the second location of the obstacle relative to the first location corresponding to the feedback system.

2. The method of claim 1 , wherein the one or more feedback processes include one or more of the following forms of feedback: (i) auditory feedback, (ii) vibrational feedback, and (iii) visual feedback.

3. The method of claim 1 , wherein the feedback system comprises a wearable feedback system including one or more of the following components: (i) at least one listening device, (ii) at least one vibration-generating device, (iii) at least one display, (iv) at least one projector, and (v) at least one light source.

4. The method of claim 1 , further comprising:

based on the environment information, determining, by the control system, a potential future contact with the obstacle,

wherein the particular feedback also indicates the potential future contact with the obstacle.

5. The method of claim 4 ,

wherein determining the potential future contact comprises determining one or more of the following factors: (i) a distance between the second location of the obstacle and the first location corresponding to the feedback system and (ii) a time period until the potential future contact occurs, and

wherein the particular feedback indicating the potential future contact with the obstacle comprises the particular feedback indicating one or more of the determined factors.

6. The method of claim 1 , further comprising:

based on the environment information, determining, by the control system, a potential future contact with the obstacle; and

in response to determining the potential future contact with the obstacle, the control system directing the aerial vehicle to move the obstacle from the second location to a third location that is selected based on the potential future contact being avoidable.

7. The method of claim 1 , further comprising:

receiving, by the control system, input data that comprises a request for navigation to a particular destination; and

applying, by the control system, another feedback process to initiate, via the feedback system, other feedback indicative of at least one instruction specifying at least one movement towards the particular destination.

8. The method of claim 1 , wherein the feedback system comprises a display device, and wherein applying the particular feedback process comprises:

directing the feedback system to display, on the display device, visual feedback information related to one or more of (i) the first location corresponding to the feedback system and (ii) the second location of the obstacle.

9. The method of claim 8 , wherein the feedback system further comprises a video-capture device configured to provide video data representative of the first location corresponding to the feedback system, and wherein directing the feedback system to display, on the display device, the visual feedback information comprises:

directing the feedback system to display, on the display device, the visual feedback information as being overlaid on at least a portion of the provided video data.

10. The method of claim 8 , wherein the visual feedback information comprises one or more of the following forms of information: (i) the environment information, (ii) at least one visual symbol that is representative of at least one instruction specifying at least one movement towards a particular destination, (iii) at least one visual symbol that is representative of the second location of the obstacle relative to the first location corresponding to the feedback system, and (iv) at least one visual symbol that is representative of a point-of-interest associated with the first location corresponding to the feedback system.

11. The method of claim 1 , wherein the aerial vehicle is a first aerial vehicle, wherein the obstacle is a first obstacle, wherein the environment information is first environment information, and wherein the particular feedback process is a first feedback process, the method further comprising:

receiving, by a control system from a second aerial vehicle, second environment information related to the first location;

based on the second environment information, determining, by the control system, a third location of a second obstacle relative to the first location corresponding to the feedback system; and

applying, by the control system, a second feedback process in accordance with the assistance mode to initiate, via the feedback system, further feedback that indicates the third location of the second obstacle relative to the first location corresponding to the feedback system.

12. The method of claim 1 , wherein the account is a first account, wherein the feedback system is a first feedback system, wherein the environment information is first environment information, and wherein the feedback process is a first feedback process, the method further comprising:

receiving, by the control system from the aerial vehicle, second environment information related to a third location corresponding to a second feedback system associated with a second account;

based on the second environment information, determining, by the control system, a fourth location of a second obstacle relative to the third location corresponding to the second feedback system; and

applying, by the control system, a second feedback process to initiate, via the second feedback system, further feedback that indicates the fourth location of the second obstacle relative to the third location corresponding to the second feedback system.

13. The method of claim 1 , wherein the control system is part of the aerial vehicle, and wherein the control system receiving environment information from the aerial vehicle comprises the control system receiving environment information from one or more sensors of the aerial vehicle.

14. The method of claim 1 , wherein the control system is part of the feedback system.

15. A non-transitory computer readable medium having stored therein instructions executable by one or more processors to cause a control system to perform functions comprising:

determining, as part of an assistance mode being provided with respect to an account, a first location corresponding to a feedback system associated with the account, wherein the assistance mode specifies one or more feedback processes to initiate feedback via the feedback system;

receiving, from an aerial vehicle that is physically separate from the feedback system, environment information related to the first location, wherein the assistance mode further specifies one or more operations for the aerial vehicle to carry out in order to obtain sensor data providing the environment information;

based on the environment information, determining a second location of an obstacle relative to the first location corresponding to the feedback system; and

applying a particular feedback process in accordance with the assistance mode to initiate, via the feedback system, particular feedback that indicates the second location of the obstacle relative to the first location corresponding to the feedback system.

16. The non-transitory computer readable medium of claim 15 , the functions further comprising:

based on the environment information, determining a potential future contact with the obstacle,

wherein the particular feedback also indicates the potential future contact with the obstacle.

17. The non-transitory computer readable medium of claim 16 ,

wherein determining the potential future contact comprises determining one or more of the following factors: (i) a distance between the second location of the obstacle and the first location corresponding to the feedback system and (ii) a time period until the potential future contact occurs, and

wherein the particular feedback indicating the potential future contact with the obstacle comprises the particular feedback indicating one or more of the determined factors.

18. A control system comprising:

one or more processors;

a non-transitory computer readable medium; and

program instructions stored on the non-transitory computer readable medium and executable by the one or more processors to:

determine, as part of an assistance mode being provided with respect to an account, a first location corresponding to a feedback system associated with the account, wherein the assistance mode specifies one or more feedback processes to initiate feedback via the feedback system;

receive, from an aerial vehicle that is physically separate from the feedback system, environment information related to the first location, wherein the assistance mode further specifies one or more operations for the aerial vehicle to carry out in order to obtain sensor data providing the environment information;

based on the environment information, determine a second location of an obstacle relative to the first location corresponding to the feedback system; and

apply a particular feedback process in accordance with the assistance mode to initiate, via the feedback system, particular feedback that indicates the second location of the obstacle relative to the first location corresponding to the feedback system.

19. The control system of claim 18 , wherein the program instructions are further executable to:

based on the environment information, determine a potential future contact with the obstacle,

wherein the particular feedback also indicates the potential future contact with the obstacle.

20. The control system of claim 19 ,

wherein the program instructions being executable to determine the potential future contact comprises the program instructions being executable to determine one or more of the following factors: (i) a distance between the second location of the obstacle and the first location corresponding to the feedback system and (ii) a time period until the potential future contact occurs, and

wherein the particular feedback indicating the potential future contact with the obstacle comprises the particular feedback indicating one or more of the determined factors.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 12, 2018
From: X DEVELOPMENT LLC
To: WING AVIATION LLC
Reel/Frame 047567/0433 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 14, 2017
From: SILLS, MAXWELL ANDREW; GORDON, ROBERT SAMUEL; WETHERBEE, IAN
To: GOOGLE INC.
Reel/Frame 041253/0877 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 14, 2017
From: GOOGLE INC.
To: X DEVELOPMENT LLC
Reel/Frame 041715/0892 →
Continuity (1)
Continuation 15002733 · Jan 21, 2016
Cited By (7)
US 12,299,557 US 12,392,583 US 12,559,131 US 12,567,282 US 12,602,971 US 12,635,744 US 12,700,296