Fitness and sports applications for an autonomous unmanned aerial vehicle
Sports and fitness applications for an autonomous unmanned aerial vehicle (UAV) are described. In an example embodiment, a UAV can be configured to track a human subject using perception inputs from one or more onboard sensors. The perception inputs can be utilized to generate values for various performance metrics associated with the activity of the human subject. In some embodiments, the perception inputs can be utilized to autonomously maneuver the UAV to lead the human subject to satisfy a performance goal. The UAV can also be configured to autonomously capture images of a sporting event and/or make rule determinations while officiating a sporting event.
1 . An unmanned aerial vehicle (UAV) comprising:
a propulsion system;
a sensor device;
a gimbal mechanism;
a camera rotatably coupled to a body of the UAV via the gimbal mechanism;
an onboard audio output device;
a microphone;
a computer system configured to:
receive an audible tracking command;
interpret the audible tracking command using natural language processing techniques;
receive perception inputs generated by the sensor device;
process the perception inputs to develop a semantic understanding of a surrounding physical environment;
fuse the audible tracking command with the semantic understanding of the surrounding physical environment to refine the audible tracking command;
identify, based on the refined tracking command, a plurality of objects within a particular area associated with a sporting event;
track multiple objects of the plurality of objects simultaneously within the particular area associated with the sporting event;
detect a moving area of interest within the particular area associated with the sporting event,
wherein the moving area of interest corresponds with an area in which most activity is occurring among the multiple tracked objects;
track the moving area of interest;
generate and continually update, based on the perception inputs, a planned trajectory configured to keep the moving area of interest within a field of view of the camera while avoiding overflying the particular area associated with the sporting event and predefined restricted regions;
control the propulsion system to autonomously maneuver the UAV and rotate the camera relative to the body of the UAV according to the planned trajectory so as to keep the moving area of interest in a field of view of the camera;
process the perception inputs to identify a characteristic of the sporting event;
select, based on the identified characteristic, a set of rules associated with the sporting event from a library of a plurality of rules associated with a plurality of different types of sporting events by recognizing a sport type based on features of the surrounding physical environment;
process the perception inputs to detect an activity occurring during the sporting event;
select, based on the activity occurring during the sporting event, a particular rule of the set of rules associated with the sporting event;
apply the particular rule to the detected activity to generate a rule determination; and
cause the onboard audio output device to emit an audible output indicative of the rule determination.
2 . The UAV of claim 1 further comprising:
a storage device, the storage device storing the library of the plurality of rules associated with the plurality of different types of sporting events.
3 . The UAV of claim 1 , wherein the particular area associated with the sporting event is a field of play.
4 . The UAV of claim 1 , wherein the moving area of interest corresponds with motion of a ball in play.
5 . The UAV of claim 1 , wherein to control the propulsion system to autonomously maneuver the UAV and rotate the camera relative to the body of the UAV so as to keep the moving area of interest in a field of view of the camera, the computer system is configured to:
generate and continually update, based on the perception inputs, a planned trajectory configured to keep the UAV within a threshold proximity of the moving area of interest while simultaneously avoiding overflying the particular area associated with the sporting event.
6 . The UAV of claim 1 , wherein the computer system is configured to:
cause display of images from the sensor device at a display device.
7 . The UAV of claim 1 , wherein the computer system is configured to:
cause a public address system at a venue hosting the sporting event to generate the audible output indicative of the rule determination.
8 . The UAV of claim 1 , wherein the computer system is configured to:
receive a plurality of other rule determinations from a plurality of other UAVs, each of the plurality of other rule determinations based on independent application of the rule to activity identified by a different one of the plurality of other UAVs; and
generate a final rule determination if the rule determination and the plurality of other rule determinations satisfy a specified matching criterion.
9 . A method comprising:
receiving, by a computer system of a UAV, perception inputs generated by one or more sensors associated with the UAV, the perception inputs including images and the one or more sensors including an image capture device;
receiving, by the computer system, audible tracking commands;
interpreting, by the computer system, the audible tracking commands using natural language processing techniques;
processing the perception inputs to develop a semantic understanding of a surrounding physical environment;
fusing the audible tracking commands with the semantic understanding of the surrounding physical environment to refine the audible tracking commands;
identifying, based on the refined tracking commands, a plurality objects within a particular area associated with a sporting event;
tracking multiple objects of the plurality of objects simultaneously within the particular area associated with the sporting event;
detecting, by the computer system, a moving area of interest within the particular area associated with the sporting event,
wherein the moving area of interest corresponds with an area in which most activity is occurring among the multiple objects;
generating and continually updating a planned trajectory configured to keep the moving area of interest within a field of view of a camera while avoiding overflying the particular area and predefined restricted regions;
identifying a characteristic of the sporting event;
selecting, based on the identified characteristic, a set of rules from a library of rules for a plurality of different types of sporting events by recognizing a sport type based on features of the surrounding physical environment;
processing, by the computer system, the perception inputs to detect an activity occurring during the sporting event;
selecting a particular rule of the set of rules;
applying, by the computer system, the selected rule to the detected activity to generate a rule determination; and
emitting, from an onboard audio output device of the UAV, an audible output indicative of the rule determination.
10 . The method of claim 9 , wherein the particular area associated with the sporting event is a field of play.
11 . The method of claim 9 , wherein the moving area of interest corresponds with the motion of a ball in play.
12 . The method of claim 9 , wherein causing the UAV to autonomously maneuver includes:
generating and continually updating, by the computer system, based on the perception inputs, a planned trajectory configured to keep the UAV within a threshold proximity of the moving area of interest while simultaneously avoiding overflying the particular area associated with the sporting event.
13 . The method of claim 9 , further comprising:
causing display, by the computer system, of images from the image capture device at a display device.
14 . The method of claim 9 , further comprising:
causing, by the computer system, a public address system at a venue hosting the sporting event to generate an audible output indicative of the rule determination.
15 . The method of 9 , further comprising:
receiving, by the computer system, a plurality of other rule determinations from a plurality of other UAVs, each of the plurality of other rule determinations based on independent application of the rule to activity identified by a different one of the plurality of other UAVs; and
generating, by the computer system, a final rule determination if the rule determination and the plurality of other rule determinations satisfy a specified matching criterion.
16 . The method of claim 9 , wherein accessing the rule associated with the sporting event includes:
processing, by the computer system, the perception inputs to identify a characteristic of the sporting event; and
selecting, by the computer system, based on the identified characteristic, the rule from a library including a plurality of rules for a plurality of types of sporting events.
17 . The method of claim 9 , further comprising:
causing display, by the computer system, of a visual output at a display device, the visual output including images from the image capture device and an indication of the rule determinization.
18 . The UAV of claim 1 , wherein to select the set of rules, the computer system is configured to recognize a sport type based on features of the physical environment and the set of rules corresponds to the recognized sport type.
19 . The UAV of claim 1 , wherein to control the propulsion system to autonomously maneuver the UAV, the computer system is configured to:
autonomously avoid predefined restricted regions within the sporting event, including areas over player zones and spectator seating.
20 . The UAV of claim 1 , wherein to generate the rule determination, the computer system is configured to receive preliminary rule determinations from other UAVs and output the rule determination only when a consensus threshold is satisfied.
21 . The UAV of claim 1 , wherein the predefined restricted regions comprise player benches and spectator seating, and the planned trajectory is generated subject to semantic keep-out classes detected from the perception inputs.
22 . The method of claim 9 , wherein the predefined restricted regions comprise player benches and spectator seating, and the planned trajectory is generated subject to semantic keep-out classes detected from the perception inputs.