Object tracking by an unmanned aerial vehicle using visual sensors
Systems and methods are disclosed for tracking objects in a physical environment using visual sensors onboard an autonomous unmanned aerial vehicle (UAV). In certain embodiments, images of the physical environment captured by the onboard visual sensors are processed to extract semantic information about detected objects. Processing of the captured images may involve applying machine learning techniques such as a deep convolutional neural network to extract semantic cues regarding objects detected in the images. The object tracking can be utilized, for example, to facilitate autonomous navigation by the UAV or to generate and display augmentative information regarding tracked objects to users.
1. An autonomous aerial vehicle comprising:
an image capture system configured to capture images of a physical environment while the autonomous aerial vehicle is in flight;
a propulsion system configured to maneuver the autonomous aerial vehicle through the physical environment;
a visual navigation system configured to:
process the captured images to extract semantic information relating to one or more physical objects in the physical environment;
process the semantic information to identify a particular class of physical objects relating to a particular one of the one or more physical objects;
identify a motion model associated with the particular class of physical objects relating to the particular one of the one or more physical objects;
determine, based on the captured images and the motion model, a predicted trajectory associated with the particular one of the one or more physical objects through three-dimensional (3D) space of the physical environment;
track an actual trajectory of the particular one of the one or more physical objects through the 3D space of the physical environment based, at least in part, on the predicted trajectory;
generate and continually update a planned trajectory for the autonomous aerial vehicle through the physical environment that follows the actual trajectory of the particular physical object instance; and
a flight control system configured to generate control commands configured to cause the propulsion system of the autonomous aerial vehicle to maneuver along the planned trajectory.
2. The autonomous aerial vehicle of claim 1 , wherein the image capture system comprises:
an array of multiple stereoscopic image capture devices placed around a perimeter of the autonomous aerial vehicle to provide stereoscopic image capture for autonomous navigation; and
a gimballed image capture device configured to capture images of the images of the physical environment while the autonomous vehicle is in flight.
3. The autonomous aerial vehicle of claim 2 , wherein the array of multiple stereoscopic image capture devices provide a full 360 degree view around the autonomous aerial vehicle.
4. The autonomous aerial vehicle of claim 1 , wherein to determine the predicted trajectory associated with the particular one of the one or more physical objects through 3D space of the physical environment, the visual navigation system is configured to:
feed the captured images into a visual-inertial state estimation system that uses a spatiotemporal factor graph to predict the trajectory, wherein the spatiotemporal factor graph probabilistically minimizes total measurement error using non-linear optimization to predict the trajectory using variable values.
5. The autonomous aerial vehicle of claim 4 , wherein the visual-inertial state estimation system estimates or predicts a pose and velocity of the particular one of the one or more physical objects at a particular time step by inputting an estimated pose and velocity at a prior time step as well as stereo depth measurements and camera image measurements obtained via at least the identified motion model.
6. The autonomous aerial vehicle of claim 1 , wherein the visual navigation system is further configured to:
process the captured images to detect the physical objects in the physical environment; and
process the captured images distinguish the physical objects from a background of the captured images.
7. The autonomous aerial vehicle of claim 6 , wherein the visual navigation system is further configured to identify the particular one of the one or more physical objects.
8. The autonomous aerial vehicle of claim 1 , wherein the semantic information includes information regarding any of a position, orientation, shape, size, scale, appearance, pixel segmentation, or activity of the detected one or more physical objects.
9. The autonomous aerial vehicle of claim 1 , wherein the particular class of physical objects is selected from a list of classes of physical objects comprising one or more of people, animals, vehicles, buildings, landscape features, and plants.
10. A method of operating an aerial vehicle for autonomously tracking physical objects in a physical environment, the method comprising:
processing captured images of a physical environment to extract semantic information relating to one or more physical objects in the physical environment;
processing the semantic information to identify a particular class of physical objects relating to a particular one of the one or more physical objects;
identifying a motion model associated with the particular class of physical objects relating to the particular one of the one or more physical objects;
estimating, based on the captured images and the motion model, a predicted trajectory of the particular one of the one or more physical objects through three-dimensional (3D) space of the physical environment;
tracking an actual trajectory of the particular one of the one or more physical objects through the 3D space of the physical environment based, at least in part, on the predicted trajectory; and
continually updating a planned trajectory for the aerial vehicle through the physical environment that follows the tracked actual trajectory of the particular physical object instance.
11. The method of claim 10 , further comprising:
receiving the captured images of the physical environment from one or more image capture devices coupled to the autonomous vehicle.
12. The method of claim 10 , further comprising:
generating control commands configured to cause a propulsion system of the aerial vehicle to maneuver along the planned trajectory.
13. The method of claim 10 , wherein estimating the predicted trajectory associated with the particular one of the one or more physical objects through 3D space of the physical environment comprises:
feeding the captured images into a visual-inertial state estimation system that includes a spatiotemporal factor graph to predict the trajectory.
14. The method of claim 13 , wherein the spatiotemporal factor graph probabilistically minimizes total measurement error using non-linear optimization to predict the trajectory using variable values.
15. The method of claim 14 , wherein the visual-inertial state estimation system estimates or predicts a pose and velocity of the particular one of the one or more physical objects at a particular time step by inputting an estimated pose and velocity at a prior time step as well as stereo depth measurements and camera image measurements obtained via at least the identified motion model.
16. The method of claim 10 , further comprising:
processing the captured images to detect the physical objects in the physical environment; and
processing the captured images distinguish the physical objects from a background of the captured images.
17. The method of claim 16 , further comprising identifying the particular one of the one or more physical objects.
18. A navigation system of an aerial vehicle, the navigation system configured to:
process captured images of a physical environment to extract semantic information relating to one or more physical objects in the physical environment;
process the semantic information to identify a particular class of physical objects relating to a particular one of the one or more physical objects;
identify a motion model associated with the particular class of physical objects relating to the particular one of the one or more physical objects;
estimate, based on the captured images and the motion model, a predicted trajectory of the particular one of the one or more physical objects through three-dimensional (3D) space of the physical environment;
track an actual trajectory of the particular one of the one or more physical objects through the 3D space of the physical environment based, at least in part, on the predicted trajectory; and
generate and continually update a planned trajectory for the autonomous aerial vehicle through the physical environment that follows the tracked actual trajectory of the particular physical object instance.
19. The navigation system of claim 18 , wherein to estimate the predicted trajectory associated with the particular one of the one or more physical objects through 3D space of the physical environment, the method comprises:
feeding the captured images into a visual-inertial state estimation system that includes a spatiotemporal factor graph to predict the trajectory, wherein the spatiotemporal factor graph probabilistically minimizes total measurement error using non-linear optimization to predict the trajectory using variable values.
20. The navigation system of claim 19 , wherein the visual-inertial state estimation system estimates or predicts a pose and velocity of the particular one of the one or more physical objects at a particular time step by inputting an estimated pose and velocity at a prior time step as well as stereo depth measurements and camera image measurements obtained via at least the identified motion model.