Object segmentation and feature tracking
Examples are described for processing images to mask dynamic objects out of images to improve feature tracking between images. A device receives an image of an environment captured by an image sensor. The image depicts at least a static portion of the environment and a dynamic object in the environment. The device identifies a portion of the image that includes a depiction of the dynamic object. For example, the device can detect a bounding box around the dynamic object, or can detect which pixels in the image correspond to the dynamic object. The device generates a masked image at least by masking the portion of the image. The device identifies features in the masked image, and uses the features from the masked image for feature tracking from other images of the environment, masked or otherwise. The device can use this feature tracking for mapping, localization, and/or relocation.
1. An apparatus for image processing, the apparatus comprising:
a memory; and
one or more processors coupled to the memory and configured to:
identify a first portion of an image of an environment captured by an image sensor, wherein the first portion of the image depicts a first object in the environment and a second portion of the image depicts a second object in the environment;
identify one or more features of the second object in a masked image, wherein the first portion of the image depicting the first object is masked in the masked image; and
track the one or more features of the second object between the masked image and one or more additional masked images of the environment, wherein the first object is masked in the one or more additional masked images.
2. The apparatus of claim 1 , wherein the one or more processors are configured to:
determine a location of a first feature of the one or more features of the second object based on tracking of the one or more features of the second object between the masked image and the one or more additional masked images of the environment; and
update a map of the environment based on the location of the first feature.
3. The apparatus of claim 2 , wherein, to update the map of the environment based on the location, the one or more processors are configured to add the location of the first feature to the map.
4. The apparatus of claim 2 , wherein, to update the map of the environment based on the location, the one or more processors are configured to modify a prior location of the first feature in the map based on the location of the first feature.
5. The apparatus of claim 1 , wherein the one or more processors are configured to:
determine a pose of the apparatus within the environment based on tracking of the one or more features of the second object between the masked image and the one or more additional masked images of the environment, wherein the pose of the apparatus within the environment includes at least one of a location of the apparatus, a pitch of the apparatus, a roll of the apparatus, or a yaw of the apparatus.
6. The apparatus of claim 1 , wherein the one or more processors are configured to:
generate a downscaled image at least by downscaling the image, wherein identifying the first portion of the image depicting the first object includes identifying a portion of the downscaled image that depicts the first object.
7. The apparatus of claim 1 , wherein the one or more processors are configured to:
generate a greyscale image at least by desaturating color in the image, wherein identifying the first portion of the image depicting the first object includes identifying a portion of the greyscale image that depicts the first object.
8. The apparatus of claim 1 , wherein, to identify the first portion of the image depicting the first object, the one or more processors are configured to:
analyze each pixel of a plurality of pixels corresponding to the image to identify a subset of the plurality of pixels that depicts at least a portion of the first object.
9. The apparatus of claim 1 , wherein, to identify the first portion of the image depicting the first object, the one or more processors are configured to identify a bounding box occupying a polygonal region of the image, wherein the first object is at least partially included within the bounding box.
10. The apparatus of claim 9 , wherein, to identify the first portion of the image depicting the first object, the one or more processors are configured to analyze each pixel of a plurality of pixels within the bounding box to identify a subset of the plurality of pixels within the bounding box that each depict a portion of the first object.
11. The apparatus of claim 10 , wherein:
to identify the bounding box, the one or more processors are configured to use at least a first trained neural network; and
to identify the subset of the plurality of pixels, the one or more processors are configured to use at least a second trained neural network.
12. The apparatus of claim 1 , wherein, to identify the first portion of the image depicting the first object, the one or more processors are configured to:
identify, using at least a first trained neural network, that the image depicts the first object; and
identify, using at least a second trained neural network in response to identification that the image depicts the first object, the first portion of the image that depicts the first object.
13. The apparatus of claim 1 , wherein the first object is a dynamic type of object and the second object is a static type of object.
14. The apparatus of claim 13 , wherein the dynamic type of object is a person, and wherein, to identify the first portion of the image depicting the first object, the one or more processors are configured to identify a depiction of a face of the person.
15. The apparatus of claim 13 , wherein the static type of object is static relative to a position of the image sensor during capture of the image, wherein the dynamic type of object moves relative to a position of the image sensor during capture of the image.
16. The apparatus of claim 1 , wherein the apparatus is one of a mobile device, a wireless communication device, a robot, a vehicle, a head-mounted display, and a camera.
17. The apparatus of claim 1 , further comprising:
the image sensor.
18. A method of image processing performed by a device, the method comprising:
identifying a first portion of an image of an environment captured by an image sensor, wherein the first portion of the image depicts a first object in the environment and a second portion of the image depicts a second object in the environment;
identifying one or more features of the second object in a masked image, wherein the first portion of the image depicting the first object is masked in the masked image; and
tracking the one or more features of the second object between the masked image and one or more additional masked images of the environment, wherein the first object is masked in the one or more additional masked images.
19. The method of claim 18 , further comprising:
determining a location of a first feature of the one or more features of the second object based on tracking of the one or more features of the second object between the masked image and the one or more additional masked images of the environment; and
updating a map of the environment based on the location of the first feature.
20. The method of claim 19 , wherein updating the map of the environment based on the location comprises adding the location of the first feature to the map.
21. The method of claim 19 , wherein updating the map of the environment based on the location comprises modifying a prior location of the first feature in the map based on the location of the first feature.
22. The method of claim 18 , further comprising:
determining a pose of the device within the environment based on tracking of the one or more features of the second object between the masked image and the one or more additional masked images of the environment, wherein the pose of the device within the environment includes at least one of a location of the device, a pitch of the device, a roll of the device, or a yaw of the device.
23. The method of claim 18 , further comprising:
generating a downscaled image at least by downscaling the image, wherein identifying the first portion of the image depicting the first object includes identifying a portion of the downscaled image that depicts the first object.
24. The method of claim 18 , further comprising:
generating a greyscale image at least by desaturating color in the image, wherein identifying the first portion of the image depicting the first object includes identifying a portion of the greyscale image that depicts the first object.
25. The method of claim 18 , wherein identifying the first portion of the image depicting the first object comprises:
analyzing each pixel of a plurality of pixels corresponding to the image to identify a subset of the plurality of pixels that depicts at least a portion of the first object.
26. The method of claim 18 , wherein identifying the first portion of the image depicting the first object comprises identifying a bounding box occupying a polygonal region of the image, wherein the first object is at least partially included within the bounding box.
27. The method of claim 26 , wherein identifying the first portion of the image depicting the first object comprises analyzing each pixel of a plurality of pixels within the bounding box to identify a subset of the plurality of pixels within the bounding box that each depict a portion of the first object.
28. The method of claim 27 , wherein:
identifying the bounding box comprises using at least a first trained neural network; and
identifying the subset of the plurality of pixels comprises using at least a second trained neural network.
29. The method of claim 18 , wherein identifying the first portion of the image depicting the first object comprises:
identifying, using at least a first trained neural network, that the image depicts the first object; and
identifying, using at least a second trained neural network in response to identification that the image depicts the first object, the first portion of the image that depicts the first object.
30. The method of claim 18 , wherein the first object is a dynamic type of object and the second object is a static type of object.
31. The apparatus of claim 1 , wherein the one or more processors are configured to:
map the environment based on the one or more features of the second object as tracked across the masked image and the one or more additional masked images.