Optimizations for dynamic object instance detection, segmentation, and structure mapping
In one embodiment, a system may access an image and generate a feature map for the image using a neural network. The system may identify regions of interest in the feature map. Regional feature maps may be generated for the regions of interest, respectively. Each of the regional feature maps has a first, a second, and a third dimension. The system may generate a first combined regional feature map by combining the regional feature maps. The combined regional feature map has a first, a second, and a third dimension. The system may generate a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers. The system may generate, for each of the regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.
1. A method comprising, by a computing system:
accessing an image;
generating a feature map for the image using a neural network;
identifying a plurality of regions of interest in the feature map;
generating a plurality of regional feature maps for the plurality of regions of interest, respectively, wherein each of the plurality of regional feature maps has a first dimension, a second dimension, and a third dimension;
generating a first combined regional feature map by combining the plurality of regional feature maps, wherein the combined regional feature map has a first dimension, a second dimension, and a third dimension;
generating a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers, wherein the processing of the first combined regional feature map is performed using a neural processing engine configured for performing convolutional operations on three-dimensional tensors; and
generating, for each of the plurality of regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.
2. The method of claim 1 , wherein the first dimension and the second dimension of the first combined regional feature map are equal to the first dimension and the second dimension of each of the plurality of regional feature maps, respectively.
3. The method of claim 2 , wherein the third dimension of the first combined regional feature map is equal to or larger than a combination of the respective third dimensions of the plurality of regional feature maps.
4. The method of claim 3 ,
wherein the third dimension of each of the plurality of regional feature maps corresponds to height size or width size;
wherein the first dimension or the second dimension corresponds to channel size.
5. The method of claim 1 , wherein the first combined regional feature map includes the plurality of regional feature maps with paddings inserted between adjacent pairs of the plurality of regional feature maps.
6. The method of claim 5 , wherein a size of the padding between each adjacent pair of the plurality of regional feature maps is at least as wide as a kernel size used by the one or more convolutional layers.
7. The method of claim 1 , wherein the information associated with the object instance is an instance segmentation mask, a keypoint mask, or a bounding box.
8. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
accessing an image;
generating a feature map for the image using a neural network;
identifying a plurality of regions of interest in the feature map;
generating a plurality of regional feature maps for the plurality of regions of interest, respectively, wherein each of the plurality of regional feature maps has a first dimension, a second dimension, and a third dimension;
generating a first combined regional feature map by combining the plurality of regional feature maps, wherein the combined regional feature map has a first dimension, a second dimension, and a third dimension;
generating a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers, wherein the processing of the first combined regional feature map is performed using a neural processing engine configured for performing convolutional operations on three-dimensional tensors; and
generating, for each of the plurality of regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.
9. The system of claim 8 , wherein the first dimension and the second dimension of the first combined regional feature map are equal to the first dimension and the second dimension of each of the plurality of regional feature maps, respectively.
10. The system of claim 9 , wherein the third dimension of the first combined regional feature map is equal to or larger than a combination of the respective third dimensions of the plurality of regional feature maps.
11. The system of claim 10 ,
wherein the third dimension of each of the plurality of regional feature maps corresponds to height size or width size; and
wherein the first dimension or the second dimension corresponds to channel size.
12. The system of claim 8 , wherein the first combined regional feature map includes the plurality of regional feature maps with paddings inserted between adjacent pairs of the plurality of regional feature maps.
13. The system of claim 12 , wherein a size of the padding between each adjacent pair of the plurality of regional feature maps is at least as wide as a kernel size used by the one or more convolutional layers.
14. One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:
accessing an image;
generating a feature map for the image using a neural network;
identifying a plurality of regions of interest in the feature map;
generating a plurality of regional feature maps for the plurality of regions of interest, respectively, wherein each of the plurality of regional feature maps has a first dimension, a second dimension, and a third dimension;
generating a first combined regional feature map by combining the plurality of regional feature maps, wherein the combined regional feature map has a first dimension, a second dimension, and a third dimension;
generating a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers, wherein the processing of the first combined regional feature map is performed using a neural processing engine configured for performing convolutional operations on three-dimensional tensors; and
generating, for each of the plurality of regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.
15. The media of claim 14 , wherein the first dimension and the second dimension of the first combined regional feature map are equal to the first dimension and the second dimension of each of the plurality of regional feature maps, respectively.
16. The media of claim 15 , wherein the third dimension of the first combined regional feature map is equal to or larger than a combination of the respective third dimensions of the plurality of regional feature maps.
17. The media of claim 16 ,
wherein the third dimension of each of the plurality of regional feature maps corresponds to height size or width size;
wherein the first dimension or the second dimension corresponds to channel size.