Apparatus and method with homographic image processing
An apparatus and method for processing an image are disclosed. The method includes: segmenting both a first image and a second image and generating segmentation mask pairs, each segmentation mask pair having a segmentation mask of the first image and a segmentation mask of the second image; generating local homography matrices of the first image with respect to the second image, based on the segmentation mask pairs, the first image, and the second image; and generating a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image.
1 . A method of processing an image, the method comprising:
segmenting both a first image and a second image to generate first segmentation masks of the first image and second segmentation masks of th e second image, and generating segmentation mask pairs by deter 1 matches between t amentation masks and the second segmentation masks, each segmentation mask pair having a first segmentation mask of the first image and a second segmentation mask of the second image;
generating local homography matrices of the first image with respect to the second image, based on the segmentation mask pairs, the first image, and the second image; and
generating a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image.
2 . The method of claim 1 , wherein the first and second image are images of a scene comprised of regions respectively corresponding to the segmentation mask pairs, and wherein each segmentation mask pair's images both correspond to the segmentation mask pair's region.
3 . The method of claim 1 ,
wherein the segmenting the first image and the second image comprises generating first initial segmentation masks of the first image and second initial segmentation masks of the second image; and
wherein generating the segmentation mask pairs comprises post-processing the first initial segmentation masks and the second initial segmentation masks.
4 . The method of claim 3 , wherein the post-processing comprises:
determining first segmentation masks according to the first initial segmentation masks and determining a second segmentation masks according to the second initial segmentation masks;
selecting N of the first segmentation masks from the first segmentation masks and selecting N of the second segmentation masks from the second segmentation masks;
merging an unselected first segmentation mask into one of the N first segmentation masks and merging an unselected second segmentation mask into one of the N second segmentation masks; and
generating the segmentation mask pairs by performing mask the matching between the N first segmentation masks and the N second segmentation masks,
wherein each of the first segmentation masks and the second segmentation masks is a segmentation mask of a connected region, and
wherein an area of each of the N first segmentation masks and the N second segmentation masks is greater than a first threshold value.
5 . The method of claim 4 , wherein the determining of the first segmentation masks according to the first initial segmentation masks and determining the second segmentation masks according to the second initial segmentation masks comprises:
in response to a case in which a first segmentation fragment, which is an initial segmentation mask having an area less than a second threshold value, exists among the first initial segmentation masks, filling, among the first initial segmentation masks, the first segmentation fragment using a mask adjacent to the first segmentation fragment and determining each connected region of the first initial segmentation masks to be a first segmentation mask after performing the filling;
determining the each connected region of the first initial segmentation masks to be a first segmentation mask in response to a case in which the first segmentation fragment does not exist;
in response to a case in which a second segmentation fragment, which is an initial segmentation mask having an area less than the second threshold value, exists among the second initial segmentation masks, filling, among the second initial segmentation masks, the second segmentation fragment using a mask adjacent to the second segmentation fragment and determining each connected region of the second initial segmentation masks to be a second segmentation mask after performing the filling; and
determining the each connected region of the second initial segmentation masks to be a second segmentation mask in response to a case in which the second segmentation fragment does not exist,
wherein the first threshold value is greater than the second threshold value.
6 . The method of claim 4 , wherein the merging of the unselected first segmentation mask into the one of the N first segmentation masks and merging the unselected second segmentation mask into the one of the N second segmentation masks comprise at least one of:
in response to a case in which at least one adjacent segmentation mask that is adjacent to the unselected first segmentation mask exists among the N first segmentation masks, merging, in the at least one adjacent segmentation mask, the unselected first segmentation mask into one adjacent segmentation mask that is closest to the unselected first segmentation mask;
in response to a case in which an adjacent segmentation mask that is adjacent to the unselected first segmentation mask does not exist among the N first segmentation masks, merging, among the N first segmentation masks, the unselected first segmentation mask into one first segmentation mask that is closest to the unselected first segmentation mask;
in response to a case in which at least one adjacent segmentation mask that is adjacent to the unselected second segmentation mask exists among the N second segmentation masks, merging, in the at least one adjacent segmentation mask, the unselected second segmentation mask into one adjacent segmentation mask that is closest to the unselected second segmentation mask; or
in response to a case in which an adjacent segmentation mask that is adjacent to the unselected second segmentation mask does not exist among the N second segmentation masks, merging, among the N second segmentation masks, the unselected second segmentation mask into one first segmentation mask that is closest to the unselected second segmentation mask.
7 . The method of claim 4 , wherein a first segmentation mask and a second segmentation mask comprised in each of the segmentation mask pairs satisfy the following conditions: the first segmentation mask and the second segmentation mask belong to a same category, have a least distance therebetween, and have an overlap that satisfies a fourth threshold.
8 . The method of claim 1 , wherein the generating of the local homography matrices of the first image with respect to the second image comprises generating the local homography matrices by applying a first neural network to the segmentation mask pairs, the first image, and the second image.
9 . The method of claim 8 , wherein the generating of the local homography matrices by applying the first neural network to the segmentation mask pairs, the first image, and the second image, comprises:
generating an encoding pyramid feature for the first image, based on a feature map of the first image and first segmentation masks in the segmentation mask pairs;
generating an encoding pyramid feature for the second image, based on a feature map of the second image and second segmentation masks in the segmentation mask pairs; and
predicting the local homography matrices based on the encoding pyramid feature for the first image and the encoding pyramid feature for the second image.
10 . The method of claim 8 , further comprising:
segmenting both a first training image and a second training image of a training image pair and generating a training segmentation mask pairs of the training image pair; and
generating the first neural network by training with the training image pair and the training segmentation mask pairs, based on the training segmentation mask pairs of the training image pair,
wherein the training segmentation mask pair comprises a segmentation mask for one region of the first training image and a segmentation mask for a region corresponding to the one region of the first training image in the second training image.
11 . The method of claim 10 , wherein the segmenting of the first training image and the second training image respectively and the generating the training segmentation mask pairs of the training image pair comprises:
segmenting the first training image and the second training image respectively from one training image pair and generating initial segmentation masks of the first training image and initial segmentation masks of the second training image; and
generating training segmentation mask pairs of the training image pair by post- processing the initial segmentation masks of the first training image and the initial segmentation masks of the second training image.
12 . The method of claim 11 , wherein the generating of the training segmentation mask pairs of the one training image pair by post-processing the initial segmentation masks of the first training image and the initial segmentation masks of the second training image comprises:
determining a first training segmentation masks of the first training image according to the initial segmentation masks of the first training image and determining a second training segmentation masks of the second training image according to the initial segmentation masks of the second training image;
selecting all first training segmentation masks having areas greater than a first threshold value from the first training segmentation masks and selecting all second training segmentation masks having areas greater than the first threshold value from the second training segmentation masks;
removing all segmentation masks of a preset category from among the selected first training segmentation masks and the selected second training segmentation masks; and
generating the training segmentation mask pairs of the one training image pair by performing mask matching on a remaining first training segmentation mask among all the selected first training segmentation masks after the removing and on a remaining second training segmentation mask among all the selected second training segmentation masks after the removing.
13 . The method of claim 1 , wherein a segmentation mask pair are formed by determining that a segmentation mask of the first image matches a segmentation mask of the second image.
14 . The method of claim 1 , wherein the generating of the resulting image obtained by aligning the first image with the second image, based on the local homography matrices, the segmentation mask pairs, the first image, and the second image, comprises:
generating distorted images by applying the respective local homography matrices to the first or second image; and
applying weights to the distorted images and fusing the weighted distorted images.
15 . The method of claim 14 , wherein the weights comprise a trained weight matrix.
16 . The method of claim 14 , wherein the trained weight matrix is trained according to training segmentation mask pairs of a pair of training images.
17 . The method of claim 16 , wherein the training segmentation mask pairs are formed by merging segment mask fragments having an area that satisfies a threshold.
18 . The method of claim 1 , wherein each homography matrix aligns a region of the first image with a corresponding region of the second image.
19 . The method of claim 18 , wherein one of the segmentation masks in one of the segmentation mask pairs is formed by merging two initial segmentation masks determined to have a same classification, the classifications of the initial segmentation masks determined from the first or second image.
20 . An electronic device comprising:
one or more processors; and
memory storing computer-executable instructions configured to cause the one or more processors to:
segment both a first image and a second image generate first segmentation masks of the first image and second s egmentation eco je, and generate a segmentation mask pairs by determin ree n masks and the second segmentation masks, each segmentation mask pair having a first segmentation mask of the first image and a second segmentation mask of the second image;
generate a local homography matrices of the first image with respect to the second image, based on the segmentation mask pairs, the first image, and the second image; and
generate a synthetic image obtained by aligning the first image with the second image, wherein the aligning is performed based on the local homography matrices, the segmentation mask pairs, the first image, and the second image.