Determining visual overlap of images by using box embeddings
An image matching system for determining visual overlaps between images by using box embeddings is described herein. The system receives two images depicting a 3D surface with different camera poses. The system inputs the images (or a crop of each image) into a machine learning model that outputs a box encoding for the first image and a box encoding for the second image. A box encoding includes parameters defining a box in an embedding space. Then the system determines an asymmetric overlap factor that measures asymmetric surface overlaps between the first image and the second image based on the box encodings. The asymmetric overlap factor includes an enclosure factor indicating how much surface from the first image is visible in the second image and a concentration factor indicating how much surface from the second image is visible in the first image.
1. A computer-implemented method comprising:
receiving a first image depicting a 3D surface;
inputting the first image into a machine learning model, the machine learning model outputting a first box encoding of the first image, the first box encoding including parameters defining a first box representing the first image in an embedding space;
retrieving a second box encoding of a second image, the second image depicting the 3D surface with a different camera pose from the first image, the second box encoding including parameters defining a second box representing the second image in the embedding space; and
determining an asymmetric overlap factor measuring asymmetric surface overlaps between the first image and the second image based on the first and second box encodings, the asymmetric overlap factor including an enclosure factor indicating how much surface from the first image is visible in the second image and a concentration factor indicating how much surface from the second image is visible in the first image.
2. The computer-implemented method of claim 1 , wherein the machine learning model is trained by:
receiving a plurality of training images depicting the 3D surface or a second 3D surface with a variety of camera poses;
determining a depth map of each training image;
forming a plurality of pairs of training images;
for each pair of training images, computing normalized surface overlap of the two training images in the pair based on the depth maps of the two training images;
forming a training set including the plurality of training images and the normalized surface overlap of each pair of training images; and
training the model based on the training set by using machine learning techniques.
3. The computer-implemented method of claim 1 , wherein the machine learning model is trained by:
receiving a plurality of training images depicting the 3D surface or a second 3D surface with a variety of camera poses;
for each training image:
partitioning the training image to form a plurality of portions, and
applying a homography to each portion to generate a patch;
forming a plurality of pairs of training images;
for each pair of training images, computing normalized surface overlap of the two training images in the pair based on the patches of the two training images;
forming a training set including the plurality of training images and the normalized surface overlap of each pair of training images; and
training the model based on the training set by using machine learning techniques.
4. The computer-implemented method of claim 1 , wherein the machine learning model is a convolutional neural network.
5. The computer-implemented method of claim 1 , wherein the parameters defining the box in the embedding space including a first parameter defining a center of the box in the embedding space and a second parameter defining dimensions of the box in the embedding space.
6. The computer-implemented method of claim 1 , wherein determining the asymmetric overlap factor measuring asymmetric surface overlaps between the first image and the second image based on the first and second box encodings comprises:
computing an intersection of the two boxes in the embedding space based on the first and second box encodings;
determining a volume of the second box in the embedding space based on the second box encoding of the second image; and
determining the enclosure factor based on a ratio of the intersection to the volume of the second box.
7. The computer-implemented method of claim 1 , wherein determining the asymmetric overlap factor measuring asymmetric surface overlaps between the first image and the second image based on the first and second box encodings comprises:
computing an intersection of the two boxes in the embedding space based on the first and second box encodings;
determining a volume of the first box in the embedding space based on the first box encoding of the first image; and
determining the concentration factor based on a ratio of the intersection to the volume of the first box.
8. A non-transitory computer-readable memory storing executable computer program instructions, the instructions executable to perform operations comprising:
receiving a first image depicting a 3D surface;
inputting the first image into a machine learning model, the machine learning model outputting a first box encoding of the first image, the first box encoding including parameters defining a first box representing the first image in an embedding space;
retrieving a second box encoding of a second image, the second image depicting the 3D surface with a different camera pose from the first image, the second box encoding including parameters defining a second box representing the second image in the embedding space; and
determining an asymmetric overlap factor measuring asymmetric surface overlaps between the first image and the second image based on the first and second box encodings, the asymmetric overlap factor including an enclosure factor indicating how much surface from the first image is visible in the second image and a concentration factor indicating how much surface from the second image is visible in the first image.
9. The non-transitory computer-readable memory of claim 8 , wherein the machine learning model is trained by:
receiving a plurality of training images depicting the 3D surface or a second 3D surface with a variety of camera poses;
determining a depth map of each training image;
forming a plurality of pairs of training images;
for each pair of training images, computing normalized surface overlap of the two training images in the pair based on the depth maps of the two training images;
forming a training set including the plurality of training images and the normalized surface overlap of each pair of training images; and
training the model based on the training set by using machine learning techniques.
10. The non-transitory computer-readable memory of claim 8 , wherein the machine learning model is trained by:
receiving a plurality of training images depicting the 3D surface or a second 3D surface with a variety of camera poses;
for each training image:
partitioning the training image to form a plurality of portions, and
applying a homography to each portion to generate a patch;
forming a plurality of pairs of training images;
for each pair of training images, computing normalized surface overlap of the two training images in the pair based on the patches of the two training images;
forming a training set including the plurality of training images and the normalized surface overlap of each pair of training images; and
training the model based on the training set by using machine learning techniques.
11. The non-transitory computer-readable memory of claim 8 , wherein the machine learning model is a convolutional neural network.
12. The non-transitory computer-readable memory of claim 8 , wherein the parameters defining the box in the embedding space including a first parameter defining a center of the box in the embedding space and a second parameter defining dimensions of the box in the embedding space.
13. The non-transitory computer-readable memory of claim 8 , wherein determining the asymmetric overlap factor measuring asymmetric surface overlaps between the first image and the second image based on the first and second box encodings comprises:
computing an intersection of the two boxes in the embedding space based on the first and second box encodings;
determining a volume of the second box in the embedding space based on the second box encoding of the second image; and
determining the enclosure factor based on a ratio of the intersection to the volume of the second box.
14. The non-transitory computer-readable memory of claim 8 , wherein determining the asymmetric overlap factor measuring asymmetric surface overlaps between the first image and the second image based on the first and second box encodings comprises:
computing an intersection of the two boxes in the embedding space based on the first and second box encodings;
determining a volume of the first box in the embedding space based on the first box encoding of the first image; and
determining the concentration factor based on a ratio of the intersection to the volume of the first box.