Generating stereo image data from monocular images

1. A computer-implemented method comprising:

receiving a first image of a scene;

determining a depth map for the first image;

converting the depth map into a disparity map; and

generating, using the disparity map, a second image from the first image, the first and second images forming a stereo image pair, wherein generating comprises:

forward warping pixels of the first image using the disparity map, and filling an occlusion hole in the second image using a texture from a third image randomly selected from a set of images including the first image and the third image.

2. The computer-implemented method of claim 1 , wherein filling the occlusion hole further comprises:

generating a fourth image by performing color transfer between the first image and the third image; and

replacing pixels of the occlusion hole in the second image with pixels at corresponding positions in the fourth image.

3. The computer-implemented method of claim 1 , wherein generating the second image further comprises resolving a collision between a first pixel and a second pixel of the first image that both correspond to a position in the second image, the resolving of the collision comprising:

comparing a first disparity in the disparity map corresponding to the first pixel to a second disparity in the disparity map corresponding to the second pixel; and

responsive to determining, based on the comparison, that the first disparity is higher than the second disparity, using the first pixel at the position in the second image.

4. The computer-implemented method of claim 1 , wherein converting the depth map into the disparity map comprises:

applying a disparity estimation function to a plurality of depth values included in the depth map, the disparity estimation function simulating a plurality of camera focal lengths.

5. The computer-implemented method of claim 4 , wherein applying the disparity estimation function comprises:

determining a disparity value of the disparity map corresponding to a depth value of the depth map by:

determining a ratio of a maximum depth value to the depth value; and

applying a scaling factor to the ratio, the scaling factor randomly sampled from a range over a minimum disparity value to a maximum disparity value.

6. The computer-implemented method of claim 1 , wherein generating the second image further comprises:

identifying a pixel in the second image as a flying pixel responsive a disparity value in the disparity map corresponding to the pixel having an edge filter response exceeding a response threshold; and

responsive to identifying the flying pixel, adjusting the disparity value corresponding to the pixel using an additional disparity value corresponding to a non-flying pixel in the disparity map.

7. The computer-implemented method of claim 1 , further comprising:

training a depth estimation model to predict depth maps for single images using the stereo image pair.

8. The computer-implemented method of claim 7 , further comprising:

receiving an additional image of an additional scene;

determining an additional depth map for the additional image using the trained stereo depth estimation model; and

displaying the additional image with virtual content using the additional depth map.

9. The computer-implemented method of claim 1 , further comprising:

training a stereo matching model to determine per-pixel displacement for stereo image pairs using the stereo image pair.

10. The computer-implemented method of claim 1 , wherein determining the depth map comprises:

generating the depth map using a pretrained depth estimation model.

11. The computer-implemented method of claim 1 , wherein determining the depth map comprises:

generating the depth map using a pretrained depth estimation model.

12. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

receiving a first image of a scene;

determining a depth map for the first image;

converting the depth map into a disparity map; and

generating, using the disparity map, a second image from the first image, the first and second images forming a stereo image pair, wherein generating comprises:

forward warping pixels of the first image using the disparity map, and filling an occlusion hole in the second image using a texture from a third image randomly selected from a set of images including the first image and the third image.

13. The non-transitory computer-readable storage medium of claim 12 , wherein filling the occlusion hole comprises:

generating a fourth image by performing color transfer between the first image and the third image; and

replacing pixels of the occlusion hole in the second image with pixels at corresponding positions in the fourth image.

14. The non-transitory computer-readable storage medium of claim 12 , wherein generating the second image further comprises resolving a collision between a first pixel and a second pixel of the first image that both correspond to a position in the second image, the resolving of the collision comprising:

comparing a first disparity in the disparity map corresponding to the first pixel to a second disparity in the disparity map corresponding to the second pixel; and

responsive to determining, based on the comparison, that the first disparity is higher than the second disparity, using the first pixel at the position in the second image.

15. The non-transitory computer-readable storage medium of claim 12 , wherein converting the depth map into the disparity map comprises:

applying a disparity estimation function to a plurality of depth values included in the depth map, the disparity estimation function simulating a plurality of camera focal lengths.

16. A computer-implemented method comprising:

receiving a first image of a scene;

determining a depth map for the first image;

converting the depth map into a disparity map; and

generating, using the disparity map, a second image from the first image, the first and second images forming a stereo image pair, wherein generating comprises:

forward warping pixels of the first image using the disparity map, and resolving a collision between a first pixel and a second pixel of the first image that both correspond to a position in the second image, wherein resolving of the collision comprises:

comparing a first disparity in the disparity map corresponding to the first pixel to a second disparity in the disparity map corresponding to the second pixel, and

responsive to determining, based on the comparison, that the first disparity is higher than the second disparity, using the first pixel at the position in the second image.

17. The computer-implemented method of claim 16 , wherein generating the second image further comprises filling an occlusion hole in the second image using a texture from a third image.

18. The computer-implemented method of claim 17 , wherein the third image is randomly selected from a set of images including the first image and the third image.

19. The computer-implemented method of claim 18 , wherein filling the occlusion hole comprises:

generating a fourth image by performing color transfer between the first image and the third image; and

replacing pixels of the occlusion hole in the second image with pixels at corresponding positions in the fourth image.

20. The computer-implemented method of claim 16 , wherein converting the depth map into the disparity map comprises:

applying a disparity estimation function to a plurality of depth values included in the depth map, the disparity estimation function simulating a plurality of camera focal lengths.

21. The computer-implemented method of claim 20 , wherein applying the disparity estimation function comprises:

determining a disparity value of the disparity map corresponding to a depth value of the depth map by:

determining a ratio of a maximum depth value to the depth value; and

applying a scaling factor to the ratio, the scaling factor randomly sampled from a range over a minimum disparity value to a maximum disparity value.

22. The computer-implemented method of claim 16 , wherein generating the second image further comprises:

identifying a pixel in the second image as a flying pixel responsive a disparity value in the disparity map corresponding to the pixel having an edge filter response exceeding a response threshold; and

responsive to identifying the flying pixel, adjusting the disparity value corresponding to the pixel using an additional disparity value corresponding to a non-flying pixel in the disparity map.

23. The computer-implemented method of claim 16 , further comprising:

training a depth estimation model to predict depth maps for single images using the stereo image pair;

receiving an additional image of an additional scene;

determining an additional depth map for the additional image using the trained depth estimation model; and

displaying the additional image with virtual content using the additional depth map.

24. The computer-implemented method of claim 16 , further comprising:

training a stereo matching model to determine per-pixel displacement for stereo image pairs using the stereo image pair.

25. A computer-implemented method comprising:

receiving a first image of a scene;

determining a depth map for the first image;

converting the depth map into a disparity map by applying a disparity estimation function to a plurality of depth values included in the depth map, the disparity estimation function simulating a plurality of camera focal lengths; and

generating, using the disparity map, a second image from the first image, the first and second images forming a stereo image pair.

26. The computer-implemented method of claim 25 , wherein generating the second image comprises forward warping pixels of the first image using the disparity map.

27. The computer-implemented method of claim 26 , wherein generating the second image further comprises filling an occlusion hole in the second image using a texture from a third image.

28. The computer-implemented method of claim 27 , wherein the third image is randomly selected from a set of images including the first image and the third image.

29. The computer-implemented method of claim 28 , wherein filling the occlusion hole further comprises:

generating a fourth image by performing color transfer between the first image and the third image; and

replacing pixels of the occlusion hole in the second image with pixels at corresponding positions in the fourth image.

30. The computer-implemented method of claim 25 , wherein generating the second image further comprises resolving a collision between a first pixel and a second pixel of the first image that both correspond to a position in the second image, the resolving of the collision comprising:

comparing a first disparity in the disparity map corresponding to the first pixel to a second disparity in the disparity map corresponding to the second pixel; and

responsive to determining, based on the comparison, that the first disparity is higher than the second disparity, using the first pixel at the position in the second image.

31. The computer-implemented method of claim 25 , wherein applying the disparity estimation function comprises:

determining a disparity value of the disparity map corresponding to a depth value of the depth map by:

determining a ratio of a maximum depth value to the depth value; and

applying a scaling factor to the ratio, the scaling factor randomly sampled from a range over a minimum disparity value to a maximum disparity value.

32. The computer-implemented method of claim 25 , wherein generating the second image further comprises:

identifying a pixel in the second image as a flying pixel responsive a disparity value in the disparity map corresponding to the pixel having an edge filter response exceeding a response threshold; and

responsive to identifying the flying pixel, adjusting the disparity value corresponding to the pixel using an additional disparity value corresponding to a non-flying pixel in the disparity map.

33. The computer-implemented method of claim 25 , further comprising:

training a depth estimation model to predict depth maps for single images using the stereo image pair;

receiving an additional image of an additional scene;

determining an additional depth map for the additional image using the trained depth estimation model; and

displaying the additional image with virtual content using the additional depth map.

34. The computer-implemented method of claim 25 , further comprising:

training a stereo matching model to determine per-pixel displacement for stereo image pairs using the stereo image pair.

35. A computer-implemented method comprising:

receiving a first image of a scene;

determining a depth map for the first image;

converting the depth map into a disparity map; and

generating, using the disparity map, a second image from the first image, the first and second images forming a stereo image pair, wherein generating comprises:

identifying a pixel in the second image as a flying pixel responsive a disparity value in the disparity map corresponding to the pixel having an edge filter response exceeding a response threshold, and

responsive to identifying the flying pixel, adjusting the disparity value corresponding to the pixel using an additional disparity value corresponding to a non-flying pixel in the disparity map.

36. The computer-implemented method of claim 35 , wherein generating the second image comprises forward warping pixels of the first image using the disparity map.

37. The computer-implemented method of claim 36 , wherein generating the second image further comprises filling an occlusion hole in the second image using a texture from a third image.

38. The computer-implemented method of claim 37 , wherein the third image is randomly selected from a set of images including the first image and the third image.

39. The computer-implemented method of claim 38 , wherein filling the occlusion hole further comprises:

generating a fourth image by performing color transfer between the first image and the third image; and

replacing pixels of the occlusion hole in the second image with pixels at corresponding positions in the fourth image.

40. The computer-implemented method of claim 35 , wherein generating the second image further comprises resolving a collision between a first pixel and a second pixel of the first image that both correspond to a position in the second image, the resolving of the collision comprising:

comparing a first disparity in the disparity map corresponding to the first pixel to a second disparity in the disparity map corresponding to the second pixel; and

responsive to determining, based on the comparison, that the first disparity is higher than the second disparity, using the first pixel at the position in the second image.

41. The computer-implemented method of claim 35 , wherein converting the depth map into the disparity map comprises:

applying a disparity estimation function to a plurality of depth values included in the depth map, the disparity estimation function simulating a plurality of camera focal lengths.

42. The computer-implemented method of claim 41 , wherein applying the disparity estimation function comprises:

determining a disparity value of the disparity map corresponding to a depth value of the depth map by:

determining a ratio of a maximum depth value to the depth value; and

applying a scaling factor to the ratio, the scaling factor randomly sampled from a range over a minimum disparity value to a maximum disparity value.

43. The computer-implemented method of claim 35 , further comprising:

training a depth estimation model to predict depth maps for single images using the stereo image pair;

receiving an additional image of an additional scene;

determining an additional depth map for the additional image using the trained depth estimation model; and

displaying the additional image with virtual content using the additional depth map.

44. The computer-implemented method of claim 35 , further comprising:

training a stereo matching model to determine per-pixel displacement for stereo image pairs using the stereo image pair.