Super-resolution apparatus and method for virtual and mixed reality
An apparatus and method for efficiently improving virtual/real interactions in augmented reality. For example, one embodiment of a method comprises: capturing a raw image including depth data; identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects; generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof; detecting interactions between the virtual objects and the real objects using the super-resolution map; and performing one or more graphics processing or general purpose processing operations based on the detected interactions.
1. A method comprising:
capturing a raw image including depth data;
identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances;
generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof;
detecting interactions between the virtual objects and the real objects using the super-resolution map; and
performing one or more graphics processing or general-purpose processing operations based on the detected interactions.
2. The method of claim 1 wherein the machine-learning techniques comprise:
performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and
evaluating results of the comparisons.
3. The method of claim 2 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics.
4. The method of claim 3 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map.
5. The method of claim 4 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model.
6. The method of claim 1 wherein identifying the one or more regions of interest further comprises:
if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest.
7. The method of claim 1 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount.
8. The method of claim 1 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions.
9. An apparatus comprising:
a camera to capture a raw image including depth data;
region detection circuitry to identify one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances;
depth super-resolution circuitry to generate a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof;
interaction detection circuitry to detect interactions between the virtual objects and the real objects using the super-resolution map; and
wherein one or more graphics processing or general-purpose processing operations are to be performed based on the detected interactions.
10. The apparatus of claim 9 wherein the machine-learning techniques comprise:
performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and
evaluating results of the comparisons.
11. The apparatus of claim 10 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics.
12. The apparatus of claim 11 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map.
13. The apparatus of claim 12 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model.
14. The apparatus of claim 9 wherein identifying the one or more regions of interest further comprises:
if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest.
15. The apparatus of claim 9 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount.
16. The apparatus of claim 9 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions.
17. A non-transitory machine-readable medium having program code stored therein which, when executed by a machine, causes the machine to perform the operations of:
capturing a raw image including depth data;
identifying one or more regions of interest based on a detected spatial proximity of one or more virtual objects and one or more real objects, wherein the spatial proximity detection includes measuring a first distance between a virtual object and a user and a second distance between a real object and the user, and determining the difference between the first and second distances;
generating a super-resolution map of the one or more regions of interest using machine-learning techniques or results thereof;
detecting interactions between the virtual objects and the real objects using the super-resolution map; and
performing one or more graphics processing or general-purpose processing operations based on the detected interactions.
18. The non-transitory machine-readable medium of claim 1 wherein the machine-learning techniques comprise:
performing comparisons between a plurality of high-resolution images and a corresponding plurality of super-resolution images generated from down-sampled versions of the high-resolution images; and
evaluating results of the comparisons.
19. The non-transitory machine-readable medium of claim 18 wherein the machine-learning techniques are implemented with a generative adversarial network (GAN) with depth field statistics.
20. The non-transitory machine-readable medium of claim 19 wherein at least one of the results of the machine-learning techniques comprise a trained model usable to generate the super-resolution map.
21. The non-transitory machine-readable medium of claim 20 wherein the machine-learning techniques incorporate a characteristic of a depth map as a loss function for the trained model.
22. The non-transitory machine-readable medium of claim 17 wherein identifying the one or more regions of interest further comprises:
if the first and second distances are within a specified threshold value, then using the virtual object and real object to define a first region of interest.
23. The non-transitory machine-readable medium of claim 17 wherein generating the super-resolution map comprises upscaling the one or more regions of interest to increase resolution by a specified amount.
24. The non-transitory machine-readable medium of claim 17 further comprising one of the graphics processing operations comprises generating graphical indication of at least one of the detected interactions.