Method for rendering of augmented reality content in combination with external display
Systems and methods are described for improved color rendering. In some embodiments, video is captured of a real-world scene, and a region of screen content is detected in the captured video. A processor selectively applies a screen-content color transformation on the region of screen content to generate processed video, and the processed video is displayed substantially in real time, for example on a video-see-through head-mounted display. The screen-content color processing may be different for different types of external displays. The screen-content color processing may also be determined based at least in part on illumination conditions. In some embodiments, the color of displayed virtual objects is adjusted based on visual parameters of the screen content.
1. A method comprising:
capturing at least one image of a real-world scene;
identifying, in the at least one image, a captured region of screen content;
determining at least one visual parameter of the screen content from the captured region of screen content;
based on the at least one visual parameter, determining a transformation for virtual content;
applying the transformation to the virtual content; and
causing display of the transformed virtual content on an augmented reality display.
2. The method of claim 1 , wherein the at least one visual parameter includes a luminance of the screen content, and wherein the transformation includes a transformation of brightness of the virtual content.
3. The method of claim 2 , wherein the determined luminance of the screen content is a maximum luminance in the captured region of screen content.
4. The method of claim 2 , wherein the transformation comprises a gain factor applied to color code values of the virtual content, and wherein the gain factor increases for increased determined luminance of the captured region of screen content.
5. The method of claim 1 , wherein the at least one visual parameter includes a white point of the screen content, and wherein the transformation includes a transformation of a white point of the virtual content.
6. The method of claim 5 , wherein determining the white point of the screen content comprises applying a grey world algorithm to the captured region of screen content.
7. The method of claim 1 , wherein the image of the real-world scene is captured using a forward-facing camera of the augmented reality display.
8. The method of claim 1 , wherein the augmented reality display is an optical-see-through head-mounted display.
9. The method of claim 1 , further comprising:
using the captured region of screen content, identifying content displayed in the captured regions of screen content; and
selecting the virtual content based on the identified content.
10. The method of claim 1 , further comprising:
identifying a type of display used to display the screen content;
determining at least one further visual parameter based on the type of display, and
further transforming the virtual content based on the further visual parameter.
11. An apparatus comprising a processor configured to perform at least:
capturing at least one image of a real-world scene;
identifying, in the at least one image, a captured region of screen content;
determining at least one visual parameter of the screen content from the captured region of screen content;
based on the at least one visual parameter, determining a transformation for virtual content;
applying the transformation to the virtual content; and
causing display of the transformed virtual content on an augmented reality display.
12. The apparatus of claim 11 , wherein the at least one visual parameter includes a luminance of the screen content, and wherein the transformation includes a transformation of brightness of the virtual content.
13. The apparatus of claim 12 , wherein the determined luminance of the screen content is a maximum luminance in the captured region of screen content.
14. The apparatus of claim 12 , wherein the transformation comprises a gain factor applied to color code values of the virtual content, and wherein the gain factor increases for increased determined luminance of the captured region of screen content.
15. The apparatus of claim 11 , wherein the at least one visual parameter includes a white point of the screen content, and wherein the transformation includes a transformation of a white point of the virtual content.