Systems and methods for display content conversion and optimization for virtual display systems
Systems and methods of converting visual content for display on a virtual display system include extracting depth information from input visual content and formatting the input visual content and the depth information. The virtual display system may produce virtual images that are multifocal virtual images. The conversion of the input visual content into the multifocal virtual images may be impacted by properties of the human vision system, physical modeling of the input visual content, user input or sensory data, or generative content.
1. A conversion system comprising:
a first computational module to receive an input two-dimensional (2D) visual content lacking depth information about the 2D visual content;
a second computational module having a first function to estimate a depth map from the input 2D visual content without reference to independent data about the depth information, the depth map corresponding to the input 2D visual content; and
a third computational module to use a color calibration to format the depth map and the input 2D visual content as content for display as a multifocal virtual image on a virtual display system, wherein the color calibration corrects a chromatic variation of a human vision system.
2. The conversion system of claim 1 , further comprising the virtual display system to display the multifocal virtual image.
3. The conversion system of claim 1 , wherein the first computational module takes as input at least one parameter of a multifocal output template.
4. The conversion system of claim 1 , further comprising a sensor to capture information about a viewer of the virtual display system, wherein the function of the second computational module uses the information to modify the content of the multifocal virtual image.
5. The conversion system of claim 1 , wherein the first function of the second computational module modifies the content of multifocal virtual image based on a property of a human vision system.
6. The conversion system of claim 1 , wherein the first function of the second computational module modifies the content of the multifocal virtual image based on a physics model of the input 2D visual content.
7. The conversion system of claim 1 , wherein the first function of the second computational module modifies the content of the multifocal virtual image using an in-painting subfunction or an artificial intelligence (AI)-generative subfunction.
8. The conversion system of claim 1 , wherein the input 2D visual content is selected from a group consisting of a 2D color image, a 2D monochrome image, a 2D video, a 2D movie, a 2D animation, and combinations thereof.
9. The conversion system of claim 1 , further comprising a graphical user interface to show a list of choices of input 2D visual content, a list of parameters to impact the function, and a locator to determine where to save the content.
10. The conversion system of claim 1 , wherein the second computational module further includes an optimization function to operate on the input 2D visual content to impact the multifocal virtual image.
11. The conversion system of claim 1 , wherein the first function of the second computational module impacts an optical property of the multifocal virtual image, the optical property being selected from a group consisting of color, intensity, polarization, directionality, and combinations thereof.
12. The conversion system of claim 1 , wherein the second computational module operates in real time, and the input 2D visual content is selected from a group consisting of a video game environment, simulation training environment, and a teleconferencing environment.
13. The conversion system of claim 1 , wherein the second computational module includes a neural network to operate on the input 2D visual content.
14. The conversion system of claim 1 , wherein one of the first computational module, the second computation module, or the third computational module (i) creates a parameterization of the depth map using a thresholding function, and (ii) assigns pixel values to every focal plane in the multifocal virtual image based on the depth map, the input visual content, and the parameterization.
15. The conversion system of claim 1 , wherein the multifocal virtual image is impacted by a multifocal shader, the multifocal shader selected from a group consisting of a feathering shader, a geometric scaling shader, an in-painting shader, a dithering shader, a blending shader, haze shader, and a texture shader.
16. The conversion system of claim 1 , wherein the multifocal virtual image is modified by a property of an extended object in the input visual content.
17. The conversion system of claim 1 , wherein the multifocal virtual image is visible in a continuous headbox spanning at least 30 cm, wherein the headbox is a continuous volume in which the multifocal virtual image able to be viewed by both eyes of an observer.
18. The conversion system of claim 1 , wherein the multifocal virtual image is impacted by a feathering shader.
19. The conversion system of claim 1 , wherein the second computational module further comprises a semantic segmentation function to operate on the input 2D visual content and the depth map to produce a segmentation of the depth map, wherein the semantic segmentation is a neural-network based operation.
20. The conversion system of claim 1 , wherein the first function is implemented at least in part by at least one of the group consisting of a feedforward neural network, a recurrent neural networks (RNN), a residual neural network, a generative adversarial networks (GANs), a modular neural networks, a transformer-based network, a semantic segmentation neural-network based operation, a feedforward network, a multi-layer perceptron network, and a convolutional neural networks (CNN).
21. A conversion system comprising:
a first computational module to receive an input two-dimensional (2D) visual content lacking depth information about the 2D visual content;
a second computational module having a function to estimate a depth map from the input 2D visual content without reference to independent data about the depth information, the depth map corresponding to the input 2D visual content; and
a third computational module (i) to use the depth map to assign each of a plurality of pixels of the input 2D visual content to a focal plane among a plurality of focal planes of and (ii) to use a color calibration function to correct a chromatic variation of a human vision system.
22. The conversion system of claim 21 , further comprising a virtual display system to display the multifocal virtual image.
23. The conversion system of claim 21 , wherein the virtual display system comprises a first light source that produces a first display content for the first focal plane, and a second light source that produces a second display content for the second focal plane.
24. The conversion system of claim 21 , wherein the information is a property selected from a group consisting of color, spatial frequency, brightness, size, shape, morphology, and combinations thereof.
25. The conversion system of claim 21 , wherein the second computational module uses an object detection function to identify the information.
26. The conversion system of claim 21 , wherein the second computational module uses a threshold function to format the information.
27. The conversion system of claim 21 , wherein the first focal plane and the second focal plane correspond respectively to a foreground and a background of the 2D visual content.
28. A conversion method comprising:
receiving an input two-dimensional (2D) visual content lacking depth information about the 2D visual content;
operating on the input visual content with a function to estimate a depth map from the input 2D visual content without reference to independent data about the depth information, the depth map corresponding to the input 2D visual content; and
formatting the input 2D visual content using the depth map and a color correction for display as a multifocal virtual image, wherein the color calibration corrects a chromatic variation of a human vision system.
29. The conversion method of claim 28 , wherein the input 2D visual content is selected from a group consisting of a color image, a monochrome image, a video, a movie, an animation, a stereoscopic pair, a three-dimensional point cloud, an anaglyph, an autostereoscopic content, and combinations thereof.
30. The conversion method of claim 28 , further comprising displaying the multifocal virtual image on a virtual display system.