Multi-plane augmented reality image generation
A set of training images are input into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images. Based on the pixel-wise phase matrix, a spatial light modulator (SLM) is actuated to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes. Each training image corresponds to one respective sub-image. A feedback image of the AR image is obtained via an image sensor. An offset is determined based on comparing the training images to the feedback image. Parameters of a loss function are updated based on the offset, and the updated parameters are provided to the neural network to obtain an updated offset.
1. A system, comprising:
a spatial light modulator (SLM) arranged to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes;
an image sensor positioned to obtain a feedback image of the AR image; and
a computer including a processor and a memory, the memory storing instructions executable by the processor programmed to:
input a set of training images into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images, wherein each training image corresponds to one respective sub-image;
actuate the SLM to output the AR image based on the pixel-wise phase matrix;
determine an offset based on comparing the training images to the feedback image; and
update parameters of a loss function based on the offset and provide the updated parameters to the neural network to obtain an updated offset.
2. The system of claim 1 , wherein the instructions further include instructions to update parameters of the loss function until the updated offset is less than a predetermined threshold.
3. The system of claim 2 , wherein the neural network is trained to output the pixel-wise phase matrix calibrated to the vehicle windshield and the SLM when the updated offset is less than the predetermined threshold.
4. The system of claim 1 , wherein the image sensor is spaced from the vehicle windshield to correspond to an expected pose of an occupant.
5. The system of claim 1 , wherein the instructions further include instructions to select the focal planes for the sub-images based on a user input.
6. The system of claim 1 , wherein each focal plane is defined by a respective distance from an expected pose of an occupant.
7. The system of claim 1 , wherein the instructions further include instructions to, while training the neural network, actuate a projector to provide the training images to the SLM.
8. The system of claim 1 , wherein the instructions further including instructions to, while training the neural network, select the training images from a plurality of training images.
9. The system of claim 1 , wherein the instructions further include instructions to:
after training the neural network, generate a three-dimensional (3D) image based on vehicle operation data; and
determine, via the trained neural network, a calibrated pixel-wise phase matrix for the 3D image.
10. The system of claim 9 , wherein the instructions further include instructions to, upon generating a set of masked images from the 3D image, input the masked images to the trained neural network that outputs the calibrated pixel-wise phase matrix for the 3D image.
11. The system of claim 10 , wherein the instructions further include instructions to generate the masked images by masking a plurality of pixels in the 3D image based on the focal planes, each masked pixel being a pixel of the 3D image having a depth corresponding to one of the focal planes.
12. The system of claim 9 , wherein the instructions further include instructions to actuate the SLM to output the AR image based on the calibrated pixel-wise phase matrix.
13. The system of claim 12 , wherein the instructions further include instructions to actuate a projector to provide the 3D image to the SLM.
14. A method, comprising:
inputting a set of training images into a neural network that outputs a pixel-wise phase matrix identifying pixels in the training images;
based on the pixel-wise phase matrix, actuating a spatial light modulator (SLM) to output, onto a vehicle windshield, an augmented reality (AR) image including a plurality of sub-images each output in one of a plurality of focal planes, wherein each training image corresponds to one respective sub-image;
obtaining a feedback image of the AR image via an image sensor;
determining an offset based on comparing the training images to the feedback image; and
updating parameters of a loss function based on the offset and providing the updated parameters to the neural network to obtain an updated offset.
15. The method of claim 14 , further comprising updating parameters of the loss function until the updated offset is less than a predetermined threshold.
16. The method of claim 15 , wherein the neural network is trained to output the pixel-wise phase matrix calibrated to the vehicle windshield and the SLM when the updated offset is less than the predetermined threshold.
17. The method of claim 14 , wherein the image sensor is spaced from the vehicle windshield to correspond to an expected pose of an occupant.
18. The method of claim 14 , further comprising selecting the focal planes for the sub-images based on a user input.
19. The method of claim 14 , wherein each focal plane is selected based on a respective distance from an expected pose of an occupant.
20. The method of claim 14 , further comprising, while training the neural network, actuating a projector to provide the training image to the SLM.