Pipeline with processor including neural network for rendering artificial reality

1. A system comprising:

a shared physical memory;

a first processor having access to the shared physical memory, the first processor configured to:

perform, during a first time period, a first rendering process to generate a first image frame of a first view of an artificial reality, and

perform, during a second time period after the first time period, a second rendering process to generate a second image frame of a second view of the artificial reality; and

a second processor including a neural network and having access to the shared physical memory, the second processor configured to perform, during a third time period overlapping a portion of the second time period, an image enhancing process on the first image frame to generate a first modified image frame.

2. The system of claim 1 , wherein the first processor is a graphics processor, and the second processor is a digital signal processor or a neural processor.

3. The system of claim 1 , wherein the first processor is configured to:

perform, during a fourth time period after the second time period, reprojection on the first modified image frame to generate a first reprojected image frame, according to a view of the artificial reality corresponding to at least a location and an orientation of the system.

4. The system of claim 3 , further comprising:

a lens, wherein the first processor is configured to apply, during the fourth time period, a predistortion on the first reprojected image frame to generate a first compensated image frame to compensate for an optical aberration due to the lens; and

an electronic display configured to display the first compensated image frame through the lens.

5. The system of claim 1 , further comprising:

a third processor configured to:

determine, during a fourth time period before the first time period, the first view of the artificial reality corresponding to at least a first location and a first orientation of the system, and

determine, during a fifth time period overlapping a portion of the first time period, the second view of the artificial reality corresponding to at least a second location and a second orientation of the system.

6. The system of claim 1 , wherein the image enhancing process includes a super resolution, and the second processor is configured to perform, during the third time period, the super resolution on a portion of the first image frame by enlarging the portion of the first image frame.

7. The system of claim 6 , wherein the portion of the first image frame corresponds to a foveated area.

8. The system of claim 7 , wherein the first processor is configured to generate the first image frame in a linear format, in response to a size of the foveated area being less than a threshold.

9. The system of claim 7 , wherein the first processor is configured to generate the first image frame in a swizzled format, in response to a size of the foveated area being larger than a threshold, the system further comprising a converter configured to convert the first image frame in the swizzled format into a linear format.

10. The system of claim 1 , wherein the first processor is configured to store the first image frame in the shared physical memory and share mapping information indicating an address of the shared physical memory storing the first image frame, and the second processor is configured to access the first image frame in the shared physical memory according to the mapping information.

11. A method comprising:

performing, during a first time period, by a first processor having access to a shared physical memory, a first rendering process to generate a first image frame of a first view of an artificial reality;

performing, during a second time period after the first time period, by the first processor, a second rendering process to generate a second image frame of a second view of the artificial reality; and

performing, during a third time period overlapping a portion of the second time period, by a second processor having access to the shared physical memory and including a neural network, an image enhancing process on the first image frame to generate a first modified image frame.

12. The method of claim 11 , wherein the first processor is a graphics processor, and the second processor is a digital signal processor or a neural processor.

13. The method of claim 11 , further comprising:

performing, during a fourth time period after the second time period, by the first processor, reprojection on the first modified image frame to generate a first reprojected image frame, according to a view of the artificial reality corresponding to at least a location and an orientation of a head mounted display worn by a user of the artificial reality.

14. The method of claim 13 , further comprising:

applying, during the fourth time period, by the first processor, a predistortion on the first reprojected image frame to generate a first compensated image frame to compensate for an optical aberration due to a lens; and

displaying, by an electronic display through the lens, the first compensated image frame.

15. The method of claim 11 , further comprising:

determining, during a fourth time period before the first time period, by a third processor, the first view of the artificial reality corresponding to at least a first location and a first orientation of a head mounted display; and

determining, during a fifth time period overlapping a portion of the first time period, by the third processor, the second view of the artificial reality corresponding to at least a second location and a second orientation of the head mounted display.

16. The method of claim 11 , wherein the image enhancing process includes a super resolution, the method further comprising performing, during the third time period, the super resolution on a portion of the first image frame to enlarge the portion of the first image frame.

17. The method of claim 16 , wherein the portion of the first image frame corresponds to a foveated area.

18. The method of claim 17 , further comprising:

comparing a size of the foveated area against a threshold; and

generating, by the first processor, the first image frame in a linear format, in response to the size of the foveated area being less than the threshold.

19. The method of claim 17 , further comprising:

comparing a size of the foveated area against a threshold;

generating, by the first processor, the first image frame in a swizzled format, in response to the size of the foveated area being larger than the threshold; and

converting, by a converter, the first image frame in the swizzled format into a linear format.

20. A system comprising:

a first processor having access to a shared physical memory, the first processor configured to:

perform, during a first time period, a first rendering process to generate a first image frame of a first view of an artificial reality, and

perform, during a second time period after the first time period, a second rendering process to generate a second image frame of a second view of the artificial reality; and

a second processor including a neural network and having access to the shared physical memory, the second processor configured to begin, prior to or when the first processor performs the second rendering process to generate the second image frame of the second view of the artificial reality, an image enhancing process on the first image frame to generate a first modified image frame.