Reducing latency in augmented reality (AR) displays

1. A method comprising:

receiving, by a display controller of a display device, a first stream of image pixels of a first frame of virtual content to be presented on a display of the display device, the first stream of image pixels received from a Graphics Processing Unit (GPU) that rendered the first frame based on an initial three-dimensional pose of the display device, the initial three-dimensional pose determined based on sensor data gathered by one or more sensors of the display device, the first stream of image pixels received by the display controller via a high-speed bulk interface that transfers data at least as fast as can be consumed by the display;

converting, by the display controller, each respective image pixel in the stream of image pixels as the respective image pixel is received from the GPU, each respective image pixel converted from a first data format used to transmit the first stream of image pixels via the high-speed bulk interface to a second data format that is compatible for display by the display, wherein each respective image pixel is converted without any portion of the first frame of virtual content being buffered by the display controller;

after each image pixel is converted into the second data format, storing the respective image pixel in one of a plurality of pixel cells of the display, wherein at least a first pixel of the first stream of pixels of the first frame is stored in a pixel cell of the display while a second pixel of the first stream of pixels of the first frame is being converted into the second data format by the display controller; and

in response to the first stream of image pixels of the first frame being converted into the second data format and stored in the pixel cells of the display, causing the display to present the first frame.

2. The method of claim 1 , further comprising:

determining, based on updated sensor data, an updated three-dimensional pose of the display device;

determining a difference between the updated three-dimension pose of the display device and the initial three-dimensional pose of the display device; and

prior to storing each image pixel in a pixel cell of the display, warping the respective image pixels based on the difference between the initial three-dimensional pose and the updated three-dimension pose.

3. The method of claim 1 , wherein the updated sensor data is gathered from an Inertial Measurement Unit (IMU) and includes at least gyroscope data indicating rotation of the display device.

4. The method of claim 1 , wherein the high-speed bulk interface is Peripheral Component Interconnect (PCI) Express.

5. The method of claim 1 , wherein the first stream of image pixels of the first frame includes a first subset of all image pixels of the first frame, the first subset of all image pixels of the first frame including only pixels in a first color.

6. The method of claim 5 , further comprising:

receiving a second stream of image pixels of a first frame of virtual content to be presenting on the display, the second stream of image pixels of the first frame including a second subset of all image pixels of the first frame, the second subset of all image pixels of the first frame including only pixels in a second color that is different than the first color, the second stream of image pixels received from the GPU that rendered the first frame based on the initial three-dimensional pose of the display device;

as the second stream of image pixel is received, converting each respective image pixel from the first data format to the second data format;

after each image pixel is converted into the second data format, storing the respective image pixel in one of the plurality of pixel cells of the display; and

in response to second stream of image pixels of the first frame being converted into the second data format and stored in the pixel cells of the display, causing the display to present the first frame.

7. The method of claim 6 , further comprising:

determining, based on updated sensor data, an updated three-dimensional pose of the display device;

determining a difference between the updated three-dimension pose of the display device and the initial three-dimensional pose of the display device; and

prior to storing each image pixel in a pixel cell of the display, warping the respective image pixels based on the difference between the initial three-dimensional pose and the updated three-dimension pose.

8. A display device comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the display device to perform operations comprising:

receiving a first stream of image pixels of a first frame of virtual content to be presented on a display of the display device, the first stream of image pixels received from a Graphics Processing Unit (GPU) that rendered the first frame based on an initial three-dimensional pose of the display device, the initial three-dimensional pose determined based on sensor data gathered by one or more sensors of the display device, the first stream of image pixels received by the display controller via a high-speed bulk interface that transfers data at least as fast as can be consumed by the display;

converting each respective image pixel in the stream of image pixels as the respective image pixel is received from the GPU, each respective image pixel converted from a first data format used to transmit the first stream of image pixels via the high-speed bulk interface to a second data format that is compatible for display by the display, wherein each respective image pixel is converted without any portion of the first frame of virtual content being buffered by the display controller;

after each image pixel is converted into the second data format, storing the respective image pixel in one of a plurality of pixel cells of the display, wherein at least a first pixel of the first stream of pixels of the first frame is stored in a pixel cell of the display while a second pixel of the first stream of pixels of the first frame is being converted into the second data format by the display controller; and

in response to the first stream of image pixels of the first frame being converted into the second data format and stored in the pixel cells of the display, causing the display to present the first frame.

9. The display device of claim 8 , the operations further comprising:

determining, based on updated sensor data, an updated three-dimensional pose of the display device;

determining a difference between the updated three-dimension pose of the display device and the initial three-dimensional pose of the display device; and

prior to storing each image pixel in a pixel cell of the display, warping the respective image pixels based on the difference between the initial three-dimensional pose and the updated three-dimension pose.

10. The display device of claim 8 , wherein the updated sensor data is gathered from an Inertial Measurement Unit (IMU) and includes at least gyroscope data indicating rotation of the display device.

11. The display device of claim 8 , wherein the high-speed bulk interface is Peripheral Component Interconnect (PCI) Express.

12. The display device of claim 8 , wherein the first stream of image pixels of the first frame includes a first subset of all image pixels of the first frame, the first subset of all image pixels of the first frame including only pixels in a first color.

13. The display device of claim 12 , the operations further comprising:

receiving a second stream of image pixels of a first frame of virtual content to be presenting on the display, the second stream of image pixels of the first frame including a second subset of all image pixels of the first frame, the second subset of all image pixels of the first frame including only pixels in a second color that is different than the first color, the second stream of image pixels received from the GPU that rendered the first frame based on the initial three-dimensional pose of the display device;

as the second stream of image pixel is received, converting each respective image pixel from the first data format to the second data format;

after each image pixel is converted into the second data format, storing the respective image pixel in one of the plurality of pixel cells of the display; and

in response to second stream of image pixels of the first frame being converted into the second data format and stored in the pixel cells of the display, causing the display to present the first frame.

14. The display device of claim 13 , operations further comprising:

determining, based on updated sensor data, an updated three-dimensional pose of the display device;

determining a difference between the updated three-dimension pose of the display device and the initial three-dimensional pose of the display device; and

prior to storing each image pixel in a pixel cell of the display, warping the respective image pixels based on the difference between the initial three-dimensional pose and the updated three-dimension pose.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a display controller of a display device, cause the display controller to perform operations comprising:

receiving a first stream of image pixels of a first frame of virtual content to be presented on a display of the display device, the first stream of image pixels received from a Graphics Processing Unit (GPU) that rendered the first frame based on an initial three-dimensional pose of the display device, the initial three-dimensional pose determined based on sensor data gathered by one or more sensors of the display device, the first stream of image pixels received by the display controller via a high-speed bulk interface that transfers data at least as fast as can be consumed by the display;

converting each respective image pixel in the stream of image pixels as the respective image pixel is received from the GPU, each respective image pixel converted from a first data format used to transmit the first stream of image pixels via the high-speed bulk interface to a second data format that is compatible for display by the display, wherein each respective image pixel is converted without any portion of the first frame of virtual content being buffered by the display controller;

after each image pixel is converted into the second data format, storing the respective image pixel in one of a plurality of pixel cells of the display, wherein at least a first pixel of the first stream of pixels of the first frame is stored in a pixel cell of the display while a second pixel of the first stream of pixels of the first frame is being converted into the second data format by the display controller; and

in response to the first stream of image pixels of the first frame being converted into the second data format and stored in the pixel cells of the display, causing the display to present the first frame.

16. The non-transitory computer-readable medium of claim 15 , the operations further comprising:

determining, based on updated sensor data, an updated three-dimensional pose of the display device;

determining a difference between the updated three-dimension pose of the display device and the initial three-dimensional pose of the display device; and

prior to storing each image pixel in a pixel cell of the display, warping the respective image pixels based on the difference between the initial three-dimensional pose and the updated three-dimension pose.

17. The non-transitory computer-readable medium of claim 15 , wherein the updated sensor data is gathered from an Inertial Measurement Unit (IMU) and includes at least gyroscope data indicating rotation of the display device.