Low latency frame delivery
Examples are described of marking specified regions of stored image frame buffer data in an image frame buffer. An imaging system can read the specified regions of the image frame buffer to identify whether the marking has been overwritten or not. The imaging system can thus efficiently identify how much of the image frame buffer has been overwritten with data from a new image frame. Based on this, the imaging system can retrieve partial image frame data from the image frame buffer and can process the partial image frame data, for instance to composite the partial image frame data with virtual content and/or to perform distortion compensation. The processed partial image frame data can be uploaded to a display buffer and displayed by a display, either as-is or once more of the frame is captured and processed. The imaging system can also perform auto-exposure using the partial image frame data.
1. An apparatus for image processing, the apparatus comprising:
a memory; and
one or more processors coupled to the memory, the one or more processors configured to:
store, in an image frame buffer, data marked with metadata at a predetermined region of the image frame buffer;
receive, from an image sensor, image frame data corresponding to an image frame;
overwrite at least some of the stored data in the image frame buffer with the image frame data;
determine that the image frame data includes at least a predetermined amount of the image frame based on a read of the predetermined region of the image frame buffer indicating that the predetermined region stores additional data that is different than the metadata; and
output at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.
2. The apparatus of claim 1 , wherein the additional data is at least a portion of the image frame data.
3. The apparatus of claim 1 , wherein, to output the first portion of the image frame data, the one or more processors are configured to process the first portion of the image frame data.
4. The apparatus of claim 3 , wherein, to process the first portion of the image frame data, the one or more processors are configured to modify at least some of the first portion of the image frame data using at least one of a distortion, a distortion compensation, and a warping.
5. The apparatus of claim 3 , wherein, to process the first portion of the image frame data, the one or more processors are configured to composite the first portion of the image frame data with virtual content.
6. The apparatus of claim 5 , wherein the one or more processors are configured to further:
identify a pose of the image sensor; and
render the virtual content based on a pose of the image sensor.
7. The apparatus of claim 1 , further comprising:
a display, wherein, to output the first portion of the image frame data, the one or more processors are configured to display the first portion of the image frame data using the display.
8. The apparatus of claim 1 , further comprising:
a display buffer, wherein, to output the first portion of the image frame data, the one or more processors are configured to store the first portion of the image frame data in the display buffer.
9. The apparatus of claim 1 , wherein, to output the first portion of the image frame data, the one or more processors are configured to:
determine one or more image statistics based on at least the first portion of the image frame data; and
determine an image capture setting based on the one or more image statistics.
10. The apparatus of claim 9 , wherein the one or more processors are configured to further:
send the image capture setting to image capture hardware, wherein the image capture hardware includes the image sensor; and
receive, from the image sensor, secondary image frame data corresponding to a second image frame, wherein the image sensor captures the second image frame based on the image capture setting.
11. The apparatus of claim 10 , wherein the image capture setting is an exposure setting.
12. The apparatus of claim 1 , wherein the one or more processors are configured to further:
mark the stored data with the metadata at the predetermined region of the image frame buffer.
13. The apparatus of claim 1 , wherein the metadata includes a pattern of colors.
14. The apparatus of claim 1 , wherein the metadata includes a frame identifier associated with the image frame.
15. The apparatus of claim 1 , wherein, to determine that the image frame data includes at least the predetermined amount of the image frame, the one or more processors are configured to:
read the predetermined region of the image frame buffer; and
determine, based on the read of the predetermined region of the image frame buffer, that the read indicates that the predetermined region stores the additional data that is different than the metadata.
16. The apparatus of claim 1 , wherein the stored data includes prior image frame data from a prior image frame, the prior image frame captured by the image sensor before capture of the image frame.
17. The apparatus of claim 1 , wherein, to output at least the first portion of the image frame data, the one or more processors are configured to output the image frame.
18. The apparatus of claim 1 , wherein the stored data is marked with second metadata at a second predetermined region of the image frame buffer.
19. The apparatus of claim 18 , wherein the one or more processors are configured to:
determine that the image frame data includes at least a second predetermined amount of the image frame based on a second read of the second predetermined region of the image frame buffer indicating that the second predetermined region stores secondary additional data that is different than the second metadata; and
output at least a second portion of the image frame data corresponding to the second predetermined amount of the image frame.
20. The apparatus of claim 1 , wherein the one or more processors are configured to:
receive a request for a view of a scene; and
expose the image sensor to light from the scene automatically in response to receipt of the request, wherein the image frame data received from the image sensor is based on exposure of the image sensor to the light from the scene.
21. The apparatus of claim 1 , further comprising:
the image sensor.
22. The apparatus of claim 1 , further comprising:
the image frame buffer.
23. The apparatus of claim 1 , wherein the apparatus is a head-mounted display.
24. A method of image processing, the method comprising:
storing, in an image frame buffer, data marked with metadata at a predetermined region of the image frame buffer;
receiving, from an image sensor, image frame data corresponding to an image frame;
overwriting at least some of the stored data in the image frame buffer with the image frame data;
determining that the image frame data includes at least a predetermined amount of the image frame based on a read of the predetermined region of the image frame buffer indicating that the predetermined region stores additional data that is different than the metadata; and
outputting at least a first portion of the image frame data corresponding to the predetermined amount of the image frame.
25. The method of claim 24 , wherein the additional data is at least a portion of the image frame data.
26. The method of claim 24 , wherein outputting the first portion of the image frame data includes processing the first portion of the image frame data.
27. The method of claim 24 , wherein outputting the first portion of the image frame data includes displaying the first portion of the image frame data using a display.
28. The method of claim 24 , wherein outputting the first portion of the image frame data includes storing the first portion of the image frame data in a display buffer.
29. The method of claim 24 , further comprising:
marking the stored data with the metadata at the predetermined region of the image frame buffer.
30. The method of claim 24 , further comprising:
receiving a request for a view of a scene; and
exposing the image sensor to light from the scene automatically in response to receipt of the request, wherein the image frame data received from the image sensor is based on exposure of the image sensor to the light from the scene.