Method and device for latency reduction of an image processing pipeline
In some implementations, a method includes: determining a complexity value for first image data associated with of a physical environment that corresponds to a first time period; determining an estimated composite setup time based on the complexity value for the first image data and virtual content for compositing with the first image data; in accordance with a determination that the estimated composite setup time exceeds the threshold time: forgoing rendering the virtual content from the perspective that corresponds to the camera pose of the device relative to the physical environment during the first time period; and compositing a previous render of the virtual content for a previous time period with the first image data to generate the graphical environment for the first time period.
1 . A method comprising:
at a device including one or more processors, non-transitory memory, and an image capture device:
obtaining, via the image capture device, first image data corresponding to a physical environment;
rendering a first slice of virtual content associated with a graphical environment; and
compositing a first portion of second image data with the rendered first slice of virtual content to generate a first composited slice of the graphical environment.
2 . The method of claim 1 , further comprising:
generating, at an image processing architecture, the first portion of the second image data based on the first image data.
3 . The method of claim 1 , further comprising:
reading a first slice of the first image data into one or more input buffers of an image processing architecture.
4 . The method of claim 1 , further comprising initiating a display scan-out of the first composited slice.
5 . The method of claim 4 , wherein initiating the display scan-out of the first composited slice comprises triggering a row-wise or column-wise display scan-out while performing one or more image processing operation on one or more subsequent slices.
6 . The method of claim 1 , further comprising
writing the first composited slice into a display buffer for a rolling display, wherein the rolling display scans-out a second composited slice during the writing the first composited slice into the display buffer.
7 . The method of claim 6 , wherein the first composited slice is written to a dedicated segment of the display buffer corresponding to a spatial region of the rolling display.
8 . The method of claim 6 , wherein the display buffer comprises a front buffer and a back buffer, and wherein the first composited slice is written to the front buffer while a previous image frame is being scanned-out from the back buffer.
9 . The method of claim 1 , wherein:
a complexity analysis is performed based on a slice of the first image data to estimate a rendering time; and
a cached render for the slice is reused based on the rendering time.
10 . The method of claim 1 , further comprising:
performing one or more image processing operations on the first slice of the first image data to obtain the first portion of the second image data.
11 . The method of claim 10 , wherein the one or more image processing operations correspond to one of: white balance correction, de-mosaicking, color correction, gamma correction, or sharpening.
12 . The method of claim 1 , wherein the first slice of the first image data corresponds to a row or line of RAW image data.
13 . The method of claim 1 , wherein the first slice of the first image data corresponds to a predefined portion of RAW image data.
14 . The method of claim 1 , wherein the first image data corresponds to RAW image data, and wherein the first composited slice corresponds to an RGB image slice.
15 . The method of claim 1 , wherein rendering the first slice of virtual content associated with the graphical environment comprises:
rendering the first slice of virtual content based on a camera pose of the image capture device relative to the virtual content.
16 . The method of claim 1 , wherein a front-end architecture digitizes analog image data into the first image data.
17 . The method of claim 1 , further comprising:
determining a focal region based on contextual information; and
performing a pixel binning operation on the first image data based on the focal region to generate a quadtree version of the first image data.
18 . The method of claim 17 , wherein the contextual information includes at least one of head pose information, body pose information, limb pose information, or gaze direction information.
19 . A device comprising:
one or more processors;
a non-transitory memory;
an image capture device including an image sensor; and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:
obtain, via the image capture device, first image data corresponding to a physical environment;
render a first slice of virtual content associated with a graphical environment; and
composite a first portion of second image data with the rendered first slice of virtual content to generate a first composited slice of the graphical environment.
20 . A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with an image capture device including an image sensor, cause the device to:
obtain, via the image capture device, first image data corresponding to a physical environment;
render a first slice of virtual content associated with a graphical environment; and
composite a first portion of second image data with the rendered first slice of virtual content to generate a first composited slice of the graphical environment.