Post-capture editing of augmented reality content
The subject technology receives, by a client device from a storage device, first image data captured by the client device at a previous time. The subject technology receives first metadata corresponding to at least a first image processing operation and a second image processing operation. The subject technology generates, in a first render pass, second image data based on the first metadata and the image processing operation performed on the first image data. The subject technology generates, in a second render pass, third image data based on the first metadata and the second image processing operation performed on the second image data. The subject technology generates second metadata comprising information corresponding to the third image data. The subject technology generates a composite AR content item comprising the second metadata, the third image data, and the first image data.
1. A method, comprising:
receiving, by a client device from a storage device, first image data captured by the client device at a previous time;
receiving first metadata corresponding to at least a first image processing operation and a second image processing operation;
generating, in a first render pass, second image data based on the first metadata and the first image processing operation performed on the first image data;
generating, in a second render pass, third image data based on the first metadata and the second image processing operation performed on the second image data;
generating second metadata comprising information corresponding to the third image data;
generating a composite AR content item comprising the second metadata, the third image data, and the first image data;
providing a multi-layer image processing pipeline, the multi-layer image processing pipeline comprising a feature layer, a render pipeline layer, and a render pass layer;
receiving, by the render pipeline layer, the first image data, the first image data comprising a first pixel buffer, a second pixel buffer, and a bitmap; and
generating, using the feature layer, cascaded render passes, the cascaded render passes comprising a sequence of respective render passes in which an order of the respective render passes is fixed based at least in part on metadata corresponding to a set of augmented reality content generators to apply for each of the respective render passes, each of the set of augmented reality content generators corresponding to a respective image processing effect or operation related to an augmented reality object or augmented reality content.
2. The method of claim 1 , wherein the first pixel buffer includes a first set of pixels in a first color space, the first color space comprising a BT.709 color space,
the second pixel buffer includes a second set of pixels in a second color space, the second color space comprising a BT.601 color space, and
the bitmap comprises a third set of pixels in a third color space, the third color space comprising an RGB color space.
3. The method of claim 2 , further comprising:
performing, by the render pipeline layer, a color conversion process based on the first pixel buffer and the second pixel buffer, the color conversion process converting the first set of pixels in the first color space to a first color converted set of pixels in the RGB color space and the second set of pixels in the second color space to a second color converted set of pixels in the RGB color space; and
generating a set of textures, based on the first color converted set of pixels and the second color converted set of pixels, for the cascaded render passes; and
generating rendering information corresponding the cascaded render passes, each of the cascaded render passes comprising a respective set of image processing operations to render at least one texture from the set of textures, the cascaded render passes comprising multiple render passes that are ordered starting from a first render pass up to a last render pass.
4. The method of claim 3 , further comprising:
determining a render destination for the set of textures, the render destination comprising a framebuffer of a rendering pipeline, the framebuffer comprising a portion of memory provided by an eyewear device that stores a final image data for display by a display of the eyewear device.
5. The method of claim 3 , further comprising:
performing, by a render engine of the render pass layer, each render pass from the cascaded render passes for display by a display of an eyewear device, the performing comprising:
rendering, using a first render pipeline, the last render pass from the cascaded render passes, the last render pass corresponding to a particular set of augmented reality content generators to apply based on the respective set of image processing operations to render at least one texture from the set of textures;
rendering, using a second render pipeline, a second render pass from the cascaded render passes, the second render pass corresponding to a second particular set of augmented reality content generators to apply based on the respective set of image processing operations to render at least one different texture from the set of textures; and
rendering, using a third render pipeline, the first render pass from the cascaded render passes, the first render pass corresponding to a third particular set of augmented reality content generators to apply based on the respective set of image processing operations to render at least one second different texture from the set of textures.
6. The method of claim 5 , wherein performing, by the render engine of the render pass layer, each render pass from the cascaded render passes is based on a ping-pong render pass process, the ping-pong render pass process comprises sending output image data of the first render pipeline as input image data to the second render pipeline, and sending output image data of the second render pipeline as input image data to the first render pipeline.
7. The method of claim 6 , further comprising:
sending output image data from the third render pipeline to a render destination comprising memory provided by an eyewear device.
8. The method of claim 5 , further comprising:
providing a final output image from the render engine of the render pass layer after performing each of the cascaded render passes.
9. The method of claim 1 , further comprising:
providing the composite AR content item for display by a display of an eyewear device.
10. A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving, by a client device from a storage device, first image data captured by the client device at a previous time;
receiving first metadata corresponding to at least a first image processing operation and a second image processing operation;
generating, in a first render pass, second image data based on the first metadata and the first image processing operation performed on the first image data;
generating, in a second render pass, third image data based on the first metadata and the second image processing operation performed on the second image data;
generating second metadata comprising information corresponding to the third image data;
generating a composite AR content item comprising the second metadata, the third image data, and the first image data;
providing a multi-layer image processing pipeline, the multi-layer image processing pipeline comprising a feature layer, a render pipeline layer, and a render pass layer;
receiving, by the render pipeline layer, the first image data, the first image data comprising a first pixel buffer, a second pixel buffer, and a bitmap; and
generating, using the feature layer, cascaded render passes, the cascaded render passes comprising a sequence of respective render passes in which an order of the respective render passes is fixed based at least in part on metadata corresponding to a set of augmented reality content generators to apply for each of the respective render passes, each of the set of augmented reality content generators corresponding to a respective image processing effect or operation related to an augmented reality object or augmented reality content.
11. The system of claim 10 , wherein the first pixel buffer includes a first set of pixels in a first color space, the first color space comprising a BT.709 color space,
the second pixel buffer includes a second set of pixels in a second color space, the second color space comprising a BT.601 color space, and
the bitmap comprises a third set of pixels in a third color space, the third color space comprising an RGB color space.
12. The system of claim 11 , wherein the operations further comprise:
performing, by the render pipeline layer, a color conversion process based on the first pixel buffer and the second pixel buffer, the color conversion process converting the first set of pixels in the first color space to a first color converted set of pixels in the RGB color space and the second set of pixels in the second color space to a second color converted set of pixels in the RGB color space; and
generating a set of textures, based on the first color converted set of pixels and the second color converted set of pixels, for the cascaded render passes; and
generating rendering information corresponding the cascaded render passes, each of the cascaded render passes comprising a respective set of image processing operations to render at least one texture from the set of textures, the cascaded render passes comprising multiple render passes that are ordered starting from a first render pass up to a last render pass.
13. The system of claim 12 , wherein the operations further comprise:
determining a render destination for the set of textures, the render destination comprising a framebuffer of a rendering pipeline, the framebuffer comprising a portion of memory provided by an eyewear device that stores a final image data for display by a display of the eyewear device.
14. The system of claim 12 , wherein the operations further comprise:
performing, by a render engine of the render pass layer, each render pass from the cascaded render passes for display by a display of an eyewear device, the performing comprising:
rendering, using a first render pipeline, the last render pass from the cascaded render passes, the last render pass corresponding to a particular set of augmented reality content generators to apply based on the respective set of image processing operations to render at least one texture from the set of textures;
rendering, using a second render pipeline, a second render pass from the cascaded render passes, the second render pass corresponding to a second particular set of augmented reality content generators to apply based on the respective set of image processing operations to render at least one different texture from the set of textures; and
rendering, using a third render pipeline, the first render pass from the cascaded render passes, the first render pass corresponding to a third particular set of augmented reality content generators to apply based on the respective set of image processing operations to render at least one second different texture from the set of textures.
15. The system of claim 14 , wherein performing, by the render engine of the render pass layer, each render pass from the cascaded render passes is based on a ping-pong render pass process, the ping-pong render pass process comprises sending output image data of the first render pipeline as input image data to the second render pipeline, and sending output image data of the second render pipeline as input image data to the first render pipeline.
16. The system of claim 15 , wherein the operations further comprise:
sending output image data from the third render pipeline to a render destination comprising memory provided by an eyewear device.
17. The system of claim 14 , wherein the operations further comprise:
providing a final output image from the render engine of the render pass layer after performing each of the cascaded render passes.
18. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
receiving, by a client device from a storage device, first image data captured by the client device at a previous time;
receiving first metadata corresponding to at least a first image processing operation and a second image processing operation;
generating, in a first render pass, second image data based on the first metadata and the first image processing operation performed on the first image data;
generating, in a second render pass, third image data based on the first metadata and the second image processing operation performed on the second image data;
generating second metadata comprising information corresponding to the third image data;
generating a composite AR content item comprising the second metadata, the third image data, and the first image data;
providing a multi-layer image processing pipeline, the multi-layer image processing pipeline comprising a feature layer, a render pipeline layer, and a render pass layer;
receiving, by the render pipeline layer, the first image data, the first image data comprising a first pixel buffer, a second pixel buffer, and a bitmap; and
generating, using the feature layer, cascaded render passes, the cascaded render passes comprising a sequence of respective render passes in which an order of the respective render passes is fixed based at least in part on metadata corresponding to a set of augmented reality content generators to apply for each of the respective render passes, each of the set of augmented reality content generators corresponding to a respective image processing effect or operation related to an augmented reality object or augmented reality content.