Recording augmented reality content on an eyewear device
The subject technology sends captured image data and metadata to a first set of computing processes for processing. The subject technology receives, asynchronously from a second set of computing processes, second metadata. The subject technology detects a signal indicating a command to stop image data capturing. The subject technology, in response to the detected signal, generating a composite AR content item based on the metadata and the captured image data. The subject technology provides the composite AR content item for rendering for display on an eyewear device.
1. A method, comprising:
sending captured image data and metadata to a first set of computing processes for processing, the first set of computing processes comprising a content recording module;
receiving, asynchronously from a second set of computing processes, second metadata;
detecting a signal indicating a command to stop image data capturing;
in response to the detected signal, generating a composite augmented reality (AR) content item based on the metadata and the captured image data, wherein generating the composite AR content item occurs without modifying a raw image format of the captured image data that results in a reduced image quality from the raw image format, wherein generating the composite AR content item comprises:
encoding, by the content recording module, each frame from the captured image data to a second image data format different from the raw image format of the captured image data, each of the encoded frames including an associated timestamp indicating when the encoded frames were captured;
providing the composite AR content item for rendering for display on an eyewear device, wherein the first set of computing processes further comprises an image data processing module, the second set of computing processes includes the image data processing module, and the metadata comprises a first set of timestamps corresponding to a first set of frames from the captured image data;
generating, by the image data processing module for each frame from the captured image data, tracking data and augmented reality content data corresponding to a particular augmented reality content generator, the second metadata comprising the tracking data and the augmented reality content data;
sending, the tracking data, the augmented reality content data, and timestamp information including at least a timestamp associated with the tracking data and the augmented reality content data of each frame to the content recording module;
receiving, by the content recording module, a first event indicating recording has stopped;
determining, by the content recording module, that the tracking data, the augmented reality content data, and the timestamp information has not been received;
receiving, by the content recording module, second captured frames;
discarding, by the content recording module, the second captured frames in response to the first event; and
receiving, by the content recording module, the tracking data, the augmented reality content data, and the timestamp information,
wherein generating, by the content recording module, the composite AR content item, the generating further comprises:
combining, based on matching timestamp information, a set of encoded frames with the tracking data and the augmented reality content data into a set of composite AR content frames, each composite AR content frame including a corresponding encoded frame, corresponding tracking data, corresponding augmented reality content data, and corresponding timestamp information, and
wherein providing the composite AR content item for rendering for display on the eyewear device comprises:
for each composite AR content frame from the set of composite AR content frames:
rendering, by a rendering module, AR content based on the corresponding tracking data and corresponding augmented reality content data, the rendering causing the AR content to be displayed within the corresponding encoded frame;
predicting, by the rendering module, a future frame to be rendered, the future frame not yet received by the rendering module and having a particular timestamp at a future time from each timestamp from the set of composite AR content frames; and
rendering, by the rendering module, the future frame for display on the eyewear device.
2. The method of claim 1 , further comprising:
prior to sending captured image data and metadata to the first set of computing processes for processing:
sending, by the image data processing module, a request to a camera module to initiate capturing of image data;
sending, by the camera module, a second request to a capture module to capture image data;
receiving, by the camera module, the captured image data and the metadata from the capture module, wherein the metadata comprises at least timestamp information for each frame from the captured image data; and
sending, by the camera module, the captured image data and the metadata to the image data processing module and the content recording module.
3. The method of claim 1 , wherein the augmented reality content data includes an augmented reality identifier, an augmented reality content generator category, an augmented reality content generator carousel index, and a carousel group.
4. The method of claim 1 , wherein the tracking data comprises a set of coordinates for a set of objects or a set of surface planes in the each frame, the set of coordinates corresponding to a subset of coordinates from a field of view from an eyewear device.
5. The method of claim 1 , further comprising:
predicting, by the rendering module, a predicted position to render AR content based at least in part on the tracking data from the set of composite AR content frames, the tracking data indicating a likelihood of the predicted position.
6. A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
sending captured image data and metadata to a first set of computing processes for processing, the first set of computing processes comprising a content recording module;
receiving, asynchronously from a second set of computing processes, second metadata;
detecting a signal indicating a command to stop image data capturing;
in response to the detected signal, generating a composite augmented reality (AR) content item based on the metadata and the captured image data, wherein generating the composite AR content item occurs without modifying a raw image format of the captured image data that results in a reduced image size or reduced image quality from the raw image format, wherein generating the composite AR content item comprises:
encoding, by the content recording module, each frame from the captured image data to a second image data format different from the raw image format of the captured image data, each of the encoded frames including an associated timestamp indicating when the encoded frames were captured;
providing the composite AR content item for rendering for display on an eyewear device, wherein the first set of computing processes further comprises an image data processing module, the second set of computing processes includes the image data processing module, and the metadata comprises a first set of timestamps corresponding to a first set of frames from the captured image data;
generating, by the image data processing module for each frame from the captured image data, tracking data and augmented reality content data corresponding to a particular augmented reality content generator, the second metadata comprising the tracking data and the augmented reality content data;
sending, the tracking data, the augmented reality content data, and timestamp information including at least a timestamp associated with the tracking data and the augmented reality content data of each frame to the content recording module;
receiving, by the content recording module, a first event indicating recording has stopped;
determining, by the content recording module, that the tracking data, the augmented reality content data, and the timestamp information has not been received;
receiving, by the content recording module, second captured frames;
discarding, by the content recording module, the second captured frames in response to the first event; and
receiving, by the content recording module, the tracking data, the augmented reality content data, and the timestamp information,
wherein generating, by the content recording module, the composite AR content item, the generating further comprises:
combining, based on matching timestamp information, a set of encoded frames with the tracking data and the augmented reality content data into a set of composite AR content frames, each composite AR content frame including a corresponding encoded frame, corresponding tracking data, corresponding augmented reality content data, and corresponding timestamp information, and
wherein providing the composite AR content item for rendering for display on the eyewear device comprises:
for each composite AR content frame from the set of composite AR content frames:
rendering, by a rendering module, AR content based on the corresponding tracking data and corresponding augmented reality content data, the rendering causing the AR content to be displayed within the corresponding encoded frame;
predicting, by the rendering module, a future frame to be rendered, the future frame not yet received by the rendering module and having a particular timestamp at a future time from each timestamp from the set of composite AR content frames; and
rendering, by the rendering module, the future frame for display on the eyewear device.
7. The system of claim 6 , wherein the operations further comprise:
prior to sending captured image data and metadata to the first set of computing processes for processing:
sending, by the image data processing module, a request to a camera module to initiate capturing of image data;
sending, by the camera module, a second request to a capture module to capture image data;
receiving, by the camera module, the captured image data and the metadata from the capture module, wherein the metadata comprises at least timestamp information for each frame from the captured image data; and
sending, by the camera module, the captured image data and the metadata to the image data processing module and the content recording module.
8. The system of claim 6 , wherein the augmented reality content data includes an augmented reality identifier, an augmented reality content generator category, an augmented reality content generator carousel index, and a carousel group.
9. The system of claim 6 , wherein the tracking data comprises a set of coordinates for a set of objects or a set of surface planes in the each frame, the set of coordinates corresponding to a subset of coordinates from a field of view from an eyewear device.
10. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
sending captured image data and metadata to a first set of computing processes for processing, the first set of computing processes comprising a content recording module;
receiving, asynchronously from a second set of computing processes, second metadata;
detecting a signal indicating a command to stop image data capturing;
in response to the detected signal, generating a composite AR content item based on the metadata and the captured image data, wherein generating the composite AR content item occurs without modifying a raw image format of the captured image data that results in a reduced image size or reduced image quality from the raw image format, wherein generating the composite AR content item comprises:
encoding, by the content recording module, each frame from the captured image data to a second image data format different from the raw image format of the captured image data, each of the encoded frames including an associated timestamp indicating when the encoded frames were captured; and
providing the composite AR content item for rendering for display on an eyewear device, wherein the first set of computing processes further comprises an image data processing module, the second set of computing processes includes the image data processing module, and the metadata comprises a first set of timestamps corresponding to a first set of frames from the captured image data;
generating, by the image data processing module for each frame from the captured image data, tracking data and augmented reality content data corresponding to a particular augmented reality content generator, the second metadata comprising the tracking data and the augmented reality content data;
sending, the tracking data, the augmented reality content data, and timestamp information including at least a timestamp associated with the tracking data and the augmented reality content data of each frame to the content recording module;
receiving, by the content recording module, a first event indicating recording has stopped;
determining, by the content recording module, that the tracking data, the augmented reality content data, and the timestamp information has not been received;
receiving, by the content recording module, second captured frames;
discarding, by the content recording module, the second captured frames in response to the first event; and
receiving, by the content recording module, the tracking data, the augmented reality content data, and the timestamp information,
wherein generating, by the content recording module, the composite AR content item, the generating further comprises:
combining, based on matching timestamp information, a set of encoded frames with the tracking data and the augmented reality content data into a set of composite AR content frames, each composite AR content frame including a corresponding encoded frame, corresponding tracking data, corresponding augmented reality content data, and corresponding timestamp information, and
wherein providing the composite AR content item for rendering for display on the eyewear device comprises:
for each composite AR content frame from the set of composite AR content frames:
rendering, by a rendering module, AR content based on the corresponding tracking data and corresponding augmented reality content data, the rendering causing the AR content to be displayed within the corresponding encoded frame;
predicting, by the rendering module, a future frame to be rendered, the future frame not yet received by the rendering module and having a particular timestamp at a future time from each timestamp from the set of composite AR content frames; and
rendering, by the rendering module, the future frame for display on the eyewear device.