Technique for recording augmented reality data
Disclosed is an improved approach for generated recordings from augmented reality systems from the perspective of a camera within the system. Instead of re-using rendered virtual content from the perspective of the user's eyes for AR recordings, additional virtual content is rendered from an additional perspective specifically for the AR recording. That additional virtual content is combined with image frames generated by a camera to form the AR recording.
1. A method for generating an image recording, comprising
capturing a camera image frame with a camera;
obtaining pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;
rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective;
warping the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and
composing the image recording by combining the camera image frame with the camera pose virtual content,
wherein a single graphics processing unit (GPU) renders the user pose virtual content and warps the user pose virtual content.
2. The method of claim 1 , further comprising:
receiving a request to display a display image frame; and
capturing the camera image frame with the camera in response to the request to display the display image frame.
3. The method of claim 1 , further comprising storing the image recording for later display.
4. The method of claim 1 , wherein a single GPU processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.
5. The method of claim 4 , wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data generated during the single GPU processing cycle is for different frames.
6. The method of claim 1 , wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.
7. The method of claim 1 , further comprising a media encoder encoding one or more image frames that combine the camera image frame with the camera pose virtual content.
8. The method of claim 7 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.
9. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for generating an image recording comprising:
capturing a camera image frame with a camera;
obtaining pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;
rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective;
warping the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and
composing the image recording by combining the camera image frame with the camera pose virtual content,
wherein a single graphics processing unit (GPU) renders the user pose virtual content and warps the user pose virtual content.
10. The computer program product of claim 9 , the method further comprising:
receiving a request to display a display image frame; and
capturing the camera image frame with the camera in response to the request to display the display image frame.
11. The computer program product of claim 9 , the method further comprising storing the image recording for later display.
12. The computer program product of claim 9 , wherein a single GPU processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.
13. The computer program product of claim 12 , wherein the user pose virtual content based on the user pose data and the camera pose virtual content based on the camera pose data generated during the single GPU processing cycle is for different frames.
14. The computer program product of claim 9 , wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.
15. The computer program product of claim 9 , wherein a media encoder encodes one or more image frames that combine the camera image frame with the camera pose virtual content.
16. The computer program product of claim 15 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.
17. A system for generating an image recording, comprising
an augmented reality display device that displays three dimensional content;
a camera that captures a camera image frame;
a single inertial measurement unit (IMU) that obtains pose data corresponding to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;
one or more graphics processing units (GPU) that:
renders user pose virtual content for immediate display to a user in the augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective, and
warps the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and
a compositor that composes the image recording by combining the camera image frame with the camera pose virtual content.
18. The system of claim 17 , further comprising a media encoder that encodes one or more image frames that combine the camera image frame with the camera pose virtual content.
19. The system of claim 18 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.
20. The system of claim 17 , wherein a single GPU processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.