Technique for recording augmented reality data
Disclosed is an improved approach for generated recordings from augmented reality systems from the perspective of a camera within the system. Instead of re-using rendered virtual content from the perspective of the user's eyes for AR recordings, additional virtual content is rendered from an additional perspective specifically for the AR recording. That additional virtual content is combined with image frames generated by a camera to form the AR recording.
1. A method for generating an image recording, comprising
capturing a camera image frame with a camera;
obtaining pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;
rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective;
warping the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and
composing the image recording by combining the camera image frame with the camera pose virtual content,
wherein a single inertial measurement unit (IMU) generates both the user pose data and the camera pose data.
2. The method of claim 1 , further comprising:
receiving a request to display a display image frame; and
capturing the camera image frame with the camera in response to the request to display the display image frame.
3. The method of claim 1 , further comprising storing the image recording for later display.
4. The method of claim 1 , wherein a single graphics processing unit (GPU) processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.
5. The method of claim 1 , wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.
6. The method of claim 1 , further comprising a media encoder encoding one or more image frames that combine the camera image frame with the camera pose virtual content.
7. The method of claim 6 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.
8. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for generating an image recording comprising:
capturing a camera image frame with a camera;
obtaining pose data that corresponds to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;
rendering user pose virtual content for immediate display to a user in an augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective;
warping the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and
composing the image recording by combining the camera image frame with the camera pose virtual content,
wherein a single inertial measurement unit (IMU) generates both the user pose data and the camera pose data.
9. The computer program product of claim 8 , the method further comprising:
receiving a request to display a display image frame; and
capturing the camera image frame with the camera in response to the request to display the display image frame.
10. The computer program product of claim 8 , the method further comprising storing the image recording for later display.
11. The computer program product of claim 8 , wherein a single graphics processing unit (GPU) processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.
12. The computer program product of claim 8 , wherein a compositor generates the image recording by combining the camera image frame with the camera pose virtual content.
13. The computer program product of claim 8 , wherein a media encoder encodes one or more image frames that combine the camera image frame with the camera pose virtual content.
14. The computer program product of claim 13 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.
15. A system for generating an image recording, comprising
an augmented reality display device that displays three dimensional content;
a camera that captures a camera image frame;
an inertial measurement unit (IMU) that obtains pose data corresponding to the camera image frame, the pose data comprising user pose data and camera pose data, wherein the user pose data is from a different perspective from the camera pose data;
a single graphics processing unit (GPU) that:
renders user pose virtual content for immediate display to a user in the augmented reality (AR) device, wherein the user pose virtual content corresponds to first virtual content from the user's perspective, and
warps the user pose virtual content based on the camera pose data to generate camera pose virtual content, where the camera pose virtual content corresponds to second virtual content from the camera's perspective; and
a compositor that composes the image recording by combining the camera image frame with the camera pose virtual content.
16. The system of claim 15 , further comprising a media encoder that encodes one or more image frames that combine the camera image frame with the camera pose virtual content.
17. The system of claim 16 , wherein the one or more image frames encoded by the media encoder comprises at least one of a single image frame or a stream of multiple image frames.
18. The system of claim 15 , wherein a single GPU processing cycle includes processing to: render the user pose virtual content based on the user pose data, and warp the user pose virtual content based on the camera pose data to generate the camera pose virtual content.