Efficient GPU/CPU pipeline for providing augmented reality content
Implementations generally relate to providing augmented reality in a web browser. In one implementation, a method includes capturing images of a physical scene with a camera of a device. The method further includes determining motion of the camera using six degrees of freedom (6DoF) markerless tracking. The method further includes overlaying virtual three-dimensional (3D) content onto a depicted physical scene in the images, resulting in augmented reality (AR) images. The method further includes rendering the AR images in a browser of the device.
1. A method for providing Augmented Reality (AR) content, the method comprising:
receiving images of a physical scene captured by a camera of a device;
processing the images in a pipeline to generate AR images, wherein each image is processed by a graphics processing unit (GPU) before being processed by a central processing unit (CPU) while the GPU is processing a subsequent image, the processing of each image including:
computing, by the GPU, feature points of the image from one or more real-world objects in the physical scene;
transmitting the feature points from the GPU to the CPU;
determining, by the CPU, changes in a position and an orientation of the camera using the feature points;
transmitting the changes in the position and the orientation of the camera from the CPU to the GPU;
overlaying, by the GPU, virtual content onto the physical scene depicted in the image using the changes in the position and the orientation of the camera, resulting in an AR image, following computing, by the GPU, the features points of the subsequent image with a timestamp subsequent to a timestamp of the image; and
generating, by the GPU, one or more buffer frames based on the image and by interpolating feature points from at least one other processed image; and
providing the AR images and the buffer frames to be displayed, wherein the buffer frames are interlaced between the AR images.
2. The method of claim 1 , wherein the processing is performed based on a web application, and wherein the web application is based at least in part on C++ code that is revised to optimize performance of generated JavaScript.
3. The method of claim 1 , wherein the processing is performed using a predetermined subset of JavaScript transpiled or compiled from source code of another programming language.
4. The method of claim 1 , further comprising switching between different computation pipelines based on one or more of device characteristics or runtime performance evaluation.
5. The method of claim 1 , wherein the feature points are extracted using a shader executing on the GPU.
6. The method of claim 1 , wherein the AR images are provided to a browser executing on the device for display within the browser.
7. The method of claim 1 , further comprising performing multithread execution of JavaScript.
8. The method of claim 7 , wherein performing the multithread execution comprises:
executing, by the CPU, a first JavaScript operation in a main thread;
executing, by the CPU, one or more second JavaScript operations in one or more background threads, wherein the main thread and the one or more background threads are performed asynchronously.
9. The method of claim 1 , further comprising:
accessing from the device one or more of sensor data, camera image data, or inertial measurements using web frameworks; and
rendering the AR images based on at least one of the sensor data, the camera image data, or the inertial measurements.
10. The method of claim 1 , further comprising:
obtaining camera data from a device library;
parsing the camera data; and
determining one or more intrinsic parameters of the camera from the camera data, wherein the one or more intrinsic parameters include at least one of a field of view, a principal point, or distortion parameters.
11. A non-transitory computer-readable medium comprising instructions for providing Augmented Reality (AR) content, the instructions, when executed by a computing system, causing the computing system to perform operations including:
receiving images of a physical scene captured by a camera of a device;
processing the images in a pipeline to generate AR images, wherein each image is processed by a graphics processing unit (GPU) before being processed by a central processing unit (CPU) while the GPU is processing a subsequent image, the processing of each image including:
computing, by the GPU, feature points of the image from one or more real-world objects in the physical scene;
transmitting the feature points from the GPU to the CPU;
determining, by the CPU, changes in a position and an orientation of the camera using the feature points;
transmitting the changes in the position and the orientation of the camera from the CPU to the GPU;
overlaying, by the GPU, virtual content onto the physical scene depicted in the image using the changes in the position and the orientation of the camera, resulting in an AR image, following computing, by the GPU, the features points of the subsequent image with a timestamp subsequent to a timestamp of the image; and
generating, by the GPU, one or more buffer frames based on the image and by interpolating feature points from at least one other processed image; and
providing the AR images to be displayed and the buffer frames to be displayed, wherein the buffer frames are interlaced between the AR images.
12. The non-transitory computer-readable medium of claim 11 , wherein the processing is performed based on a web application, and wherein the web application is based at least in part on C++ code that is revised to optimize performance of generated JavaScript.
13. The non-transitory computer-readable medium of claim 11 , wherein the processing is performed using a predetermined subset of JavaScript transpiled or compiled from source code of another programming language.
14. The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise switching between different computation pipelines based on one or more of device characteristics or runtime performance evaluation.
15. The non-transitory computer-readable medium of claim 11 , wherein the feature points are extracted using a shader executing on the GPU.
16. The non-transitory computer-readable medium of claim 11 , wherein the AR images are provided to a browser executing on the device for display within the browser.
17. The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise performing multithread execution of JavaScript.
18. The non-transitory computer-readable medium of claim 17 , wherein performing the multithread execution comprises:
executing, by the CPU, a first JavaScript operation in a main thread;
executing, by the CPU, one or more second JavaScript operations in one or more background threads, wherein the main thread and the one or more background threads are performed asynchronously.
19. The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise:
accessing from the device one or more of sensor data, camera image data, or inertial measurements using web frameworks; and
rendering the AR images based on at least one of the sensor data, the camera image data, or the inertial measurements.
20. The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise:
obtaining camera data from a device library;
parsing the camera data; and
determining one or more intrinsic parameters of the camera from the camera data, wherein the one or more intrinsic parameters include at least one of a field of view, a principal point, or distortion parameters.