IP Library Granted Patent US 10,909,764
Granted Patent B2
US 10,909,764 · App. 16/451,965 · Granted Feb 2, 2021

Providing augmented reality target images in a web browser

Inventors: Erik Murphy-Chutorian (Palo Alto, CA); Nicholas Butko (Cupertino, CA)
G06T19/006G06F8/4441G06F16/957G06T1/20G06T7/246G06T7/80G06T2207/30204G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,909,764
App. No.
16/451,965
Granted
Feb 2, 2021
Kind
B2
Abstract

Implementations generally relate to providing augmented reality in a web browser. In one implementation, a method includes capturing images of a physical scene with a camera of a device. The method further includes determining motion of the camera using six degrees of freedom (6DoF) marker-based tracking. The method further includes determining positions of markers relative to the camera using 6DoF marker-based tracking. The method further includes overlaying virtual content onto a depicted physical scene in the images, resulting in augmented reality (AR) images. The method further includes rendering the AR images in a browser of the device.

Claims (58)

1. A system comprising:

one or more processors;

logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors and when executed operable to perform operations comprising:

capturing images of a physical scene with a camera of a device;

obtaining camera data from a device library via a browser of the device;

parsing the camera data;

determining one or more intrinsic parameters of the camera, wherein the one or more intrinsic parameters comprise at least one or more of a field of view, a principal point, and distortion parameters;

determining motion of the camera using six degrees of freedom (6DoF) marker-based tracking;

determining positions of markers relative to the camera using 6DoF marker-based tracking;

overlaying virtual content onto a depicted physical scene in the images, resulting in augmented reality (AR) images; and

rendering the AR images in the browser of the device.

2. The system of claim 1 , wherein the logic when executed performs the operations based at least in part on a web application that is executed in the browser, and wherein the web application is based at least in part on C++ code that is revised to optimize performance of generated JavaScript.

3. The system of claim 1 , wherein the logic when executed is further operable to perform operations based at least in part on a predetermined subset of JavaScript transpiled or compiled from source code of another programming language.

4. The system of claim 1 , wherein the logic when executed is further operable to perform operations comprising:

processing the images using a computer program executed on a graphics processing unit;

extracting feature points from the images; and

performing runtime optimizations to JavaScript frameworks.

5. The system of claim 1 , wherein the logic when executed is further operable to perform operations comprising performing multithread execution of JavaScript in the browser.

6. The system of claim 1 , wherein the logic when executed is further operable to perform operations comprising:

determining in advance or at runtime an execution strategy to use to optimize performance; and

processing image frames in a pipeline using at least one central processing unit (CPU) and at least one graphics processing unit (GPU), wherein the least one CPU processes a first image frame that was previously processed by the at least one GPU, and wherein the at least one CPU processes the first image frame and the at least one GPU processes a second image frame simultaneously.

7. The system of claim 1 , wherein the logic when executed is further operable to perform operations comprising:

accessing from the device one or more of sensor data, camera image data, and inertial measurements using web frameworks; and

rendering the AR images based at least in part on one or more of the sensor data, the camera image data, and the inertial measurements.

8. The system of claim 1 , wherein the logic when executed is further operable to perform operations comprising rendering the virtual content using a rendering engine and a JavaScript content library.

9. A non-transitory computer-readable storage medium carrying program instructions thereon, the instructions when executed by one or more processors are operable to perform operations comprising:

capturing images of a physical scene with a camera of a device;

obtaining camera data from a device library via a browser of the device;

parsing the camera data;

determining one or more intrinsic parameters of the camera, wherein the one or more intrinsic parameters comprise at least one or more of a field of view, a principal point, and distortion parameters;

determining motion of the camera using six degrees of freedom (6DoF) marker-based tracking;

determining positions of markers relative to the camera using 6DoF marker-based tracking;

overlaying virtual content onto a depicted physical scene in the images, resulting in augmented reality (AR) images; and

rendering the AR images in the browser of the device.

10. The computer-readable storage medium of claim 9 , wherein the instructions when executed perform the operations based at least in part on a web application that is executed in the browser, and wherein the web application is based at least in part on C++ code that is revised to optimize performance of generated JavaScript.

11. The computer-readable storage medium of claim 9 , wherein the instructions when executed are further operable to perform operations based at least in part on a predetermined subset of JavaScript transpiled or compiled from source code of another programming language.

12. The computer-readable storage medium of claim 9 , wherein the instructions when executed are further operable to perform operations comprising:

processing the images using a computer program executed on a graphics processing unit;

extracting feature points from the images; and

performing runtime optimizations to JavaScript frameworks.

13. The computer-readable storage medium of claim 9 , wherein the instructions when executed are further operable to perform operations comprising performing multithread execution of JavaScript in the browser.

14. The computer-readable storage medium of claim 9 , wherein the instructions when executed are further operable to perform operations comprising:

determining in advance or at runtime an execution strategy to use to optimize performance; and

processing image frames in a pipeline using at least one central processing unit (CPU) and at least one graphics processing unit (GPU), wherein the least one CPU processes a first image frame that was previously processed by the at least one GPU, and wherein the at least one CPU processes the first image frame and the at least one GPU processes a second image frame simultaneously.

15. The computer-readable storage medium of claim 9 , wherein the instructions when executed are further operable to perform operations comprising:

accessing from the device one or more of sensor data, camera image data, and inertial measurements using web frameworks; and

rendering the AR images based at least in part on one or more of the sensor data, the camera image data, and the inertial measurements.

16. The computer-readable storage medium of claim 9 , wherein the instructions when executed are further operable to perform operations comprising rendering the virtual content using a rendering engine and a JavaScript content library.

17. A computer-implemented method comprising:

capturing images of a physical scene with a camera of a device;

obtaining camera data from a device library via a browser of the device;

parsing the camera data;

determining one or more intrinsic parameters of the camera, wherein the one or more intrinsic parameters comprise at least one or more of a field of view, a principal point, and distortion parameters;

determining motion of the camera using six degrees of freedom (6DoF) marker-based tracking;

determining positions of markers relative to the camera using 6DoF marker-based tracking;

overlaying virtual content onto a depicted physical scene in the images, resulting in augmented reality (AR) images; and

rendering the AR images in the browser of the device.

18. The method of claim 17 , wherein the method performs the steps based at least in part on a web application that is executed in the browser, and wherein the web application is based at least in part on C++ code that is revised to optimize performance of generated JavaScript.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 16, 2025
From: NIANTIC, INC.
To: NIANTIC SPATIAL, INC.
Reel/Frame 071555/0833 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 16, 2023
From: 8TH WALL, INC.; 8TH WALL, LLC
To: NIANTIC, INC.
Reel/Frame 064614/0970 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 25, 2019
From: MURPHY-CHUTORIAN, ERIK; BUTKO, NICHOLAS
To: 8TH WALL INC.
Reel/Frame 049583/0117 →
Continuity (1)
Related Publication 20200410759A1 · Dec 31, 2020