Granted Patent B2

US 12,244,782 · App. 17/519,121 · Granted Mar 4, 2025

Inventors: Sankar Jayaram (Los Gatos, CA); Wayne O. Cochran (Ridgefield, WA); John Harrison (Hillsboro, OR); Timothy P. Heidmann (Los Altos, CA); Thomas Sahara (Atlanta, GA); John Buddy Scott (Lawrenceville, CA)

Assignee: Quintar, Inc.

H04N13/111G06T19/006G06V10/255G06V20/46H04N13/344

Abstract

Claims (60)

1. A method, comprising:

receiving, by a first augmented reality viewing device, a coordinate transformation between a coordinate system of the first augmented reality viewing device and a coordinate system for augmented reality content to be displayed by the first augmented reality viewing device;

determining, by the first augmented reality viewing device and in the coordinate system of the first augmented reality viewing device, a specified location within a field of view of the first augmented reality viewing device;

receiving, by the first augmented reality viewing device, augmented reality content, including video of an event at a venue, in the coordinate system for augmented reality content;

transforming, by the first augmented reality viewing device, the received augmented reality content to the coordinate system of the first augmented reality viewing device; and

displaying the received augmented reality content by the first augmented reality viewing device at the specified location in the coordinate system of the first augmented reality viewing device.

2. The method of claim 1 , wherein the first augmented reality viewing device is at a location remote from the venue of the event.

3. The method of claim 1 , wherein the video of the event is three dimensional video and the augmented reality content further includes three dimensional graphics, wherein displaying the received augmented reality content at the specified location the coordinate system of the first augmented reality viewing device includes:

displaying the three dimensional video of the event at the specified location in the coordinate system of the first augmented reality viewing device; and

overlaying the three dimensional graphics over the three dimensional video.

4. The method of claim 1 , wherein the video of the event is three dimensional video and the three dimensional video is received as frames of stereoscopic, each frame comprising a left image and a right image, wherein displaying the received augmented reality content includes:

separating out the left image and the right image of each of the frames; and

concurrently displaying the left image to a left eye of a viewer using the first augmented reality viewing device and displaying the right image to a right eye of the viewing.

5. The method of claim 1 , further comprising:

specifying the location within the field of view of the first augmented reality viewing device by a reference object within the field of view of the first augmented reality viewing device.

6. The method of claim 1 , further comprising:

specifying the location within the field of view of the first augmented reality viewing device by manually indicating a location within the field of view of the first augmented reality viewing device.

7. The method of claim 1 , further comprising:

determining a viewer using the first augmented reality viewing device has changed position; and

in response to determining that the viewer has changed position, changing the location at which the received augmented reality content is displayed.

8. The method of claim 1 , wherein the first augmented reality viewing device receives the augmented reality content from one or more servers, the method further comprising:

determining a viewer using the first augmented reality viewing device has changed position; and

in response to determining that the viewer has changed position, requesting from the one or more servers video of the event with a different view of the event than previously received video of the event.

9. The method of claim 1 , wherein the video of the event is monoscopic video.

10. A system, comprising:

one or more servers configured to receive data from and transmit data to one or more augmented reality viewing devices and to:

receive video from a plurality of views of an event at an event venue in a first coordinate system;

receive a request for video for a selected one of the views from a first augmented reality viewing device;

provide to the first augmented reality viewing device the requested video from the selected view in the first coordinate system; and

provide to the first augmented reality viewing device a coordinate transformation between the coordinate system of the first augmented reality viewing device and the first coordinate system.

11. The system of claim 10 , the one or more servers further configured to:

perform segmentation on the received video, wherein the requested video provided to the first augmented reality viewing device is segmented.

12. The system of claim 11 , wherein, to performing segmentation on the received video, the one or more servers further configured to:

mask out a portion of individual frames of the received video corresponding to a geographic area of interest within the individual frames;

determine objects of interest in the individual frames that extend outside of the masked out portion;

segment out objects of interest from the individual frames; and

composite the masked out portion and the segmented out objects of interest.

13. The system of claim 12 , wherein masking out the portion of the individual frames includes a manual process.

14. The system of claim 12 , wherein masking out the portion of the individual frames includes an automated process.

15. The system of claim 10 , wherein the first augmented reality viewing device is at a location other than the event venue.

16. The system of claim 15 , wherein the video is three dimensional (3D) video and the one or more servers further configured to:

provide to a second augmented reality viewing device the 3D video the event, wherein the second augmented reality viewing device is at the location other than the event venue;

provide to the second augmented reality viewing device a coordinate transformation between the coordinate system of the second augmented reality viewing device and the first coordinate system; and

provide to one or both of the first augmented reality viewing device and the second augmented reality viewing device information to coordinate display of the 3D video by the first augmented reality viewing device and the second augmented reality viewing device.

17. The system of claim 16 , wherein the 3D video of the event provided to the second augmented reality viewing device is a different view of the 3D video of the event than the view of the 3D video of the event provided to the first augmented reality viewing device.

18. The system of claim 10 , the one or more servers further configured to:

receiving at the system a request for overlay augmented reality graphics for the video from the first augmented reality viewing device; and

providing from the system to the first augmented reality viewing device the requested overlay augmented reality graphics.

19. A method, comprising:

receiving, at a system of one or more servers, video from a plurality of views of an event at an event venue in a first coordinate system;

receiving at the system a request for video for a selected one of the views from a first of the augmented reality viewing devices;

receiving at the system a request for overlay augmented reality graphics for the video from the first augmented reality viewing device;

providing from the system to the first augmented reality viewing device the requested video from the selected view in the first coordinate system;

providing from the system to the first augmented reality viewing device the requested overlay augmented reality graphics in the first coordinate system; and

providing from the system to the first augmented reality viewing device a coordinate transformation between a coordinate system of the first augmented reality viewing device and the first coordinate system.

20. The method of claim 19 , further comprising:

masking out a portion of individual frames of the received video corresponding to a geographic area of interest within the individual frames;

determining objects of interest in the individual frames that extend outside of the masked out portion;

segmenting out objects of interest from the individual frames; and

compositing the masked out portion and the segmented out objects of interest for the individual frames, wherein the requested video provided to the first augmented reality viewing device are the composited individual frames.

Assignments (1)

Continuity (6)

References Cited (185)

Cited By (2)

US 12,469,155 US 12,675,962