IP Library Granted Patent US 10,754,511
Granted Patent B2
US 10,754,511 · App. 14/323,807 · Granted Aug 25, 2020

Multi-view audio and video interactive playback

Inventors: Neil Birkbeck (Sunnyvale, CA); Isasi Inguva (Mountain View, CA); Damien Kelly (Sunnyvale, CA); Andrew Crawford (Mountain View, CA); Hugh Denman (San Francisco, CA); Perry Tobin (Santa Cruz, CA); Steve Benting (San Mateo, CA); Anil Kokaram (Sunnyvale, CA); Jeremy Doig (Los Gatos, CA)
Assignee: GOOGLE LLC
G06F3/0484G11B27/10G11B27/34H04N21/2387H04N21/242H04N21/45H04N21/8455H04N21/8456H04N21/8549
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,754,511
App. No.
14/323,807
Filed
Jul 3, 2014
Granted
Aug 25, 2020
Kind
B2
Examiner
HUYNH, THU V
Art Unit
2177
USPC
715/203
Abstract

An interactive multi-view module identifies a plurality of media items associated with a real-world event, each of the plurality of media items comprising a video portion and an audio portion. The interactive multi-view module synchronizes the audio portions of each of the plurality of media items according to a common reference timeline, determines a relative geographic position associated with each of the plurality of media items and presents the plurality of media items in an interactive multi-view player interface based at least on the synchronized audio portions and the relative geographic positions.

Claims (60)

1. A method comprising:

identifying, by a processing device, a plurality of media items associated with a real-world event, wherein each of the plurality of media items is created by capturing the real-world event from a particular geographic position and comprises a video portion and an audio portion;

synchronizing the audio portions of each of the plurality of media items according to a common reference timeline;

determining a relative geographic position associated with each of the plurality of media items; and

providing the plurality of media items for presentation in an interactive multi-view player interface based at least on the synchronized audio portions and multiple relative geographic positions,

wherein the interactive multi-view player interface allows a user of a plurality of users to switch between different viewpoints in space across the plurality of media items, and to seek, on a plurality of event bars, to different points in time across the plurality of media items and beyond a single media item of the plurality of media items associated with the multiple relative geographic positions,

wherein the plurality of event bars comprises a video density bar indicating a number of media items available at a given point in time across the plurality of media items and an event highlight bar indicating a popularity of content portions of respective media items at the given point in time across the plurality of media items, and

wherein the popularity of the content portions of the respective media items at the given point in time is determined using one or more factors comprising a number of times the respective media items were replayed at the given point in time from different viewpoints by the plurality of users.

2. The method of claim 1 , further comprising:

generating a single edited media item associated with the real-world event, by combining the plurality of media items based on the synchronized audio portions.

3. The method of claim 1 , wherein identifying the plurality of media items comprises determining media items having metadata that identifies the real-world event.

4. The method of claim 1 , wherein synchronizing the audio portions of each of the plurality of media items comprises determining a temporal offset for each of the plurality of media items that increases a correlation for a frequency-based audio spectrogram of each audio portion, wherein the temporal offsets represent locations on the common reference timeline.

5. The method of claim 1 , wherein determining the relative geographic position associated with each of the plurality of media items comprises determining relative positions of separate cameras used to capture each of the plurality of media items at a time of the real-world event based on a plurality of points visible in each of the plurality of media items.

6. The method of claim 1 , wherein providing the plurality of media items for presentation in the interactive multi-view player interface comprises:

displaying an icon representing each of the plurality of media items, wherein each icon is displayed in a location based on the corresponding relative geographic position;

receiving a user selection of an icon representing a first media item of the plurality of media items and beginning playback of the first media item; and

receiving, at a time during playback of the first media item corresponding to a first point on the common reference timeline, a user selection of an icon representing a second media item of the plurality of media items and beginning playback of the second media item at a time in the second media item corresponding to the first point on the common reference timeline.

7. The method of claim 6 , further comprising:

predicting a subsequent media item to be selected by the user; and

buffering the subsequent media item prior to a user selection of the subsequent media item.

8. A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising:

identifying, by the processing device, a plurality of media items associated with a real-world event, wherein each of the plurality of media items is created by capturing the real-world event from a particular geographic position and comprises a video portion and an audio portion;

synchronizing the audio portions of each of the plurality of media items according to a common reference timeline;

determining a relative geographic position associated with each of the plurality of media items; and

providing the plurality of media items for presentation in an interactive multi-view player interface based at least on the synchronized audio portions and multiple relative geographic positions,

wherein the interactive multi-view player interface allows a user of a plurality of users to switch between different viewpoints in space across the plurality of media items, and to seek, on a plurality of event bars, to different points in time across the plurality of media items and beyond a single media item of the plurality of media items associated with the multiple relative geographic positions,

wherein the plurality of event bars comprises a video density bar indicating a number of media items available at a given point in time across the plurality of media items and an event highlight bar indicating a popularity of content portions of respective media items at the given point in time across the plurality of media items, and

wherein the popularity of the content portions of the respective media items at the given point in time is determined using one or more factors comprising a number of times the respective media items were replayed at the given point in time from different viewpoints by the plurality of users.

9. The non-transitory machine-readable storage medium of claim 8 , wherein the operations further comprise:

generating a single edited media item associated with the real-world event, by combining the plurality of media items based on the synchronized audio portions.

10. The non-transitory machine-readable storage medium of claim 8 , wherein identifying the plurality of media items comprises determining media items having metadata that identifies the real-world event.

11. The non-transitory machine-readable storage medium of claim 8 , wherein synchronizing the audio portions of each of the plurality of media items comprises determining a temporal offset for each of the plurality of media items that increases a correlation for a frequency-based audio spectrogram of each audio portion, wherein the temporal offsets represent locations on the common reference timeline.

12. The non-transitory machine-readable storage medium of claim 8 , wherein determining the relative geographic position associated with each of the plurality of media items comprises determining relative positions of separate cameras used to capture each of the plurality of media items at a time of the real-world event based on a plurality of points visible in each of the plurality of media items.

13. The non-transitory machine-readable storage medium of claim 8 , wherein providing the plurality of media items for presentation in the interactive multi-view player interface comprises:

displaying an icon representing each of the plurality of media items, wherein each icon is displayed in a location based on the corresponding relative geographic position;

receiving a user selection of an icon representing a first media item of the plurality of media items and beginning playback of the first media item; and

receiving, at a time during playback of the first media item corresponding to a first point on the common reference timeline, a user selection of an icon representing a second media item of the plurality of media items and beginning playback of the second media item at a time in the second media item corresponding to the first point on the common reference timeline.

14. The non-transitory machine-readable storage medium of claim 13 , wherein the operations further comprise:

predicting a subsequent media item to be selected by the user; and

buffering the subsequent media item prior to a user selection of the subsequent media item.

15. A server computer system comprising:

a processing device;

a memory coupled to the processing device; and

an interactive multi-view module, executable by the processing device from the memory, to:

identify a plurality of media items associated with a real-world event, wherein each of the plurality of media items is created by capturing the real-world event from a particular geographic position and comprises a video portion and an audio portion;

synchronize the audio portions of each of the plurality of media items according to a common reference timeline;

determine a relative geographic position associated with each of the plurality of media items; and

provide the plurality of media items for presentation in an interactive multi-view player interface based at least on the synchronized audio portions and multiple relative geographic positions,

wherein the interactive multi-view player interface allows a user of a plurality of users to switch between different viewpoints in space across the plurality of media items, and to seek, on a plurality of event bars, to different points in time across the plurality of media items and beyond a single media item of the plurality of media items associated with the multiple relative geographic positions,

wherein the plurality of event bars comprises a video density bar indicating a number of media items available at a given point in time across the plurality of media items and an event highlight bar indicating a popularity of content portions of respective media items at the given point in time across the plurality of media items, and

wherein the popularity of the content portions of the respective media items at the given point in time is determined using one or more factors comprising a number of times the respective media items were replayed at the given point in time from different viewpoints by the plurality of users.

16. The server computer system of claim 15 , wherein the interactive multi-view module further to:

generate a single edited media item associated with the real-world event, by combining the plurality of media items based on the synchronized audio portions.

17. The server computer system of claim 15 , wherein to identify the plurality of media items, the interactive multi-view module to determine media items having metadata that identifies the real-world event.

18. The server computer system of claim 15 , wherein to synchronize the audio portions of each of the plurality of media items, the interactive multi-view module to determine a temporal offset for each of the plurality of media items that increases a correlation for a frequency-based audio spectrogram of each audio portion, wherein the temporal offsets represent locations on the common reference timeline.

19. The server computer system of claim 15 , wherein to determine the relative geographic position associated with each of the plurality of media items, the interactive multi-view module to determine relative positions of separate cameras used to capture each of the plurality of media items at a time of the real-world event based on a plurality of points visible in each of the plurality of media items.

20. The server computer system of claim 15 , wherein to provide the plurality of media items for presentation in the interactive multi-view player interface, the interactive multi-view module to:

display an icon representing each of the plurality of media items, wherein each icon is displayed in a location based on the corresponding relative geographic position;

receive a user selection of an icon representing a first media item of the plurality of media items and begin playback of the first media item; and

receive, at a time during playback of the first media item corresponding to a first point on the common reference timeline, a user selection of an icon representing a second media item of the plurality of media items and begin playback of the second media item at a time in the second media item corresponding to the first point on the common reference timeline.

Assignments (2)
CHANGE OF NAME Recorded Oct 5, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044129/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 3, 2014
From: BIRKBECK, NEIL; INGUVA, ISASI; KELLY, DAMIEN; CRAWFORD, ANDREW; DENMAN, HUGH; TOBIN, PERRY; BENTING, STEVE; KOKARAM, ANIL; DOIG, JEREMY
To: GOOGLE INC.
Reel/Frame 033242/0380 →
Continuity (2)
Provisional Application 61906588 · Nov 20, 2013
Related Publication 20150143239A1 · May 21, 2015
Cited By (1)
US 12,520,105