IP Library Granted Patent US 10,560,657
Granted Patent B2
US 10,560,657 · App. 16/402,601 · Granted Feb 11, 2020

Systems and methods for intelligently synchronizing events in visual content with musical features in audio content

Inventor: Jean-Baptiste Noel (Le Vesinet, FR)
Assignee: GoPro, Inc.
H04N5/9305G11B27/031G11B27/34H04N5/91
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,560,657
App. No.
16/402,601
Granted
Feb 11, 2020
Kind
B2
Abstract

Systems for synchronizing events or transitions in visual content with musical features in audio content are configured to obtain audio and visual content; determine a minimum, maximum, and/or a target display duration for items of visual content; determine a first playback-time in the first audio content to associate with a start-of-display time for the first visual content; identify a first timeframe in the first audio content corresponding to a range of acceptable end-of-display times for the first visual content; identify musical features within the first timeframe; identify a candidate musical feature among the identified musical features in accordance with a hierarchy; and/or define a first candidate end-of-display time that aligns with the playback time of the candidate musical feature. A set of candidate end-of-display times are defined for multiple visual content items in a single multimedia project, the set identified by seeking a solution that increases rank among the hierarchy.

Claims (68)

1. A system for synchronizing visual content with musical features in audio content, the system comprising:

one or more physical computer processors configured by computer readable instructions to:

obtain a first media file containing first visual information defining first visual content;

obtain a minimum display duration, a maximum display duration, and a target display duration for display of the first visual content;

obtain an audio file containing audio information representing audio content;

determine a first playback-time in the audio content to associate with a start-of-display time for display of the first visual content;

identify a range of acceptable end-of-display times for the first visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the first visual content, the range of acceptable end-of-display times for the first visual content corresponding to a first timeframe in the audio content;

identify a target end-of-display time of the first visual content based on the start-of-display time and the target display duration of the first visual content;

identify relative importance of musical features within the audio content during the first timeframe based on a musical feature hierarchy;

select a first musical feature from among the musical features within the audio content during the first timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the first visual content as compared to other musical features; and

determine a first end-of-display time for the first visual content that aligns with a playback time of the first musical feature.

2. The system of claim 1 , wherein the start-of-display time for the first visual content is defined by a user.

3. The system of claim 1 , wherein the one or more physical computer processors are further configured by computer readable instructions to:

obtain a second media file containing second visual information defining second visual content;

determine a minimum display duration, a maximum display duration, and a target display duration for display of the second visual content;

determine a second playback-time in the audio content to associate with a start-of-display time for display of the second visual content;

identify a range of acceptable end-of-display times for the second visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the second visual content, the range of acceptable end-of-display times for the second visual content corresponding to a second timeframe in the audio content;

identify a target end-of-display time of the second visual content based on the start-of-display time and the target display duration of the second visual content;

identify relative importance of musical features within the audio content during the second timeframe based on the musical feature hierarchy;

select a second musical feature from among the musical features within the audio content during the second timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the second visual content as compared to other musical features; and

determine a second end-of-display time for the second visual content that aligns with a playback time of the second musical feature.

4. The system of claim 3 , wherein the second timeframe in the audio content is adjacent to the first timeframe.

5. The system of claim 3 , wherein a combination of a first duration from the start-of- display time of the first visual content until the first end-of-display time for the first visual content and a second duration from the start-of-display-time of the second visual content until the second end-of-display time for the second visual content matches a total playback time of the audio content.

6. The system of claim 3 , wherein the first musical feature and the second musical feature are selected such that an average of the proximity in time between the playback time of the first musical feature and the target-end of display time for the first visual content, and proximity in time between the playback time of the second musical feature and the target-end of display time for the second visual content is minimized.

7. The system of claim 3 , wherein the start-of-display time and the end-of-display time for the first visual content is defined by a user.

8. A method for synchronizing transitions in visual content with musical features in audio content, the method comprising:

obtaining a first media file containing first visual information defining first visual content;

determining a minimum display duration, a maximum display duration, and a target display duration for display of the first visual content;

obtaining an audio file containing audio information representing audio content;

determining a first playback-time in the audio content to associate with a start-of-display-time for display of the first visual content;

identifying a range of acceptable end-of-display times for the first visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the first visual content, the range of acceptable end-of-display times for the first visual content corresponding to a first timeframe in the audio content;

identifying a target end-of-display time of the first visual content based on the start-of-display time and the target display duration of the first visual content;

identifying relative importance musical features within the audio content during the first timeframe based on a musical feature hierarchy;

selecting a first musical feature from among the musical features within the audio content during the first timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the first visual content as compared to other musical features; and

determining a first end-of-display time for the first visual content that aligns with a playback time of the first musical feature.

9. The method of claim 8 , wherein start-of-display time for the first visual content is defined by a user.

10. The method of claim 8 , further comprising:

obtaining a second media file containing second visual information defining second visual content;

determining a minimum display duration, a maximum display duration, and a target display duration for display of the second visual content;

determining a second playback-time in the audio content to associate with a start-of-display-time for display of the second visual content;

identifying a range of acceptable end-of-display times for the second visual content based on the start-of-display time, the minimum display duration, and/or the maximum display duration of the second visual content, the range of acceptable end-of-display times for the second visual content corresponding to a second timeframe in the audio content;

identifying a target end-of-display time of the second visual content based on the start-of-display time and the target display duration of the second visual content;

identifying relative importance of musical features within the audio content during the second timeframe based on the musical feature hierarchy;

selecting a second musical feature from among the musical features within the audio content during the second timeframe based on (1) its relative importance in accordance with the musical feature hierarchy, and (2) its relative proximity in time to the target end-of-display time of the second visual content as compared to other musical features; and

determining a second end-of-display time for the second visual content that aligns with a playback time of the second musical feature.

11. The method of claim 10 , wherein the second timeframe in the audio content is adjacent to the first timeframe.

12. The method of claim 10 , wherein a combination of a first duration from the start-of-display time of the first visual content until the first end-of-display time for the first visual content and a second duration from the start-of-display-time of the second visual content until the second end-of-display time for the second visual content matches a total playback time of the audio content.

13. The method of claim 10 , wherein the first musical feature and the second musical feature are selected such that an average of the proximity in time between the playback time of the first musical feature and the target-end of display time for the first visual content, and proximity in time between the playback time of the second musical feature and the target-end of display time for the second visual content is minimized.

14. The method of claim 10 , wherein the start-of-display time and the end-of-display time for the first visual content is defined by a user.

15. A system for synchronizing transitions in visual content with musical features in audio content, the system comprising:

one or more physical computer processors configured by computer readable instructions to:

obtain multiple media files containing visual information representing visual content;

define a display order for the visual content of the multiple media files;

define a minimum display duration, a maximum display duration, and a target display duration for display of the visual content of the multiple media files;

obtain an audio file containing audio information representing audio content;

define a first playback-time in the audio content as a start-of-display time for display of the visual content in accordance with the display order;

identify a range of acceptable end-of-display times for individual ones of the visual content of the multiple media files based on the start-of-display time, the minimum display duration, and/or the maximum display duration for the individual ones of the visual content of the multiple media files, the ranges of acceptable end-of-display times for the visual content of the multiple media files corresponding to timeframes in the audio content;

identify a target end-of-display times of the individual ones of the visual content of the multiple media files based on the start-of-display time and the target display duration of the individual ones of the visual content of the multiple media files;

identify relative importance of musical features within the audio content during the timeframes based on a musical feature hierarchy;

select a musical feature within individual ones of the timeframes based on (1) its relative importance as compared to the other identified musical features, and (2) its relative proximity in time to a corresponding target end-of-display time of the visual content as compared to other musical features; and

determine an end-of-display time for the individual ones of the visual content of the multiple media files, wherein the end-of-display time aligns with a playback time of the musical feature within a corresponding timeframe.

16. The system of claim 15 , wherein the start-of-display time is defined by a user.

17. The system of claim 15 , wherein the multiple media files include a first media file and a second media file, the first media file containing first visual information defining first visual content and the second media file containing second visual information defining second visual content.

18. The system of claim 17 , wherein:

a first musical feature is selected for the first visual content and a second musical feature is selected for the second visual content; and

the first musical feature and the second musical feature are selected such that an average of the proximity in time between a playback time of the first musical feature and the target-end of display time for the first visual content, and proximity in time between a playback time of the second musical feature and the target-end of display time for the second visual content is minimized.

19. The system of claim 17 , wherein the timeframes in the audio content include a first timeframe for the first visual content and a second timeframe for the second visual content, the second timeframe adjacent to the first timeframe.

20. The system of claim 17 , wherein a combination of a first duration from the start-of-display time of the first visual content until the first end-of-display time for the first visual content and a second duration from the start-of-display-time of the second visual content until the second end-of-display time for the second visual content matches a total playback time of the audio content.

Assignments (5)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 3, 2019
From: NOEL, JEAN-BAPTISTE
To: GOPRO, INC.
Reel/Frame 049072/0971 →
Continuity (3)
Continuation 15436422 · Feb 17, 2017
Provisional Application 62418712 · Nov 7, 2016
Related Publication 20190289254A1 · Sep 19, 2019
Cited By (2)
US 12,236,981 US 12,634,563