Audio-visual effects system for augmentation of captured performance based on content thereof
Visual effects schedules are applied to audiovisual performances with differing visual effects applied in correspondence with differing elements of musical structure. Segmentation techniques applied to one or more audio tracks (e.g., vocal or backing tracks) are used to compute some of the components of the musical structure. In some cases, applied visual effects schedules are mood-denominated and may be selected by a performer as a component of his or her visual expression or determined from an audiovisual performance using machine learning techniques.
1. A system including an audiovisual processing pipeline comprising:
a video effects planner that, based on a selected video style recipe, maps computationally extracted audio features corresponding to a machine-readable encoding of an audiovisual performance to particular sets of visual effects; and
a video effects renderer that applies differing visual effects of the particular sets of visual effects to the audiovisual performance encoding in correspondence with differing audio features of the computationally extracted audio features.
2. The system of claim 1 , further comprising:
a segmentation engine that computationally extracts the audio features from one or more tracks corresponding to the machine-readable encoding of the audiovisual performance.
3. The system of claim 2 ,
wherein at least one of the tracks includes captured vocal audio; and
wherein the segmentation engine operates on one or more of:
the captured vocal audio track;
a backing track against which the vocal audio track was captured; and
a MIDI file corresponding to the captured vocal audio track and/or the backing track,
to extract the audio features including at least musical section boundaries coded for temporal alignment with the captured vocal audio track.
4. The system of claim 3 ,
wherein the applied visual effects encode differing visual effects for differing musical structure elements of the audiovisual performance encoding and provide visual effect transitions in temporal alignment with at least some of the coded musical section boundaries.
5. The system of claim 4 , wherein the visual effects encoded for respective musical structure elements of the audiovisual performance encode one or more of:
a particle-based effect or lens flare;
transitions between, or layouts of, distinct source videos;
animations or motion of a frame within a source video
vector graphics or images of patterns or textures; and
color, saturation or contrast.
6. The system of claim 1 ,
wherein the selected video style recipe is selected from amongst a plurality of video style recipes based on a computationally-determined mood for the audiovisual performance.
7. The system of claim 1 ,
wherein the selected video style recipe is selected based on a user interface selection by the vocal audio performer prior to, or coincident with, capture of the vocal audio.
8. The system of claim 1 ,
wherein the selected video style recipe is selected based on a user interface selection by the vocal audio performer after an initial, post vocal audio capture, rendering of the audiovisual performance.
9. The system of claim 1 , further comprising:
a communications interface configured to stream the audiovisual performance to an audience at one or more remote client devices.
10. The system of claim 9 ,
wherein the streamed audiovisual performance is mixed with an encoding of a backing track against which the vocal audio was captured.
11. The system of claim 9 ,
wherein the streamed audiovisual performance includes the applied set of visual effects.
12. The system of claim 9 ,
wherein the streamed audiovisual performance is supplied with an identification of the applied set of visual effects for video effect rendering at one or more of the remote client devices.
13. The system of claim 1 ,
wherein at least some of the video style recipes map to mood-denominated sets of visual effects, and
wherein for a particular mood-denominated set of visual effects applied by the renderer, mood values are parameterized as a two-dimensional quantity, wherein a first dimension of the mood parameterization codes an emotion and wherein second dimension of the mood parameterization codes intensity.
14. The system of claim 13 ,
wherein the intensity dimension of the mood parameterization based on one or more of (i) a time-varying audio signal strength or vocal energy density measure computationally determined from a vocal audio track and (ii) beats, tempo, signal strength or energy density of a backing audio track against which the vocal audio track was captured.
15. The system of claim 1 ,
wherein operation of the segmentation engine is based at least in part on a computational determination of vocal intensity of a vocal audio track with at least some segmentation boundaries constrained to temporally align with beats or tempo computationally extracted from a corresponding audio backing track against which the vocal audio track was captured.
16. The system of claim 1 ,
embodied, at least in part, as a content server or service platform to which geographically-distributed, network-connected, vocal capture devices are communicatively coupled.
17. The system of claim 1 ,
embodied, at least in part, as a network-connected, vocal capture device communicatively coupled to stream the audiovisual performance to one or more client devices.
18. A method comprising: processing an audiovisual performance in an audiovisual processing pipeline to augment a machine-readable encoding of the audiovisual performance with temporally aligned visual effects,
the processing including executing video effects planning code that, based on a selected video style recipe, maps computationally extracted audio features corresponding to a machine-readable encoding of the audiovisual performance to particular sets of visual effects and executing video effects rendering code that applies differing visual effects of the particular sets of visual effects to the audiovisual performance encoding in correspondence with differing audio features of the computationally extracted audio features.
19. The method of claim 18 , further comprising:
segmenting the machine-readable encoding of the audiovisual performance to computationally extract the corresponding audio features.
20. The method of claim 19 ,
wherein the machine-readable encoding of the audiovisual performance includes one or more tracks including at least one captured vocal audio track; and
wherein the segmenting operates on one or more of:
the captured vocal audio track;
a backing track against which the vocal audio track was captured; and
a MIDI file corresponding to the captured vocal audio track and/or the backing track,
to extract the audio features including at least musical section boundaries coded for temporal alignment with the captured vocal audio track.
21. The method of claim 18 ,
wherein the applied visual effects encode differing visual effects for differing musical structure elements of the audiovisual performance encoding and provide visual effect transitions in temporal alignment with at least some of the coded musical section boundaries.
22. The method of claim 18 , further comprising:
after an audiovisual rendering of the audiovisual performance with a first set of visual effects applied, selecting a second video style recipe from amongst a plurality of video style recipes, the video style recipe mapping to a second set of visual effects differing first set of visual effects; and
in the video effects rendering code of the audiovisual processing pipeline, applying the second set visual effects to at least a portion of the audiovisual performance encoding.
23. The method of claim 22 ,
wherein at least some of the plurality of video style recipes map to mood-denominated sets of visual effects, and
wherein the applied second set visual effects is a mood-denominated set of visual effects.
24. The method of claim 18 , further comprising:
capturing the audiovisual performance at a network-connected vocal capture device communicatively coupled to a content server or service platform from which the musical structure encoding is supplied.
25. The method of claim 18 ,
wherein the audiovisual performance capture is performed at the network-connected vocal capture device in accordance with a Karaoke-style operational mechanic in which lyrics are visually presented in correspondence with audible rendering of a backing track.