Audio-visual effects system for augmentation of captured performance based on content thereof
Visual effects schedules are applied to audiovisual performances with differing visual effects applied in correspondence with differing elements of musical structure. Segmentation techniques applied to one or more audio tracks (e.g., vocal or backing tracks) are used to compute some of the components of the musical structure. In some cases, applied visual effects schedules are mood-denominated and may be selected by a performer as a component of his or her visual expression or determined from an audiovisual performance using machine learning techniques.
1. A method comprising:
accessing a machine readable encoding of a first audiovisual performance captured as vocal audio with performance synchronized video and having an associated musical structure encoding that includes at least musical section boundaries coded for temporal alignment with the first audiovisual performance encoding;
applying a first visual effect schedule to at least a portion of the first audiovisual performance encoding, wherein the applied visual effect schedule encodes differing visual effects for differing musical structure elements of the first audiovisual performance encoding and provides visual effect transitions in temporal alignment with at least some of the coded musical section boundaries.
2. The method of claim 1 , further comprising:
segmenting at least an audio track of the first audiovisual performance encoding to provide the associated musical structure encoding.
3. The method of claim 1 ,
wherein the associated musical structure encoding includes group part or musical section metadata.
4. The method of claim 1 ,
wherein the differing visual effects differ in either degree or type or both degree and type.
5. The method of claim 1 , further comprising:
selecting the first visual effect schedule from amongst a plurality of visual effect schedules.
6. The method of claim 5 ,
wherein the selecting is based on a computationally-determined mood for at least the captured vocal audio.
7. The method of claim 5 ,
wherein the selecting is based on a user interface selection by the vocal audio performer prior to, or coincident with, capture of the vocal audio.
8. The method of claim 5 , further comprising:
selecting a second visual effect schedule from amongst the plurality of mood-denominated visual effect schedules, the second visual effect schedule differing from the first visual effect schedule; and
applying the second visual effect schedule to at least a portion of the first audiovisual performance encoding.
9. The method of claim 8 , further comprising:
supplying an identification of the applied visual effect schedule for video effect rendering at one or more of the remote client devices.
10. The method of claim 5 , further comprising:
streaming, to an audience at one or more remote client devices, the first audiovisual performance.
11. The method of claim 10 ,
wherein the streamed first audiovisual performance is mixed with an encoding of a backing track against which the vocal audio was captured.
12. The method of claim 10 ,
wherein the streamed first audiovisual performance is streamed with the first visual effect schedule applied.
13. The method of claim 5 , further comprising:
to, from, or via a content server or service platform, transferring the first audiovisual performance together with at least an identifier for the one or more applied visual effect schedules.
14. The method of claim 5 ,
wherein the selecting is based on a user interface selection during, or prior to, audiovisual rendering of the first audiovisual performance.
15. The method of claim 14 , further comprising:
after the audiovisual rendering of the first audiovisual performance, selecting a second visual effect schedule from amongst the plurality of mood-denominated visual effect schedules, the second visual effect schedule differing from the first visual effect schedule; and
applying the second visual effect schedule to at least a portion of the first audiovisual performance encoding.
16. The method of claim 5 ,
wherein for a particular mood-denominated visual effect schedule, mood values are parameterized as a two-dimensional quantity, wherein a first dimension of the mood parameterization codes an emotion and wherein second dimension of the mood parameterization codes intensity.
17. The method of claim 16 , further comprising:
determining an intensity dimension of the mood parameterization based on one or more of: (i) a time-varying audio signal strength or vocal energy density measure computationally determined from the vocal audio and (ii) beats, tempo, signal strength or energy density of a backing audio track.
18. The method of claim 1 , further comprising:
segmenting the first audiovisual performance encoding to identify the differing musical structure elements.
19. The method of claim 18 ,
wherein the segmenting is based at least in part on a computational determination of vocal intensity with at least some segmentation boundaries constrained to temporally align with beats or tempo computationally extracted from a corresponding audio backing track.
20. The method of claim 18 ,
wherein the segmenting is based at least in part on a similarity analysis computationally performed on a temporally-aligned lyrics track to classify particular portions of first audiovisual performance encoding as verse or chorus.
21. The method of claim 1 , further comprising:
segmenting an audio backing track associated with the first audiovisual performance encoding to identify the differing musical structure elements.
22. The method of claim 1 , wherein the differing visual effects encoded by the applied visual effect schedule include for a given element thereof, one or more of:
a particle-based effect or lens flare;
transitions between, or layouts of, distinct source videos;
animations or motion of a frame within a source video
vector graphics or images of patterns or textures; and
color, saturation or contrast.
23. The method of claim 1 , wherein the differing visual effects encoded by the applied visual effect schedule include for a given element thereof, one or more of:
visual augmentation or ornamentation of a video image of a vocal performer's face, head or body; and
background subtraction.
24. The method of claim 1 ,
wherein the associated musical structure encodes musical sections of differing types; and
wherein the applied visual effect schedule defines differing visual effects for different ones of the encoded musical sections.
25. The method of claim 1 ,
wherein the associated musical structure encodes events or transitions; and
wherein the applied visual effect schedule defines differing visual effects for different ones of the encoded events or transitions.
26. The method of claim 1 , wherein the machine readable encoding further encodes at least part of a second audiovisual performance captured as second vocal audio with performance synchronized video, the first and second audiovisual performances constituting a group performance.
27. The method of claim 26 ,
wherein the associated musical structure encodes group parts, and
wherein the applied visual effect schedule is temporally selective for particular performance synchronized video in correspondence with the encoded musical structure.
28. The method of claim 26 ,
wherein the first and second audiovisual performances present as a duet.
29. The method of claim 26 ,
wherein the applied visual effect schedule codes, for at least some musical structure elements, color matching of performance synchronized video for respective performers in the group performance.
30. The method of claim 26 ,
wherein the applied visual effect schedule codes, for at least some musical structure elements, a visual blur or blend at an interface between performance synchronized video for respective performers in the group performance.
31. The method of claim 26 ,
wherein the first and second audiovisual performances are captured against a common backing track.
32. The method of claim 1 , further comprising:
capturing the first audiovisual performance at a network-connected vocal capture device communicatively coupled to a content server or service platform from which the musical structure encoding is supplied.
33. The method of claim 32 ,
wherein the audiovisual performance capture is performed at the network-connected vocal capture device in accordance with a Karaoke-style operational mechanic in which lyrics are visually presented in correspondence with audible rendering of a backing track.
34. The method of claim 1 ,
performed, at least in part, on a content server or service platform to which geographically-distributed, network-connected, vocal capture devices are communicatively coupled.
35. The method of claim 1 ,
performed, at least in part, on a network-connected, vocal capture device communicatively coupled to a content server or service platform.
36. The method of claim 1 ,
embodied, at least in part, as a computer program product encoding of instructions executable on a content server or service platform to which a plurality of geographically-distributed, network-connected, vocal capture devices are communicatively coupled.
37. A system comprising:
a geographically distributed set of network-connected devices configured to capture audiovisual performances including vocal audio with performance synchronized video; and
a service platform configured to (i) receive encodings of the captured audiovisual performances and, in correspondence with associated musical structure encodings that include at least musical section boundaries coded for temporal alignment with the audiovisual performance encodings, to (ii) apply particular visual effect schedules to at least a portion of the audiovisual performance encodings,
wherein the applied visual effect schedules encode differing visual effects for differing musical structure elements of the audiovisual performance encodings and provide visual effect transitions in temporal alignment with at least some of the coded musical section boundaries.
38. The system of claim 37 ,
wherein the service platform is configured to, responsive to user or performer selections, change the applied visual effect schedules and supply resulting captured audiovisual performances to respective ones of the geographically distributed network-connected devices for audiovisual rendering thereon.
39. The system of claim 38 ,
wherein the applied visual effect schedules are selected from amongst a plurality of visual effect schedules.
40. A system comprising:
at least a guest and host pairing of network-connected devices configured to capture at least vocal audio; and
the host device configured to (i) receive from the guest device an encoding of a respective encoding of at least vocal audio and, in correspondence with an associated musical structure encoding that includes at least musical section boundaries coded for temporal alignment with an audiovisual performance encoding, to (ii) apply a selected visual effect schedules to the audiovisual performance encoding,
wherein the applied visual effect schedules encode differing visual effects for differing musical structure elements of the audiovisual performance encoding and provide visual effect transitions in temporal alignment with at least some of the coded musical section boundaries.
41. The system of claim 40 ,
the host and guest devices coupled as local and remote peers via communication network with non-negligible peer-to-peer latency for transmissions of audiovisual content, the host device communicatively coupled as the local peer to receive a media encoding of a mixed audio performance constituting vocal audio captured at the guest device, and the guest device communicatively coupled as the remote peer to supply the media encoding captured from a first one of the performers and mixed with a backing audio track.
42. The system of claim 40 ,
wherein associated musical structure encoding is computationally determined at the host device based on segmenting at least an audio track received from the guest device.
43. The system of claim 40 ,
the host device configured to render the audiovisual performance coding as a mixed audiovisual performance, including vocal audio and performance synchronized video from the first and a second one of the performers, and transmit the audiovisual performance coding as an apparently live broadcast with the selected visual effect schedule applied.