IP Library Granted Patent US 10,565,972
Granted Patent B2
US 10,565,972 · App. 15/337,866 · Granted Feb 18, 2020

Audiovisual media application platform with wireless handheld audiovisual input

Inventors: Benjamin Hersh (San Francisco, CA); John Shimmin (San Francisco, CA); Jeannie Yang (San Francisco, CA); Perry R. Cook (Jacksonville, OR)
Assignee: Smule, Inc.
G10H1/368G06F3/0346G06F3/0484G06F3/04883G06F3/162G06F3/165G10H1/365G10H1/366H04W12/06G06F3/167G10H2220/011G10H2220/096H04W8/005
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,565,972
App. No.
15/337,866
Granted
Feb 18, 2020
Kind
B2
Abstract

Embodiments described herein relate generally to systems comprising a display device, a display device-coupled computing platform, a mobile device in communication with the computing platform, and a content server in which methods and techniques of capture and/or processing of audiovisual performances are described and, in particular, description of techniques suitable for use in connection with display device connected computing platforms for rendering vocal performance captured by a handheld computing device.

Claims (75)

1. A method comprising:

communicatively pairing a touchscreen-enabled portable computing device and a display device-coupled computing platform for paired communications over a wireless network;

the display device-coupled computing platform hosting a karaoke-style media application executing thereon that presents lyrics on the display device in temporal correspondence with an audible rendering of a backing audio track; and

via the wireless network, transmitting from the touchscreen-enabled portable computing device to the computing platform paired therewith a vocal audio stream captured using a microphone of the portable computing device, the vocal audio stream being supplied from the paired-with portable computing device as an audio input to the media application executing on the display device-coupled computing platform, wherein the captured vocal audio is combined for audiovisual rendering with the backing audio track and performance synchronized video.

2. The method of claim 1 ,

wherein the karaoke-style media application executing on the display device-coupled computing platform further presents vocal pitch cues on the display device in temporal correspondence with the presented lyrics and audible rendering of the backing audio track.

3. The method of claim 1 , further comprising:

capturing performance synchronized video using a camera of the portable computing device; and

via the wireless network, transmitting the captured performance synchronized video from the touchscreen-enabled portable computing device to the computing platform paired therewith.

4. The method of claim 1 , further comprising:

using the display device-coupled computing platform, audiovisually rendering the captured vocal audio in temporal correspondence with the backing audio track and the performance synchronized video.

5. The method of claim 1 , further comprising:

pairing a second touchscreen-enabled portable computing device with the display device-coupled computing platform for communications over the wireless network; and

via the wireless network, transmitting from the touchscreen-enabled portable computing device to the computing platform paired therewith a second vocal audio stream captured using a microphone of the second portable computing device, the second vocal audio stream being supplied as a second audio input to the media application executing on the display device-coupled computing platform, wherein the captured second vocal audio is also combined for audiovisual rendering with the backing audio track and performance synchronized video.

6. The method of claim 1 , wherein the pairing includes:

detecting, via a discovery protocol conveyed over the wireless network, local network presence of the display device-coupled computing platform;

displaying on the touchscreen of the portable computing device, an indication of the detected local network presence;

capturing on the touchscreen of the portable computing device, a user gesture to couple the portable computing device to the display device-coupled computing platform as the audio input to a media application executing on the display device-coupled computing platform.

7. The method of claim 1 , further comprising:

displaying on the touchscreen of the portable computing device, an indication to the user of the portable computing device to reorient the portable computing device from a generally right-side-up orientation to a generally upside-down orientation, wherein the generally upside-down orientation orients the microphone toward the user's mouth.

8. The method of claim 1 , further comprising:

detecting, using one or more of an accelerometer and a gyroscope of the portable computing device, an orientation of the portable computing device; and

responsive to a detected change in orientation, transitioning to a vocal capture mode.

9. The method of claim 8 , further comprising:

responsive to a detected change in orientation, audibly rendering a success indication.

10. The method of claim 1 , further comprising:

displaying, on the touchscreen of the portable computing device, an on-off switch-type user interface control; and

responsive to user gestures on the touchscreen changing between (i) a first user interface state wherein the captured vocal audio stream captured is transmitted to the display device-coupled computing platform over the wireless network and (ii) a second user interface state wherein the vocal audio stream is not transmitted.

11. The method of claim 1 , further comprising:

pitch correcting, at the portable computing device, captured vocals prior to transmitting the captured vocal audio to the display device-coupled computing platform.

12. The method of claim 1 , further comprising:

pitch correcting, at the display device-coupled computing platform, the captured vocal audio received from the portable computing device; and

audibly rendering, at the display device-coupled computing platform, the pitch-corrected vocal audio.

13. The method of claim 12 ,

wherein the audible rendering includes vocals of an additional vocalist other than the user, and

further comprising visually displaying on the display coupled to the display device-coupled computing platform performance synchronized video of the additional vocalist.

14. The method of claim 1 , further comprising:

displaying, on the touchscreen of the portable computing device, an audio reverberation user interface control; and

responsive to user gestures on the touchscreen and to a resulting state of the audio reverberation user interface control, changing signal processing applied to the vocal audio at either or both of the portable computing device and the display device-coupled computing platform.

15. The method of claim 1 , further comprising:

displaying, on the touchscreen of the portable computing device, an audio volume user interface control; and

responsive to user gestures on the touchscreen and to a resulting state of the audio volume user interface control, changing signal processing applied to the vocal audio stream at either or both of the portable computing device and the display device-coupled computing platform.

16. The method of claim 1 , further comprising:

displaying, on the touchscreen of the portable computing device, an audio effects user interface control; and

responsive to user gestures on the touchscreen and to a resulting state of the audio effects user interface control, changing signal processing applied to the vocal audio stream at either or both of the portable computing device and the display device-coupled computing platform.

17. The method of claim 1 ,

wherein the audible rendering at the display device-coupled computing platform is in correspondence with a temporally synchronized backing track selected by a user of the portable computing device.

18. The method of claim 1 , further comprising:

in connection with the pairing, associating the media application executing on the display device-coupled computing platform with a preexisting account that identifies the user for one or more additional media applications executable on the portable computing device.

19. The method of claim 1 , further comprising:

initiating creation of an account that identifies the user for the media application executing on the display device-coupled computing platform and for one or more additional media applications executable on the portable computing device.

20. The method of claim 1 , further comprising:

authenticating, using either or both of the portable computing device and the display device-coupled computing platform, an identity of the user.

21. A computer program product encoded in non-transitory media and including instructions executable in connection with the method of claim 1 to capture using the microphone of the portable computing device and to transmit via the wireless network the vocal audio stream.

22. A computer program product encoded in non-transitory media and including instructions executable in connection with the method of claim 1 to couple the captured audio stream received from the portable computing device into the audio input of the media application executing on the display device-coupled computing platform.

23. A system comprising:

a display device-coupled computing platform hosting a karaoke-style media application executing thereon that presents lyrics on the display device in temporal correspondence with an audible rendering of a backing audio track; and

a portable computing device communicatively paired with the display device-coupled computing platform, wherein the portable computing device is configured as an audio input device that captures user vocals via a microphone and audio transducer of the portable computing device and transmits the captured user vocals using paired communications to the display device-coupled computing platform as an audio input to the karaoke-style media application executing on the display device-coupled computing platform which, in turn, audibly renders the captured vocal audio in combination with the backing audio track and performance synchronized video.

24. The system of claim 23 , further comprising:

pitch correction code executable on the display device-coupled computing platform to pitch correct the captured user vocals streamed thereto.

25. The system of claim 23 , further comprising:

pitch correction code executable on the paired portable computing device to pitch correct the captured user vocals streamed therefrom.

26. The system of claim 23 ,

wherein pitch correction code is executable in connection with a vocal score temporally synchronized with the backing track and lyrics to correct or shift the captured user vocals in accordance with one or more pitch tracks of the vocal score.

27. The system of claim 23 , wherein the media application includes a karaoke application executing, at least in part, on the display device-coupled computing platform.

28. The system of claim 23 , wherein the display device-coupled computing platform is coupled to a television.

29. The system of claim 23 , wherein the display device-coupled computing platform comprises a set-top box or media streamer.

30. The system of claim 23 , wherein the display device-coupled computing platform is coupled to one or more speakers; and

wherein the display device-coupled computing platform is configured to audibly render a backing track via the one or more speakers and to, in correspondence therewith, visually present lyrics on the display.

31. The system of claim 23 , wherein the audible rendering includes vocals of an additional vocalist, and wherein the visual presentation includes on the display performance synchronized video of the additional vocalist.

32. The system of claim 23 , wherein the portable computing device is selected from the group of: a mobile phone or media player; a personal digital assistant or book reader; and a compute pad.

33. A system, comprising:

a display device;

a computing platform coupled to the display device and configured to present at least a portion of a karaoke application user interface on the display, the computing platform networked to communicate audiovisual content to and/or from a content server; and

a mobile device having a touchscreen display configured to present at least a portion of the karaoke application user interface, the mobile device communicatively paired with the computing platform, and the mobile device configured for vocal capture for paired wireless communication of captured vocals to.

Assignments (3)
SECURITY INTEREST Recorded Feb 26, 2020
From: SMULE, INC.
To: WESTERN ALLIANCE BANK
Reel/Frame 052022/0440 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 24, 2017
From: HERSH, BENJAMIN
To: SMULE, INC.
Reel/Frame 042487/0454 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 28, 2017
From: SHIMMIN, JOHN; YANG, JEANNIE; COOK, PERRY R.
To: SMULE, INC.
Reel/Frame 042186/0023 →
Continuity (4)
Continuation In Part 15173041 · Jun 3, 2016
Continuation In Part 14928727 · Oct 30, 2015
Provisional Application 62247743 · Oct 28, 2015
Related Publication 20170124999A1 · May 4, 2017