IP Library › Granted Patent US 9,818,216
Granted Patent B2
US 9,818,216 · App. 14/680,374 · Granted Nov 14, 2017

Audio-based caricature exaggeration

Inventors: Matan Sela (Haifa, IL); Yonathan Aflalo (Tel Aviv, IL); Ron Kimmel (Haifa, IL)
Assignee: Technion Research and Development Foundation Limited
G06T13/205G06T13/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,818,216
App. No.
14/680,374
Granted
Nov 14, 2017
Kind
B2
Abstract

Computerized, audio-based caricature exaggeration which includes: receiving a three-dimensional model of an object; receiving an audio sequence; generating a video frame sequence, said generating comprising computing a caricature of the object, wherein (a) the computing is with a different exaggeration factor for each of multiple ones of the video frames, and (b) the different exaggeration factor is based on one or more parameters of the audio sequence; and synthesizing the audio sequence and the video frame sequence into an audiovisual clip.

Claims (48)

1. A method comprising using at least one hardware processor for:

receiving a three-dimensional model of an object, wherein the three-dimensional model is embodied as a digital file that comprises a representation of the object;

receiving an audio sequence embodied as a digital file that comprises a musical composition;

generating a video frame sequence, wherein the generating comprises computing a caricature of the object by applying a computerized caricaturization algorithm to the three-dimensional model, wherein the computing comprises:

scaling gradient fields of surface coordinates of the three-dimensional model by a function of a Gaussian curvature of the surface,

finding a regular surface whose gradient fields fit the scaled gradient fields, and

amplifying the scaling according to local discrepancies between the object and a reference object, wherein the reference object is not a scaled down version of the object,

wherein (a) the computing is with a different exaggeration factor for each of multiple ones of the video frames, and (b) the different exaggeration factor is based on one or more parameters of the musical composition of the audio sequence; and

synthesizing the audio sequence and the video frame sequence into an audiovisual clip.

2. The method according to claim 1 , further comprising using the at least one hardware processor for determining the one or more parameters for each of multiple periods of the audio sequence.

3. The method according to claim 2 , wherein the one or more parameters are selected from the group consisting of: amplitude, frequency and tempo.

4. The method according to claim 1 , wherein the generating further comprises altering a view angle of the caricature along the video frame sequence.

5. The method according to claim 1 , wherein the exaggeration factor is applied uniformly, to the entirety of the three-dimensional model.

6. The method according to claim 1 , wherein the exaggeration factor is applied non-uniformly, only to one or more portions of the three-dimensional model, which portions amount to less than the entirety of the three-dimensional model.

7. The method according to claim 1 , wherein the computing of the caricature of the object comprises:

constructing a look-up table comprised of (a) different visualizations of the caricature, each computed with one of the different exaggeration factors, and (b) the exaggeration factor for each of the different visualizations; and

using each caricature visualization from the look-up table when the exaggeration factor of that caricature visualization is determined to be suitable for the one or more parameters of the audio sequence.

8. The method according to claim 1 , wherein the computing of the caricature of the object further comprises amplifying the scaling according to local discrepancies between the object and a scaled down version of the object.

9. A computer program product comprising a non-transitory computer-readable storage medium having program code embodied thereon, the program code executable by at least one hardware processor for:

receiving a three-dimensional model of an object, wherein the three-dimensional model is embodied as a digital file that comprises a representation of the object;

receiving an audio sequence embodied as a digital file that comprises a musical composition;

generating a video frame sequence, wherein the generating comprises computing a caricature of the object by applying a computerized caricaturization algorithm to the three-dimensional model, wherein the computing comprises:

scaling gradient fields of surface coordinates of the three-dimensional model by a function of a Gaussian curvature of the surface,

finding a regular surface whose gradient fields fit the scaled gradient fields, and

amplifying the scaling according to local discrepancies between the object and a reference object, wherein the reference object is not a scaled down version of the object,

wherein (a) the computing is with a different exaggeration factor for each of multiple ones of the video frames, and (b) the different exaggeration factor is based on one or more parameters of the musical composition of the audio sequence; and

synthesizing the audio sequence and the video frame sequence into an audiovisual clip.

10. The computer program product according to claim 9 , wherein:

the program code is further executable by said at least one hardware processor for determining the one or more parameters for each of multiple periods of the audio sequence; and

the one or more parameters are selected from the group consisting of: amplitude, frequency and tempo.

11. The computer program product according to claim 9 , wherein the generating further comprises altering a view angle of the caricature along the video frame sequence.

12. The computer program product according to claim 9 , wherein the computing of the caricature of the object further comprises amplifying the scaling according to local discrepancies between the object and a scaled down version of the object.

13. A system comprising:

(a) a non-transitory computer-readable storage medium having program code embodied thereon, the program code comprising instructions for:

receiving a three-dimensional model of an object, wherein the three-dimensional model is embodied as a digital file that comprises a representation of the object,

receiving an audio sequence embodied as a digital file that comprises a musical composition,

generating a video frame sequence, wherein the generating comprises computing a caricature of the object by applying a computerized caricaturization algorithm to the three-dimensional model, wherein the computing comprises:

scaling gradient fields of surface coordinates of the three-dimensional model by a function of a Gaussian curvature of the surface,

finding a regular surface whose gradient fields fit the scaled gradient fields, and

amplifying the scaling according to local discrepancies between the object and a reference object, wherein the reference object is not a scaled down version of the object,

wherein (i) the computing is with a different exaggeration factor for each of multiple ones of the video frames, and (ii) the different exaggeration factor is based on one or more parameters of the musical composition of the audio sequence, and

synthesizing the audio sequence and the video frame sequence into an audiovisual clip; and

(b) at least one hardware processor configured to execute the instructions.

14. The system according to claim 13 , wherein:

the program code is further executable by said at least one hardware processor for determining the one or more parameters for each of multiple periods of the audio sequence; and

the one or more parameters are selected from the group consisting of: amplitude, frequency and tempo.

15. The system according to claim 13 , wherein the generating further comprises altering a view angle of the caricature along the video frame sequence.

16. The system according to claim 13 , wherein the computing of the caricature of the object further comprises amplifying the scaling according to local discrepancies between the object and a scaled down version of the object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 10, 2017
From: SELA, MATAN; AFLALO, YONATHAN; KIMMEL, RON
To: TECHNION RESEARCH AND DEVELOPMENT FOUNDATION LIMITED
Reel/Frame 043825/0312 →
Continuity (2)
Provisional Application 61976510 · Apr 8, 2014
Related Publication 20150287229A1 · Oct 8, 2015