IP Library › Granted Patent US 10,147,217
Granted Patent B2
US 10,147,217 · App. 15/729,217 · Granted Dec 4, 2018

Audio-based caricature exaggeration

Inventors: Matan Sela (Haifa, IL); Yonathan Aflalo (Tel Aviv, IL); Ron Kimmel (Haifa, IL)
Assignee: Technion Research and Development Foundation Limited
G06T13/205G06T13/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,147,217
App. No.
15/729,217
Granted
Dec 4, 2018
Kind
B2
Abstract

A method that uses at least one hardware processor for receiving a three-dimensional model of an object, receiving an audio sequence embodied as a digital file that comprises a musical composition, generating a video frame sequence, and synthesizing the audio sequence and the video frame sequence into an audiovisual clip. The three-dimensional model is embodied as a digital file that comprises a representation of the object. The generating step comprises computing a caricature of the object by applying a computerized caricaturization algorithm to the three-dimensional model. The computing has scaling gradient fields of surface coordinates of the three-dimensional model by a function of a Gaussian curvature of the surface, and finding a regular surface whose gradient fields fit the scaled gradient fields. The computing is with a different exaggeration factor for each of multiple ones of the video frames, and the different exaggeration factor is based on one or more parameters of the musical composition of the audio sequence.

Claims (57)

1. A method comprising using at least one hardware processor for:

receiving a three-dimensional model of an object, wherein the three-dimensional model is embodied as a digital file that comprises a representation of the object;

receiving an audio sequence embodied as a digital file that comprises a musical composition;

generating a video frame sequence, wherein the generating comprises computing a caricature of the object by applying a computerized caricaturization algorithm to the three-dimensional model, wherein the computing comprises:

scaling gradient fields of surface coordinates of the three-dimensional model by a function of a Gaussian curvature of the surface, and

finding a regular surface whose gradient fields fit the scaled gradient fields,

wherein (a) the computing is with a different exaggeration factor for each of multiple ones of the video frames, and (b) the different exaggeration factor is based on one or more parameters of the musical composition of the audio sequence; and

synthesizing the audio sequence and the video frame sequence into an audiovisual clip;

wherein the applying of the computerized caricaturization algorithm is only to the three-dimensional model of the object and not to a reference three-dimensional model which is not the three-dimensional model of the object.

2. The method according to claim 1 , further comprising using said at least one hardware processor for determining the one or more parameters for each of multiple periods of the audio sequence, wherein the one or more parameters are selected from the group consisting of: amplitude, frequency and tempo.

3. The method according to claim 1 , wherein the generating further comprises altering a view angle of the caricature along the video frame sequence.

4. The method according to claim 1 , wherein the exaggeration factor is applied uniformly, to the entirety of the three-dimensional model.

5. The method according to claim 1 , wherein the exaggeration factor is applied non-uniformly, only to one or more portions of the three-dimensional model, which portions amount to less than the entirety of the three-dimensional model.

6. The method according to claim 1 , wherein the computing of the caricature of the object further comprises:

constructing a look-up table comprised of (a) different visualizations of the caricature, each computed with one of the different exaggeration factors, and (b) the exaggeration factor for each of the different visualizations; and

using each caricature visualization from the look-up table when the exaggeration factor of that caricature visualization is determined to be suitable for the one or more parameters of the audio sequence.

7. The method according to claim 1 , wherein the computing of the caricature of the object further comprises amplifying the scaling according to local discrepancies between the object and a scaled down version of the object.

8. A computer program product comprising a non-transitory computer-readable storage medium having program code embodied thereon, the program code executable by at least one hardware processor for:

receiving a three-dimensional model of an object, wherein the three-dimensional model is embodied as a digital file that comprises a representation of the object;

receiving an audio sequence embodied as a digital file that comprises a musical composition;

generating a video frame sequence, wherein the generating comprises computing a caricature of the object by applying a computerized caricaturization algorithm to the three-dimensional model, wherein the computing comprises:

scaling gradient fields of surface coordinates of the three-dimensional model by a function of a Gaussian curvature of the surface, and

finding a regular surface whose gradient fields fit the scaled gradient fields,

wherein (a) the computing is with a different exaggeration factor for each of multiple ones of the video frames, and (b) the different exaggeration factor is based on one or more parameters of the musical composition of the audio sequence; and

synthesizing the audio sequence and the video frame sequence into an audiovisual clip;

wherein the applying of the computerized caricaturization algorithm is only to the three-dimensional model of the object and not to a reference three-dimensional model which is not the three-dimensional model of the object.

9. The computer program product according to claim 8 , wherein:

the program code is further executable by said at least one hardware processor for determining the one or more parameters for each of multiple periods of the audio sequence; and

the one or more parameters are selected from the group consisting of: amplitude, frequency and tempo.

10. The computer program product according to claim 8 , wherein the generating further comprises altering a view angle of the caricature along the video frame sequence.

11. The computer program product according to claim 8 , wherein the exaggeration factor is applied uniformly, to the entirety of the three-dimensional model.

12. The computer program product according to claim 8 , wherein the exaggeration factor is applied non-uniformly, only to one or more portions of the three-dimensional model, which portions amount to less than the entirety of the three-dimensional model.

13. The computer program product according to claim 8 , wherein the computing of the caricature of the object further comprises:

constructing a look-up table comprised of (a) different visualizations of the caricature, each computed with one of the different exaggeration factors, and (b) the exaggeration factor for each of the different visualizations; and

using each caricature visualization from the look-up table when the exaggeration factor of that caricature visualization is determined to be suitable for the one or more parameters of the audio sequence.

14. The computer program product according to claim 8 , wherein the computing of the caricature of the object further comprises amplifying the scaling according to local discrepancies between the object and a scaled down version of the object.

15. A system comprising:

(a) a non-transitory computer-readable storage medium having program code embodied thereon, the program code comprising instructions for:

receiving a three-dimensional model of an object, wherein the three-dimensional model is embodied as a digital file that comprises a representation of the object,

receiving an audio sequence embodied as a digital file that comprises a musical composition,

generating a video frame sequence, wherein the generating comprises computing a caricature of the object by applying a computerized caricaturization algorithm to the three-dimensional model, wherein the computing comprises:

scaling gradient fields of surface coordinates of the three-dimensional model by a function of a Gaussian curvature of the surface, and

finding a regular surface whose gradient fields fit the scaled gradient fields,

wherein (i) the computing is with a different exaggeration factor for each of multiple ones of the video frames, and (ii) the different exaggeration factor is based on one or more parameters of the musical composition of the audio sequence, and

synthesizing the audio sequence and the video frame sequence into an audiovisual clip; and

(b) at least one hardware processor configured to execute the instructions;

wherein the applying of the computerized caricaturization algorithm is only to the three-dimensional model of the object and not to a reference three-dimensional model which is not the three-dimensional model of the object.

16. The system according to claim 15 , wherein:

the program code is further executable by said at least one hardware processor for determining the one or more parameters for each of multiple periods of the audio sequence; and

the one or more parameters are selected from the group consisting of: amplitude, frequency and tempo.

17. The system according to claim 15 , wherein the generating further comprises altering a view angle of the caricature along the video frame sequence.

18. The system according to claim 15 , wherein the exaggeration factor is applied uniformly, to the entirety of the three-dimensional model.

19. The system according to claim 15 , wherein the exaggeration factor is applied non-uniformly, only to one or more portions of the three-dimensional model, which portions amount to less than the entirety of the three-dimensional model.

20. The system according to claim 15 , wherein the computing of the caricature of the object further comprises:

constructing a look-up table comprised of (a) different visualizations of the caricature, each computed with one of the different exaggeration factors, and (b) the exaggeration factor for each of the different visualizations; and

using each caricature visualization from the look-up table when the exaggeration factor of that caricature visualization is determined to be suitable for the one or more parameters of the audio sequence.

21. The system according to claim 15 , wherein the computing of the caricature of the object further comprises amplifying the scaling according to local discrepancies between the object and a scaled down version of the object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 10, 2017
From: SELA, MATAN; AFLALO, YONATHAN; KIMMEL, RON
To: TECHNION RESEARCH AND DEVELOPMENT FOUNDATION LIMITED
Reel/Frame 043827/0214 →
Continuity (3)
Continuation 14680374 · Apr 7, 2015
Provisional Application 61976510 · Apr 8, 2014
Related Publication 20180033181A1 · Feb 1, 2018