Write-a-movie: visualize your story from script
A script-to-movie generation method for a computing device includes obtaining a movie script, generating a video according to the movie script, optimizing the generated video until a pass condition is satisfied, and outputting the optimized video.
1. A script-to-movie generation method for a computing device, comprising:
obtaining a movie script;
generating a video according to the movie script;
optimizing the generated video until a pass condition is satisfied; and
outputting the generated video,
wherein generating the video according to the movie script includes:
generating a list of first actions according to the movie script;
generating stage performance based on each action in the list of first actions; and
using one or more cameras to capture the video of the stage performance.
2. The method according to claim 1 , wherein optimizing the generated video until the pass condition is satisfied includes:
evaluating a total aesthetic distortion D on the video captured by the one or more cameras from the stage performance;
generating a list of second actions based on the video captured by the one or more cameras from the stage performance;
determining a fidelity error E between the list of first actions and the list of second actions; and
iteratively optimizing camera setting and character performance to minimize the total aesthetic distortion D such that the pass condition is satisfied, wherein the pass condition includes that the fidelity error E is smaller than or equal to a pre-configured fidelity error threshold Th E is satisfied or a count of minimization iterations reaches a pre-configured count threshold.
3. The method according to claim 2 , wherein:
each action in the list of first actions and the list of second actions includes attributes such as a subject, an action, an object, an action duration, a subject start position, a subject end position, a subject emotion, and an action style.
4. The method according to claim 2 , wherein:
the list of first actions is represented by a chronological action list {a i |i=1, 2, . . . , N}; and
the list of second actions is represented by a chronological action list {a′ i |i=1, 2, . . . , N};
wherein a i or a′ i is ith action object that contains information for one or more virtual characters in a scene of the stage performance and N is the total number of action objects performed by various characters in various scenes of the stage performance.
5. The method according to claim 4 , wherein:
the stage performance is represented by {p t |t=1, 2, . . . , T}, wherein p t is the stage performance by a character at time t and T is a total performance time; and
the stage performance corresponding to a i is represented by
{
p
t
a
i
,
…
,
p
t
+
1
a
i
,
…
,
p
t
+
l
a
i
a
i
}
,
wherein l a i is the action duration of a i , and t a i and l a i are fixed values derived from the list of first actions.
6. The method according to claim 5 , wherein:
the camera setting optimized for {p t |t=1, 2, . . . , T} is represented by {c t |t=1, 2, . . . , T}; and
a video corresponding to {p t |t=1, 2, . . . , T} is represented by {f t |t=1, 2, . . . , T}.
7. The method according to claim 6 , wherein evaluating the total aesthetic distortion D on the video captured by the one or more cameras from the stage performance includes:
evaluating character visibility V(c t ) for a character k in the camera setting {c t |t=1, 2, . . . , T} by calculating Σ k=0 K =I(c t ,k)·r k , wherein r k is a ratio of a size of the character k in a current video frame over a total size of the current video frame, l(c t , k) refers to a relationship between the character k and the camera c t , and a low value of l(c t , k) indicates that the character k is the more significant character of the camera c t ;
evaluating a character action A(c t ) to be 0 when the character k bounded to the camera c t performs an action at time t or be 1 otherwise;
evaluating a camera configuration C(c t , ã t ) for the camera c t by calculating ϕ C (p c i ,d c i ,ã t ), wherein p c i is a camera position, d c i is a shooting direction, ã t is an action object at time t, and ϕ C ( ) is a distortion cost function for the camera configuration;
evaluating screen continuity S(c t , c t-1 ) including a summary of each single character position change in the current video frame by calculating Σ k=0 K v(k,c t )·ϕ S (p(k,c t )−p(k,c t-1 )), wherein p(k,c t ) and p(k,c t-1 ) refer to positions of the character k in the current video frame captured by the camera c t and the camera c t-1 , v(k, c t )=1 when the character k appears in a view of the camera c t , v(k,c t )=0 otherwise, and ϕ S ( ) is a penalty of the character position change as a non-linear function of a distance between the character position p(k,c t ) and the character position p(k,c t-1 );
evaluating moving continuity M(c t , c t-1 ) including a character moving direction change caused by the character action before or after the view of the camera c t changes by calculating Σ k=0 K v(k,c t )·ϕ M (m(k,c t )−m(k,c t-1 )), wherein m(k,c t ) and m(k,c t-1 ) refer to the character moving direction vectors in the current video frame at time t and time t-1 captured by the camera c t and the camera c t-1 , and ϕ M ( ) is a penalty of the character moving direction change as a non-linear function of a difference between the character moving direction vector m(k,c t ) and the character moving direction vector m(k,c t-1 ); and
evaluating a shot duration distortion U(ū, c t , c t-1 , . . . , c t-q ) by calculating ϕ U (ū, c t , c t-1 , . . . , c t-q ) , wherein ū is an average shot duration set for each scene, q is the longest allowable shot duration, and ϕ U ( ) is a penalty of the shot duration for the video frames in a range of [t−q, . . . , t] that change cameras;
wherein the total aesthetic distortion D=Σ t=0 T [ω 0 ·V (c t )+ω 1 ·C(c t ,ã t )+ω 2 ·A(c t )+ω 3 ·S (c t , c t-1 )+ω 4 ·M(c t ,c t-1 )]+Σ t=q T (1−ω 0 −ω 1 −ω 2 −ω 3 −ω 4 )U(ū, c t , c t-1 , . . . , c t-q ), and ω 0 , ω 1 , ω 2 , ω 3 , and ω 4 are weights for each distortion component in a range between 0 and 1.
8. The method according to claim 7 , wherein determining the fidelity error E between the list of first actions and the list of second actions includes:
determining an action difference d i between textual description of the list of first actions and the list second actions by calculating
1
-
G
(
a
i
)
·
G
(
a
i
′
)
G
(
a
i
)
·
G
(
a
i
′
)
,
wherein G( ) is a GloVe word embedding model;
defining a function W(t)=d t when time t is equal to starting time of a i and W(t)=0 otherwise; and
calculating Σ t=0 T W(t), wherein T is a total performance time.
9. The method according to claim 8 , wherein optimizing the camera setting to minimize the total aesthetic distortion D includes:
optimizing camera placement for different shot sizes, different profile angles, and different camera heights by placing a plurality of cameras surrounding and bounding with each character, each camera maintaining a relative position from the bounded character.
10. The method according to claim 9 , wherein iteratively optimizing camera setting and character performance to minimize the total aesthetic distortion D such that the pass condition is satisfied includes:
min(Σ t=0 T [ω 0 ·V (c t )+ω 1 ·C(c t ,ã t )+ω 2 ·A(c t )+ω 3 ·S(c t , c t-1 )+ω 4 ·M(c t ,c t-1 )]+Σ t=q T (1−ω 0 −ω 1 −ω 2 −ω 3 −ω 4 )U(ū, c t , c t-1 , . . . , c t-q )) such that Σ t=0 T W(t)≤Th E .
11. The method according to claim 10 , wherein:
defining J λ (c t , a t )=D+λ·E, wherein λ is a Lagrange multiplier; and
min(Σ t=0 T [ω 0 ·V (c t )+ω 1 ·C(c t ,ã t )+ω 2 ·A(c t )+ω 3 ·S(c t , c t-1 )+ω 4 ·M(c t ,c t-1 )]+Σ t=q T (1−ω 0 −ω 1 −ω 2 −ω 3 −ω 4 )U(ū, c t , c t-1 , . . . , c t-q )) such that Σ t=0 T W(t)≤Th E is reduced to min J λ (c t , a T ).
12. The method according to claim 11 , wherein:
defining z k =(c k ,a k );
defining a cost function G T (Z T-q , . . . , Z T ) to represent a minimum of the fidelity error E and the total aesthetic distortion D for all video frames; and
min
z
J
λ
(
z
)
=
min
z
T
-
q
,
…
,
z
T
G
T
(
z
T
-
q
,
…
,
z
T
)
.
13. The method according to claim 12 , wherein:
G
k
(
z
k
-
q
,
…
,
z
k
)
=
min
z
k
-
q
-
1
,
…
,
z
k
-
1
G
k
-
1
(
z
k
-
q
-
1
,
…
,
z
k
-
1
)
+
∑
t
=
0
T
W
(
t
)
+
λ
·
{
∑
t
=
0
T
[
ω
0
·
V
(
c
t
)
+
ω
1
·
C
(
c
t
,
a
~
t
)
+
ω
2
·
A
(
c
t
)
+
ω
3
·
S
(
c
t
,
c
t
-
1
)
+
ω
4
·
M
(
c
t
,
c
t
-
1
)
]
+
∑
t
=
q
T
(
1
-
ω
0
-
ω
1
-
ω
2
-
ω
3
-
ω
4
)
U
(
u
_
,
c
t
,
c
t
-
1
,
…
,
c
t
-
q
)
}
.
wherein:
each future step of an optimization process is independent of a past step thereof;
the optimization process is converted into a graph theory problem of finding a shortest path in a directed acyclic graph; and
a computation complexity of the optimization process is O(T×|Z| q+1 ), and I smore efficient than an exhaustive search algorithm with exponential computation complexity.
14. A script-to-movie generation device comprising:
a memory storing program instructions; and
a processor coupled with the memory and configured to execute the program instructions to:
obtaining a movie script;
generating a video according to the movie script, comprising:
generate a list of first actions according to the movie script;
generate stage performance based on each action in the list of first actions; and
use one or more cameras to capture the video of the stage performance;
optimizing the generated video until a pass condition is satisfied; and
outputting the generated video.
15. The device according to claim 14 , wherein the processor is further configured to:
evaluate a total aesthetic distortion D on the video captured by the one or more cameras from the stage performance;
generate a list of second actions based on the video captured by the one or more cameras from the stage performance;
determine a fidelity error E between the list of first actions and the list of second actions; and
iteratively optimize camera setting and character performance to minimize the total aesthetic distortion D such that the pass condition is satisfied, wherein the pass condition includes that the fidelity error E is smaller than or equal to a pre-configured fidelity error threshold Th E is satisfied or a count of minimization iterations reaches a pre-configured count threshold.
16. The device according to claim 15 , wherein:
each action in the list of first actions and the list of second actions includes attributes such as a subject, an action, an object, an action duration, a subject start position, a subject end position, a subject emotion, and an action style.
17. The device according to claim 15 , wherein:
the list of first actions is represented by a chronological action list {a i |i=1, 2, . . . , N}; and
the list of second actions is represented by a chronological action list {a′ i |i=1, 2, . . . , N};
wherein a i or a′ i is ith action object that contains information for one or more virtual characters in a scene of the stage performance and N is the total number of action objects performed by various characters in various scenes of the stage performance.
18. The device according to claim 17 , wherein:
the stage performance is represented by {p t |t=1, 2, . . . , T}, wherein p t is the stage performance by a character at time t and T is a total performance time; and
the stage performance corresponding to a i is represented by
{
p
t
a
i
,
…
,
p
t
+
1
a
i
,
…
,
p
t
+
l
a
i
a
i
}
,
wherein l a i is the action duration of a i , and t a i and are fixed values derived from the list of first actions.