Method and system for optical flow based motion vector estimation for picture rate up-conversion
Certain aspects of a method and system for optical flow based motion vector estimation for picture rate up-conversion (PRUC) may include generating one or more motion vectors based on extracted picture rate up-conversion (PRUC) data by minimizing a cost function. The cost function may be constrained by any combination of a block matching constraint, a smoothness constraint and a bias constraint. The PRUC data may be extracted from a compressed video data stream while the compressed video data stream is being decompressed by a video decompression engine. The PRUC data may comprise local block motion vectors, block coding modes, quantization levels, quantized residual data and decoded pictures. A plurality of interpolated pictures may be generated based on extracting the PRUC data.
1. A method for processing video data, the method comprising:
generating one or more motion vectors based on extracted picture rate up-conversion (PRUC) data by minimizing a cost function, wherein said cost function is constrained by one or more of:
a block matching constraint, a smoothness constraint and/or a bias constraint, wherein said block matching constraint is represented as
E
(
x
,
y
,
t
;
v
x
,
v
y
,
Δ
t
)
=
(
∂
P
∂
x
v
x
+
∂
P
∂
y
v
y
+
∂
P
∂
t
Δ
t
)
2
,
where P is a brightness distribution in an image space (x,y) and time t, and v x and v y are motion vectors in said image space (x,y), wherein ∂P/∂x, ∂P/∂y comprise partial derivatives of P in the image space (x,y) and ∂P/∂t comprises a partial derivative of Pin time t.
2. The method according to claim 1 , wherein said smoothness constraint is represented as
S
(
v
⇀
)
=
(
∂
v
x
∂
x
)
2
+
(
∂
v
x
∂
y
)
2
+
(
∂
v
y
∂
x
)
2
+
(
∂
v
y
∂
y
)
2
+
(
∂
v
x
∂
t
)
2
+
(
∂
v
y
∂
t
)
2
,
where {right arrow over (v)} is said generated one or more motion vectors.
3. The method according to claim 2 , wherein said bias constraint is represented as where B( ν , ν ref )=(v x −v x ref ) 2 +(v y −v y ref ) 2 , where {right arrow over (v)} ref , v x ref , and v y ref are reference motion vectors.
4. The method according to claim 3 , wherein said cost function is represented as
J
(
v
_
,
w
S
,
w
B
)
=
∫
Ω
(
E
+
w
S
·
S
+
w
B
·
B
)
ⅆ
Ω
,
where ν ″= O , w S is an adaptive smoothness parameter, S is said smoothness constraint, w B is an adaptive bias parameter and B is said bias constraint and Ω⊂R 2 .
5. The method according to claim 4 , comprising dynamically updating said adaptive smoothness parameter w S and said adaptive bias parameter w B .
6. The method according to claim 3 , wherein said cost function is represented as
J
(
v
_
,
w
S
,
w
B
)
=
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
(
P
(
x
i
,
y
j
,
k
)
-
P
(
x
i
+
v
x
,
y
j
+
v
y
,
k
-
1
)
)
2
+
w
S
·
(
(
v
x
-
v
x
,
(
p
-
1
,
q
)
k
)
2
+
(
v
y
-
v
y
,
(
p
-
1
,
q
)
k
)
2
+
(
v
x
-
v
x
,
(
p
,
q
-
1
)
k
)
2
+
(
v
y
-
v
y
,
(
p
,
q
-
1
)
k
)
2
+
(
v
x
-
v
x
,
(
p
,
q
)
k
-
1
)
2
+
(
v
y
-
v
y
,
(
p
,
q
)
k
-
1
)
2
)
+
w
B
·
(
(
v
x
-
v
x
ref
)
2
+
(
v
y
-
v
y
ref
)
2
)
,
where w S is an adaptive smoothness parameter, w B is an adaptive bias parameter, k is an index of a picture number and v b,(c,d) a is a previously estimated motion vector in a picture a for b direction and at a grid location (c,d).
7. The method according to claim 6 , wherein a sufficient condition for a unique solution for said cost function is ν =H −1 · Q , where
H
=
(
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
(
∂
P
∂
x
)
2
+
(
w
S
4
+
w
B
)
,
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
x
·
∂
P
∂
y
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
y
·
∂
P
∂
x
,
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
(
∂
P
∂
y
)
2
+
(
w
S
4
+
w
B
)
)
,
and
Q
_
=
(
(
w
S
·
(
v
_
x
(
x
)
+
v
_
x
(
y
)
)
8
+
2
w
B
v
x
ref
)
-
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
x
·
∂
P
∂
t
(
w
S
·
(
v
_
y
(
x
)
+
v
_
y
(
y
)
)
8
+
2
w
B
v
y
ref
)
-
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
y
·
∂
P
∂
t
)
.
8. The method according to claim 7 , comprising solving said cost function using
v
_
^
(
ρ
)
=
v
_
^
(
ρ
-
1
)
-
μ
2
·
∂
J
(
v
_
,
w
S
,
w
B
;
k
)
∂
v
_
(
ρ
)
❘
v
_
^
(
ρ
-
1
)
=
v
_
^
(
ρ
-
1
)
+
μ
2
·
(
Q
⇀
-
H
·
v
_
^
(
ρ
-
1
)
,
where
v
_
^
(
ρ
)
=
(
v
^
x
(
ρ
)
v
^
y
(
ρ
)
)
,
ρ is an iteration step and parameter μ is a step size for iteration.
9. The method according to claim 7 , comprising solving said cost function using
v
_
^
(
k
)
=
v
_
^
(
k
-
1
)
+
μ
2
·
(
Q
_
-
H
·
v
_
^
(
k
-
1
)
)
,
where parameter μ is a step size for iteration.
10. A system for processing video data, the system comprising:
one or more circuits that enables generation of one of more motion vectors based on extracted picture rate up-conversion (PRUC) data by minimizing a cost function, wherein said cost function is constrained by one or more of: a block matching constraint, a smoothness constraint and/or bias constraint, wherein said block matching constraint is represented as
E
(
x
,
y
,
t
;
v
x
,
v
y
,
Δ
t
)
=
(
∂
P
∂
x
v
x
+
∂
P
∂
y
v
y
+
∂
P
∂
t
Δ
t
)
2
,
where P is a brightness distribution in an image space (x,y) and time t, and v x and v y are motion vectors in said image space (x,y), wherein ∂P/∂x, ∂P/∂y comprise partial derivatives of P in the image space (x,y), and ∂P/∂t comprises a partial derivative of P in time t.
11. The system according to claim 10 , wherein said smoothness constraint is represented as
S
(
v
_
)
=
(
∂
v
x
∂
x
)
2
+
(
∂
v
x
∂
y
)
2
+
(
∂
v
y
∂
x
)
2
+
(
∂
v
y
∂
y
)
2
+
(
∂
v
x
∂
t
)
2
+
(
∂
v
y
∂
t
)
2
,
where {right arrow over (v)} is said generated one or more motion vectors.
12. The system according to claim 11 , wherein said bias constraint is represented as B( ν , ν ref )=(v x −v x ref ) 2 +(v y −v y ref ) 2 , where {right arrow over (v)} ref , v x ref and v y ref are reference motion vectors.
13. The system according to claim 12 , wherein said cost function is represented as
J
(
v
_
,
w
S
,
w
B
)
=
∫
Ω
(
E
+
w
S
·
S
+
w
B
·
B
)
ⅆ
Ω
,
where ν ″= O , w S is an adaptive smoothness parameter, S is said smoothness constraint, w B is an adaptive bias parameter and B is said bias constraint and Ω⊂ 2 .
14. The system according to claim 13 , wherein said one or more circuits enables dynamic updating of said adaptive smoothness parameter w S and said adaptive bias parameter w B .
15. The system according to claim 12 , wherein said cost function is represented as
J
(
v
_
,
w
S
,
w
B
)
=
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
(
P
(
x
i
,
y
j
,
k
)
-
P
(
x
i
+
v
x
,
y
j
+
v
y
,
k
-
1
)
)
2
+
w
S
·
(
(
v
x
-
v
x
,
(
p
-
1
,
q
)
k
)
2
+
(
v
y
-
v
y
,
(
p
-
1
,
q
)
k
)
2
+
(
v
x
-
v
x
,
(
p
,
q
-
1
)
k
)
2
+
(
v
y
-
v
y
,
(
p
,
q
-
1
)
k
)
2
+
(
v
x
-
v
x
,
(
p
,
q
)
k
-
1
)
2
+
(
v
y
-
v
y
,
(
p
,
q
)
k
-
1
)
2
)
+
w
B
·
(
(
v
x
-
v
x
ref
)
2
+
(
v
y
-
v
y
ref
)
2
)
,
where w S is an adaptive smoothness parameter, w B is an adaptive bias parameter, k is an index of a picture number and v b,(c,d) a is a previously estimated motion vector in a picture a for b direction and at a grid location (c,d).
16. The system according to claim 15 , wherein a sufficient condition for a unique solution for said cost function is ν =H −1 · Q , where
H
=
(
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
(
∂
P
∂
x
)
2
+
(
w
S
4
+
w
B
)
,
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
x
·
∂
P
∂
y
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
y
·
∂
P
∂
x
,
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
(
∂
P
∂
y
)
2
+
(
w
S
4
+
w
B
)
)
,
and
Q
_
=
(
(
w
S
·
(
v
_
x
(
x
)
+
v
_
x
(
y
)
)
8
+
2
w
B
v
x
ref
)
-
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
x
·
∂
P
∂
t
(
w
S
·
(
v
_
y
(
x
)
+
v
_
y
(
y
)
)
8
+
2
w
B
v
y
ref
)
-
2
∑
i
=
pm
pm
+
m
-
1
∑
j
=
qn
qn
+
n
-
1
∂
P
∂
y
·
∂
P
∂
t
)
.
17. The system according to claim 16 , wherein said one or more circuits enables solving of said cost function using
v
_
^
(
ρ
)
=
v
_
^
(
ρ
-
1
)
-
μ
2
·
∂
J
(
v
_
,
w
S
,
w
B
;
k
)
∂
v
_
(
ρ
)
|
v
_
^
(
ρ
-
1
)
=
v
_
^
(
ρ
-
1
)
+
μ
2
·
(
Q
_
-
H
·
v
_
^
(
ρ
-
1
)
)
,
where
v
_
^
(
ρ
)
=
(
v
^
x
(
ρ
)
v
^
y
(
ρ
)
)
,
ρ is an iteration step and parameter μ is a step size for iteration.
18. The system according to claim 16 , wherein said one or more circuits enables solving of said cost function using
v
_
^
(
k
)
=
v
_
^
(
k
-
1
)
+
μ
2
·
(
Q
_
-
H
·
v
_
^
(
k
-
1
)
)
,
where parameter μ is a step size for iteration.