Lip-password based speaker verification system
A lip-based speaker verification system for identifying a speaker using a modality of lip motions; wherein an identification key of the speaker comprising one or more passwords; wherein the one or more passwords are embedded into lip motions of the speaker; wherein the speaker is verified by underlying dynamic characteristics of the lip motions; and wherein the speaker is required to match the one or more passwords embedded in the lip motions with registered information in a database. That is, in the case where the target speaker saying the wrong password or even in the case where an impostor knowing and saying the correct password, the nonconformities will be detected and the authentications/accesses will be denied.
1. A lip-based speaker verification system for identifying a speaker, comprising one or more computer processors for executing a process of verification of identity of the speaker using one modality of lip motions;
wherein an identification key of the speaker comprising one or more passwords;
wherein the one or more passwords are embedded into lip motions of the speaker;
wherein the speaker is verified by underlying dynamic characteristics of the lip motions and extracted area-based features wherein the extracted area-based features further comprise teeth, tongue and oral cavity during the utterance; and wherein the speaker is required to match the one or more passwords embedded in the lip motions with registered information in a database such that the matching between the dynamic characteristics of the speaker lip motions and the extracted area-based features with the one or more passwords is verified by using one or more multi-boosted hidden Markov models (HMMs);
wherein the process comprises the steps of:
(1) extracting visual features for each lip frame;
(2) performing lip motion segmentation of D to yield D={D 1 , D 2 , . . . , D p } where D denotes the one or more passwords, and p is the number of password components;
(3) for each value of m=1, . . . , p, performing the steps of:
(3.1) getting a training set D m T ={X 1 T , X 2 T , . . . , X N a T } of the speaker and D m I ={X 1 1 , X 2 1 , . . . , X N b I } of an imposer, and forming a novel training set using a data sharing scheme (DSS);
(3.2) initializing w i,j T , w i,j I , r and ε 0 respectively with:
w
i
,
j
T
=
2
N
a
(
N
a
-
1
)
for
1
≤
i
≤
j
≤
N
a
;
w
i
,
j
I
=
1
N
a
N
b
for 1≦i≦N a and 1≦j≦N b ; r=0; and ε 0 =0; where w i,j T (1≦i<j≦K) denotes the weight of a coupled training sample {O i , O j } of the speaker, and O k (1≦k≦K) is a kth observation sequence, and K is the number of observations;
(3.3) while r≦R and ε r <0.5 where ε r is a weighted classification error and R is a pre-determined number for each HMM to boost around, performing the steps of:
(3.3.1) updating w r,i,j T and w r,i,j I with
w
r
,
i
,
j
T
=
w
r
,
i
,
j
T
∑
i
′
,
j
′
w
i
′
,
j
′
T
+
∑
i
′
,
j
′
w
i
′
,
j
′
I
and
w
r
,
i
,
j
I
=
w
r
,
i
,
j
I
∑
i
′
,
j
′
w
i
′
,
j
′
T
+
∑
i
′
,
j
′
w
i
′
,
j
′
I
,
respectively, to thereby normalize the weights w r,i,j T and w r,i,j I ;
(3.3.2) sampling feature vectors of P rsm % dimensionality in positive data set D m T by a random sampling method (RSM), where P rsm is an RSM sampling percentage;
(3.3.4) building an HMM λ m r (T) via
a
^
i
,
j
=
∑
k
=
1
K
ω
k
P
k
∑
t
=
1
l
k
-
1
α
t
k
(
i
)
a
i
,
j
b
j
(
O
t
+
1
k
)
β
t
+
1
k
(
j
)
∑
k
=
1
K
ω
k
P
k
∑
t
=
1
l
k
-
1
α
t
k
(
i
)
β
t
k
(
j
)
and
b
^
j
(
1
)
=
∑
k
=
1
K
ω
k
P
k
∑
t
=
1
l
k
-
1
s
.
t
.
O
t
=
v
1
α
t
k
(
i
)
β
t
k
(
j
)
∑
k
=
1
K
ω
k
P
k
∑
t
=
1
l
k
-
1
α
t
k
(
i
)
β
t
k
(
j
)
where: l k is the length of O k ; α t k (i) is a forward variable for O k ; β t k (j) is a backward variable for O k ; ω k is a normalized weight for O k ; α i,j is the (i, j)th element of a state transition matrix of the HMM; b j (l) is the (j,l)th element of a symbol emission matrix of the HMM; and P k =P(O k |λ m r (T));
(3.3.4) calling WeakLearner learning with respect to
h
(
O
s
)
=
{
+
1
,
if
LLR
(
O
s
)
or
NLL
(
O
s
)
≥
τ
-
1
,
otherwise
where LLR denotes a log likelihood ratio and NLL denotes a normalized log likelihood;
(3.3.5) training a threshold τ m to minimize ε r given by
ε r =Σ i,j w i,j T e r,i,j T +Σ i,j w i,j I e r,i,j I ,
where: e r,i,j T =1 if h m r (X i T , X j T , λ m r (T))≧τ m and e r,i,j T =0 otherwise; and e r,i,j I =1 if h m r (X i T , X j I , λ m r (T))<τ m and e r,i,j I =0 otherwise;
(3.3.6) setting
α
m
r
=
1
2
log
[
1
-
ɛ
r
ɛ
r
]
;
(3.3.7) updating the weights by
w r+1,i,j T =w r,i,j T ·exp(2α m r e r,i,j T )
and
w r+1,i,j I =w r,i,j I ·exp(2α m r e r,i,j I ); and
(3.3.8) setting r=r+1;
(3.4) obtaining a similarity score between X p T and X q , where X p T is from the data set of the speaker:
h
^
m
(
X
p
T
,
X
q
)
=
∑
w
=
1
r
α
m
w
h
m
w
(
X
p
T
,
X
q
,
λ
m
w
(
T
)
)
.
2. The system of claim 1 , wherein the one or more passwords are synchronously embedded into the lip motions, such that the verification of the speaker's identity by the underlying dynamic characteristics of the lip motions and the matching of the one or more passwords embedded in the lip motion is performed simultaneously.
3. The system according to claim 2 ;
wherein the modality of lip motions is completely insensitive to background noise;
wherein acquisition of lip motions of the speaker is insusceptible to distance between the speaker's lips and an acquisition means of the system;
wherein the system is usable by a mute person; and
wherein the system can operate in silence and in an obscure manner.
4. The system according to claim 2 , wherein the system is included in one or more security systems.
5. The system according to claim 2 , wherein the system is implemented across one or more computing hardware platform in one or more locations.
6. The system according to claim 2 , wherein the process of verification of identity of the speaker is implemented in software that is executable on one or more hardware platform.
7. The system according to claim 1 ;
wherein the modality of lip motions is completely insensitive to background noise;
wherein acquisition of lip motions of the speaker is insusceptible to distance between the speaker's lips and an acquisition means of the system;
wherein the system is usable by a mute person; and
wherein the system can operate in silence and in an obscure manner.
8. The system according to claim 1 , wherein the system is included in one or more security systems.
9. The system according to claim 1 , wherein the system is implemented across one or more computing hardware platform in one or more locations.
10. The system according to claim 1 , wherein the process of verification of identity of the speaker is implemented in software that is executable on one or more hardware platform.
11. The system according to claim 1 , wherein the process further comprises the steps of:
(4) given a test lip-password sequence V={v 1 , v 2 , . . . , v p }, verifying each subunit via
h
^
m
i
n
m
=
min
X
i
T
∈
D
m
T
h
^
m
(
X
i
T
,
v
m
)
after the step (3) is performed; and
(5) outputting: a result that the sequence V is uttered by the speaker if ĥ min m ≦τ for m=1, . . . , p; or otherwise another result that the sequence V is not uttered by the speaker.