Training machine learning by sequential conditional generalized iterative scaling
View Patent ↗A system and method facilitating training machine learning systems utilizing sequential conditional generalized iterative scaling is provided. The invention includes an expected value update component that modifies an expected value based, at least in part, upon a feature function of an input vector and an output value, a sum of lambda variable and a normalization variable. The invention further includes an error calculator that calculates an error based, at least in part, upon the expected value and an observed value. The invention also includes a parameter update component that modifies a trainable parameter based, at least in part, upon the error. A variable update component that updates at least one of the sum of lambda variable and the normalization variable based, at least in part, upon the error is also provided.
1. A system for training a machine learning system, comprising:
a training component that performs an iterative loop indexed on feature functions and the iterative loop including:
an expected value update component that, for a plurality of outputs and for a plurality of instances in which a single feature function is non-zero, modifies an expected value based, at least in part, upon the single feature function of an input vector and an output, a sum of lambda variable and a normalization variable;
an error calculator that calculates an error based, at least in part, upon the expected value and an observed value;
a parameter update component that modifies a trainable parameter based, at least in part, upon the error; and,
a variable update component that, for the plurality of outputs and for the plurality of instances in which the feature function is non-zero, sequentially updates at least one of the sum of lambda variable and the normalization variable based, at least in part, upon the error.
2. The system of claim 1 , the error calculation further employing, at least in part, the following equation:
observed
value
[
i
]
=
expected
value
[
i
]
⨯
ⅇ
δ
i
max
i
,
y
f
#
,
(
x
_
j
,
y
)
+
λ
i
+
δ
i
σ
2
where λ i is the trainable parameter,
δ i is the error,
σ is a standard deviation,
ƒ # =max j,y Σ i ƒ i ( x , y),
where i iterates from 1 to I wherein I is the number of training instances,
ƒ i ( x j , y) is the feature function,
j is the current training instance,
x j is the input vector, and,
y is the output for the function in x.
3. The system of claim 1 , the error calculation further employing, at least in part, the following equation:
observed
value
[
i
]
=
∑
j
,
y
P
λ
_
(
x
_
j
,
y
)
⨯
f
i
(
x
j
_
,
y
)
⨯
exp
(
δ
i
f
i
(
x
j
_
,
y
)
)
where
λ is a trainable parameter vector,
δ i is the error,
ƒ i ( x j , y) is the feature function,
P 80 ( x j , y) is a probability model with parameters λ ,
x j is the input vector, and,
y is the output for the function in x.
4. The system of claim 1 , modification of the expected value being based, at least in part, upon the following equation:
expected value=expected value+ƒ i ( x j , y ) e s[j,y] /z[j]
where ƒ i ( x j , y) is the feature function,
x j is the input vector,
y is the output for the function in x,
s [j, y] is the sum of lambda variable, and,
z [j] is the normalization variable.
5. The system of claim 1 , the error being based, at least in part, upon the following equation:
δ
i
=
1
max
j
,
y
f
i
(
x
_
j
,
y
)
log
(
observed
value
[
i
]
/
expected
value
[
i
]
)
where ƒ i ( x j , y) is the feature function,
δ i is the error,
observer value [i] is the current training data value
expected value [i] is the initially taken as 0 and incremented by a function for each subsequent iteration,
x j is the vector, and,
y is the output for the function in x.
6. The system of claim 1 , modification of the trainable parameter being based, at least in part, upon the following equation:
λ i =λ i +δ i
where λ i is the trainable parameter, and,
δ i is the error.
7. The system of claim 1 , updating of the sum of lambda variable and the normalization variable being based upon the following equation:
z[j]=z[j]−e s[j,y]
s[j,y]=s[j,y]+δ i
z[j]=z[j]+e s[j,y]
where s [j,y] is the sum of lambda variable,
z [j] is the normalization variable, and,
δ i is the error.
8. The system of claim 1 , further comprising a training data store that stores at least one of the observed value and the input vector.
9. The system of claim 8 , at least one of the observed value and the input vector being stored in a sparse representation.
10. The system of claim 1 , further comprising a parameter store that stores at least one trainable parameter.
11. The system of claim 1 , further comprising a training data store comprised of at least a sparse matrix of instances with non-zero values for each feature function.
12. A method for training a machine learning system executed by a computing device, comprising:
for each feature function:
updating an expected value based, at least in part, upon a feature function of an input vector and an output value, a sum of lambda variable and a normalization variable, wherein the input vector is based at least in part, upon information received from a computer readable medium;
calculating an error based, at least in part, upon the expected value and an observed value;
modifying a trainable parameter, based, at least in part, upon the error; and,
updating and storing on a computer readable medium at least one of the sum of lambda variable and the normalization variable based, at least in part, upon the error.
13. The method of claim 12 , the trainable parameter is a class trainable parameter or a word trainable parameter.
14. The method of claim 13 , the class trainable parameters being trained before the word trainable parameter are trained.
15. The method of claim 12 , further comprising at least one of word clustering, smoothing and improved iterative scaling.
16. The method of claim 12 , further comprising updating at least one of the sum of lambda variable and the normalization variable based, at least in part, upon the error, for each output, for each instance that the feature function is not zero.
17. The method of claim 12 , further comprising at least one of the following acts:
performing general initialization;
resetting an expected value;
determining whether there are more outputs; and,
determining whether there are more feature functions.
18. The method of claim 12 , storing at least one of the observed value and the input vector in a sparse representation.
19. A training system for a machine learning system, comprising:
means for performing an iterative loop indexed on feature functions, wherein the iterative loop includes:
means for modifying an expected value for a plurality of outputs and for a plurality of instances in which a feature function is non-zero based, at least in part, upon the feature function of an input vector and an output value, a sum of lambda variable and a normalization variable;
means for calculating an error based, at least in part, upon the expected value and an observed value;
means for modifying a trainable parameter based, at least in part, upon the error; and,
means for updating at least one of the sum of lambda variable and the normalization variable for the plurality of outputs and for the plurality of instances in which the feature function is non-zero based, at least in part, upon the error.