Gradient based training method for a support vector machine
A training method for a support vector machine, including executing an iterative process on a training set of data to determine parameters defining the machine, the iterative process being executed on the basis of a differentiable form of a primal optimization problem for the parameters, the problem being defined on the basis of the parameters and the data set.
1. A training method for a support vector machine to perform data classification for a relationship between a training set of data, the method executed by a computer system, including executing an iterative process by a processor on the training set of data read from a data input device to determine parameters defining said machine represented by:
y=sgn ( w·x+βb ),
where y is the classification output which is output by a data output device, x is the input data read from the data input device, βis 0 or 1, the vector w and bias b, being parameters defining a decision surface, said iterative process being executed by the processor based on a derivative optimization function for said parameters and said data set.
2. A method as claimed in claim 1 , wherein said method is adapted for generation of a kernel learning machine.
3. A method as claimed in claim 1 , wherein said method is adapted to generate a regularisation network.
4. A method as claimed in claim 1 , wherein for classification, said parameters are obtained by minimizing the differentiable objective function:
Ψ
(
w
,
b
)
=
1
2
w
·
w
+
C
∑
i
=
1
n
L
(
1
-
y
,
(
w
·
x
i
+
β
b
)
)
where C >0 is a free parameter, x i . i=1, . . . , n, are data points of the training set, y i ,=±1, i=1, . . . , n, are known labels, n is the number of data points and L is a differentiable loss function such that L(ξ) =0 for ξ<0.
5. A method as claimed in claim 4 , wherein said iterative process operates on a derivative of the objective function Ψuntil the vectors converge to a vector w for the machine.
6. A method as claimed in claim 1 , wherein for ε-insensitive regression, the vector w and bias b, being parameters defining a decision surface, are obtained by minimizing the differentiable objective function
Ψ
(
w
,
b
)
=
1
2
w
·
w
+
C
∑
i
=
1
n
L
(
y
i
-
w
·
x
i
+
β
b
-
ɛ
)
where the ε>0 is a free parameter, C>0 is a free parameter, βis 0 or 1, x i , i=1, . . . , n, are the training data points of the data set, y i =±1, i=1, . . . , n, are known labels, n is the number of data points and L is a differentiable loss function such that L (ξ)=0 for ξ<0.
7. A support vector machine for performing a classification task, the support vector machine comprising:
an input device reading training data points;
a processor calculating classification output y for the classification task given by
y
=
y
(
x
)
=
∑
i
=
1
n
y
i
α
i
k
(
x
i
,
x
j
)
+
β
b
where x εR″ is a data point to be classified and x i , are the training data points, k is a kernel function, and a i , are coefficients determined by the processor calculating
a i =CL ′(1 −y i n i βb )
where L′ (ξ) is the derivative of the loss and the values n i , are determined by the processor iteratively executing
η
j
t
+
1
=
η
j
t
-
δ
(
η
j
t
-
C
∑
i
=
1
n
L
′
(
1
-
y
i
η
i
t
-
y
i
β
b
t
)
y
i
k
(
x
i
,
x
j
)
)
,
b
t
+
1
=
β
b
t
+
δ
β
C
∑
i
=
1
n
L
′
(
1
-
y
i
η
j
t
-
y
i
β
b
t
)
y
i
.
where ξ>0 is a free parameter representing a learning rate and/or, by the processor iteratively executing in the homogeneous case (β=0):
η
j
t
+
1
=
C
∑
i
=
1
n
L
′
(
1
-
y
i
η
j
t
)
y
i
k
(
x
i
,
x
j
)
where i, j=1, n, n are the number of data points, t represents an iteration and L′ is the derivative of a loss function L.
8. A support vector machine for ε-regression, the support vector machine comprising:
a data input device reading training data points; and
a processor calculating an a classification output y given by
y
(
x
)
=
∑
i
=
1
n
β
i
k
(
x
,
x
i
)
+
β
b
where x ε R′″ is a data point to be evaluated and x i , are the training data points, k is a kernel function, β=0 or 1,and β i , and bias b are coefficients determined by the processor calculating
β i =CL ′( |y i -n j -β b |-ε) sgn ( y i - i -β b )
where ε is a free parameter and the values n j and b are determined by the processor iteratively executing
η
j
t
+
1
=
η
j
t
-
δ
(
η
j
t
-
C
∑
i
=
1
n
L
′
(
y
i
-
η
i
t
-
β
b
-
ɛ
)
sgn
(
y
i
-
η
i
t
-
β
b
)
k
(
x
i
,
x
j
)
)
b
t
+
1
=
b
t
+
δ
β
C
∑
i
=
1
n
L
′
(
y
i
-
η
i
t
-
β
b
-
ɛ
)
sgn
(
y
i
-
η
i
t
-
β
b
)
where ξ>0 is a free parameter representing a learning rate and/or, by the processor iteratively executing in the homogeneous case (β=0):
η
j
t
+
1
=
C
∑
i
=
1
n
L
′
(
y
i
-
η
j
t
-
ɛ
)
sgn
(
y
i
-
η
i
t
)
k
(
x
i
,
x
j
)
where i, j =1, . . . , n, n being the number of data points and t represents an iteration and L′ is the derivative of a loss function L.
9. A regularization network of a computer system comprising:
a data input device for reading a set of training data points, x;
a processor calculating a classification output y by solving the equation
y
(
x
)
=
∑
i
=
1
n
β
i
k
(
x
,
x
i
)
+
β
b
where x ε R′″ is a data point to be evaluated and x i , are training data points of the set of training data points, k is a kernel function, β=0 or 1, and β i and bias b are coefficients: and
wherein the processor determines the coefficients β i by the equation:
β l =CL ′( |y i -n i -β b |-ε)
where ε is a free parameter and the values n i , and b are determined by the processor iteratively executing
n
j
t
+
1
=
n
j
t
-
δ
(
n
j
t
-
C
∑
i
=
t
n
L
′
(
❘
ly
i
-
n
j
t
-
β
b
)
k
(
x
~
i
,
x
~
j
)
b
t
+
1
=
b
t
+
δ
β
C
∑
i
=
1
n
L
′
(
❘
ly
i
-
n
j
t
-
β
b
)
where δ>0 is a free parameter representing a learning rate and/or, by the processor iteratively executing in the homogeneous case (β=0):
n
j
t
+
1
=
C
∑
i
=
t
n
L
′
(
ly
i
-
n
j
t
-
β
b
)
k
(
x
~
i
,
x
~
j
)
where i, j=1, . . . , n, n being the number of data points and t represents an iteration and L′ is the derivative of a loss function L.
10. A non-transitory computer readable medium having stored thereon instructions for performing data classification for the relationship between a training set of data, the stored instructions comprising machine executable code, which when executed by at least one machine processor, causes the machine to:
execute an iterative process on the training set of data to determine parameters defining said machine represented by:
y=sgn ( w.x+βb ),
where y is the output which is output by a data output device, x is the input data read from the data input device, β is 0 or 1, the vector w and bias b, being parameters defining a decision surface, said iterative process being executed on a derivative optimization function for said parameters and said data set.