Reinforcement learning based adaptive state observation for brain- machine interface
View Patent ↗A reinforcement learning (RL) based adaptive state observation model usable for implementing a brain machine interface (BMI) is proposed for decoding a brain signal to determine a movement action and controlling a machine to perform the movement action. In the model, the brain signal is processed by a neural network (NN) for applying a nonlinear mapping defined by NN weights to the brain signal to thereby yield a transformed brain signal. The NN learns the nonlinear mapping by RL, allowing the weights to be adaptively and continuously updated to follow nonlinearity and non-stationarity of the brain signal. The transformed brain signal is processed by a Kalman filter (KF) to yield a control signal for controlling the machine to perform the movement action, thereby utilizing the KF to provide smooth generation of the control signal while blocking adverse influence of nonlinearity and non-stationarity of the brain signal to the KF.
1 . A computer-implemented method for controlling a machine to perform a movement action determined by a brain signal of a subject, the method comprising:
processing the brain signal with a neural network (NN) for applying a nonlinear mapping defined by a plurality of weights of the NN to the brain signal to thereby yield a transformed brain signal;
updating the plurality of weights by a reinforcement learning (RL) process such that the NN learns the nonlinear mapping by RL; and
processing the transformed brain signal with a Kalman filter (KF) to yield a control signal for controlling the machine to perform the movement action, wherein the control signal is computed as x t|t given by
x t|t =x t|t-1 +K ( y t −x t|t-1 )
where:
x t|t is a posterior estimation of a mean of a state x at time step t, wherein the state x is a state of the KF, the KF being used to model the machine;
x t|t-1 is a prior estimation of the mean of the state x at time step t; and
K is a Kalman gain given by K=P t|t-1 (P t|t-1 +R) −1 where is a prior estimation of the covariance of the state x at time step t, and R is a covariance matrix of a Gaussian noise term in y t , in which y t is the transformed brain signal generated by the NN at time step t.
2 . The method of claim 1 , wherein the RL process comprises:
equating the transformed brain signal to be a plurality of scores respectively associated with a plurality of candidate movement actions competing to be the movement action, an individual score of a respective candidate movement action being indicative to a probability that the respective candidate movement action is the movement action;
computing the plurality of weights according to at least the plurality of scores; and
updating the NN with the computed plurality of weights for configuring the nonlinear mapping.
3 . The method of claim 2 , wherein the computing of the plurality of weights according to at least the plurality of scores comprises:
computing, from the plurality of scores, a plurality of probabilities associated with the plurality of candidate movement actions, wherein an individual probability associated with the respective candidate movement action is the probability that the respective candidate movement action is the movement action;
selecting, from the plurality of candidate movement actions, a winner in competing to be the movement action according to the plurality of probabilities;
determining a reward due to selecting the winner as the movement action according to whether or not the winner is the movement action; and
computing the plurality of weights according to at least the reward.
4 . The method of claim 3 , wherein:
P(a t =k′), the probability that a k′th candidate movement action in the plurality of candidate movement actions is the movement action, is computed by
P
(
a
t
=
k
′
)
=
exp
(
α
y
k
′
)
∑
k
=
1
N
exp
(
α
y
k
)
where N is a total number of candidate movement actions in the plurality of candidate movement actions, y t =[y 1 , . . . , y N ] T is the transformed brain signal with y k being a kth component of y t , and α≥0 is a controlling parameter; and
the winner is selected to be a k*th candidate movement action from P (a t =k*) among all N probability values of P(a t =k′), k′=1, . . . , N.
5 . The method of claim 4 , wherein r t , the reward due to selecting the k*th candidate movement action as the movement action, is given by r t =1 if the k*th candidate movement action is the movement action, and r t =0 if not.
6 . The method of claim 5 , wherein:
the NN is a three-layer NN comprising:
an input layer having D z nodes for receiving the brain signal, the brain signal having D z components;
a hidden layer having J hidden units; and
an output layer having N nodes for outputting the transformed brain signal, the transformed brain signal having N components; and
the plurality of weights is computed by
v jk *←v jk *+γh j f (δ), j= 1, . . . , J,
and
w ij ←w ij +γz ti h j f (δ) v jk* (1− h j ), i= 1, . . . , D z and j= 1, . . . , J,
where:
w ij is a weight from an ith node of the input layer to a jth hidden unit;
v jk* is a weight from the jth hidden unit to a k*th node of the output layer;
‘←’ is an assignment operator meaning that the variable at the left hand side of the operator is assigned with a value computed by the expression shown on the right hand side of the operator;
γ is a learning rate;
δ is an error function computed by δ=r t −P(a t =k*);
f(δ) is an error expansive function given by
f
(
δ
)
=
{
δ
1
-
δ
,
δ
≥
0
,
-
1
,
δ
<
0
;
z ti is an ith component of z t , the brain signal obtained at time step t; and
h j is a value of the jth hidden unit, given by
h
j
=
1
1
+
exp
(
-
∑
i
=
1
D
z
w
ij
z
ti
)
.
7 . A system for capturing a brain signal of a subject and performing a movement action determined by the brain signal, the system comprising:
a sensing device for capturing the brain signal from the subject;
a machine for performing the movement action; and
a computer configured to execute a computing process of processing the brain signal to determine the movement action and controlling the machine to perform the movement action according to the method of claim 1 .
8 . The system of claim 7 , wherein the sensing device is an electroencephalogram (EEG) sensing device.
9 . The system of claim 7 , wherein the machine is a prosthesis.
10 . The system of claim 7 , wherein the machine is a second computer configured to generate the movement action on a virtual object for virtual-reality applications.
11 . A brain-machine interface (BMI) apparatus for capturing a brain signal of a subject and controlling a machine to perform a movement action determined by the brain signal, the BMI apparatus comprising:
a sensing device for capturing the brain signal from the subject; and
a computer configured to execute a computing process of processing the brain signal to determine the movement action and controlling the machine to perform the movement action according to the method of claim 1 .
12 . The BMI apparatus of claim 11 , wherein the sensing device is an electroencephalogram (EEG) sensing device.
13 . A computer-implemented method for controlling a machine to perform a movement action determined by a brain signal of a subject, the method comprising:
processing the brain signal with a neural network (NN) for applying a nonlinear mapping defined by a plurality of weights of the NN to the brain signal to thereby yield a transformed brain signal;
updating the plurality of weights by a reinforcement learning (RL) process such that the NN learns the nonlinear mapping by RL; and
processing the transformed brain signal with a Kalman filter (KF) to yield a control signal for controlling the machine to perform the movement action;
wherein the RL process comprises:
equating the transformed brain signal to be a plurality of scores respectively associated with a plurality of candidate movement actions competing to be the movement action, an individual score of a respective candidate movement action being indicative to a probability that the respective candidate movement action is the movement action;
computing the plurality of weights according to at least the plurality of scores; and
updating the NN with the computed plurality of weights for configuring the nonlinear mapping;
wherein the computing of the plurality of weights according to at least the plurality of scores comprises:
computing, from the plurality of scores, a plurality of probabilities associated with the plurality of candidate movement actions, wherein an individual probability associated with the respective candidate movement action is the probability that the respective candidate movement action is the movement action;
selecting, from the plurality of candidate movement actions, a winner in competing to be the movement action according to the plurality of probabilities;
determining a reward due to selecting the winner as the movement action according to whether or not the winner is the movement action; and
computing the plurality of weights according to at least the reward;
wherein P(a t =k′), the probability that a k′th candidate movement action in the plurality of candidate movement actions is the movement action, is computed by
P
(
a
t
=
k
′
)
=
exp
(
α
y
k
′
)
∑
k
=
1
N
exp
(
α
y
k
)
where N is a total number of candidate movement actions in the plurality of candidate movement actions, y t =[y 1 , . . . , y N ] T is the transformed brain signal with y k being a kth component of y t , and α≥0 is a controlling parameter; and
wherein the winner is selected to be a k*th candidate movement action from P(a t =k*) among all N probability values of P(α t =k′), k′=1, . . . , N.
14 . The method of claim 13 , wherein r t , the reward due to selecting the k*th candidate movement action as the movement action, is given by r t =1 if the k*th candidate movement action is the movement action, and r t =0 if not.
15 . The method of claim 14 , wherein:
the NN is a three-layer NN comprising:
an input layer having D z nodes for receiving the brain signal, the brain signal having D z components;
a hidden layer having/hidden units; and
an output layer having N nodes for outputting the transformed brain signal, the transformed brain signal having N components; and
the plurality of weights is computed by
v jk *←v jk *+γh j f (δ), j= 1, . . . , J,
and
w ij ←w ij +γz ti h j f (δ) v jk* (1− h j ), i= 1, . . . , D z and j= 1, . . . , J,
where:
w ij is a weight from an ith node of the input layer to a jth hidden unit;
v jk* is a weight from the jth hidden unit to a k*th node of the output layer;
‘←’ is an assignment operator meaning that the variable at the left hand side of the operator is assigned with a value computed by the expression shown on the right hand side of the operator;
γ is a learning rate;
δ is an error function computed by δ=r t −P(α t =k*);
f(δ) is an error expansive function given by
f
(
δ
)
=
{
δ
1
-
δ
,
δ
≥
0
,
-
1
,
δ
<
0
;
z ti is an ith component of z t , the brain signal obtained at time step t; and
h j is a value of the jth hidden unit, given by
h
j
=
1
1
+
exp
(
-
∑
i
=
1
D
z
w
i
j
z
ti
)
.
16 . A system for capturing a brain signal of a subject and performing a movement action determined by the brain signal, the system comprising:
a sensing device for capturing the brain signal from the subject;
a machine for performing the movement action; and
a computer configured to execute a computing process of processing the brain signal to determine the movement action and controlling the machine to perform the movement action according to the method of claim 13 .
17 . The system of claim 16 , wherein the sensing device is an electroencephalogram (EEG) sensing device.
18 . The system of claim 16 , wherein the machine is a prosthesis.
19 . The system of claim 16 , wherein the machine is a second computer configured to generate the movement action on a virtual object for virtual-reality applications.
20 . A brain-machine interface (BMI) apparatus for capturing a brain signal of a subject and controlling a machine to perform a movement action determined by the brain signal, the BMI apparatus comprising:
a sensing device for capturing the brain signal from the subject; and
a computer configured to execute a computing process of processing the brain signal to determine the movement action and controlling the machine to perform the movement action according to the method of claim 13 .