Spectrum access method and system using prior knowledge-based double-action reinforcement learning
View Patent ↗The present disclosure provides a spectrum access method and system using prior knowledge-based double-action reinforcement learning, and belongs to the technical field of electromagnetic spectrum. The method includes evaluating and screening prior knowledge, initializing a Q-table, and confirming a current state; and performing Q-learning by: firstly, decomposing an action space into two dimensions with an action in one dimension defined as a channel chosen by an agent, and an action in the other dimension defined as a number of time slots of an access channel, and choosing actions in turn according to the dimensions; then performing spectrum access according to the actions chosen; and finally, updating the Q-table in combination with biased information, wherein the biased information is a reward value. The system is configured to implement the proposed method. By adoption of the method, better performance is achieved, and the efficiency of spectrum access can be improved.
1. A spectrum access method using prior knowledge-based double-action reinforcement learning, comprising:
evaluating and screening prior knowledge, initializing a Q-table, and confirming a current state; and
decomposing an action space of the Q-table into two dimensions with an action in one dimension defined as a channel chosen by an agent, and an action in the other dimension defined as a number of time slots of an access channel, and choosing actions in turn according to the dimensions; then performing spectrum access according to the actions chosen; and finally, updating the Q-table in combination with biased information, wherein the biased information is a reward value corresponding to a channel capacity and a channel bandwidth;
wherein said choosing actions in turn according to the dimensions specifically comprises: setting two dimension parameters x and y which are random numbers of [0,1];
when x<ε, choosing an action in one dimension according to a random probability, otherwise, calculating a mean value of the Q-table in the dimension, and choosing an action in the dimension according to a maximum value, wherein F is a constant; or
when y<ε, choosing an action in the other dimension according to a random probability, otherwise, calculating an action in the dimension according to a maximum value.
2. The method according to claim 1 , wherein said updating the Q-table in combination with biased information specifically comprises:
acquiring a reward value;
determining a state value at a next moment;
updating the Q-table; and
adjusting the Q-table using the biased information.
3. The method according to claim 2 , wherein said adjusting the Q-table using the biased information specifically comprises:
in case the reward value is taken as the biased information, and the reward value is less than zero, if Q(s, a 1 , a 2 )>0, making Q(s, a 1 , a 2 )=const1×Q(s, a 1 , a 2 ); otherwise, making Q(s, a 1 , a 2 )=(1/const1)×Q(s, a 1 , a 2 ), wherein const1 denotes a constant, and a 1 and a 2 respectively denote actions in the two dimensions; and
in case the reward value is less than a statistical average reward value R , if Q(s, a 1 , a 2 )>0, making Q(s, a 1 , a 2 )=const2×Q(s, a 1 , a 2 ); otherwise, making Q(s, a 1 , a 2 )=(1/const2)×Q(s, a 1 , a 2 ), wherein const2 denotes a constant.
4. The method according to claim 3 , wherein said acquiring a reward value specifically comprises: taking a ratio of a channel capacity to a utility function as a reward which is expressed in the following formula:
R
=
u
1
·
u
2
·
(
c
u
·
W
-
1
k
1
·
Th
)
wherein c denotes a channel capacity, u denotes a Kosca utility function, W denotes a channel bandwidth, k 1 denotes a constant, Th denotes a jamming threshold, u 1 denotes a reward value balancing coefficient on the order of magnitudes, and u 2 denotes a duration guidance coefficient.
5. The method according to claim 4 , wherein a Q-value is updated according to the following formula:
Q ( s,a 1 ,a 2 )=(1−α)× Q ( s,a 1 ,a 2 )+α×[ r+γ a max Q ( s′,a 1 ,a 2 )]
wherein α denotes a learning rate, γ denotes an attenuation factor, s and s′ denote a state at a current state and a state at a next moment, respectively, and a denotes an optimal action set in the state at the next moment, wherein α comprises a 1 and a 2 .
6. The method according to claim 1 , wherein said evaluating and screening prior knowledge specifically comprises:
defining accuracy of the prior knowledge according to Pearson's correlation coefficient, and choosing information with high accuracy as prior information.
7. The method according to claim 1 , wherein said evaluating and screening prior knowledge specifically comprises: defining accuracy of the prior knowledge according to Pearson's correlation coefficient, and choosing information with high accuracy as prior information.
8. The method according to claim 2 , wherein said evaluating and screening prior knowledge specifically comprises:
defining accuracy of the prior knowledge according to Pearson's correlation coefficient, and choosing information with high accuracy as prior information.
9. The method according to claim 3 , wherein said evaluating and screening prior knowledge specifically comprises:
defining accuracy of the prior knowledge according to Pearson's correlation coefficient, and choosing information with high accuracy as prior information.
10. The method according to claim 4 , wherein said evaluating and screening prior knowledge specifically comprises:
defining accuracy of the prior knowledge according to Pearson's correlation coefficient, and choosing information with high accuracy as prior information.
11. The method according to claim 5 , wherein said evaluating and screening prior knowledge specifically comprises:
defining accuracy of the prior knowledge according to Pearson's correlation coefficient, and choosing information with high accuracy as prior information.
12. The method according to claim 6 , wherein a state at any moment involves a difference value and a mean value of first two time slots of each channel.
13. A spectrum access system using prior knowledge-based double-action reinforcement learning, the system comprising:
a prior information evaluating and state confirming unit configured to evaluate and screen prior knowledge, initialize a Q-table, and confirm a current state; and
a learning unit configured to decompose an action space of the Q-table into two dimensions with an action in one dimension defined as a channel chosen by an agent, and an action in the other dimension defined as a number of time slots of an access channel, and choose actions in turn according to the dimensions; then perform spectrum access according to the actions chosen; and finally, update the Q-table in combination with biased information, wherein the biased information is a reward value corresponding to a channel capacity and a channel bandwidth;
wherein said choosing actions in turn according to the dimensions specifically comprises: setting two dimension parameters x and y which are random numbers of [0,1];
when x<ε, choosing an action in one dimension according to a random probability, otherwise, calculating a mean value of the Q-table in the dimension, and choosing an action in the dimension according to a maximum value, wherein F is a constant; or
when y<ε, choosing an action in the other dimension according to a random probability, otherwise, calculating an action in the dimension according to a maximum value.
14. The spectrum access system according to claim 13 , wherein said updating the Q-table in combination with biased information specifically comprises:
acquiring a reward value;
determining a state value at a next moment;
updating the Q-table; and
adjusting the Q-table using the biased information.
15. The spectrum access system according to claim 14 , wherein said adjusting the Q-table using the biased information specifically comprises:
in case the reward value is taken as the biased information, and the reward value is less than zero, if Q(s, a 1 , a 2 )>0, making Q(s, a 1 , a 2 )=const1×Q(s, a 1 , a 2 ); otherwise, making Q(s, a 1 , a 2 )=(1/const1)×Q(s, a 1 , a 2 ), wherein const1 denotes a constant, and a 1 and a 2 respectively denote actions in the two dimensions; and
in case the reward value is less than a statistical average reward value R , if Q(s, a 1 , a 2 )>0, making Q(s, a 1 , a 2 )=const2×Q(s, a 1 , a 2 ); otherwise, making Q(s, a 1 , a 2 )=(1/const2)×Q(s, a 1 , a 2 ), wherein const2 denotes a constant.
16. The spectrum access system according to claim 15 , wherein said acquiring a reward value specifically comprises: taking a ratio of a channel capacity to a utility function as a reward which is expressed in the following formula:
R
=
u
1
·
u
2
·
(
c
u
·
W
-
1
k
1
·
Th
)
wherein c denotes a channel capacity, u denotes a Kosca utility function, W denotes a channel bandwidth, k 1 denotes a constant, Th denotes a jamming threshold, u 1 denotes a reward value balancing coefficient on the order of magnitudes, and u 2 denotes a duration guidance coefficient.
17. The spectrum access system according to claim 16 , wherein
a Q-value is updated according to the following formula:
Q
(
s
,
a
1
,
a
2
)
=
(
1
-
α
)
×
Q
(
s
,
a
1
,
a
2
)
+
α
×
[
r
+
γ
max
a
Q
(
s
′
,
a
1
,
a
2
)
]
wherein α denotes a learning rate, γ denotes an attenuation factor, s and s′ denote a state at a current state and a state at a next moment, respectively, and a denotes an optimal action set in the state at the next moment, wherein a comprises a 1 and a 2 .
18. The spectrum access system according to claim 13 , wherein said evaluating and screening prior knowledge specifically comprises: defining accuracy of the prior knowledge according to Pearson's correlation coefficient, and choosing information with high accuracy as prior information.