Methods and apparatus for implementing reinforcement learning

1 . A method of operation for a node implementing reinforcement learning, RL, wherein the node instructs actions in an environment in accordance with a policy generated by a RL agent, wherein the RL agent models the environment and encodes a state of the environment using a set of features, the method comprising:

obtaining an intent, wherein the intent specifies one or more criteria to be satisfied by the environment;

determining a Markov Decision Process, MDP that encodes states of the environment using a subset of the set of features used by the RL agent;

generating a finite state automaton that represents the intent as a series of logic states;

computing a product of MDP output states and logic states, wherein the product contains potential combinations of a MDP output state and a logic state;

determining probabilities of actions from one or more suggested actions obtained from the policy causing a criterion to be violated using value iteration and Depth First Search (DFS) algorithms;

selecting an action to be performed on the environment from the one or more suggested actions obtained from the policy, the selection being based on the product of MDP output states and logic states; and

controlling an antenna tilt based on the selected action.

2 . An apparatus for implementing reinforcement learning, RL, wherein the apparatus is configured to instruct actions in an environment in accordance with a policy generated by a RL agent, wherein the RL agent models the environment and encodes a state of the environment using a set of features, and wherein the apparatus comprises a processing circuitry and a memory containing instructions executable by the processing circuitry, whereby the apparatus is operable to:

obtain an intent, wherein the intent specifies one or more criteria to be satisfied by the environment;

determine a Companion-Markov Decision Process, MDP CMDP, that encodes states of the environment using a subset of the set of features used by the RL agent;

generate a finite state automaton that represents the intent as a series of logic states;

compute a product of MDP CMDP output states and logic states, wherein the product contains all of the potential combinations of a MDP CMDP output state and a logic state;

determine probabilities of actions from one or more suggested actions obtained from the policy causing a criterion to be violated using value iteration and Depth First Search (DFS) algorithms;

select an action to be performed on the environment by the apparatus from the one or more suggested actions obtained from the policy, the selection being based on the product of MDP output states and logic states; and

controlling an antenna tilt based on the selected action.

3 . The apparatus of claim 2 , configured to determine the MDP CMDP based on the features that influence the one or more criteria to be satisfied by the environment.

4 . The apparatus of claim 3 , wherein the determined MDP CMDP encodes states of the environment using the features that influence the one or more criteria to be satisfied by the environment as specified by the intent.

5 . The apparatus of claim 2 , wherein each of the determined MDP CMDP output states is linked to an action from among the suggested actions obtained from the policy.

6 . The apparatus of claim 5 configured, when selecting an action, to determine if any actions, from the one or more suggested actions obtained from the policy, have a probability above a predetermined threshold of causing a criterion from among the one or more criteria to be satisfied by the environment to be violated, based on the combinations of a MDP CMDP output state and a logic state.

7 . The apparatus of claim 6 configured such that, if a combination of a given MDP output state and given logic state has a probability above the predetermined threshold of causing a criterion from among the one or more criteria to be satisfied by the environment to be violated, actions linked to the given MDP output state are prevented from being selected.

8 . The apparatus of claim 7 , wherein the predetermined threshold is a 1% chance, or wherein the predetermined threshold is a 0% chance.

9 . The apparatus of claim 6 configured such that, if it is determined that none of the suggested actions obtained from the policy have a probability below the predetermined threshold, the apparatus:

either selects a default safe action; or

relaxes the criteria in the intent and repeats the determining, generating, computing and selecting steps using the relaxed criteria; or

increases the predetermined threshold and repeats the selecting step; or

selects the action having the lowest probability of causing a criterion from among the one or more criteria to be satisfied by the environment to be violated.

10 . The apparatus of claim 2 , configured to determine the MDP CMDP by selecting a pre-generated MDP from a group of MDPs, based on the one or more criteria specified by the intent.

11 . The apparatus of claim 2 , configured to determine the MDP by deriving a MDP based on the one or more criteria specified by the intent.

12 . The apparatus of claim 2 , wherein the environment is at least a part of a telecommunications network.

13 . The apparatus of claim 12 , wherein a base station of the telecommunications network comprises the apparatus.

14 . The apparatus of claim 12 , wherein the set of features encoding a state comprise one or more of: base station configuration measurements; signal to interference plus noise ratios, SINR; capacity measurements; coverage measurements; and Quality of Service, QoA measurements.

15 . The apparatus of claim 12 , wherein the suggested actions comprise one or more of: antenna configuration alterations; and transmission parameter adjustments.

16 . The apparatus of claim 2 , wherein the product contains all of the potential combinations of a MDP output state and a logic state.