Neuro-symbolic reinforcement learning with first-order logic
A computer-implemented method for reinforcement learning with Logical Neural Networks (LNNs) is provided including receiving a plurality of observation text sentences from a target environment, extracting one or more propositional logic values from the plurality of observation text sentences, finding a class for each propositional logic value by using external knowledge, converting each propositional logic value into a first-order logic by replacing a part in the propositional logic value with a variable word, the part indicating the class, selecting a LNN based on the class among LNNs prepared in advance for each class, each LNN receiving the one or more propositional logic values as a status input and outputting an action with a score indicating a degree of preference for taking the action, and performing a highest score action to the target environment to obtain a next state of the target environment and a reward for the highest score action.
1. A computer-implemented method for reinforcement learning (RL) with Logical Neural Networks (LNNs), the method comprising:
receiving a plurality of observation text sentences from a target environment;
extracting one or more propositional logic values from the plurality of observation text sentences;
finding a class for each propositional logic value by using external knowledge;
converting, by an FOL converter, each propositional logic value into a first-order logic (FOL) by replacing a part in the propositional logic value with a variable word, the part indicating the class;
selecting a LNN based on the class among LNNs prepared in advance for each class, each LNN receiving the one or more propositional logic values as a status input and outputting an action with a score indicating a degree of preference for taking the action; and
performing a highest score action to the target environment to obtain a next state of the target environment and a reward for the highest score action.
2. The computer-implemented method of claim 1 , further comprising storing the current state, the highest score action, the next state, and the reward in a replay buffer to update each selected LNN.
3. The computer-implemented method of claim 1 , further comprising generating the FOL based on a history of the states of the target environment.
4. The computer-implemented method of claim 1 , wherein the FOL converter uses a semantic parser or natural language processing method.
5. The computer-implemented method of claim 1 , wherein the LNN includes one or more conjunction gates, one or more disjunction gates, one or more negation nodes, one or more universal quantifiers, and one or more existential quantifiers.
6. The computer-implemented method of claim 1 , wherein an action policy is trained in the LNN from the FOL.
7. The computer-implemented method of claim 1 , wherein the reward for the highest score action is used for adding one or more new nodes and for updating a weight value for each connection.
8. A computer program product for reinforcement learning (RL) with Logical Neural Networks (LNNs), the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
receive a plurality of observation text sentences from a target environment;
extract one or more propositional logic values from the plurality of observation text sentences;
find a class for each propositional logic value by using external knowledge;
convert, by an FOL converter, each propositional logic value into a first-order logic (FOL) by replacing a part in the propositional logic value with a variable word, the part indicating the class;
select a LNN based on the class among LNNs prepared in advance for each class, each LNN receiving the one or more propositional logic values as a status input and outputting an action with a score indicating a degree of preference for taking the action; and
perform a highest score action to the target environment to obtain a next state of the target environment and a reward for the highest score action.
9. The computer program product of claim 8 , wherein the current state, the highest score action, the next state, and the reward are stored in a replay buffer to update each selected LNN.
10. The computer program product of claim 8 , wherein the FOL is generated based on a history of the states of the target environment.
11. The computer program product of claim 8 , wherein the FOL converter uses a semantic parser or natural language processing method.
12. The computer program product of claim 8 , wherein the LNN includes one or more conjunction gates, one or more disjunction gates, one or more negation nodes, one or more universal quantifiers, and one or more existential quantifiers.
13. The computer program product of claim 8 , wherein an action policy is trained in the LNN from the FOL.
14. The computer program product of claim 8 , wherein the reward for the highest score action is used for adding one or more new nodes and for updating a weight value for each connection.
15. A system for reinforcement learning (RL) with Logical Neural Networks (LNNs), the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
receive a plurality of observation text sentences from a target environment;
extract one or more propositional logic values from the plurality of observation text sentences;
find a class for each propositional logic value by using external knowledge;
convert, by an FOL converter, each propositional logic value into a first-order logic (FOL) by replacing a part in the propositional logic value with a variable word, the part indicating the class;
select a LNN based on the class among LNNs prepared in advance for each class, each LNN receiving the one or more propositional logic values as a status input and outputting an action with a score indicating a degree of preference for taking the action; and
perform a highest score action to the target environment to obtain a next state of the target environment and a reward for the highest score action.
16. The system of claim 15 , wherein the current state, the highest score action, the next state, and the reward are stored in a replay buffer to update each selected LNN.
17. The system of claim 15 , wherein the FOL is generated based on a history of the states of the target environment.
18. The system of claim 15 , wherein the FOL converter uses a semantic parser or natural language processing method.
19. The system of claim 15 , wherein the LNN includes one or more conjunction gates, one or more disjunction gates, one or more negation nodes, one or more universal quantifiers, and one or more existential quantifiers.
20. The system of claim 15 , wherein an action policy is trained in the LNN from the FOL.