IP Library › Granted Patent US 12,346,802
Granted Patent B2
US 12,346,802 · App. 17/110,400 · Granted Jul 1, 2025

Neuro-symbolic next state prediction based on multiple potential causes

Inventors: Akifumi Wachi (Tokyo, JP); Ryosuke Kohita (Tokyo, JP); Daiki Kimura (Tokyo, JP)
Assignee: International Business Machines Corporation
G06N3/08G06N3/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,346,802
App. No.
17/110,400
Granted
Jul 1, 2025
Kind
B2
Abstract

Next state prediction technology that performs the following computer based operations: receiving state information that includes information indicative of a current state of an environment; processing the state information to predict a future state of the environment, with the processing being performed by a hybrid computer system that includes both of the following: (i) neural network software module(s) that include machine learning functionality, and (ii) symbolic rule based software modules; and using the prediction of the next state of the environment as an input with respect to taking a further action (for example, activating a hardware device or effecting a communication to a human or another device).

Claims (24)

1. A computer-implemented method (CIM) performed by a hybrid computing system comprising a neural network that includes machine learning functionality and a symbolic rule based software module that includes a symbolic reasoner, wherein the CIM comprises steps of:

receiving a current state information set including a plurality of attribute values characterizing a current and historical status of an environment with the plurality of attribute values including at least a temperature attribute value characterizing a temperature value in the environment;

receiving first action information including information indicative of a first action being taken in the environment;

applying the neural network of the hybrid computing system to the current state information data set and the first action information that includes the information indicative of the first action being taken in the environment to make a first prediction of a future temperature value in the environment, with the neural network including a pre-trained multi-label classifier;

applying the symbolic reasoner of the hybrid computing system to the current state information data set to make a second prediction of the future temperature value in the environment, with the symbolic reasoner including a plurality of software based rules, with the application of the symbolic reasoner including applying each rule of the plurality of software based rules to a symbolic representation of the current state information; and

making a third prediction of the future temperature value based on the first prediction and the second prediction.

2. A computer-implemented method (CIM) performed by a hybrid computing system comprising a neural network that includes machine learning functionality and a symbolic rule based software module that includes a symbolic reasoner, wherein the CIM comprises steps of:

receiving a current state information set including a plurality of attribute values characterizing a current and historical status of an environment with the plurality of attribute values including at least a light and/or color attribute value characterizing a light and/or color related value in the environment;

receiving first action information including information indicative of a first action being taken in the environment;

applying the neural network of the hybrid computing system to the current state information data set and the first action information that includes the information indicative of the first action being taken in the environment to make a first prediction of a future light and/or color value in the environment, with the neural network including a pre-trained multi-label classifier;

applying the symbolic reasoner of the hybrid computing system to the current state information data set to make a second prediction of the future light and/or color value in the environment, with the symbolic reasoner including a plurality of software based rules, with the application of the symbolic reasoner including applying each rule of the plurality of software based rules to a symbolic representation of the current state information; and

making a third prediction of the future light and/or color value based on the first prediction and the second prediction.

3. A computer-implemented method (CIM) performed by a hybrid computing system comprising a neural network that includes machine learning functionality and a symbolic rule based software module that includes a symbolic reasoner, wherein the CIM comprises steps of:

receiving a current state information set including a plurality of attribute values characterizing a current and historical status of text document;

receiving first action information including information indicative of a first action being taken on the text document;

applying the neural network of the hybrid computing system to the current state information data set and the first action information that includes the information indicative of the first action being taken in the environment to make a first prediction of a future word and/or phrase that will be added to the text document, with the neural network including a pre-trained multi-label classifier;

applying the symbolic reasoner of the hybrid computing system to the current state information data set to make a second prediction of the future word and/or phrase that will be added to the text document, with the symbolic reasoner including a plurality of software based rules, with the application of the symbolic reasoner including applying each rule of the plurality of software based rules to a symbolic representation of the current state information; and

making a third prediction of the future word and/or phrase that will be added to the text document based on the first prediction and the second prediction.

4. The computer-implemented method of claim 1 , wherein the current state information further comprises an image.

5. The computer-implemented method of claim 2 , wherein the current state information further comprises an image.

6. The computer-implemented method of claim 3 , wherein the current state information further comprises an image.

7. The computer-implemented method of claim 1 , wherein the symbolic rule based software module is void of any machine learning component.

8. The computer-implemented method of claim 2 , wherein the symbolic rule based software module is void of any machine learning component.

9. The computer-implemented method of claim 3 , wherein the symbolic rule based software module is void of any machine learning component.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 3, 2020
From: WACHI, AKIFUMI; KOHITA, RYOSUKE; KIMURA, DAIKI
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 054526/0672 →
Continuity (1)
Related Publication 20220180166A1 · Jun 9, 2022
References Cited (34)
US 11694061B2 · Byrnes · 2023 [cited by examiner]
US 20160234972A1 · Billet · 2016 [cited by examiner]
US 20170235735A1 · Ignatyev · 2017 [cited by examiner]
US 20190244112A1 · Bowling · 2019 [cited by examiner]
US 20190251449A1 · Dai · 2019 [cited by examiner]
US 20190272465A1 · Kimura · 2019 [cited by applicant]
US 20190385061A1 · Chaudhury · 2019 [cited by applicant]
“Multi-Label Image Classification with Neural Network,” Towards Data Science, Verma, pp. 1-2 (Year: 2019). [cited by examiner]
“Joint Event Extraction Based on Hierarchical Event Schemas From FrameNet,” IEEE, Li et al; Li (Year: 2019). [cited by examiner]
“Neural-Symbolic Learning Systems: Foundations and Applications,” Springer-Verlag Landan Ltd, d'Avila Garcez et al; d'Avila Garcez (Year: 2002). [cited by examiner]
“Generating Sequences With Recurrent Neural Networks,” arXiv, Graves (Year: 2014). [cited by examiner]
“Neural-Symvolic Learning Systems: Foudnations and Applications,” Springer-Verlag Landan Ltd, d'Avila Garcez et al; d'Avila Garcez (Year: 2002). [cited by examiner]
Angeli et al., “Leveraging Linguistic Structure for Open Domain Information Extraction”, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference… [cited by applicant]
Cote et al., “TextWorld: A Learning Environment for Text-based Games”, arXiv:1806.11532v2 [cs.LG] Nov. 8, 2019, 29 pages. [cited by applicant]
Coumans et al., “Pybullet Quickstart Guide”, 2016-2019, 87 pages, <http://pybullet.org>. [cited by applicant]
Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, arXiv:1810.04805v2 [cs.CL] May 24, 2019, 16 pages. [cited by applicant]
Gammulle et al., “Predicting the Future: A Jointly Learnt Model for Action Anticipation”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 27-Nov. 2, 2019, DOI: 10.1109/ICCV.2019.00566, Seoul, Kore… [cited by applicant]
Granroth-Wilding et al., “What Happens Next? Event Prediction Using a Compositional Neural Network Model”, Copyright © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org), 7 pages. [cited by applicant]
Hassanzadeh et al., “Answering Binary Causal Questions Through Large-Scale Text Mining: An Evaluation Using Cause-Effect Pairs from Human Experts”, Proceedings of the Twenty-Eighth International Joint Conference on Arti… [cited by applicant]
He et al., “Mask R-CNN”, 2017 IEEE International Conference on Computer Vision (ICCV), Oct. 22-29, 2017, DOI: 10.1109/ICCV.2017.322, Added to IEEE Xplore: Dec. 25, 2017, Conference Location: Venice, Italy, 9 pages. [cited by applicant]
Iten et al., “Discovering physical concepts with neural networks,” arXiv:1807.10300v3 [quant-ph] Jan. 23, 2020, 18 pages. [cited by applicant]
Johnson, et al., “CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21-26, 2017, Added to IEEE Xplore: … [cited by applicant]
Kitaev et al., “Constituency Parsing with a Self-Attentive Encoder”, arXiv:1805.01052v1 [cs.CL] May 2, 2018, 12 pages. [cited by applicant]
Mao et al., “The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision”, Published as a conference paper at ICLR 2019, arXiv:1904.12584v1 [cs.CV] Apr. 26, 2019, 28 pages. [cited by applicant]
Petruck, Miriam R. L., “Background to FrameNet”, International Computer Science Institute, LREC 2016, May 24, 2016, 67 pages. [cited by applicant]
Polceanu, Mihai, “ORPHEUS: Reasoning and Prediction with Heterogeneous rEpresentations Using Simulation”, Artificial Intelligence [cs.AI], Universite de Bretagne Occidentale (UBO), 2015. English, 128 pages. [cited by applicant]
Radinsky et al., “Learning to Predict from Textual Data”, Journal of Artificial Intelligence Research 45 (2012) 641-684, Submitted Nov. 2012; published Dec. 2012. [cited by applicant]
Rajani et al., “Explain Yourself! Leveraging Language Models for Commonsense Reasoning”, arXiv:1906.02361v1 [cs.CL] Jun. 6, 2019, 11 pages. [cited by applicant]
Ruppenhofer et al., “FrameNet II: Extended Theory and Practice”, Printed Sep. 14, 2010, 119 pages. [cited by applicant]
Schneider, et al., “The NLTK FrameNet API: Designing for Discoverability with a Rich Linguistic Resource”, Proceedings of the 2017 EMNLP System Demonstrations, pp. 1-6, Copenhagen, Denmark, Sep. 7-11, 2017, © 2017 Assoc… [cited by applicant]
Talmor et al., “CommonSenseQA: A Question Answering Challenge Targeting Commonsense Knowledge”, arXiv:1811.00937v2 [cs.CL] Mar. 15, 2019, 10 pages. [cited by applicant]
Yi et al., “CLEVRER: Collision Events for Video Representation and Reasoning”, Published as a conference paper at ICLR 2020, arXiv:1910.01442v2 [cs.CV] Mar. 8, 2020, 19 pages. [cited by applicant]
Yi et al., “Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding”, 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada, 12 pages. [cited by applicant]
Zolfaghari et al., “Learning Representations for Predicting Future Activities”, arXiv:1905.03578v1 [cs.CV] May 9, 2019, 14 pages. [cited by applicant]