IP Library › Granted Patent US 12,493,788
Granted Patent B2
US 12,493,788 · App. 17/313,555 · Granted Dec 9, 2025

Generating knowledge graphs from conversational data

Inventors: Shubhashis Sengupta (Bangalore, IN); Anutosh Maitra (Bangalore, IN); Roshni Ramesh Ramnani (Bangalore, IN); Zishan Ahmad (Patna, IN); Pushpak Bhattacharyya (Bihta, IN); Asif Ekbal (Patna, IN)
Assignee: Accenture Global Solutions Limited
G06N3/08G06N3/045G06N5/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,493,788
App. No.
17/313,555
Granted
Dec 9, 2025
Kind
B2
Abstract

Techniques for building knowledge graphs from conversational data are disclosed. The systems include a high-performance relation classifier developed with active learning and requiring minimal supervision. The classifier is used to classify relation triples extracted from conversational text, which are then used to populate the knowledge graph. A heuristic for constructing the knowledge graph is also disclosed. The proposed embodiments provide a way to efficiently build and/or augment knowledge graphs and improve the quality of the generated responses by a dialogue agent despite a sparsity of data.

Claims (71)

1 . A computer implemented method of relation classification for generating knowledge graphs, the method comprising:

extracting a first relation triple from conversational data including one or more entities, comprising:

for a first utterance in the conversational data,

in response to a determination that a speaker is a user, splitting the first utterance into one or more sentences,

for a first sentence of the one or more sentences, in response to a determination that a verb is absent in the first sentence, extracting the first relation triple using an intent classifier based triple extraction, and

for the first sentence of the one or more sentences, in response to a determination that a verb is present in the first sentence, extracting the first relation triple using a triple extraction which is different than the intent classifier based triple extraction;

marking each entity of the one or more entities in the first relation triple with Bidirectional Encoder Representations from Transformers (BERT) unused tokens to produce a first enriched sentence;

tokenizing, using a BERT tokenizer, the first enriched sentence with one or more tokens;

assigning token indices to each of the one or more tokens to produce an indexed sentence;

extracting a representation of a classification token (CLS) from the indexed sentence;

classifying the first enriched sentence into at least a first relation class based on the representation of the classification token;

populating a knowledge graph with the first relation triple based on the first relation class;

determining whether a second enriched sentence for a second relation triple in the conversational data is classified in the first relation class; and

replacing the first relation triple with the second relation triple in the populated knowledge graph in response to determining that the second enriched sentence is classified to be in the first relation class.

2 . The method of claim 1 , further comprising extracting any entities, relations, and entity-relation triples from the conversational data, including the first relation triple and the second relation triple.

3 . The method of claim 1 , wherein a BERT classifier is used to classify the first enriched sentence and the second enriched sentence.

4 . The method of claim 3 , further comprising training the BERT classifier on seed data comprising a plurality of relation triples and their corresponding utterances and a test set comprising annotated utterance samples.

5 . The method of claim 4 , further comprising:

running the trained BERT classifier on un-annotated data; and

obtaining prediction probabilities using final SoftMax layer of the BERT classifier.

6 . The method of claim 5 , further comprising:

calculating entropy values of each of the prediction probabilities; and

adding samples from the test set associated with a largest entropy value to the seed data.

7 . The method of claim 1 , further comprising:

detecting negated arguments in the conversational data using semantic role labelling (SRL);

identifying the second relation triple in the knowledge graph that includes entities matching any of the detected negated arguments or portions thereof; and

removing the second relation triple from the knowledge graph.

8 . A non-transitory computer-readable medium storing instructions executable by one or more computers which, upon such execution, cause the one or more computers to:

extract a first relation triple from conversational data including one or more entities, comprising:

for a first utterance in the conversational data,

in response to a determination that a speaker is a user, splitting the first utterance into one or more sentences,

for a first sentence of the one or more sentences, in response to a determination that a verb is absent in the first sentence, extracting the first relation triple using an intent classifier based triple extraction, and

for the first sentence of the one or more sentences, in response to a determination that a verb is present in the first sentence, extracting the first relation triple using a triple extraction which is different than the intent classifier based triple extraction;

mark each entity of the one or more entities in the first relation triple with Bidirectional Encoder Representations from Transformers (BERT) unused tokens to produce a first enriched sentence;

tokenize, using a BERT tokenizer, the first enriched sentence with one or more tokens;

assign token indices to each of the one or more tokens to produce an indexed sentence;

extract a representation of a classification token (CLS) from the indexed sentence;

classify the first enriched sentence into at least a first relation class based on the representation of the classification token; and

populate a knowledge graph with the first relation triple based on the first relation class;

determine whether a second enriched sentence for a second relation triple in the conversational data is classified in the first relation class; and

replace the first relation triple with the second relation triple in the populated knowledge graph in response to determining that the second enriched sentence is classified to be in the first relation class.

9 . The non-transitory computer-readable medium of claim 8 , wherein a BERT classifier is used to classify the first enriched sentence and the second enriched sentence.

10 . The non-transitory computer-readable medium of claim 9 , wherein the instructions further cause the one or more computers to train the BERT classifier on seed data comprising a plurality of relation triples and their corresponding utterances and a test set comprising annotated utterance samples.

11 . The non-transitory computer-readable medium of claim 10 , wherein the instructions further cause the one or more computers to:

run the trained BERT classifier on un-annotated data; and

obtain prediction probabilities using final SoftMax layer of the BERT classifier.

12 . The non-transitory computer-readable medium of claim 8 , wherein the instructions further cause the one or more computers to:

calculate entropy values of each of the prediction probabilities; and

add samples from a test set associated with a largest entropy value to seed data.

13 . A system for generating responses to queries, comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to:

extract a first relation triple from conversational data including one or more entities, comprising:

for a first utterance in the conversational data,

in response to a determination that a speaker is a user, splitting the first utterance into one or more sentences,

for a first sentence of the one or more sentences, in response to a determination that a verb is absent in the first sentence, extracting the first relation triple using an intent classifier based triple extraction, and

for the first sentence of the one or more sentences, in response to a determination that a verb is present in the first sentence, extracting the first relation triple using a triple extraction which is different than the intent classifier based triple extraction;

mark each entity of the one or more entities in the first relation triple with Bidirectional Encoder Representations from Transformers (BERT) unused tokens to produce a first enriched sentence;

tokenize, using a BERT tokenizer, the first enriched sentence with one or more tokens;

assign token indices to each of the one or more tokens to produce an indexed sentence;

extract a representation of a classification token (CLS) from the indexed sentence;

classify the first enriched sentence into at least a first relation class based on the representation of the classification token; and

populate a knowledge graph with the first relation triple based on the first relation class;

determine whether a second enriched sentence for a second relation triple in the conversational data is classified in the first relation class; and

replace the first relation triple with the second relation triple in the populated knowledge graph in response to determining that the second enriched sentence is classified to be in the first relation class.

14 . The system of claim 13 , wherein a BERT classifier is used to classify the first enriched sentence and the second enriched sentence.

15 . The system of claim 14 , wherein the instructions further cause the one or more computers to train the BERT classifier on seed data comprising a plurality of relation triples and their corresponding utterances and a test set comprising annotated utterance samples.

16 . The system of claim 15 , wherein the instructions further cause the one or more computers to:

run the trained BERT classifier on un-annotated data; and

obtain prediction probabilities using final SoftMax layer of the BERT classifier.

17 . The system of claim 16 , wherein the instructions further cause the one or more computers to:

calculate entropy values of each of the prediction probabilities; and

add samples from the test set associated with a largest entropy value to the seed data.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 10, 2022
From: AHMAD, ZISHAN; BHATTACHARYYA, PUSHPAK; EKBAL, ASIF
To: ACCENTURE GLOBAL SOLUTIONS LIMITED
Reel/Frame 058611/0681 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 6, 2021
From: SENGUPTA, SHUBHASHIS; MAITRA, ANUTOSH; RAMNANI, ROSHNI RAMESH
To: ACCENTURE GLOBAL SOLUTIONS LIMITED
Reel/Frame 056162/0303 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 6, 2021
From: AHMAD, ZISHAN; BHATTACHARYA, PUSHPAK; EKBAL, ASIF
To: ACCENTURE GLOBAL SOLUTIONS LIMITED
Reel/Frame 056162/0429 →
Priority Claims (1)
IN 202041049720 · Nov 13, 2020 · national
Continuity (1)
Related Publication 20220156582A1 · May 19, 2022
References Cited (41)
US 10678816B2 · Peng et al. · 2020 [cited by applicant]
US 11475310B1 · Teig · 2022 [cited by examiner]
US 20040059564A1 · Zhou · 2004 [cited by examiner]
US 20070073533A1 · Thione · 2007 [cited by examiner]
US 20160062980A1 · Boguraev · 2016 [cited by examiner]
US 20170024375A1 · Hakkani-Tur et al. · 2017 [cited by applicant]
US 20170288940A1 · Lagos · 2017 [cited by examiner]
US 20190317994A1 · Singh · 2019 [cited by examiner]
US 20200133962A1 · Kuo · 2020 [cited by examiner]
US 20210349925A1 · Panineerkandy · 2021 [cited by examiner]
US 20230367977A1 · Nagata · 2023 [cited by examiner]
Zhou et al., “Pre-trained Contextualized Representation for Chinese Conversation Topic Classification,” 2019, In 2019 IEEE International Conference on Intelligence and Security Informatics (ISI), pp. 122-127, doi.org/10… [cited by examiner]
Elhammadi et al., “A High Precision Pipeline for Financial Knowledge Graph Construction,” 2020, Proceedings of the 28th International Conference on Computational Linguistics, pp. 967-977 (Year: 2020). [cited by examiner]
Wu et al., “Enriching Pre-trained Language Model with Entity Information for Relation Classification,” 2019, arXiv:1905.08284v1 (Year: 2019). [cited by examiner]
Z. Ahmad et al., “Active Learning Based Relation Classification for Knowledge Graph Construction from Conversation Data”, ICONIP 2020, Bangkok Thailand, Nov. 18-22, 2020, Proceedings, Part IV ; https://www.researchgate.… [cited by applicant]
Agichtein, E., Gravano, L.: Snowball: Extracting relations from large plain-text collections. In: Proceedings of the fifth ACM conference on Digital libraries. pp. 85-94 (2000). [cited by applicant]
Angeli, G., Premkumar, M.J.J., Manning, C.D.: Leveraging linguistic structure for open domain information extraction. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7… [cited by applicant]
Bao, J., Duan, N., Yan, Z., Zhou, M., Zhao, T.: Constraint-based question answering with knowledge graph. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.… [cited by applicant]
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD international conference on Mana… [cited by applicant]
Bordes, A., Gabrilovich, E.: Constructing and mining web-scale knowledge graphs: Kdd 2014 tutorial. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 1967-1967 (… [cited by applicant]
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). [cited by applicant]
Dong, X., Gabrilovich, E., Heitz, G., Horn, W., Lao, N., Murphy, K., Strohmann, T., Sun, S., Zhang, W.: Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In: Proceedings of the 20th ACM SIGKDD int… [cited by applicant]
Graupmann, J., Schenkel, R., Weikum, G.: The SphereSearch engine for unified ranked retrieval of heterogeneous XML and Web documents. In: Proceedings of the 31st international conference on Very large data bases. pp. 52… [cited by applicant]
Guo, Q., Zhuang, F., Qin, C., Zhu, H., Xie, X., Xiong, H., He, Q.: A survey on knowledge graph-based recommender systems. arXiv preprint arXiv:2003.00911 (2020). [cited by applicant]
Hixon, B., Clark, P., Hajishirzi, H.: Learning knowledge graphs for question answering through conversational dialog. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computati… [cited by applicant]
Koncel-Kedziorski, R., Bekal, D., Luan, Y., Lapata, M., Hajishirzi, H.: Text generation from knowledge graphs with graph transformers. arXiv preprint arXiv:1904.02342 (2019). [cited by applicant]
Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., Van Kleef, P., Auer, S., et al.: DBpedia—a large-scale, multilingual knowledge base extracted from wikipedia. Sem… [cited by applicant]
Lenat, D.B.: CYC: A large-scale investment in knowledge infrastructure. Communications of the ACM 38(11), 33-38 (1995). [cited by applicant]
Lin, H., Yan, J., Qu, M., Ren, X.: Learning dual retrieval module for semisupervised relation extraction. In: The World Wide Web Conference. pp. 1073-1083 (2019). [cited by applicant]
Mitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., et al.: Never-ending learning. Communications of the ACM 61(5), 103-115 (2018). [cited by applicant]
Nakashole, N., Theobald, M., Weikum, G.: Scalable knowledge harvesting with high precision and high recall. In: Proceedings of the fourth ACM international conference on Web search and data mining. pp. 227-236 (2011). [cited by applicant]
Ng, V., Cardie, C.: Improving machine learning approaches to coreference resolution. In: Proceedings of the 40th annual meeting on association for computational linguistics. pp. 104-111. Association for Computational Li… [cited by applicant]
Peng, C.: A bert based relation classfication network for inter-personal relationship extraction. CCKS2019-shared task (2019). [cited by applicant]
Shi, P., Lin, J.: Simple BERT models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255 (2019). [cited by applicant]
Suchanek, F.M., Kasneci, G., Weikum, G.: YAGO: a core of semantic knowledge unifying Wordnet and Wikipedia. In: Proceedings of the 16th international conference on World Wide Web. pp. 697-706 (2007). [cited by applicant]
Turki, H., Shafee, T., Taieb, M.A.H., Aouicha, M.B., Vrandečić, D., Das, D.,Hamdi, H.: Wikidata: A large-scale collaborative ontological medical database. Journal of biomedical informatics 99, 103292 (2019). [cited by applicant]
Vu, T., Nguyen, T.D., Nguyen, D.Q., Phung, D., et al.: A capsule network-based embedding model for knowledge graph completion and search personalization. In:Proceedings of the 2019 Conference of the North American Chapt… [cited by applicant]
Wang, L., Cao, Z., De Melo, G., Liu, Z.: Relation classification via multi-level attention CNNs. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers). pp. 129… [cited by applicant]
Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S., Hitzler, P.: Quality assessment methodologies for linked open data. Submitted to Semantic Web Journal 15, 16 (2013). [cited by applicant]
Zeng, D., Liu, K., Lai, S., Zhou, G., Zhao, J., et al.: Relation classification via convolutional deep neural network (2014). [cited by applicant]
Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., Xu, B.: Attention-based bidirectional long short-term memory networks for relation classification. In: Proceedings of the 54th annual meeting of the association for … [cited by applicant]