IP Library › Granted Patent US 12,462,104
Granted Patent B2
US 12,462,104 · App. 18/420,131 · Granted Nov 4, 2025

Methods and apparatus for natural language understanding in conversational systems using machine learning processes

Inventors: Pratik Sridatt Jayarao (Pittsburgh, PA); Arpit Sharma (Santa Clara, CA); Deepa Mohan (San Jose, CA)
Assignee: Walmart Apollo, LLC
G06F40/295G06F40/211G06F40/284G06F40/56G06N3/04G06N20/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,462,104
App. No.
18/420,131
Granted
Nov 4, 2025
Kind
B2
Abstract

This application relates to apparatus and methods for natural language understanding in conversational systems using machine learning processes. In some examples, a computing device receives a request that identifies textual data. The computing device applies a natural language model to the textual data to generate first embeddings. In some examples, the natural language model is trained on retail data, such as item descriptions and chat session data. The computing device also applies a dependency based model to the textual data to generate second embeddings. Further, the computing device concatenates the first and second embeddings, and applies an intent and entity classifier to the concatenated embeddings to determine entities, and an intent, for the request. The computing device may generate a response to the request based on the determined intent and entities.

Claims (63)

1 . A system comprising:

a processor; and

a non-transitory memory storing instructions that, when executed, cause the processor to:

receive input data comprising a plurality of characters;

apply a first machine learning model to the input data to generate first output embeddings based at least in part by:

tokenizing the input data into a plurality of tokens,

generating a sequence output for each token of the plurality of tokens and a pooled output for a first token of the plurality of tokens,

applying a first linear layer to the sequence output to generate the first output embedding for each token of the plurality of tokens, and

applying a second linear layer to the pooled output to generate an intent embedding;

apply, in parallel with the first machine learning, a second machine learning model to the input data to generate second output embeddings, wherein the second machine learning model comprises:

an embedding model configured to generate syntactic dependencies based on the input data,

a single layer transformer encoder configured to generate encoded data from the syntactic dependencies, and

a linear neural network configured to generate the second output embeddings based on the encoded data;

generate concatenated embeddings by a stepwise concatenation of the first output embeddings and the second output embeddings; and

apply a classifier model to the concatenated embeddings and the intent embedding to generate classification data.

2 . The system of claim 1 , wherein the classification data are stored in a database communicatively coupled to the processor and characterize at least one entity and an intent of the plurality of characters.

3 . The system of claim 1 , wherein:

the input data is received in a request from a second computing device; and

the processor is configured to generate a response to the request based on the classification data.

4 . The system of claim 1 , wherein the instructions, when executed, further cause the processor to:

train the first machine learning model based on catalog data identifying one or more attributes of a plurality of items and chat session data.

5 . The system of claim 1 , wherein the first machine learning model includes a softmax function.

6 . The system of claim 1 , wherein the second machine learning model comprises a two-layered neural network configured to receive a current intent.

7 . A method comprising:

receiving input data comprising a plurality of characters;

applying a first machine learning model to the input data to generate first output embeddings based at least in part by:

tokenizing the input data into a plurality of tokens,

generating a sequence output for each token of the plurality of tokens and a pooled output for a first token of the plurality of tokens,

applying a first linear layer to the sequence output to generate the first output embedding for each token of the plurality of tokens, and

applying a second linear layer to the pooled output to generate an intent embedding;

applying, in parallel with the first machine learning, a second machine learning model to the input data to generate second output embeddings, wherein the second machine learning model comprises:

an embedding model configured to generate syntactic dependencies based on the input data,

a single layer transformer encoder configured to generate encoded data from the syntactic dependencies, and

a linear neural network configured to generate the second output embeddings based on the encoded data;

generating concatenated embeddings by a stepwise concatenation of the first output embeddings and the second output embeddings; and

applying a classifier model to the concatenated embeddings and the intent embedding to generate classification data.

8 . The method of claim 7 , wherein the classification data are stored in a database and characterize at least one entity and an intent of the plurality of characters.

9 . The method of claim 7 , wherein:

the input data is received in a request from a second computing device; and

the method further comprises generating a response to the request based on the classification data.

10 . The method of claim 9 , further comprising:

training the first machine learning model based on catalog data identifying one or more attributes of a plurality of items and chat session data.

11 . The method of claim 7 , wherein the first machine learning model includes a softmax function.

12 . The method of claim 7 , wherein the second machine learning model comprises a two-layered neural network configured to receive a current intent.

13 . A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause a device to perform operations comprising:

receiving input data comprising a plurality of characters;

applying a first machine learning model to the input data to generate first output embeddings based at least in part by:

tokenizing the input data into a plurality of tokens,

generating a sequence output for each token of the plurality of tokens and a pooled output for a first token of the plurality of tokens,

applying a first linear layer to the sequence output to generate the first output embedding for each token of the plurality of tokens, and

applying a second linear layer to the pooled output to generate an intent embedding;

applying, in parallel with the first machine learning, a second machine learning model to the input data to generate second output embeddings, wherein the second machine learning model comprises:

an embedding model configured to generate syntactic dependencies based on the input data,

a single layer transformer encoder configured to generate encoded data from the syntactic dependencies, and

a linear neural network configured to generate the second output embeddings based on the encoded data;

generating concatenated embeddings by a stepwise concatenation of the first output embeddings and the second output embeddings; and

applying a classifier model to the concatenated embeddings and the intent embedding to generate classification data.

14 . The non-transitory computer readable medium of claim 13 , wherein the classification data are stored in a database and characterize at least one entity and an intent of the plurality of characters.

15 . The non-transitory computer readable medium of claim 13 , wherein:

the input data is received in a request from a second computing device; and

the operations further comprise generating a response to the request based on the classification data.

16 . The non-transitory computer readable medium of claim 13 , wherein the instructions, when executed by the at least one processor, cause the device to further perform operations comprising:

training the first machine learning model based on catalog data identifying one or more attributes of a plurality of items and chat session data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 23, 2024
From: JAYARAO, PRATIK SRIDATT; SHARMA, ARPIT; MOHAN, DEEPA
To: WALMART APOLLO, LLC
Reel/Frame 066218/0445 →
Continuity (2)
Continuation 17187762 · Feb 27, 2021
Related Publication 20240160846A1 · May 16, 2024
References Cited (53)
US 7092871B2 · Pentheroudakis et al. · 2006 [cited by applicant]
US 8620836B2 · Ghani et al. · 2013 [cited by applicant]
US 11144726B2 · Chatterjee · 2021 [cited by examiner]
US 20180121788A1 · Hashimoto · 2018 [cited by examiner]
US 20180232438A1 · Gu et al. · 2018 [cited by applicant]
US 20200050949A1 · Sundaraman et al. · 2020 [cited by applicant]
US 20200143265A1 · Jonnalagadda · 2020 [cited by examiner]
US 20200265116A1 · Chatterjee · 2020 [cited by examiner]
US 20200311198A1 · Poon et al. · 2020 [cited by applicant]
US 20200388396A1 · Lindvall · 2020 [cited by applicant]
US 20200410337A1 · Huang et al. · 2020 [cited by applicant]
US 20210117623A1 · Aly et al. · 2021 [cited by applicant]
US 20210133535A1 · Zhao · 2021 [cited by applicant]
US 20210142181A1 · Liu et al. · 2021 [cited by applicant]
US 20210255862A1 · Volkovs · 2021 [cited by examiner]
US 20220067280A1 · Wagner · 2022 [cited by examiner]
US 20220092267A1 · Hou et al. · 2022 [cited by applicant]
US 20220129633A1 · Shah et al. · 2022 [cited by applicant]
US 20220164626A1 · Bird et al. · 2022 [cited by applicant]
US 20220198327A1 · Wang et al. · 2022 [cited by applicant]
US 20220229993A1 · Vu et al. · 2022 [cited by applicant]
US 20220237377A1 · Zhang et al. · 2022 [cited by applicant]
CN 111640425A · 2019 [cited by applicant]
CN 112613326B · 2022 [cited by applicant]
Wu, S., Zhang, D., Zhang, Z., Yang, N., Li, M., & Zhou, M. (2018). Dependency-to-dependency neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(11), 2132-2141. (Year: 2018). [cited by examiner]
A. Vaswani et al., “Attention is all you Need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), Year: 2017, 15 pages. [cited by applicant]
S. Wu et al., “Dependency-to-Dependency Neural Machine Translation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 36, No. 11, Nov. 2018, 10 pages. [cited by applicant]
H. Tang et al., “Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul. 5-10, 2020,… [cited by applicant]
Spacy, “SpacCy 101: Everything you need to know,” Retrieved Feb. 29, 2020 via Wayback Machine, Explosion (Year: 2020) 44 pages. [cited by applicant]
Z. Xingran “Cross-lingual Word Sense Disambiguation using mBERT Embeddings with Syntactic Dependencies,” Department of Linguistics and Philology, Uppsala University, arXiv:2012.05300v1 [cs.CL], Dec. 9, 2020, 9 pages. [cited by applicant]
R. Cui et al., “Tweets can tell: activity recognition using hybrid gated recurrent neural networks,” Social Network Analysis and Mining (2020) 10:16, https://doi.org/10.107/s13278-020-0628-0, (Year: 2020), 15 pages. [cited by applicant]
Z. Jie et al., “Dependency-Guided LSTM-CRF for Named Entity Recognition,” StatNLP Research Group, Singapore University of Technology and Design, arXiv:1909.10148v1 [cs.CL], Sep. 23, 2019, 13 pages. [cited by applicant]
T. Bunk et al., “DIET: Lightweight Language Understanding for Dialogue Systems,” arXiv:2004.09936v3 [cs.CL], May 11, 2020, 9 pages. [cited by applicant]
Google Research, “Use BERT fine-tuned model for Tensorflow serving #146,” Nov. 19, 2018, 47 pages. [cited by applicant]
N. Zivkovic, “Deploying Machine Learning Models, pt. 3:gRPC and TensorFlow Serving,” Rukiscode.net, Feb. 24, 2020, 17 pages. [cited by applicant]
S. MacAvaney et al., “A Deeper Look into Dependency-Based Word Embeddings,” Proceedings of NAACL-HLT 2018: Student Research Workshop, New Orleans, Louisiana, Jun. 2-4, 2018 pp. 40-45. [cited by applicant]
D. Sundararaman et al., “Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding,” Department of Electrical and Computer Engineering, Duke University, Durham, NC, ArXiv:1911… [cited by applicant]
Q. Liu et al., “A Survey on Contextual Embeddings,” arXiv:2003.072782v2 [cs.CL], Apr. 13, 2020, 13 pages. [cited by applicant]
R. Horev, “Bert: State of the Art NPL Model, Explained,” https://www.kdnuggets.com/2018/12/bert-sota-nlp-model-explained.html, (Year: 2018), 8 pages. [cited by applicant]
S. Mukherjee et al., “Discriminative Pre-training for Low Resource Title Compression in Conversational Grocery,” SIGIR eCom'20, Jul. 30, 2020, Virtual Event, China, 7 pages. [cited by applicant]
E. Alsentzer et al., “Publicly Available Clinical BERT Embeddings,” https://arxiv.org/abs/1904.03323, Jun. 20, 2019, 7 pages. [cited by applicant]
I. Beltagy et al., “SCIBERT: A Pretrained Language Model for Scientific Text,” https://arxiv.org/abs/1903.10676, Sep. 10, 2019, 6 pages. [cited by applicant]
J. Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” Proceeding of NAACL-HTL 2019, Jun. 2-7, 2019, pp. 4171-4186. [cited by applicant]
E. Elwany et al., “BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding,” https://arxiv.org/abs/1911.00473, Nov. 1, 2019, 4 pages. [cited by applicant]
Google, Inc., “TensorFlow code for pre-trained models for BERT,” Mar. 11, 2020, 15 pages. [cited by applicant]
J. Lee et al., “BioBERT: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, (Year: 2019), pp. 1-7. [cited by applicant]
O. Levy et al., “Dependency-Based Word Embeddings,” Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Jun. 23-25, 2014, pp. 302-308. [cited by applicant]
D. Mohan, “Joint Intent Classification and Entity Recognition for Conversational Commerce,” Walmart Global Tech Blog, Dec. 11, 2019, 10 pages. [cited by applicant]
S. Rongali et al., “Continual Domain-Tuning for Pretrained Language Models,” https://arxiv.org/abs/2004.02288, Mar. 19, 2021, 15 pages. [cited by applicant]
V. Sanh et al., “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” https://arxiv.org/abs/1910.01108, Mar. 1, 2020, 5 pages. [cited by applicant]
C. Sung et al., “Pre-Training BERT on Domain Resources for Short Answer Grading,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu… [cited by applicant]
A. Vaswani et al., “Attention Is All You Need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), (Year: 2017), 11 pages. [cited by applicant]
Y. You et al., “Large Batch Optimization for Deep Learning: Training Bert in 76 Minutes,” https://arxiv.org/abs/1904.00962, Jan. 3, 2020, 37 pages. [cited by applicant]