IP Library › Granted Patent US 12,235,790
Granted Patent B2
US 12,235,790 · App. 17/670,368 · Granted Feb 25, 2025

Deep neural network-based decision network

Inventors: Alexander Rosenberg Johansen (San Francisco, CA); Bryan McCann (San Francisco, CA); James Bradbury (San Francisco, CA); Richard Socher (Menlo Park, CA)
Assignee: Salesforce, Inc.
G06F15/76G06F18/217G06F18/24G06F18/241G06F18/24133G06F40/169G06F40/30G06N3/044G06N3/045G06N3/047G06N3/048G06N3/084G06N5/04G06N20/00G06V10/764G06V10/776G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,235,790
App. No.
17/670,368
Granted
Feb 25, 2025
Kind
B2
Abstract

The technology disclosed proposes using a combination of computationally cheap, less-accurate bag of words (BoW) model and computationally expensive, more-accurate long short-term memory (LSTM) model to perform natural processing tasks such as sentiment analysis. The use of cheap, less-accurate BoW model is referred to herein as “skimming”. The use of expensive, more-accurate LSTM model is referred to herein as “reading”. The technology disclosed presents a probability-based guider (PBG). PBG combines the use of BoW model and the LSTM model. PBG uses a probability thresholding strategy to determine, based on the results of the BoW model, whether to invoke the LSTM model for reliably classifying a sentence as positive or negative. The technology disclosed also presents a deep neural network-based decision network (DDN) that is trained to learn the relationship between the BoW model and the LSTM model and to invoke only one of the two models.

Claims (19)

1. A system for performing a selection task using a decision neural network-based classifier, comprising: a memory storing a plurality of processor-executable instructions for training and operating the decision neural network-based classifier; and a processor that reads the plurality of processor-executable instructions from the memory to perform operations comprising: constructing a decision training set for the decision neural network-based classifier by: identifying, based on a confusion matrix generated from performance of a non-recurrent network-based classifier and a recurrent neural network-based classifier, a first subset of inputs inferred by the recurrent neural network-based classifier and a second subset of inputs comprising inputs not in the first subset, and including, in the decision training set, the first subset of inputs labeled with a first model class label identifying the recurrent neural network-based classifier and the second subset of inputs labeled with a second model class label identifying the non-recurrent neural network-based classifier; generating, by the decision neural network based classifier, a plurality of selection outputs in response to inputs from the decision training set; computing a training objective by comparing the plurality of selection outputs with labels from the first subset and the second subset; and updating the decision neural network based classifier based on the training objective wherein the operations further comprise: training the non-recurrent and recurrent neural network-based classifiers to perform the machine classification task using a training set, the training set comprising training inputs annotated with task class labels defined for the machine classification task; using the trained non-recurrent and recurrent neural network-based classifiers to perform the machine classification task on a validation set, the validation set comprising validation inputs annotated with the task class labels; and

training the decision neural network-based classifier using the decision training set to output probabilities for the first and second model class labels on an input-by-input basis, the output probabilities specifying respective likelihoods of selecting the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier.

2. The system of claim 1 , wherein the operations further comprise: at an inference stage, selecting, by the decision neural network-based classifier, on an input-by-input basis, between the non-recurrent neural network-based classifier and the recurrent neural network-based classifier to perform a machine classification task; and based on output probabilities of the trained decision neural network-based classifier, performing the machine classification task on the given input using either the trained recurrent neural network-based classifier or the trained non-recurrent neural network-based classifier.

3. The system of claim 2 , wherein the operation further comprise: selecting the trained recurrent neural network-based classifier when an output probability associated with the first model class label is higher than that of the second model class label; or selecting the trained non-recurrent neural network-based classifier when an output probability associated with the second model class label is higher than that of the first model class label.

4. The system of claim 1 , wherein the non-recurrent network-based classifier and the recurrent neural network-based classifier are separately trained on the decision training set.

5. The system of claim 1 , wherein the decision training set comprises a plurality of sentences annotated with a sentiment label.

6. The system of claim 1 , wherein the confusion matrix identifies at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier; and validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.

7. The system of claim 1 , wherein the first subset includes validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.

8. The system of claim 1 , wherein the second subset includes at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; and validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier.

9. The system of claim 1 , wherein an operation of updating the decision neural network based classifier comprises: backpropagating gradients only for fully-connected layers and a new classification layer while keeping weights of the trained non-recurrent neural network-based classifier fixed.

10. A method for performing a selection task using a decision neural network-based classifier, the method comprising: constructing a decision training set for the decision neural network-based classifier by: identifying, based on a confusion matrix generated from performance of a non-recurrent network-based classifier and a recurrent neural network-based classifier, a first subset of inputs inferred by the recurrent neural network-based classifier and a second subset of inputs comprising inputs not in the first subset, and including, in the decision training set, the first subset of inputs labeled with a first model class label identifying the recurrent neural network-based classifier and the second subset of inputs labeled with a second model class label identifying the non-recurrent neural network-based classifier; generating, by the decision neural network based classifier, a plurality of selection outputs in response to inputs from the decision training set; computing a training objective by comparing the plurality of selection outputs with labels from the first subset and the second subset; and updating the decision neural network based classifier based on the training objective; training the non-recurrent and recurrent neural network-based classifiers to perform the machine classification task using a training set, the training set comprising training inputs annotated with task class labels defined for the machine classification task; using the trained non-recurrent and recurrent neural network-based classifiers to perform the machine classification task on a validation set, the validation set comprising validation inputs annotated with the task class labels; and training the decision neural network-based classifier using the decision training set to output probabilities for the first and second model class labels on an input-by-input basis, the output probabilities specifying respective likelihoods of selecting the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier.

11. The method of claim 10 , further comprising: at an inference stage, selecting, by the decision neural network-based classifier, on an input-by-input basis, between the non-recurrent neural network-based classifier and the recurrent neural network-based classifier to perform a machine classification task; and based on output probabilities of the trained decision neural network-based classifier, performing the machine classification task on the given input using either the trained recurrent neural network-based classifier or the trained non-recurrent neural network-based classifier.

12. The method of claim 11 , further comprising: selecting the trained recurrent neural network-based classifier when an output probability associated with the first model class label is higher than that of the second model class label; or selecting the trained non-recurrent neural network-based classifier when an output probability associated with the second model class label is higher than that of the first model class label.

13. The method of claim 10 , wherein the non-recurrent network-based classifier and the recurrent neural network-based classifier are separately trained on the decision training set.

14. The method of claim 10 , wherein the decision training set comprises a plurality of sentences annotated with a sentiment label.

15. The method of claim 10 , wherein the confusion matrix identifies at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.

16. The method of claim 10 , wherein the first subset includes validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier, and the second subset includes at least one of: validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; and validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier.

17. The method of claim 10 , wherein updating the decision neural network based classifier comprises: backpropagating gradients only for fully-connected layers and a new classification layer while keeping weights of the trained non-recurrent neural network-based classifier fixed.

18. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for performing a selection task using a decision neural network-based classifier, the instructions being executed by a processor to perform operations comprising: constructing a decision training set for the decision neural network-based classifier by: identifying, based on a confusion matrix generated from performance of a non-recurrent network-based classifier and a recurrent neural network-based classifier, a first subset of inputs inferred by the recurrent neural network-based classifier and a second subset of inputs comprising inputs not in the first subset, and including, in the decision training set, the first subset of inputs labeled with a first model class label identifying the recurrent neural network-based classifier and the second subset of inputs labeled with a second model class label identifying the non-recurrent neural network-based classifier; generating, by the decision neural network based classifier, a plurality of selection outputs in response to inputs from the decision training set; computing a training objective by comparing the plurality of selection outputs with labels from the first subset and the second subset; and updating the decision neural network based classifier based on the training objective; training the non-recurrent and recurrent neural network-based classifiers to perform the machine classification task using a training set, the training set comprising training inputs annotated with task class labels defined for the machine classification task; using the trained non-recurrent and recurrent neural network-based classifiers to perform the machine classification task on a validation set, the validation set comprising validation inputs annotated with the task class labels; and training the decision neural network-based classifier using the decision training set to output probabilities for the first and second model class labels on an input-by-input basis, the output probabilities specifying respective likelihoods of selecting the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier.

Assignments (2)
CHANGE OF NAME Recorded Aug 4, 2026
From: SALESFORCE.COM, INC.
To: SALESFORCE, INC.
Reel/Frame 076118/0548 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 2, 2022
From: JOHANSEN, ALEXANDER ROSENBERG; MCCANN, BRYAN; BRADBURY, JAMES; SOCHER, RICHARD
To: SALESFORCE.COM, INC.
Reel/Frame 060978/0957 →
Continuity (3)
Continuation 15853570 · Dec 22, 2017
Provisional Application 62471934 · Mar 15, 2017
Related Publication 20220164635A1 · May 26, 2022
References Cited (101)
US 8121367B2 · Socher et al. · 2012 [cited by applicant]
US 8355550B2 · Zhang et al. · 2013 [cited by applicant]
US 9224386B1 · Weber · 2015 [cited by examiner]
US 9436895B1 · Jones et al. · 2016 [cited by applicant]
US 9489630B2 · Achin · 2016 [cited by examiner]
US 9772998B2 · Anisimovich · 2017 [cited by examiner]
US 9836484B1 · Bialynicka-Birula et al. · 2017 [cited by applicant]
US 10282663B2 · Socher et al. · 2019 [cited by applicant]
US 10346721B2 · Albright et al. · 2019 [cited by applicant]
US 20130018824A1 · Ghani et al. · 2013 [cited by applicant]
US 20140156568A1 · Ganguly · 2014 [cited by examiner]
US 20160350653A1 · Socher et al. · 2016 [cited by applicant]
US 20170024645A1 · Socher et al. · 2017 [cited by applicant]
US 20170032280A1 · Socher · 2017 [cited by applicant]
US 20170052971A1 · Boyer et al. · 2017 [cited by applicant]
US 20170140240A1 · Socher · 2017 [cited by applicant]
US 20180032526A1 · Cudak et al. · 2018 [cited by applicant]
US 20180082171A1 · Merity et al. · 2018 [cited by applicant]
US 20180082197A1 · Aravamudan et al. · 2018 [cited by applicant]
US 20180089553A1 · Liu et al. · 2018 [cited by applicant]
US 20180096219A1 · Socher · 2018 [cited by applicant]
US 20180121787A1 · Hashimoto et al. · 2018 [cited by applicant]
US 20180121788A1 · Hashimoto et al. · 2018 [cited by applicant]
US 20180121799A1 · Hashimoto et al. · 2018 [cited by applicant]
US 20180129931A1 · Bradbury et al. · 2018 [cited by applicant]
US 20180129937A1 · Bradbury et al. · 2018 [cited by applicant]
US 20180129938A1 · Xiong et al. · 2018 [cited by applicant]
US 20180143966A1 · Lu et al. · 2018 [cited by applicant]
US 20180144208A1 · Lu et al. · 2018 [cited by applicant]
US 20180144248A1 · Lu et al. · 2018 [cited by applicant]
US 20180247636A1 · Arik et al. · 2018 [cited by applicant]
US 20180268287A1 · Johansen et al. · 2018 [cited by applicant]
US 20180268298A1 · Johansen et al. · 2018 [cited by applicant]
US 20180300400A1 · Paulus · 2018 [cited by applicant]
US 20190130896A1 · Zhou et al. · 2019 [cited by applicant]
WO 2018170175A1 · 2018 [cited by applicant]
Mahasseni et al., Budget-aware Deep Semantic, Jul. 26, 2017, IEEE Conference on Computer Vision and Pattern Recognition (pp. 1029-1038). (Year: 2017). [cited by examiner]
“Baluja et al., Adversarial Transformation Networks: Learning to Generate Adversarial Examples, Mar. 28, 2017, In International Conference on Computer Analysis of Images and Patterns (pp. 96-108)” (Year: 2017). [cited by examiner]
“Windasari et al., Sentiment Analysis on Twitter Posts: An analysis of Positive or Negative Opinion on GoJek, Oct. 9, 2017, In 2017 4th international conference on information technology, computer, and electrical engine… [cited by examiner]
“Arlot et al., A survey of cross-validation procedures for model selection, 2010, Statistics Surveys, vol. 4, pp. 40-79” (Year: 2010). [cited by examiner]
“Vo et al., Multi-channel LSTM-CNN model for Vietnamese sentiment analysis, Oct. 2017, 2017 9th International Conference on Knowledge and Systems Engineering(KSE), pp. 24-29” (Year: 2017). [cited by examiner]
“Sheikh et al., Learning Word Importance with the Neural-Bag-of-Words Model, 2016, In Proceedings of the 1st Workshop on Representation Learning for NLP (pp. 222-229).” (Year: 2016). [cited by examiner]
“Dos Santos et al., Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts, 2014, Proceedings of COLING 2014: Technical Papers, pp. 49-78” (Year: 2014). [cited by examiner]
“Wang et al., Combination of Convolutional and Recurrent Neural Network Sentiment Analysis of Short Texts, 2016, Proceedings of COLING 2016: Technical Papers, pp. 2428-2437” (Year: 2016). [cited by examiner]
“Bradbury et al., Quasi Recurrent Neural Networks, Nov. 5, 2016, arXiv preprint arXiv:1611.01576.” (Year: 2016). [cited by examiner]
“Kohavi et al., A study of cross-validation and bootstrap for accuracy estimation and model selection, 1995, I(nternational Joint Conference on Artificial Intelligence (IJCAI)” (Year: 1995). [cited by examiner]
“Hockey et al., Comparison of Grammar-Based and Statistical Language Models trained on the Same Data, 2005, In Proceedings of the AAAI Workshop on Spoken Language Understanding” (Year: 2005). [cited by examiner]
“Kucuktunc et al., A large-Scale Sentiment Analysis for Yahoo! Answers), 2012, In Proceedings of the fifth ACM international conference on Web search and data mining, pp. 633-642.” (Year: 2012). [cited by examiner]
“Nam et al., Learning Multi-Domain Convolutional Neural Networks for Visual Tracking, 2016, Proceedings of the IEEE conference on computer vision and pattern recognition” (Year: 2016). [cited by examiner]
“Nogueira et al., Towards better exploiting convolutional neural networks for remote sensing scene classification, 2016, Pattern Recognition, vol. 61, pp. 539-556” (Year: 2016). [cited by examiner]
“Yin et al., Comparative study of CNN and RNN for Natural Language Processing, Feb. 7, 2017, arXiv preprint arXiv:1702.01923.” (Year: 2017). [cited by examiner]
“Collobert et al., A unified architecture for natural language processing: deep neural networks with multitask learning, 2008, InProceedings of the 25th international conference on Machine learning Jul. 5, 2008 (pp. 160… [cited by examiner]
“Koo et al., Simple Semi-dependency parsing, 2008, Proceeding of ACL-08, pp. 595-603” (Year: 2008). [cited by examiner]
“Deng et al., Machine Learning Paradigms for Speech Recognition: An Overview, 2013, IEEE Transactions on Audio, Speech, and Language Processing, 21(5), pp. 1060-1089.” (Year: 2013). [cited by examiner]
“Gambhir et al., Recent automatic text summarization techniques: a survey, Jan. 2017, Artificial Intelligence Review, 47(1), pp. 1-66.” (Year: 2017). [cited by examiner]
“Andreas et al., Learning to Compose Neural Networks for Question Answering, 2016, arXiv preprint arXiv:1601.01705” (Year: 2016). [cited by examiner]
“You et al., Image captioning with semantic attention, 2016, In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4651-4659).” (Year: 2016). [cited by examiner]
“Wang et al., Tacotron: A fully end-to-end Text-to-Speech synthesis model, Mar. 2017, arXiv preprint arXiv:1703.10135, 164.” (Year: 2017). [cited by examiner]
“Barry et al., Sentiment Analysis of Online Review Using Bag-of-Words and LSTM Approaches, Dec. 7, 2017,In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4651-4659).” (Year: 2017). [cited by examiner]
Billy Peralta, “Embedded local Feature Selection within Mixture of Experts”, 2013 (Year: 2013). [cited by examiner]
Ruck et al., “The Multilayer Perception as an Approximation to a Bayes Optimal Discriminant Function,” IEEE Transactions on Neural Networks, vol. 1, No. 4, Dec. 1990. pp. 1-3. [cited by applicant]
Schuster et al., “Bidirectional Recurrent Neural Networks,” IEEE Transactions of Signal Processing, vol. 45, No. 11, Nov. 1997. pp. 1-9. [cited by applicant]
Sennrich et al., “Neural Machine Translation of Rare Words with Subword Units,” Research Presented in this Publication was Conducted in Cooperation with Samsung Electronics Polska sp. zo.o.—Samsung R&D Institute, Poland… [cited by applicant]
Shazeer et al., “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” https://arxiv.org/abs/1701.06538, Jan. 23, 2017. pp. 1-19. [cited by applicant]
Silver et al., “Mastering the Game of Go with Deep Neural Networks and Tree Search,” Nature, 529(7587):484-487, doi: 10.1038/nature16961. Jan. 2016. pp. 1-37. [cited by applicant]
Socher, Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Oct. 21, 2013 (Year: 2013). [cited by applicant]
Srivastava et al., “Dropout: A Simple Way to Prevent Neural Networks from Overfilling,” Journal of Machine Learning Research. University of Toronto. Jun. 14, 2014. pp. 1-30. [cited by applicant]
Stalidis, Machine Learning Sentiment Prediction based on Hybrid Document Representation, arxiv.org, Nov. 29, 2015 (Year: 2015). [cited by applicant]
Stuner, Cascading BLSTM Networks For Handwritten Word Recognition, 2016 23rd International Conference on Pattern Recognition (ICPR), Dec. 8, 2016 (Year: 2016). [cited by applicant]
Tai, K.S. et al., “Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks”, arXiv.org, v2, pp. 1-11, May 30, 2015. [cited by applicant]
Tang, D. et al., “Document Modeling with Gated Recurrent Neural Network for Sentiment Classification”, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1422-1432, Sep. 21, 2015. [cited by applicant]
Van Der Maaten et al., “Visualizing Data Using T-SNE,” Journal of Machine Leaming Research, (2579-2605). Nov. 9, 2008. pp. 1-27. [cited by applicant]
Wu et al., “Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine ranslation,” URL: http://arxiv.org/abs/1609.08144. eprint arXiv:1609.08144. Oct. 8, 2016. pp. 1-23. [cited by applicant]
Wu, Aspect-based Opinion Summarization with Convolutional Neural Networks, 2016 IEEE (Year: 2016). [cited by applicant]
Zoltan Tuske, Integrating Gaussian mixtures into deep neural networks: Softmax layer with hidden variables, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Year: 2015). [cited by applicant]
A. Almahairi et al: “Dynamic capacity networks”, Proceedings of Machine Learning Research, vol. 48, Jun. 22, 2016 (Jun. 22, 2016), pp. 2549-2558, XP055479670. [cited by applicant]
Abdelazeem, A Greedy Approach for Building Classification Cascades, 2008 Seventh International Conference on Machine Learning and Applications (Year: 2008). [cited by applicant]
Amodel et al., “Deep Speech 2: End-to-End Speech Recognition in English and Mandarin,” Published as a Conference Paper at the International Conference on Machine Leaming. CoRR, abs/1512.02595. URL: http://arxiv.orgbs/15… [cited by applicant]
Bradbury, Quasi-Recurrent Neural Networks, arxiv.org, Nov. 21, 2016 (Year: 2016). [cited by applicant]
Canadian Examiner's Report for App. No. 3,052,212 dated Nov. 9, 2020, 7 pages. [cited by applicant]
D. Krueger et al: “Zoneout: regularizing RNNs by randomly preserving hidden activations” , arXiv:1606.01305v2, Jun. 13, 2016 (Jun. 13, 2016), XP080705872. [cited by applicant]
Figurnov et al., “Spatially Adaptive Computation Time for Residual Networks,” Conference on Computer Vision and Pattern Recognition (CVPR), 2017. CoRR, abs/1612.02297, 2016. URL: http://arxiv.org/abs/1612.02297. Jul. 2,… [cited by applicant]
Gers et al., “Learning to Forget: Continual Prediction With LSTM,” Neural Computation, 12(10):2451-2471. ISSN 0899-7667. Oct. 2000. pp. 1-19. [cited by applicant]
Graves et al., “Adaptive Computational Time for Recurrent Neural Networks,” Corr, abs/1603.08983, 2016. URL: http:/arxiv.org/abs/1603.08983. Feb. 21, 2017. pp. 1-19. [cited by applicant]
Graves et al., “Supervised Sequence Labelling with Recurrent Neural Networks,” Springer Berlin Heidelberger, 2012, pp. 1-137. [cited by applicant]
Hannun et al., “Deep Speeh: Scaling Up End to End Speech Recognition,” CoRR, abs/1412.5567, 2014. URL: http:/arxiv.org/abs/1412.5567. Dec. 19, 2014. pp. 1-12. [cited by applicant]
He et al., “Deep Residual Leaming for Image Recognition,” CoRR, abs/1512.03385. URL: http://arxiv.org/abs/1512.03385. Dec. 10, 2015. pp. 1-12. [cited by applicant]
He et al., “Identity Mappings in Deep Residual Networks,” ECCV: European Conference on Computer Vision, Jul. 25, 2016. pp. 1-15. [cited by applicant]
Hochreiter et al., “Long Short-Term Memory: Neural Computation,” Neural Computation, vol. 9, Issue 8, Nov. 15, 1997, pp. 1-32. [cited by applicant]
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US18/22504, mailed on Jun. 28, 2019, 24 pages. [cited by applicant]
International Search Report and Written Opinion dated Aug. 10, 2018 issued in related PCT App. No. PCT/US2018/022504 (16 pages). [cited by applicant]
Jacobs et al., “Adaptive Mixtures of Local Experts,” Neural Computation 3, 79-87, 1991. Massachusetts Institute of Technology, pp. 1-49. [cited by applicant]
Jordan et al., “Hierarchical Mixtures of Experts and the EM Algorithm,” Proceedings of the 1993 International Joint Conference on Neural Networks. pp. 1-6. [cited by applicant]
Kingma et al., “ADAM: A Method for Stochastic Optimization,” Published as a conference paper at International Conference on Learning Representations (ICLR). Jan. 30, 2017. arXiv:1412.6980. pp. 1-15. [cited by applicant]
L. Denoyer et al., “Deep sequential neural network,” arXiv:1410.0510v1, Oct. 2, 2014, XP055469696, Retrieved from the Internet: URL:https://arxiv.org/abs/1410.0510v.1 [retrieved on Jun. 22, 2018], pp. 1-9 (9 pages). [cited by applicant]
L. Liu et al., “Dynamic deep neural networks: optimizing accuracy-efficiency trade-offs by selective execution,” arXiv:1701.00299v1, Jan. 2, 2017, XP080739001, Retrieved from the internet: URL:https://arxiv.org/abs/1701… [cited by applicant]
Lee et al., “Fully Character-Level Neural Machine Translation Without Explicit Segmentation,” Transactions of the Association for Computational Linguistics (TACL). arXiv:1610.03017v3. Jun. 13, 2017. pp. 1-13. [cited by applicant]
Merity et al., “Pointer Sentinel Mixture Models,” Published as a Conference paper at the International Conference on Learning Representations (ICLR). Sep. 26, 2016. hllps://arxiv.org/abs/1609.07843v1. pp. 1-13. [cited by applicant]
Mikolov et al., “Efficient Estimation of Word Representations in Vector Space,” International Conference on Learning Representation, 2013 (ICLR). Cornell University Library. Jan. 16, 2013. arXiv:1301.3781. pp. 1-12. [cited by applicant]
N. Shazeer et al: “Outrageously large neural networks: the sparsely-gated mixture-of-experts layer” , arXiv:1701.06538vl, Jan. 23, 2017 (Jan. 23, 2017), XP080750950. [cited by applicant]
Pennington et al., “GloVe: Global Vectors for Word Representation”, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar. Oct. 25-29, 2014. https://nlp.stanford.edu… [cited by applicant]