IP Library › Granted Patent US 12,248,879
Granted Patent B2
US 12,248,879 · App. 17/102,264 · Granted Mar 11, 2025

Systems providing a learning controller utilizing indexed memory and methods thereto

Inventors: Omar Florez Choque (Oakland, CA); Erik Mueller (Chevy Chase, MD)
Assignee: CAPITAL ONE SERVICES, LLC
G06N3/084G06N20/00H04L9/0643
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,248,879
App. No.
17/102,264
Granted
Mar 11, 2025
Kind
B2
Abstract

A system includes one or more memory devices storing instructions, and one or more processors configured to execute the instructions to perform steps of a method. A method can include receiving observations and a corresponding class label, determining a candidate key based on the observations, determining a current memory state of a memory module based on a similarity of stored keys to the candidate key, generating a measurement vector based on the current memory state, concatenating the candidate key and measurement vector to form a state vector, determining, based on the state vector and an action distribution policy, an action of a plurality of actions such that the determined action maximizes an expected reduction in entropy as compared to the remaining actions of the plurality actions, executing the determined action, determining a value of the determined action, and updating, based on the value, the action distribution policy.

Claims (69)

1. A system comprising:

a processor; and

memory having stored thereon:

a hash key vector and a label vector associated with the hash key vector; and

instructions that, when executed by the processor, cause the system to:

convert a current state of the memory into a memory state vector;

identify one or more saved keys of the hash key vector that are within a predetermined similarity to a candidate key, the candidate key being based at least in part on an input data vector;

implement a hash structure configured to produce collisions and comprising less of the one or more saved keys than observances, the hash structure being stored in an indexed memory;

perform, via a machine learning model, an entropy reduction estimation to determine a first estimated entropy associated with reading the at least one of the one or more identified saved keys, a second estimated entropy associated with updating the at least one of the one or more identified saved keys to include data indicative of the candidate key, and a third estimated entropy associated with inserting a new key indicative of the candidate key; and

dynamically train the machine learning model to incrementally and continuously learn new classes without retraining the machine learning model by using a trainable controller configured to learn from the indexed memory by executing, based at least in part on the entropy reduction estimation and based on the collisions from the hash structure to predict indices for new classes in a set of available entries of the hash structure, one of reading the at least one of the one or more identified saved keys, updating the at least one of the one or more identified saved keys in the memory to include data indicative of the candidate key, and inserting in the memory a new key indicative of the candidate key.

2. The system of claim 1 , wherein the system is configured to encode received data using a convolutional neural network.

3. The system of claim 1 , wherein the system is configured to measure a current state of the memory using a multi-layer perceptron network.

4. The system of claim 1 , wherein the instructions, when executed by the processor, further cause the system to:

prior to querying the memory for any saved keys that are within the predetermined similarity to the candidate key:

normalize the input data vector and the memory state vector; and

concatenate the input data vector and the memory state vector to create a state vector.

5. The system of claim 1 , wherein the instructions, when executed by the processor, further cause the system to, prior to performing the entropy reduction estimation, identify a single saved key that is most similar to the candidate key.

6. The system of claim 1 , wherein the instructions, when executed by the processor, further cause the system to perform the entropy reduction estimation for each of the one or more identified saved keys that is above the predetermined similarity to the candidate key.

7. The system of claim 1 , wherein the instructions, when executed by the processor, further cause the system to determine, based at least in part on a calculation of a back-propagation algorithm, a reduction in entropy in the system.

8. The system of claim 1 , wherein inserting a new key comprises adding the new key to the hash key vector and adding a new label to the label vector, the new label being associated with the new key.

9. The system of claim 1 , wherein updating the at least one of the one or more identified saved keys to include data indicative of the candidate key comprises averaging a plurality of most similar saved keys and the candidate key to generate an updated key and replacing an oldest key among the one or more identified saved keys with the updated key.

10. The system of claim 1 , wherein updating the at least one of the one or more identified saved keys to include data indicative of the candidate key comprises averaging a single most similar saved key and the candidate key to generate an updated key and replacing the single most similar saved key with the updated key.

11. The system of claim 1 , wherein:

the memory further comprises an age vector associated with the hash key vector, and

the instructions, when executed by the processor, further cause the system to:

increment the age vector by one each time a controller executes a query function associated with the hash key vector; and

set the age vector equal to zero each time the controller executes an update function associated with the hash key vector or an insert function associated with the hash key vector.

12. A method for continuously teaching a learning model comprising:

receiving a set of observations;

generating a candidate key based on the set of observations;

implementing a hash structure configured to produce collisions to predict new classes and comprising more of the observations than one or more stored keys in a stored key vector, the hash structure being stored in an indexed memory;

dynamically train the learning model to incrementally and continuously learn new classes without retraining the learning model by using a trainable controller configured to learn from the indexed memory by determining whether the one or more stored keys in the stored key vector are similar to the candidate key by:

determining an average vector of the one or more stored keys in the stored key vector; and

determining that the average vector has a similarity to the candidate key that is above a predetermined threshold;

determining, based on the determined similarity of one or more stored keys to the candidate key, a current memory state of a memory comprising the stored key vector;

generating a measurement vector based on the current memory state, wherein the measurement vector is based on a memory state scalar associated with the average vector;

concatenating the candidate key and the measurement vector to form a state vector;

determining, based on the state vector and an action distribution policy, an action of a plurality of actions, wherein the plurality of actions comprises querying a memory, updating the memory, inserting the candidate key to the memory, or combinations thereof,

executing the determined action;

determining a value of the determined action; and

updating, based on the value, the action distribution policy.

13. The method of claim 12 , further comprising encoding received observations using a convolutional neural network.

14. The method of claim 12 , further comprising determining a memory state scalar based on the average vector, wherein the measurement vector is based on the memory state scalar.

15. The method of claim 12 , further comprising measuring the current memory state of the memory using a multi-layer perceptron network.

16. The method of claim 12 , further comprising:

for at least one of the one or more stored keys, determining a first estimated entropy associated with reading the at least one of the one or more stored keys;

for the at least one of the one or more stored keys, determining, based on a comparison of the candidate key to at least one of the one or more stored keys, a second estimated entropy associated with updating the at least one of the one or more stored keys to include data indicative of the candidate key; and

determining a third estimated entropy associated with inserting a new key indicative of the candidate key.

17. The method of claim 16 , wherein updating the at least one of the one or more stored keys to include data indicative of the candidate key comprises averaging a plurality of most similar stored keys and the candidate key to generate an updated key and replacing an oldest key among the one or more stored keys with the updated key.

18. The method of claim 12 , wherein the memory further comprises an age vector associated with the state vector, and the method further comprises incrementing the age vector by one each time a controller executes a query function associated with the state vector and set the age vector equal to zero each time the controller executes an update function associated with the state vector or an insert function associated with the state vector.

19. A method for continuously teaching a learning model comprising:

receiving a set of observations;

generating a candidate key based on the set of observations;

implementing a hash structure configured to produce collisions to predict new classes, wherein the hash structure comprises more of the observations than stored keys in a stored key vector, and wherein the hash structure is stored in an indexed memory;

dynamically train the learning model to incrementally and continuously learn new classes without retraining the learning model by using a trainable controller configured to learn from the indexed memory by determining whether any of the stored keys in the stored key vector are similar to the candidate key by:

determining an average vector of the stored keys in the stored key vector; and

determining that the average vector has a similarity to the candidate key that is above a predetermined threshold; and

determining a memory state scalar based on the average vector;

determining, based on the determined similarity of stored keys to the candidate key, a current memory state of a memory comprising the stored key vector;

generating a measurement vector based on the current memory state wherein the measurement vector is based on the memory state scalar;

normalizing the candidate key and the measurement vector;

concatenating the candidate key and the measurement vector to form a state vector;

determining, based on the state vector and an action distribution policy, an action of a plurality of actions, wherein the plurality of actions comprises querying a memory, updating the memory, inserting the candidate key to the memory, or combinations thereof, and

executing the determined action, wherein the memory further comprises an age vector associated with the state vector, and the method further comprises incrementing the age vector by one each time a controller executes a query function associated with the state vector and set the age vector equal to zero each time the controller executes an update function associated with the state vector or an insert function associated with the state vector.

20. The method of claim 19 , further comprising determining an expected return value for executing the determined action in a state corresponding to the state vector and following the action distribution policy by:

approximating the expected return value based on weights of a neural network implementing a controller configured to execute the method;

minimizing a loss function associated with execution of the determined action;

determining an update to the action distribution policy based on the minimized loss function, wherein updating the action distribution policy comprises updating the action distribution policy to include the update; and

updating, based on the expected return value, the action distribution policy.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 23, 2020
From: CHOQUE, OMAR FLOREZ; MUELLER, ERIK
To: CAPITAL ONE SERVICES, LLC
Reel/Frame 054450/0501 →
Continuity (2)
Continuation 16250272 · Jan 17, 2019
Related Publication 20210150368A1 · May 20, 2021
References Cited (35)
US 9264451B2 · Chari · 2016 [cited by applicant]
US 10217023B1 · Rubin · 2019 [cited by examiner]
US 20100318478A1 · Yoshiike · 2010 [cited by applicant]
US 20180189676A1 · Predovic · 2018 [cited by applicant]
US 20180247549A1 · Martin · 2018 [cited by examiner]
Chen et al., “Semi-Supervised Deep Learning with Memory,” in Proc. Eur. Conf. Computer Vision 268-83 (2018). (Year: 2018). [cited by examiner]
Bahdanau, D. et al., Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015; May 19, 2016. [cited by applicant]
Goodfellow, I. et al., Generative Adversarial Nets, Department d'informatique et de recherche operationnelle, Universite de Montreal; Jun. 10, 2014. [cited by applicant]
Ioffe, S. et al., Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; Mar. 2, 2015. [cited by applicant]
Johnson, M. et al., Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation; Aug. 21, 2017. [cited by applicant]
Kaiser, L. et al., Learning to Remember Rare Events, ICLR 2017; Mar. 9, 2017. [cited by applicant]
Kingman, D. et al., Adam: A Method for Stochastic Optimization, ICLR 2015; Jan. 30, 2017. [cited by applicant]
Kirkpatrick, J. et al., Overcoming catastrophic forgetting in neural networks, DeepMind, London, UK, Bioengineering Depoartment, imperial College London; Jan. 25, 2017. [cited by applicant]
Koch, G. et al., Siamese Neural Networks for One-shot Image Recognition, Department of Computer Science, University of Toronto; 2015. [cited by applicant]
Krizhevsky, A. et al., ImageNet Classification with Deep Convolutional Neural Networks; Jun. 2017. [cited by applicant]
Lake, B. et al., One shot learning of simple visual concepts, Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology; 2011. [cited by applicant]
Mnih, V. et al., Asynchronus Methods for Deep Reinforcement Learning; Jun. 16, 2016. [cited by applicant]
Pauleve, L. et al., Locality sensitive hashing: a comparison of has function types and querying mechanisms, HAL archives-ouvertes; Apr. 16, 2010. [cited by applicant]
Qian, N., On the Momentum Term in Gradient Descent Learning Algorithms, Neural Networks; Jan. 1999. [cited by applicant]
Qian, Q. et al., Distance Metric Learning Using Dropout: A Structured Regularization Approach, KDD '14 Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, Aug. 2014. [cited by applicant]
Santoro, A. et al., Meta-Learning with Memory-Augmented Neural Networks, ICML'16 Proceedings of the 33rd International Conference on International Conference on Machine Learning—vol. 48, pp. 1842-1850; Jun. 2016. [cited by applicant]
Vinyals, O. et al., Matching Networks for One Shot Learning, NIPS'16 Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3637-3645; Dec. 2016. [cited by applicant]
Wei, Z. et al., Reinforcement Learning to Rank with Markov Decision Process, SIGIR'17; Aug. 2017. [cited by applicant]
Williams, R. et al., Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Machine Learning, 8, pp. 229-256; 1992. [cited by applicant]
Zhu, S. et al., Multi-labelled Classification Using Maximum Entropy Method, SIGIR'05; Aug. 2005. [cited by applicant]
A. Long, “A bunch of tips and tricks for training deep neural networks,” 2018 (Year: 2018). [cited by applicant]
J. Oh, “Efficient Deep Reinforcement Learning via Planning, Generalization, and Improved Exploration,” 2018 (Year: 2018). [cited by applicant]
H. Yin et al., “Hashing over Predicted Future Frames for Informed Exploration of Deep Reinforcement Learning,” 2018 (Year: 2018). [cited by applicant]
S. Wiedemann et al., “Entropy-Constrained Training of Deep Neural Networks,” 2018 (Year: 2018). [cited by applicant]
Silver et al., “Deterministic Policy Gradient Algorithims,” in Proc. 31st Instl. Conf. Machine Learning (2014). (Year: 2014). [cited by applicant]
Liang et al., “Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition 848-57 (2017). (Year: 2017). [cited by applicant]
Arel et al., Reinforcement learning-Based Multi-Agent System for Network Traffic Signal Control,: in 4.2 IET Intelligent Transport Sys. 128-35 (2009). (Year: 2009). [cited by applicant]
Santoro et al., “Relational Recurrent Neural Networks,” in Advances in Neural Info. Processing Sys. 7299-7310 (2018). Year: 2018. [cited by applicant]
Frady et al., “A Theory of Sequence Indexing and Working Memory in Recurrent Neural Networks,” in 30.6 Neural Computation 1449-1513 (2018). (Year: 2018). [cited by applicant]
Wayne et al., “Unsupervised Predictive memory in a Goal-Directed Agent,” in ArXiv preprint arXiv:1803.10760 (2018). (Year: 2018). [cited by applicant]