IP Library › Granted Patent US 12,437,004
Granted Patent B2
US 12,437,004 · App. 18/412,034 · Granted Oct 7, 2025

Deep multi-modal pairwise ranking model for crowdsourced food data

Inventors: Surender Reddy Yerva (Austin, TX); Iman Barjasteh (Austin, TX); Patrick Howell (Austin, TX); Chul Lee (Austin, TX); Hesamoddin Salehian (Austin, TX)
Assignee: MyFitnessPal, Inc.
G06F16/906G06F16/90344G06F16/908G06N3/049G06N20/10G16H20/60
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,437,004
App. No.
18/412,034
Granted
Oct 7, 2025
Kind
B2
Abstract

A method and system for providing more relevant search results and recommendation from a food database is disclosed. The method includes receiving a query, a first candidate food, and a second candidate food. The method includes generating vectors based on the query and food names of the first and second candidate foods using at least one embedding function of a machine learning model. The method includes determining nutrition content vectors from the nutritional data of the first and second candidate foods. The method includes generating a nutrition content vector based on the query using another embedding function of the machine learning model. The method includes determining which of the first and second candidate food is more relevant to the query based at least in part on the nutrition content vectors. The method includes providing search results or recommendation based on the determined relevance.

Claims (60)

1. A method of operating a health tracking system having a processor and a database configured to store a plurality of data records, each of the plurality of data records comprising at least a descriptive string and nutritional data regarding a respective consumable item, the method comprising:

receiving, with the processor, a query string;

retrieving, with the processor, a first data record of the plurality of data records and a second data record of the plurality of data records from the database;

generating, with the processor, (i) a first nutrition information vector from the nutritional data of the first data record and (ii) a second nutrition information vector from the nutritional data of the second data record;

generating, with the processor, at least one feature vector using at least one first embedding function of a machine learning model, the at least one first embedding function being learned in a training process of the machine learning model;

generating, with the processor, a third nutrition information vector based on the query string, using a second embedding function of the machine learning model, the second embedding function being learned in the training process of the machine learning model, wherein the at least one first embedding function and the second embedding function each include a different Long Short Term Memory (LSTM); and

determining, with the processor, which of the first data record and the second data record is more relevant to the query string based at least in part on the first nutrition information vector, the second nutrition information vector, and the third nutrition information vector, and the at least one feature vector.

2. The method according to claim 1 , wherein the at least one feature vector includes (i) a first feature vector based on the descriptive string of the first data record, (ii) a second feature vector based on the descriptive string of the second data record, and (iii) a third feature vector based on the query string.

3. The method according to claim 2 , wherein the act of determining which of the first data record and the second data record is more relevant to the query string includes:

determining, with the processor, (i) a first distance between the first feature vector and the third feature vector and (ii) a second distance between the second feature vector and the third feature vector, using a first distance function; and

determining, with the processor, (i) a third distance between the first nutrition information vector and the third nutrition information vector and (ii) a fourth distance between the second nutrition information vector and the third nutrition information vector, using a second distance function.

4. The method according to claim 3 , wherein the act of determining which of the first data record and the second data record is more relevant to the query string includes:

determining, with the processor, a first total distance as a sum of the first distance and the third distance;

determining, with the processor, a second total distance as a sum of the second distance and the fourth distance; and

determining, with the processor, which of the first data record and the second data record is more relevant to the query string based on a comparison of the first total distance and the second total distance, the first data record being more relevant to the query string if the first total distance is less than the second total distance, the second data record being more relevant to the query string if the second total distance is less than the first total distance.

5. The method according to claim 2 , wherein the act of generating the first feature vector, the second feature vector, and the third feature vector includes:

generating, with the processor, (i) a first numeric matrix representing words contained in the descriptive string of the first data record, (ii) a second numeric matrix representing words contained in the descriptive string of the second data record, and (iii) a third numeric matrix representing words contained in the query string; and

generating, with the processor, (i) the first feature vector based on the first numeric matrix, (ii) the second feature vector based on the second numeric matrix, and (iii) the third feature vector based on the third numeric matrix, using the at least one first embedding function of the machine learning model.

6. The method according to claim 5 , wherein each of the first numeric matrix, the second numeric matrix, and the third numeric matrix are composed of a plurality of one-hot vectors, each representing individual words.

7. The method according to claim 1 , wherein the act of generating the first nutrition information vector and the second nutrition information vector includes:

forming, with the processor, the first nutrition information vector with values equal to an energy content from the first data record, a fat content from the first data record, a carbohydrate content from the first data record, and a protein content from the first data record; and

forming, with the processor, the second nutrition information vector with values equal to an energy content from the second data record, a fat content from the second data record, a carbohydrate content from the second data record, and a protein content from the second data record.

8. The method according to claim 7 , wherein the act of generating the first nutrition information vector and the second nutrition information vector includes:

normalizing, with the processor, the energy content, the fat content, the carbohydrate content, and the protein content of the first nutrition information vector and of the second nutrition information vector on one of (i) a per-unit-mass basis, (ii) a per-unit-weight basis, and (iii) a per-unit-volume basis.

9. A method of operating a health tracking system having a processor and a database configured to store a plurality of data records, each of the plurality of data records comprising at least a descriptive string and nutritional data regarding a respective consumable item, the method comprising:

receiving, with the processor, a query string;

retrieving, with the processor, a first data record of the plurality of data records and a second data record of the plurality of data records from the database;

generating, with the processor, (i) a first nutrition information vector from the nutritional data of the first data record and (ii) a second nutrition information vector from the nutritional data of the second data record;

generating, with the processor, a third nutrition information vector based on the query string, using an embedding function of a machine learning model, the embedding function being learned in a training process of the machine learning model;

determining, with the processor, which of the first data record and the second data record is more relevant to the query string based in part on the first nutrition information vector, the second nutrition information vector, and the third nutrition information vector; and

training, with the processor, the machine learning model using a plurality of training inputs, each training input including (i) a training query string, (ii) a first descriptive string and first nutritional data labeled as corresponding to a relevant candidate, and (iii) a second descriptive string and second nutritional data labeled as corresponding to an irrelevant candidate, parameter values of the at least one embedding function and of the embedding function being learned during the training.

10. The method of claim 9 further comprising generating, with the processor, at least one feature vector using at least one first embedding function of the machine learning model, the at least one first embedding function being learned in the training process of the machine learning model, wherein determining which of the first data record and the second data record is more relevant to the query string is further based on the at least one feature vector.

11. The method of claim 10 wherein (i) a first feature vector is defined based on the descriptive string of the first data record, (ii) a second feature vector is defined based on the descriptive string of the second data record, and (iii) a third feature vector is defined based on the query string.

12. The method of claim 11 further comprising:

generating, with the processor, (i) a first numeric matrix representing words contained in the descriptive string of the first data record, (ii) a second numeric matrix representing words contained in the descriptive string of the second data record, and (iii) a third numeric matrix representing words contained in the query string; and

generating, with the processor, (i) the first feature vector based on the first numeric matrix, (ii) the second feature vector based on the second numeric matrix, and (iii) the third feature vector based on the third numeric matrix, using the at least one embedding function of the machine learning model.

13. The method of claim 9 wherein the act of generating the first nutrition information vector and the second nutrition information vector includes:

forming, with the processor, the first nutrition information vector with values equal to an energy content from the first data record, a fat content from the first data record, a carbohydrate content from the first data record, and a protein content from the first data record; and

forming, with the processor, the second nutrition information vector with values equal to an energy content from the second data record, a fat content from the second data record, a carbohydrate content from the second data record, and a protein content from the second data record.

14. The method of claim 9 wherein the act of generating the first nutrition information vector and the second nutrition information vector includes:

normalizing, with the processor, the energy content, the fat content, the carbohydrate content, and the protein content of the first nutrition information vector and of the second nutrition information vector on one of (i) a per-unit-mass basis, (ii) a per-unit-weight basis, and (iii) a per-unit-volume basis.

15. A method of operating a health tracking system having a processor and a database configured to store a plurality of data records, each of the plurality of data records comprising at least a descriptive string and nutritional data regarding a respective consumable item, the method comprising:

receiving, with the processor, a query string;

retrieving, with the processor, a first data record of the plurality of data records and a second data record of the plurality of data records from the database;

generating, with the processor, (i) a first nutrition information vector from the nutritional data of the first data record and (ii) a second nutrition information vector from the nutritional data of the second data record;

generating, with the processor, a third nutrition information vector based on the query string, using an embedding function of a machine learning model, the embedding function being learned in a training process of the machine learning model;

determining, with the processor, which of the first data record and the second data record is more relevant to the query string based at least in part on the first nutrition information vector, the second nutrition information vector, and the third nutrition information vector; and

transmitting, with a transceiver of the health tracking system, a list of data records of the plurality of data records to an electronic device of a user of the health tracking system, the list of data records at least including the first data record and the second data record, a relative sorting of the first data record and the second data record in the list of data records depending on the determination of which of the first data record and the second data record is more relevant to the query string.

16. The method of claim 15 wherein (i) a first feature vector is defined based on the descriptive string of the first data record, (ii) a second feature vector is defined based on the descriptive string of the second data record, and (iii) a third feature vector is defined based on the query string.

17. The method of claim 16 wherein the act of determining which of the first data record and the second data record is more relevant to the query string includes:

determining, with the processor, (i) a first distance between the first feature vector and the third feature vector and (ii) a second distance between the second feature vector and the third feature vector, using a first distance function; and

determining, with the processor, (i) a third distance between the first nutrition information vector and the third nutrition information vector and (ii) a fourth distance between the second nutrition information vector and the third nutrition information vector, using a second distance function.

18. The method of claim 16 further comprising:

generating, with the processor, (i) a first numeric matrix representing words contained in the descriptive string of the first data record, (ii) a second numeric matrix representing words contained in the descriptive string of the second data record, and (iii) a third numeric matrix representing words contained in the query string; and

generating, with the processor, (i) the first feature vector based on the first numeric matrix, (ii) the second feature vector based on the second numeric matrix, and (iii) the third feature vector based on the third numeric matrix, using the at least one embedding function of the machine learning model.

19. The method of claim 15 wherein the act of generating the first nutrition information vector and the second nutrition information vector includes:

forming, with the processor, the first nutrition information vector with values equal to an energy content from the first data record, a fat content from the first data record, a carbohydrate content from the first data record, and a protein content from the first data record; and

forming, with the processor, the second nutrition information vector with values equal to an energy content from the second data record, a fat content from the second data record, a carbohydrate content from the second data record, and a protein content from the second data record.

20. The method of claim 19 wherein the act of generating the first nutrition information vector and the second nutrition information vector includes:

normalizing, with the processor, the energy content, the fat content, the carbohydrate content, and the protein content of the first nutrition information vector and of the second nutrition information vector on one of (i) a per-unit-mass basis, (ii) a per-unit-weight basis, and (iii) a per-unit-volume basis.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 15, 2024
From: YERVA, SURENDER REDDY; BARJASTEH, IMAN; HOWELL, PATRICK; LEE, CHUL; SALEHIAN, HESAMODDIN
To: UNDER ARMOUR, INC.
Reel/Frame 068301/0437 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 15, 2024
From: UNDER ARMOUR, INC.
To: UA CONNECTED FITNESS, INC.
Reel/Frame 068301/0508 →
CHANGE OF NAME Recorded Aug 15, 2024
From: UA CONNECTED FITNESS, INC.
To: MYFITNESSPAL, INC.
Reel/Frame 068301/0613 →
PATENT SECURITY AGREEMENT Recorded Jul 26, 2024
From: MYFITNESSPAL, INC.
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 068173/0300 →
Continuity (4)
Continuation 17459404 · Aug 27, 2021
Continuation 16354863 · Mar 15, 2019
Provisional Application 62643919 · Mar 16, 2018
Related Publication 20240232266A1 · Jul 11, 2024
References Cited (34)
US 20150228062A1 · Joshi et al. · 2015 [cited by applicant]
US 20190027060A1 · Ishii · 2019 [cited by applicant]
US 20190057306A1 · Xue et al. · 2019 [cited by applicant]
US 20190171707A1 · Rapaport · 2019 [cited by applicant]
Hang, L., “A short introduction to learning to rank,” IEICE Transactions on Information and Systems, 2011, pp. 1854-1862, vol. 94, No. 10. [cited by applicant]
Li, H., “Learning to rank for information retrieval and natural language processing,” Synthesis Lectures on Human Language Technologies, vol. 7, No. 3, pp. 1-121, 2014. [cited by applicant]
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P., “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, 1998, pp. 2278-2324, vol. 86, No. 11. [cited by applicant]
Krizhevsky, A., Sutskever, I., and Hinton, G. E., “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097-1105. [cited by applicant]
Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., and Wu, Y., “Learning fine-grained image similarity with deep ranking,” in Proceedings of the IEEE Conference on Computer Vision and Pattern… [cited by applicant]
Zhao, F., Huang, Y., Wang, L., and Tan, T., “Deep semantic ranking based hashing for multi-label image retrieval,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1556-1564. [cited by applicant]
Zhao, X., Li, X., and Zhang, Z., “Multimedia retrieval via deep learning to rank,” IEEE Signal Processing Letters, 2015, pp. 1487-1491, vol. 22, No. 9. [cited by applicant]
Severyn, A., and Moschitti, A., “Learning to rank short text pairs with convolutional deep neural networks,” in Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retri… [cited by applicant]
Lu, Z., and Li, H., “A deep architecture for matching short texts,” in Advances in Neural Information Processing Systems, 2013, pp. 1367-1375. [cited by applicant]
Rigutini, L., Papini, T., Maggini, M., and Bianchini, M., “A neural network approach for learning 409 object ranking,” in International Conference on Artificial Neural Networks, 2008, pp. 899-908, Springer. [cited by applicant]
Gong, Y., Jia, Y., Leung, T., Toshev, A., and Ioffe, S., “Deep convolutional ranking for multilabel image annotation,” 2013. [cited by applicant]
Zhang, X., Zhao, J., and Lecun, Y., “Character-level convolutional networks for text classification,” in Advances in Neural Information Processing Systems, 2015, pp. 649-657. [cited by applicant]
Cao, Z., Wei, F., Dong, L., Li, S., and Zhou, M., “Ranking with recursive neural networks and its application to multi-document summarization.,” in AAAI, 2015, pp. 2153-2159. [cited by applicant]
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., and Yuille, A., “Deep captioning with multimodal recurrent neural networks (m-rnn),” 2014. [cited by applicant]
Kiros, R., Salakhutdinov, R., and Zemel, R. S., “Unifying visual-semantic embeddings with multimodal neural language models,” 2014. [cited by applicant]
Sutskever, I., Vinyals, O., and Le, Q. V., “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, 2014, pp. 3104-3112. [cited by applicant]
Sundermeyer, M., Schluter, R., and Ney, H., “Lstm neural networks for language modeling.,” in Interspeech, 2012, pp. 194-197. [cited by applicant]
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., and Xu, W., “Cnn-rnn: A unified framework for multi-label image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016,… [cited by applicant]
Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Mikolov, T., et al., “Devise: A deep visual-semantic embedding model,” in Advances in neural information processing systems, 2013, pp. 2121-2129. [cited by applicant]
Vinyals, O., Toshev, A., Bengio, S., and Erhan, D., “Show and tell: A neural image caption generator,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3156-3164. [cited by applicant]
Lynch, C., Aryafar, K., and Attenberg, J., “Images don't lie: Transferring deep visual semantic features to large-scale multimodal learning to rank,” 2015. [cited by applicant]
Howell, P. D., Martin, L. D., Salehian, H., Lee, C., Eastman, K. M., and Kim, J., “Analyzing taste preferences from crowdsourced food entries,” in Proceedings of the 6th International Conference on Digital Health Confer… [cited by applicant]
Srivastava, A., Jermyn, I., and Joshi, S., “Riemannian analysis of probability density functions with applications in vision,” in Computer Vision and Pattern Recognition, 2007. CVPR'07. 441 IEEE Conference on, 2007, pp.… [cited by applicant]
Lee, J., “Riemannian geometry: An introduction to curvature, No. 176 in graduate texts in 443 mathematics,” 1997. [cited by applicant]
Moakher, M., “A differential geometric approach to the geometric mean of symmetric positive definite matrices,” SIAM Journal on Matrix Analysis and Applications, 2005, pp. 735-747 vol. 26, No. 3. [cited by applicant]
Hochreiter, S., and Schmidhuber, J., “Long short-term memory,” Neural computation, 1997, pp. 1735-1780, vol. 9, No. 8. [cited by applicant]
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J., “Efficient estimation of word representations in vector space,” in Proceedings of Workshop at ICLR, 2013. [cited by applicant]
F. Chollet, “Keras: Theano-based deep learning library,” Code: https://github. com/fchollet.459 Documentation: http://keras. io, 2015. [cited by applicant]
Wang, Y., Wang, L., Li, Y., He, D., and Liu, T.-Y., “A theoretical analysis of ndcg type ranking measures,” in Conference on Learning Theory, 2013, pp. 25-54. [cited by applicant]
Matching Restaurant Menus to Crowdsourced Food Data a Scalable Machine Learning Approach by Hesam Salehian, 2017 ACM. 978-1-4503-4887-4/17/08 (Year:2017). [cited by applicant]