IP Library › Granted Patent US 12,198,046
Granted Patent B2
US 12,198,046 · App. 17/073,147 · Granted Jan 14, 2025

Real-time visualization of machine learning models

Inventors: Wei Xia (Seattle, WA); Weixin Wu (Seattle, WA); Meng Wang (Seattle, WA); Ranju Das (Seattle, WA)
Assignee: Amazon Technologies, Inc.
G06N3/08G06N3/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,198,046
App. No.
17/073,147
Filed
Oct 16, 2020
Granted
Jan 14, 2025
Kind
B2
Art Unit
2125
USPC
706/11
Abstract

A visualization tool for machine learning models obtains metadata from a first training node at which a multi-layer machine learning model is being trained. The metadata includes a parameter of an internal layer of the model. The tool determines a plurality of metrics from the metadata, including respective loss function values corresponding to several training iterations of the model. The tool indicates the loss function values and the internal layer parameter values via a graphical interface.

Claims (53)

1. A computer-implemented method, comprising:

obtaining, via an interactive interface, (a) a first visualization control request pertaining to a first machine learning model indicated by the interactive interface, and (b) a second visualization control request pertaining to a second machine learning model indicated by the interactive interface;

executing, at a first node of a multi-node training environment, a first training process to train the first machine learning model using a first training data set, wherein the first training process includes a first set of training iterations;

executing, at a second node of the multi-node training environment, a second training process to train the second machine learning model using the first training data set, wherein the second training process includes a second set of training iterations;

obtaining a replay request pertaining to the first machine learning model via the interactive interface, wherein the replay request indicates a plurality of training iterations performed during the training of the first machine learning model; and

causing, in response to the replay request, a video representation to be presented via the interactive interface, wherein the video representation is dynamically updated with data pertaining to the training of the first machine learning model across the plurality of training iterations during the first training process, wherein the data is obtained from the first node, and wherein the video representation presents an iteration-to-iteration comparison of the first and second training processes.

2. The computer-implemented method as recited in claim 1 , further comprising:

obtaining, via the interactive interface, an indication of a selection of a particular internal layer of the second machine learning model from a plurality of internal layers of the second machine learning model, wherein at least a portion of the data with which the interactive interface is dynamically updated is obtained from the particular internal layer.

3. The computer-implemented method as recited in claim 1 , wherein at least a portion of the data with which the interactive interface is dynamically updated comprises a loss function.

4. The computer-implemented method as recited in claim 1 , wherein at least a portion of the data with which the interactive interface is dynamically updated comprises a test score.

5. The computer-implemented method as recited in claim 1 , wherein at least a portion of the data with which the interactive interface is dynamically updated comprises an amount of time or resources consumed for individual iterations of the first training process.

6. The computer-implemented method as recited in claim 1 , further comprising:

generating a reduced-dimension mapping of a multi-dimensional result obtained from the first machine learning model; and

causing the reduced-dimensional mapping to be presented via the interactive interface.

7. The computer-implemented method as recited in claim 1 , further comprising:

causing a resource usage metric of the second machine learning model to be presented via the interactive interface.

8. A system, comprising:

one or more computing devices;

wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices cause the one or more computing devices to:

obtain, via an interactive interface, (a) a first visualization control request pertaining to a first machine learning model indicated by the interactive interface, and (b) a second visualization control request pertaining to a second machine learning model indicated by the interactive interface;

determine, based at least in part on the first and second visualization control requests, that dynamic updates of data pertaining to training of the second machine learning model is to be presented via the interactive interface;

execute, at a first node of a multi-node training environment, a first training process to train the first machine learning model using a first training data set, wherein the first training process includes a first set of training iterations;

execute, at a second node of the multi-node training environment, a second training process to train the second machine learning model using the first training data set, wherein the second training process includes a second set of training iterations;

cause, based at least in part on the second visualization control request, a video representation to be presented via at least a portion of the interactive interface, wherein the video representation is dynamically updated with data pertaining to the training of the second machine learning model during the second training process, wherein the data is obtained from the second node, and wherein the video representation presents an iteration-to-iteration comparison of the first and second training processes.

9. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices further cause the one or more computing devices to:

obtain a log entry associated with the second machine learning model, wherein the log entry comprises a result of a first transformation applied to data in the first training data set produced by the second machine learning model; and

apply a reverse transformation to the result of the first transformation, wherein the data with which the interactive interface is dynamically updated comprises a result of the reverse transformation.

10. The system as recited in claim 8 , wherein at least a portion of the data with which the interactive interface is dynamically updated comprises a gradient value.

11. The system as recited in claim 8 , wherein at least a portion of the data with which the interactive interface is dynamically updated comprises a weight associated with a layer of the second machine learning model.

12. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices further cause the one or more computing devices to:

obtain, from the first node of a multi-node training environment, a first set of log records pertaining to the first set of training iterations of the first machine learning model;

obtain, from the second node of the multi-node training environment, a second set of log records pertaining to the second set of training iterations of the second machine learning model;

perform correlation analysis to identify a first subset of the first set of log records which represent a particular training iteration that matches, with respect to an ordinal position of the particular training iteration within the first set of training iterations of the first machine learning model, another training iteration in the second set of training iterations represented by a second subset of the second set of log records; and

include, in the iteration-to-iteration comparison of the first and second training processes presented via the interactive interface, a result of comparison of contents of the first subset and the second subset of log entries.

13. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices further cause the one or more computing devices to:

cause an alert indication to be displayed via the interactive interface in response to a determination that data associated with the second machine learning model satisfies a threshold criterion.

14. The system as recited in claim 8 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices further cause the one or more computing devices to:

obtain, via the interactive interface, a third visualization control request pertaining to a third machine learning model; and

determine, based at least in part on the third visualization control request, that dynamic updates of data pertaining to training of the third machine learning model is not be presented via the interactive interface.

15. A computer-implemented method, comprising:

identifying a plurality of machine learning models, including a first machine learning model and a second machine learning model, for which representations of one or more parameters are to be displayed via one or more programmatic interfaces, wherein individual ones of the plurality of machine learning models are trained using a first training data set;

executing, at a first node of a multi-node training environment, a first training process to train the first machine learning model using the first training data set, wherein the first training process includes a first set of training iterations;

executing, at a second node of the multi-node training environment, a second training process to train the second machine learning model using the first training data set, wherein the second training process includes a second set of training iterations;

obtaining a replay request pertaining to the first machine learning model via the one or more programmatic interfaces, wherein the replay request indicates a plurality of training iterations performed during the training of the first machine learning model; and

causing, in response to the replay request, a video representation to be presented via the one or more programmatic interfaces, wherein the video representation is dynamically updated with data pertaining to the training of the first machine learning model across the plurality of training iterations during the first training process, wherein the data is obtained from the first node, and wherein the video representation presents an iteration-to-iteration comparison of the first and second training processes.

16. The computer-implemented method as recited in claim 15 , wherein the first parameter comprises one of: (a) a weight assigned to a subcomponent of the first machine learning model in a training iteration, (b) a bias assigned to a subcomponent of the first machine learning model in a training iteration, (d) a value of a loss function of the first machine learning model, (c) a result obtained from a test iteration of the first machine learning model, or (d) an output of a feature processing filter of the first machine learning model.

17. The computer-implemented method as recited in claim 15 , further comprising:

causing a warning to be displayed via the one or more programmatic interfaces in response to a determination that data associated with the first machine learning model satisfies a threshold criterion.

18. The computer-implemented method as recited in claim 15 , further comprising:

causing a recommendation to terminate training of the first machine learning model to be presented via the one or more programmatic interfaces.

19. The computer-implemented method as recited in claim 15 , wherein the first machine learning model comprises a neural network.

20. The computer-implemented method as recited in claim 15 , further comprising:

causing a resource usage metric of the second machine learning model to be presented via the one or more programmatic interfaces.

Continuity (2)
Continuation 15074203 · Mar 18, 2016
Related Publication 20210034980A1 · Feb 4, 2021
References Cited (39)
US 6038337A · Lawrence et al. · 2000 [cited by applicant]
US 8234228B2 · Weston et al. · 2012 [cited by applicant]
US 9552549B1 · Gong · 2017 [cited by applicant]
US 9928448B1 · Merler · 2018 [cited by applicant]
US 10176245B2 · Lim · 2019 [cited by applicant]
US 10410096B2 · Dijkman · 2019 [cited by applicant]
US 10510000B1 · Commons · 2019 [cited by applicant]
US 10810491B1 · Xia · 2020 [cited by examiner]
US 20120158623A1 · Bilenko · 2012 [cited by examiner]
US 20150324690A1 · Chilimbi · 2015 [cited by examiner]
US 20150339570A1 · Scheffler · 2015 [cited by examiner]
US 20150379424A1 · Dirac et al. · 2015 [cited by applicant]
US 20170124487A1 · Szeto · 2017 [cited by examiner]
US 20170344884A1 · Lin · 2017 [cited by applicant]
US 20180121768A1 · Lin · 2018 [cited by applicant]
US 20180260668A1 · Shen · 2018 [cited by applicant]
US 20180285682A1 · Najibi · 2018 [cited by applicant]
CN 104112113 · 2014 [cited by applicant]
EP 2833295 · 2015 [cited by applicant]
Brescia, Massimo, et al. “DAME: A web oriented infrastructure for scientific data mining & exploration.” arXiv preprint arXiv: 1010.4843 (2010). (Year: 2010). [cited by examiner]
Simard, Patrice, et al. “Ice: enabling non-experts to build models interactively for large-scale lopsided problems.” arXiv preprint arXiv:1409.4814 (2014). (Year: 2014). [cited by examiner]
Amershi, Saleema, et al. “Power to the people: The role of humans in interactive machine learning.” Ai Magazine 35.4 (2014): 105-120. (Year: 2014). [cited by examiner]
Berral, Josep Lluís, et al. “ALOJA: a framework for benchmarking and predictive analytics in Hadoop deployments.” IEEE Transactions on Emerging Topics in Computing 5.4 (2015): 480-493. (Year: 2015). [cited by examiner]
Martin Abadi, et al.“TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems”, Nov. 9, 2015, Google AI: 1-19. (Year: 2015). [cited by examiner]
Bolei Zhou, et al. “Learning Deep Features for Discriminative Localization”, Dec. 14, 2015, arXiv: 1-10 (Year: 2015). [cited by examiner]
McGregor, Sean, et al. “Facilitating testing and debugging of Markov Decision Processes with interactive visualization.” 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 2015. (Year: 2… [cited by examiner]
Samek, Wojciech, et al. “Evaluating the visualization of what a Deep Neural Network has learned.” arXiv preprint arXiv:1509.06321 (2015): 1-13 (Year: 2015). [cited by examiner]
Kulesza, Todd, et al. “Principles of explanatory debugging to personalize interactive machine learning.” Proceedings of the 20th international conference on intelligent user interfaces. 2015: 126-137 (Year: 2015). [cited by examiner]
U.S. Appl. No. 15/045,030, filed Feb. 16, 2016, Chatterjee, et al. [cited by applicant]
Amazon Web Services, Amazon Machine Learning Developer Guide, Feb. 9, 2016, pp. 1-140. [cited by applicant]
Neural Network Basics, “A Basic Introduction to Feedforward Backpropagation Neural Networks,” David Leverington, Associate Professor of Geosciences, downloaded from http://www.webpages.ttu.edu/dleverin/neural_network/ne… [cited by applicant]
“Gradient-Based Learning Applied to Document Recognition,” Yann LeCun, et al., Proc. of the IEEE, Nov. 1998, pp. 1-46. [cited by applicant]
“ImageNet Classification with Deep Convolutional Neural Networks,” Alex Krizhevsky, et al., 2012, pp. 1-9. [cited by applicant]
Wikipedia, “Convolutional neural network,” Jan. 19, 2016, pp. 1-9. [cited by applicant]
“Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis,” Patrice Y. Simard, et al., Microsoft Research, One Microsoft Way, Redmond WA 98052, 2003, pp. 1-6. [cited by applicant]
U.S. Appl. No. 15/719,402, filed Sep. 28, 2017, Mohammed Hidayath Ansari et al. [cited by applicant]
Wang, et al., “Learning Deep Representations for Scene Labeling with Semantic Context Guided Supervision,” Jun. 9, 2017, https://arxiv.org/pdf/1706.02493.pdf, pp. 1-13. [cited by applicant]
Yan et al., “Semantic Indexing with Deep Learning; A Case Study,” 2016, Open Access/Big Data Analytics, BioMed Central, pp. 1-13. [cited by applicant]
Liberda, et al., Image Segmentation and Convolutional Neural Networks as Tools for Indoor Scene Understanding, 2016, Bachelor's Thesis at Signals and systems—SSYX02-16-31, https://publications.lib.chalmers.se/records/fu… [cited by applicant]