IP Library Granted Patent US 12,379,836
Granted Patent B2
US 12,379,836 · App. 17/818,852 · Granted Aug 5, 2025

Interactive graphical user interfaces for deployment and application of neural network models using cross-device node-graph pipelines

Inventors: Ruofei Du (San Francisco, CA); Na Li (Palo Alto, CA); Jing Jin (Mountain View, CA); Maria Mandlis (Vancouver, CA); Scott Joseph Miles (American Canyon, CA); Ping Yu (San Carlos, CA)
Assignee: Google LLC
G06F3/04847G06F3/0482G06F3/0486G06N3/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,379,836
App. No.
17/818,852
Granted
Aug 5, 2025
Kind
B2
Abstract

A method includes providing an interactive graphical user interface comprising a first menu providing one or more input options, a second menu providing one or more machine learning models, and a third menu providing one or more output formats. The method also includes generating a graph in a portion of the interactive graphical user interface by detecting one or more user selections of an input option, a machine learning model, and an output format, displaying nodes corresponding to the input option, the machine learning model, the output format, and displaying edges connecting the first node to the second node, and the second node to the third node. The method additionally includes applying the machine learning model to an input associated with the input option to generate an output in the output format. The method further includes providing, by the interactive graphical user interface, the output in the output format.

Claims (62)

1. A computer-implemented method, comprising:

providing, by a computing device, an interactive graphical user interface comprising a first menu providing one or more input options, a second menu providing one or more machine learning models, and a third menu providing one or more output formats;

generating a graph in a portion of the interactive graphical user interface, wherein the generating of the graph comprises:

detecting one or more user selections of an input option from the first menu, a machine learning model from the second menu, and an output format from the third menu,

responsive to the one or more user selections, displaying, in the portion, a first node of the graph corresponding to the input option, a second node of the graph corresponding to the machine learning model, a third node of the graph corresponding to the output format, a first edge of the graph connecting the first node to the second node, and a second edge of the graph connecting the second node to the third node,

detecting another user selection of a second machine learning model from the second menu, and

responsive to the other user selection, displaying, in the portion, a fourth node of the graph corresponding to the second machine learning model, a third edge of the graph connecting the first node to the fourth node, and a fourth edge of the graph connecting the fourth node to the third node;

applying the machine learning model to an input associated with the input option to generate an output in the output format;

applying the second machine learning model to the input to generate a second output in the output format; and

providing, by the interactive graphical user interface, the output and the second output in the output format.

2. The computer-implemented method of claim 1 , further comprising:

receiving, by a second portion of the interactive graphical user interface, the input associated with the input option.

3. The computer-implemented method of claim 1 , further comprising:

receiving, by a drop-down menu linked to the third node, the output in the output format.

4. The computer-implemented method of claim 1 , further comprising:

providing, by the interactive graphical user interface, the input.

5. The computer-implemented method of claim 1 , wherein the one or more user selections comprises dragging and dropping an item from a menu into the portion.

6. The computer-implemented method of claim 1 , further comprising:

enabling a user to edit one or more parameters associated with one or more of the input, the machine learning model, or the output.

7. The computer-implemented method of claim 1 , wherein the generating of the graph further comprises:

detecting another user selection of a second output format from the third menu;

responsive to the other user selection, displaying, in the portion, a fourth node of the graph corresponding to the second output format, and a third edge of the graph connecting the second node to the fourth node; and

applying the machine learning model to the input to generate a second output in the second output format.

8. The computer-implemented method of claim 1 , wherein the other user selection comprises dragging and dropping the second machine learning model from the second menu into the portion.

9. The computer-implemented method of claim 1 , wherein the other user selection comprises uploading the second machine learning model from a library of the user.

10. The computer-implemented method of claim 1 , wherein the displaying of the first edge is responsive to a user indication connecting the first node to the second node.

11. The computer-implemented method of claim 1 , further comprises:

providing the user with a selectable edge that enables the user to confirm a connection of the first node to the second node, and

wherein the displaying of the first edge is performed upon receiving user confirmation to connect the first node to the second node.

12. The computer-implemented method of claim 1 , wherein the generating of the graph further comprises:

predicting, by a trained graph predictive model, one or more of a next node or a next edge of the graph; and

recommending the one or more of the next node or the next edge to a user.

13. The computer-implemented method of claim 1 , further comprising:

training the graph predictive model based on a plurality of graphs deployed on a plurality of computing devices.

14. The computer-implemented method of claim 1 , wherein the graph is an editable graph, and further comprising:

enabling a user to update the graph by performing one or more of adding, removing, or replacing a node, an edge, or both; and

updating the output in substantial real-time based on an update to the graph.

15. The computer-implemented method of claim 1 , wherein the providing of the output comprises providing the output to an end-user application.

16. The computer-implemented method of claim 1 , wherein the input option comprises one or more of an image, a video, an audio, or text.

17. The computer-implemented method of claim 1 , wherein the interactive graphical user interface is hosted on a platform and shared across a plurality of computing devices, and wherein one or more of the generating of the graph, the applying of the machine learning model, or the providing of the output is synchronized across the plurality of computing devices.

18. A computing device, comprising:

one or more processors; and

data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising:

providing, by a computing device, an interactive graphical user interface comprising a first menu providing one or more input options, a second menu providing one or more machine learning models, and a third menu providing one or more output formats;

generating a graph in a portion of the interactive graphical user interface, wherein the generating of the graph comprises:

detecting one or more user selections of an input option from the first menu, a machine learning model from the second menu, and an output format from the third menu,

responsive to the one or more user selections, displaying, in the portion, a first node of the graph corresponding to the input option, a second node of the graph corresponding to the machine learning model, a third node of the graph corresponding to the output format, a first edge of the graph connecting the first node to the second node, and a second edge of the graph connecting the second node to the third node,

detecting another user selection of a second machine learning model from the second menu, and

responsive to the other user selection, displaying, in the portion, a fourth node of the graph corresponding to the second machine learning model, a third edge of the graph connecting the first node to the fourth node, and a fourth edge of the graph connecting the fourth node to the third node;

applying the machine learning model to an input associated with the input option to generate an output in the output format;

applying the second machine learning model to the input to generate a second output in the output format; and

providing, by the interactive graphical user interface, the output and the second output in the output format.

19. An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions comprising:

providing, by a computing device, an interactive graphical user interface comprising a first menu providing one or more input options, a second menu providing one or more machine learning models, and a third menu providing one or more output formats;

generating a graph in a portion of the interactive graphical user interface, wherein the generating of the graph comprises:

detecting one or more user selections of an input option from the first menu, a machine learning model from the second menu, and an output format from the third menu,

responsive to the one or more user selections, displaying, in the portion, a first node of the graph corresponding to the input option, a second node of the graph corresponding to the machine learning model, a third node of the graph corresponding to the output format, a first edge of the graph connecting the first node to the second node, and a second edge of the graph connecting the second node to the third node,

detecting another user selection of a second machine learning model from the second menu, and

responsive to the other user selection, displaying, in the portion, a fourth node of the graph corresponding to the second machine learning model, a third edge of the graph connecting the first node to the fourth node, and a fourth edge of the graph connecting the fourth node to the third node;

applying the machine learning model to an input associated with the input option to generate an output in the output format;

applying the second machine learning model to the input to generate a second output in the output format; and

providing, by the interactive graphical user interface, the output and the second output in the output format.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 10, 2022
From: DU, RUOFEI; LI, NA; JIN, JING; MANDLIS, MARIA; MILES, SCOTT JOSEPH; YU, PING
To: GOOGLE LLC
Reel/Frame 060771/0436 →
Continuity (1)
Related Publication 20240053877A1 · Feb 15, 2024
References Cited (42)
US 5168441A · Onarheim · 1992 [cited by examiner]
US 5490246A · Brotsky · 1996 [cited by examiner]
US 10402380B1 · Carr · 2019 [cited by examiner]
US 11182697B1 · Murakonda · 2021 [cited by examiner]
US 20170316114A1 · Bourhani · 2017 [cited by examiner]
US 20190347341A1 · Carr · 2019 [cited by examiner]
US 20190384640A1 · Swamy · 2019 [cited by examiner]
US 20200156210A1 · Sullivan · 2020 [cited by examiner]
US 20200233860A1 · Laino · 2020 [cited by examiner]
US 20200380301A1 · Siracusa · 2020 [cited by examiner]
US 20210065002A1 · Samek · 2021 [cited by examiner]
US 20210342125A1 · Burnett · 2021 [cited by examiner]
US 20220138004A1 · Nandakumar · 2022 [cited by applicant]
US 20220245460A1 · Qin · 2022 [cited by examiner]
US 20220300850A1 · Mendez · 2022 [cited by examiner]
US 20220343250A1 · Tremblay · 2022 [cited by examiner]
US 20220360545A1 · Mingels · 2022 [cited by examiner]
US 20230108774A1 · Archer · 2023 [cited by examiner]
Taygan “Azure Machine Learning Designer” (May 24, 2020). Available at https://www.taygan.co/blog/2020/05/16/azure-machine-learning-designer (last accessed Dec. 22, 2024). (Year: 2020). [cited by examiner]
G. Kruppai and A. Kiss, “Building Neural Networks as Dataflow Graphs,” 2019 IEEE 15th International Scientific Conference on Informatics, Poprad, Slovakia, 2019, pp. 000261-000266, (Year: 2019). [cited by examiner]
I. Khodnenko, S. V. Ivanov, K. Prokofiev and A. Lantseva, “A Lightweight Visual Programming tool for Machine Learning and Data Manipulation,” 2020 International Conference on Computational Science and Computational Inte… [cited by examiner]
International Searching Authority, International Search Report and Written Opinion mailed Oct. 30, 2023, issued in connection with International Patent Application No. PCT/US2023/028575, filed Jul. 25, 2023, 15 pages. [cited by applicant]
Rifat, Taygan, “Zero Code AI with Drag-n-Drop Machine Learning,” https://www.youtube.com/watch?v=Xw7UYcoFH8&ab_channel=TayganRifat, Jun. 26, 2020, 2 pages. [cited by applicant]
Tan et al., “HumanGPS: Geodesic PreServing Feature for Dense Human Correspondences”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021). [cited by applicant]
Brooks et al., “BodyPix model demo application for Google Coral”, google-coral/project-bodypix, https://github.com/google-coral/project-bodypix (2021), retrieved Aug. 9, 2022. [cited by applicant]
MoveNet: Ultra fast and accurate pose detection model, https://www.tensorflow.org/hub/tutorials/movenet (2022), retrieved Aug. 9, 2022. [cited by applicant]
Pytorch Team, “MobilenetV2”, https://pytorch.org/hub/pytorch_vision_mobilenet_v2, retrieved Aug. 9, 2022. [cited by applicant]
“Tensorflow/tfjs-models”, https://github.com/tensorflow/tfjs-models/tree/master/coco-ssd, retrieved Aug. 9, 2022. [cited by applicant]
Cochard, “BlazeFace: A Machine Learning Model for Fast Detection of Face Positions and Key Points” (2021), https://medium.com/axinc-ai/blazeface-a-machine-learning-model-for-fast-detection-of-face-positions-and-key-poin… [cited by applicant]
DeepLabCut Documentation: DeepLabCut: a software package for animal pose estimation, https://deeplabcut.github.io/DeepLabCut/docs/intro.html, retrieved Aug. 9, 2022. [cited by applicant]
Cochard, “BlazePose: A 3D Pose Estimation Model”, (2021), https://medium.com/axinc-ai/blazepose-a-3d-pose-estimation-model-d8689d06b7c4, retrieved Aug. 9, 2022. [cited by applicant]
Google LLC, “MediaPipe”, (2020), https://google.github.io/mediapipe/, retrieved Aug. 9, 2022. [cited by applicant]
Tyka et al., “Google-coral/project-posenet”, (2021), https://github.com/google-coral/project-posenet, retrieved August 9. 2022. [cited by applicant]
Collins et al., “LaMDA: our breakthrough conversation technology”, (2021), https://blog.google/technology/ai/lamda/, retrieved Aug. 9, 2022. [cited by applicant]
Tensorflow, “Universal Sentence Encoder”, https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder, retrieved Aug. 9, 2022. [cited by applicant]
Tensorflow, “An end-to-end open source maching learning platform”, https://www.tensorflow.org/, retrieved Aug. 9, 2022. [cited by applicant]
WebM: an open web mdeia project, https://www.webmproject.org/, retrieved Aug. 9, 2022. [cited by applicant]
Heinzman, “What is a GIF, and How Do You Use Them?”, How-To Geek, (2019), https://www.howtogeek.com/441185/what-is-a-gif-and-how-do-you-use-them/, retrieved Aug. 9, 2022. [cited by applicant]
Welcome to Colaboratory, https://colab.research.google.com/, retrieved Aug. 9, 2022. [cited by applicant]
“ECMA-404 the JSON Data Interchange Standard”, Introducing JSON, https://www.json.org/json-en.html, retrieved Aug. 9, 2022. [cited by applicant]
“Shear mapping”, Wikipedia, https://en.wikipedia.org/wiki/Shear_mapping, retrieved Aug. 9, 2022. [cited by applicant]
Casiez et al., “Tuning the filter”, https://cristal.univ-lille.fr/˜casiez/1euro/, retrieved Aug. 9, 2022. [cited by applicant]