IP Library › Granted Patent US 12,450,038
Granted Patent B2
US 12,450,038 · App. 17/515,917 · Granted Oct 21, 2025

Generation of graphical user interface prototypes

Inventors: Yuan Jie Song (Shanghai, CN); Xiao Feng Ji (Shanghai, CN); Dan Zhang (Shanghai, CN); Jun Qian Zhou (Shanghai, CN); Meng Chai (Shanghai, CN)
Assignee: International Business Machines Corporation
G06F8/38G06N3/045G06N3/0475G06N3/088
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,450,038
App. No.
17/515,917
Filed
Nov 1, 2021
Granted
Oct 21, 2025
Kind
B2
Art Unit
2179
USPC
715/762
Abstract

A method of generating a prototype of a graphical user interface (GUI). The method includes acquiring a draft wireframe representing a GUI design, the draft wireframe including one or more wireframe components, and decomposing the draft wireframe into one or more component slices, each component slice including a respective wireframe component of the one or more wireframe components. The method also includes generating a component slice sequence including the one or more component slices and at least one additional component slice selected based on the draft wireframe, constructing a wireframe based on the component slice sequence, and generating a prototype of the GUI design based on the constructed wireframe.

Claims (34)

1. A method of generating a prototype of a graphical user interface (GUI), comprising:

acquiring a draft wireframe representing a GUI design, the draft wireframe including one or more wireframe components;

decomposing the draft wireframe into one or more component slices, each component slice including a respective wireframe component of the one or more wireframe components;

generating a component slice sequence including the one or more component slices and at least one additional component slice selected based on the draft wireframe;

constructing a wireframe based on the component slice sequence; and

generating a prototype of the GUI design based on the constructed wireframe,

wherein the machine learning model is a generative adversarial model that includes a generator configured to generate the component slice sequence, the generator configured to be trained by a first discriminator and a second discriminator,

wherein the first discriminator trains the generator based on discriminating between real component slice sequences from a training data set and component slice sequences output by the generator, and

wherein the second discriminator trains the generator based on discriminating between real wireframes from the training data set and wireframes constructed from the component slice sequences output by the generator.

2. The method of claim 1 , wherein the additional component slice is selected based on a target platform and a target function.

3. The method of claim 1 , wherein generating the component slice sequence and constructing the wireframe is performed at least in part by a machine learning model.

4. The method of claim 1 , wherein generating the component slice sequence and constructing the wireframe is performed at least in part by a GUI wireframe generative adversarial model (GWGAN) and generating the prototype is performed at least in part by a GUI style generative adversarial model (GSGAN).

5. A system for generating a prototype of a graphical user interface (GUI), the system comprising one or more processors and computer-readable instructions stored in tangible storage medium, the computer-readable instructions when executed by the one or more processors performs operations comprising:

acquiring a draft wireframe representing a GUI design, the draft wireframe including one or more wireframe components;

decomposing the draft wireframe into one or more component slices, each component slice including a respective wireframe component of the one or more wireframe components;

generating a component slice sequence including the one or more component slices and at least one additional component slice selected based on the draft wireframe;

constructing a wireframe based on the component slice sequence; and

generating a prototype of the GUI design based on the constructed wireframe,

wherein the machine learning model is a generative adversarial model, generating the component slice sequence is performed using a machine learning model including a generator, the generator configured to be trained by a first discriminator and a second discriminator,

wherein the first discriminator is configured to train the generator based on discriminating between real component slice sequences from a training data set and component slice sequences output by the generator, and

wherein second discriminator is configured to train the generator based on discriminating between real wireframes from the training data set and wireframes constructed from the component slice sequences output by the generator.

6. The system of claim 5 , wherein the additional component slice is selected based on a target platform and a target function.

7. The system of claim 5 , wherein generating the component slice sequence and constructing the wireframe is performed at least in part by a machine learning model.

8. The system of claim 5 , wherein generating the component slice sequence and constructing the wireframe is performed at least in part by a GUI wireframe generative adversarial model (GWGAN), and generating the prototype is performed at least in part by a GUI style generative adversarial model (GSGAN).

9. A computer program product, the computer program product comprising:

a tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:

acquiring a draft wireframe representing a GUI design, the draft wireframe including one or more wireframe components;

decomposing the draft wireframe into one or more component slices, each component slice including a respective wireframe component of the one or more wireframe components;

generating a component slice sequence including the one or more component slices and at least one additional component slice selected based on the draft wireframe;

constructing a wireframe based on the component slice sequence; and

generating a prototype of the GUI design based on the constructed wireframe, wherein generating the component slice sequence is performed using a machine learning model including a generator, the generator configured to be trained by a first discriminator and a second discriminator, the first discriminator configured to train the generator based on discriminating between real component slice sequences from a training data set and component slice sequences output by the generator, and where the second discriminator is configured to train the generator based on discriminating between real wireframes from the training data set and wireframes constructed from the component slice sequences output by the generator.

10. The computer program product of claim 9 , wherein generating the component slice sequence and constructing the wireframe is performed at least in part by a machine learning model.

11. The computer program product of claim 10 , wherein the machine learning model is a generative adversarial model.

12. The computer program product of claim 10 , wherein generating the component slice sequence and constructing the wireframe is performed at least in part by a GUI wireframe generative adversarial model (GWGAN), and generating the prototype is performed at least in part by a GUI style generative adversarial model (GSGAN).

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2021
From: SONG, YUAN JIE; JI, XIAO FENG; ZHANG, DAN; ZHOU, JUN QIAN; CHAI, MENG
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 057982/0193 →
Continuity (1)
Related Publication 20230138367A1 · May 4, 2023
References Cited (21)
US 10504268B1 · Huang · 2019 [cited by examiner]
US 11366963B1 · Morrison · 2022 [cited by examiner]
US 20070130529A1 · Shrubsole · 2007 [cited by applicant]
US 20190250891A1 · Kumar et al. · 2019 [cited by applicant]
US 20190317739A1 · Turek · 2019 [cited by examiner]
US 20200151508A1 · Yang · 2020 [cited by examiner]
US 20210303886A1 · Hassan · 2021 [cited by examiner]
US 20210406673A1 · Pardeshi · 2021 [cited by examiner]
US 20230106159A1 · Morrison · 2023 [cited by examiner]
CN 101005681A1 · 2007 [cited by applicant]
CN 102043582B · 2012 [cited by applicant]
Kevin Moran et al.; Automatic Generation of Graphical User Interface Prototypes from Unrestricted Natural Language Requirements; IEEE Transactions on Software Engineering; Jun. 7, 2018. [cited by applicant]
Kevin Moran et al.; Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps; IEEE Transactions on Software Engineering; Jun. 7, 2018. [cited by applicant]
Nick Babich; Wireframing Automation and Artificial Intelligence for UX Design; Adobe; Apr. 16, 2020. [cited by applicant]
Tianming Zhao et al.; GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks; arXiv; Jan. 27, 2021. [cited by applicant]
Bao et al., “Tracking and analyzing cross-cutting activities in developers' daily work”, In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering, IEEE, 2015, pp. 277-282. [cited by applicant]
Chen et al., “From UI design image to GUI skeleton: A neural machine translator to bootstrap mobile GUI implementation.”, In Proceedings of the 40th International Conference on Software Engineering. ACM, May 27-Jun. 3, … [cited by applicant]
Deka et al., “Rico: A mobile app dataset for building data-driven design applications”, In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Oct. 2017, pp. 845-854. [cited by applicant]
Kumar et al., “Webzeitgeist: Design mining the web”, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2013, pp. 3083-3092. [cited by applicant]
Ritchie et al., “d.tour: Style-based exploration of design example galleries”, In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. ACM, 2011, pp. 165-174. [cited by applicant]
Yu et al., “SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient”, https://arxiv.org/pdf/1609.05473, Aug. 2017, 11 pages. [cited by applicant]