Granted Patent B2

US 12,190,080 · App. 17/884,616 · Granted Jan 7, 2025

Inventors: Mouleswara Reddy Chintakunta (Allagadda, IN); Omar Odibat (Cedar Park, TX); Pritpal S. Arora (Bangalore, IN)

Assignee: Kyndryl, Inc.

G06F8/20G06F3/013G06F40/279

Abstract

Claims (79)

1. A computer-implemented method for automated user experience mockup generation, comprising:

obtaining a user experience theme description;

obtaining a new feature image set;

generating a plurality of user experience designs corresponding to the new feature image set by performing an iterative process using a generative adversarial network (GAN) based on the theme description and the new feature image set;

sequentially displaying, using an electronic display device, the plurality of user experience designs to a user and tracking an eye gaze of the user with respect to a display of the plurality of user experience designs and determining where on the electronic display the user is viewing the user experience designs of the plurality of user experience designs;

for the user experience designs of the plurality of user experience designs, identifying a corresponding set of user interface elements based on the eye gaze, wherein the corresponding set of user interface elements identified is a set of elements that are gazed at for a duration exceeding a predetermined threshold duration;

comparing the identified corresponding set of user interface elements to a designer created user experience design; and

returning the user interface elements of the identified corresponding set of user interface elements as feedback input to the GAN.

2. The method of claim 1 , further comprising:

obtaining an audio recording of a meeting;

performing speech-to-text on the audio recording to generate a transcript;

performing natural language processing on the transcript to generate user experience attributes; and

wherein the returning the user interface elements of the identified corresponding set of user interface elements is based on inputting the user experience attributes into the GAN.

3. The method of claim 1 , further comprising:

obtaining a text document describing a user experience feature;

performing natural language processing on the text document to generate one or more user experience attributes; and

wherein the returning the user interface of the identified corresponding set of user interface elements is based on inputting the one or more user experience attributes into the GAN.

4. The method of claim 1 , wherein the feedback input to the GAN includes a user experience element type error.

5. The method of claim 4 , wherein the user experience element type error is based on a Euclidean distance computation.

6. The method of claim 4 , wherein the user experience element type error is associated with a user experience element type that includes a graph type.

7. The method of claim 1 , wherein the feedback input to the GAN includes a user experience element position error.

8. The method of claim 7 , wherein computing the user experience element position error comprises the comparing the identified corresponding set of user interface elements to the designer created user experience design by:

inputting the plurality of user experience designs into a first Siamese subnetwork to identify a first plurality of user experience elements;

inputting the designer created user experience design into a second Siamese subnetwork to identify a second plurality of user experience elements;

identifying corresponding user experience elements within the first plurality of user experience elements and the second plurality of user experience elements;

computing a distance between the corresponding user experience elements; and

indicating the user experience element position error in response to the distance exceeding a predetermined threshold.

9. The method of claim 1 , further comprising:

ranking the plurality of user experience designs;

identifying a user experience design from the plurality of user experience designs as having a highest rank; and

selecting the identified user experience design as a starting point for creating a finalized user experience (UX) design.

10. The method of claim 1 , wherein the theme description includes a button shape.

11. The method of claim 1 , wherein the new feature image set is obtained by:

obtaining a hand-drawn image set; and

converting the hand-drawn image set to an XML file.

12. An electronic computation device comprising:

a processor;

a memory coupled to the processor, the memory containing instructions, that when executed by the processor, cause the electronic computation device to:

obtain a user experience theme description;

obtain a new feature image set;

generate a plurality of user experience designs corresponding to the new feature image set by performing an iterative process using a generative adversarial network (GAN) based on the theme description and the new feature image set;

sequentially display, using an electronic display device, the plurality of user experience designs to a user and track an eye gaze of the user with respect to a display of the plurality of user experience designs on the electronic display device and determining where on the electronic display the user is viewing for the user experience design of the plurality of user experience designs;

for the user experience design of the plurality of user experience designs, identify a corresponding set of user interface elements based on the eye gaze, wherein the corresponding set of user interface elements identified is a set of elements that are gazed at for a duration exceeding a predetermined threshold duration;

comparing the identified corresponding set of user interface elements to a designer created user experience design; and

return the user interface elements of the identified corresponding set of user interface elements that were identified based on the eye gaze as feedback input to the GAN.

13. The electronic computation device of claim 12 , wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to:

obtain an audio recording of a designer meeting;

perform speech-to-text on the audio recording to generate a transcript;

perform natural language processing on the transcript to generate user experience attributes; and

return the user interface elements of the identified corresponding set of user interface elements based on inputting the user experience attributes into the GAN.

14. The electronic computation device of claim 12 , wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to:

obtain a text document describing a user experience feature;

perform natural language processing on the text document to generate one or more user experience attributes; and

return the user interface elements of the identified corresponding set of user interface elements based on inputting the one or more user experience attributes into the GAN.

15. The electronic computation device of claim 12 , wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to generate a user experience element type error based on a Euclidean distance computation.

16. The electronic computation device of claim 12 , wherein the memory further comprises instructions, that when executed by the processor, cause the electronic computation device to generate a user experience element position error by:

inputting the plurality of user experience designs into a first Siamese subnetwork to identify a first plurality of user experience elements;

inputting a designer finalized user experience design into a second Siamese subnetwork to identify a second plurality of user experience elements;

identifying corresponding user experience elements within the first plurality of user experience elements and the second plurality of user experience elements;

computing a distance between the corresponding user experience elements; and

indicating the user experience element position error in response to the distance exceeding a predetermined threshold.

17. A computer program product for an electronic computation device comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the electronic computation device to:

obtain a user experience theme description;

obtain a new feature image set;

generate a plurality of user experience designs corresponding to the new feature image set by performing an iterative process using a generative adversarial network (GAN) based on the theme description and the new feature image set;

track an eye gaze of a user with respect to a display of the plurality of user experience designs on an electronic display device;

for the plurality user experience designs, identify a corresponding set of user interface elements based on the eye gaze, wherein the corresponding set of user interface elements identified is a set of elements that are gazed at for a duration exceeding a predetermined threshold duration;

comparing the identified corresponding set of user interface elements to a designer created user experience design; and

return the user interface elements of the identified corresponding set of user interface elements that were identified based on the eye gaze as feedback input to the GAN of the identified corresponding set of user interface elements.

18. The computer program product of claim 17 , wherein the computer readable storage medium includes program instructions executable by the processor to cause the electronic computation device to:

obtain an audio recording of a designer meeting;

perform speech-to-text on the audio recording to generate a transcript;

perform natural language processing on the transcript to generate user experience attributes; and

return the user interface elements of the identified corresponding set of user interface elements based on inputting the user experience attributes into the GAN.

19. The computer program product of claim 17 , wherein the computer readable storage medium includes program instructions executable by the processor to cause the electronic computation device to:

obtain a text document describing a user experience feature;

perform natural language processing on the text document to generate one or more user experience attributes; and

return the user interface elements of the identified corresponding set of user interface elements based on inputting the one or more user experience attributes into the GAN.

20. The computer program product of claim 17 , wherein the computer readable storage medium includes program instructions executable by the processor to cause the electronic computation device to generate a user experience element type error based on a Euclidean distance computation.

Assignments (1)

Continuity (1)

References Cited (21)