Editing user interfaces using free text
Systems, methods and non-transitory computer readable media for editing user interfaces from free text and/or sketches are provided. Digital data associated with a design of a user interface may be received. A presentation of the user interface based on the design may be caused. A selection of a portion of the user interface may be received. A textual input in a natural language and/or a sketch may be received. The textual input and/or the sketch may be analyzed to determine at least one change to a portion of the design of the user interface corresponding to the selected portion of the user interface. The determined at least one change may be implemented to generate a modified design of the user interface. A second presentation of the user interface based on the modified design may be caused.
1 . A non-transitory computer readable medium storing computer implementable instructions that when executed by at least one processor cause the at least one processor to perform operations for editing user interfaces using natural language inputs, the operations comprising:
receiving digital data associated with a design of a user interface, the user interface includes a plurality of elements;
using the digital data to cause a first presentation of the user interface to an individual, the first presentation is based on the design;
receiving from the individual a selection of a portion of the user interface, the selected portion of the user interface includes at least two but not all elements of the plurality of elements;
receiving from the individual an input in a natural language, the input is associated with a desire of the individual to modify a portion of the design of the user interface corresponding to the selected portion of the user interface, the input includes at least one word indicative of a subgroup of the plurality of elements, the subgroup includes at least one element included in the selected portion and at least one element not included in the selected portion;
based on the at least one word included in the input, identifying the at least one element indicated by the at least one word and included in the selected portion;
analyzing the input to determine at least one change to the identified at least one element indicated by the at least one word and included in the selected portion;
implementing the determined at least one change to generate a modified design of the user interface; and
causing a second presentation of the user interface to the individual, the second presentation is based on the modified design.
2 . The non-transitory computer readable medium of claim 1 , wherein the at least one word indicative of the subgroup is a pronoun.
3 . The non-transitory computer readable medium of claim 1 , wherein the input includes a verb, and the operations further comprise:
based on the verb, selecting a type of change; and
including a change of the selected type in the at least one change, wherein the selected type of change includes at least one of adding a new user interface element to the user interface or removing an existing user interface element from the user interface.
4 . The non-transitory computer readable medium of claim 1 , wherein the input includes a preposition, and the operations further comprise:
based on the preposition, selecting a new location for a particular element of the identified at least one element indicated by the at least one word and included in the selected portion, wherein the new location is a relative spatial position of the particular element with respect to a second element included in the user interface; and
including a change of a location of the particular element to the new location in the at least one change.
5 . The non-transitory computer readable medium of claim 1 , wherein the input includes a preposition, and the operations further comprise:
based on the preposition, selecting a new timing for a particular animation associated with a particular element of the identified at least one element indicated by the at least one word and included in the selected portion; and
including a change to a timing of the particular animation based on the new timing in the at least one change, wherein the new timing defines whether the particular animation occurs while a user action is performed or after the user action is performed.
6 . The non-transitory computer readable medium of claim 1 , wherein the input includes an adjective, and the operations further comprise:
based on the adjective, determining a particular design style;
selecting a new color scheme associated with the determined particular design style for a particular element of the identified at least one element indicated by the at least one word and included in the selected portion; and
including a change of a color scheme of the particular element to the new color scheme in the at least one change.
7 . The non-transitory computer readable medium of claim 1 , wherein the input includes an adjective, and the operations further comprise:
based on the adjective, selecting a new layout for the elements included in the selected portion of the user interface; and
including in the at least one change a change of a layout of the identified at least one element indicated by the at least one word and included in the selected portion to the new layout, wherein the adjective is indicative of a geometric structure of the elements, and the selected new layout is at least one of a symmetrical structure or a linear structure.
8 . The non-transitory computer readable medium of claim 1 , wherein the input includes a conjunction, the identified at least one element indicated by the at least one word and included in the selected portion includes at least a first element and a second element, and the operations further comprise:
based on the conjunction, determining a change to a spatial relation between the first element and the second element, wherein based on the conjunction, the change to a spatial relation includes one of moving both the first element and the second element to a target location or moving only one of the first element or the second element to the target location; and
including the determined change to the spatial relation between the first element and the second element in the at least one change.
9 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
after the second presentation, receiving from the individual a second input in the natural language, the second input is associated with a second desire of the individual to modify the portion of the design of the user interface corresponding to the selected portion of the user interface;
analyzing the second input to determine a second at least one change to the portion of the design of the user interface; and
implementing the determined second at least one change to generate a second modified design of the user interface.
10 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
after the second presentation, receiving from the individual a second input in the natural language, the second input includes an interjection;
based on the interjection, determining a degree of success in the determination of the at least one change to the portion of the design of the user interface; and
based on the determined degree of success, generating a second modified design of the user interface.
11 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
receiving from the individual a second input in the natural language; and
analyzing the second input to select the portion of the user interface, wherein selecting the portion comprises using a large language model (LLM) to analyze the second textual input and a source code associated with the user interface.
12 . The non-transitory computer readable medium of claim 1 , wherein the determination of the at least one change to the portion of the design of the user interface is further based on a characteristic of a target audience, wherein the characteristic comprises a behavior pattern associated with the target audience.
13 . The non-transitory computer readable medium of claim 12 , wherein the operations further comprise analyzing historic activities of individuals associated with the target audience to determine the characteristic of the target audience.
14 . The non-transitory computer readable medium of claim 1 , wherein the determination of the at least one change to the portion of the design of the user interface is further based on a particular design style, and wherein the operations further comprise analyzing at least one other user interface associated with a specific brand to determine the particular design style.
15 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise using a multimodal machine learning model to analyze the input to determine the at least one change to the portion of the design of the user interface.
16 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
identifying a first mathematical object in a mathematical space, the first mathematical object corresponds to a word of the input;
analyzing the digital data to identify a second mathematical object in a mathematical space;
calculating a function of the first mathematical object and the second mathematical object to obtain a third mathematical object in the mathematical space; and
basing the determination of the at least one change to the portion of the design of the user interface on the third mathematical object.
17 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
receiving from the individual a sketch, the sketch is associated with the desire of the individual to modify the portion of the design of the user interface corresponding to the selected portion of the user interface; and
analyzing the input and the sketch to determine the at least one change to the portion of the design of the user interface, wherein the at least one change comprises changing a level of details associated with at least part of the user interface.
18 . The non-transitory computer readable medium of claim 17 , wherein the sketch includes a plurality of pixel values, and the operations further comprise:
calculating a convolution of at least part of the pixel values of the sketch to obtain a result value;
identifying a first mathematical object in a mathematical space, the first mathematical object corresponds to a word of the input;
calculating a function of the result value and the first mathematical object to obtain a second mathematical object in the mathematical space; and
basing the determination of the at least one change to the portion of the design of the user interface on the second mathematical object.
19 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
receiving from the individual a pointing input indicative of a particular position in the first presentation of the user interface; and
using the particular position indicated by the pointing input to select the portion of the user interface.
20 . A system for editing user interfaces using natural language inputs, the system comprising:
at least one processing unit configured to perform operations, the operations comprise:
receiving digital data associated with a design of a user interface, the user interface includes a plurality of elements;
using the digital data to cause a first presentation of the user interface to an individual, the first presentation is based on the design;
receiving from the individual a selection of a portion of the user interface, the selected portion of the user interface includes at least two but not all elements of the plurality of elements;
receiving from the individual an input in a natural language, the input is associated with a desire of the individual to modify a portion of the design of the user interface corresponding to the selected portion of the user interface, the input includes at least one word indicative of a subgroup of the plurality of elements, the subgroup includes at least one element included in the selected portion and at least one element not included in the selected portion;
based on the at least one word included in the input, identifying the at least one element indicated by the at least one word and included in the selected portion;
analyzing the input to determine at least one change to the identified at least one element indicated by the at least one word and included in the selected portion;
implementing the determined at least one change to generate a modified design of the user interface; and
causing a second presentation of the user interface to the individual, the second presentation is based on the modified design.
21 . A method for editing user interfaces using natural language inputs, the method comprising:
receiving digital data associated with a design of a user interface, the user interface includes a plurality of elements;
using the digital data to cause a first presentation of the user interface to an individual, the first presentation is based on the design;
receiving from the individual a selection of a portion of the user interface, the selected portion of the user interface includes at least two but not all elements of the plurality of elements;
receiving from the individual an input in a natural language, the input is associated with a desire of the individual to modify a portion of the design of the user interface corresponding to the selected portion of the user interface, the input includes at least one word indicative of a subgroup of the plurality of elements, the subgroup includes at least one element included in the selected portion and at least one element not included in the selected portion;
based on the at least one word included in the input, identifying the at least one element indicated by the at least one word and included in the selected portion;
analyzing the input to determine at least one change to the identified at least one element indicated by the at least one word and included in the selected portion;
implementing the determined at least one change to generate a modified design of the user interface; and
causing a second presentation of the user interface to the individual, the second presentation is based on the modified design.