Selecting layouts for user interfaces using free text
Systems, methods and non-transitory computer readable media for selecting layouts for user interfaces using free text and/or sketches are provided. In some examples, an indication of a plurality of visual elements may be received. Further, a textual input in a natural language and/or a sketch may be received. The textual input and/or the sketch may be associated with a desire of an individual to arrange the plurality of visual elements. Further, the textual input and/or the sketch may be analyzed to obtain a layout for the plurality of visual elements. Further, a visual presentation of the plurality of visual elements based on the layout may be generated.
1 . A non-transitory computer readable medium storing computer implementable instructions that when executed by at least one processor cause the at least one processor to perform operations for selecting layouts using natural language inputs, the operations comprising:
receiving an indication of a plurality of visual elements;
receiving an input in a natural language, the input is associated with a desire of an individual to arrange the plurality of visual elements, the input includes at least a word;
analyzing the input to obtain a layout for the plurality of visual elements, the layout includes a positioning of a first visual element of the plurality of visual elements closer to a second visual element of the plurality of visual elements than to a third visual element of the plurality of visual elements based on the word; and
generating a visual presentation of the plurality of visual elements based on the layout.
2 . The non-transitory computer readable medium of claim 1 , wherein the plurality of visual elements is a plurality of visual user interface elements, and the operations further comprise generating a user interface that includes the plurality of visual user interface elements based on the layout.
3 . The non-transitory computer readable medium of claim 2 , wherein the visual presentation is a three-dimensional presentation of the plurality of visual user interface elements, and wherein, for each visual user interface element of the plurality of visual user interface elements, a three-dimensional position of the respective visual user interface element is based on the layout.
4 . The non-transitory computer readable medium of claim 2 , wherein the input includes a first portion and a second portion, the first portion includes a first pronoun, the second portion includes a second pronoun, the second pronoun differs from the first pronoun, and the operations further comprise:
selecting a first visual user interface element from the plurality of visual user interface elements based on the first pronoun;
selecting a second visual user interface element from the plurality of visual user interface elements based on the second pronoun, the second visual user interface element differs from the first visual user interface element;
analyzing the first portion to select a first position for the first visual user interface element;
analyzing the second portion to select a second position for the second visual user interface element, the second position differs from the first position; and
including in the layout a positioning of the first visual user interface element at the first position and a positioning of the second visual user interface element at the second position.
5 . The non-transitory computer readable medium of claim 2 , wherein the input includes at least a verb associated with a prospective user of the user interface, and wherein the operations further comprise basing the obtaining of the layout on the verb.
6 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
receiving an indication of a draft layout;
analyzing the input to determine at least one change to the draft layout; and
implementing the determined at least one change to obtain the layout for the plurality of visual elements.
7 . The non-transitory computer readable medium of claim 1 , wherein the input includes a first portion and a second portion, the first portion includes a first noun, the second portion includes a second noun, the second noun differs from the first noun, and the operations further comprise:
selecting a fourth visual element of the plurality of visual elements based on the first noun;
selecting a fifth visual element of the plurality of visual elements based on the second noun, the fifth visual element differs from the fourth visual element;
analyzing the first portion using a large language model to select a first position for the fourth visual element;
analyzing the second portion to select a second position for the fifth visual element, the second position differs from the first position; and
including in the layout a positioning of the fourth visual element at the first position and a positioning of the fifth visual element at the second position.
8 . The non-transitory computer readable medium of claim 1 , wherein the input includes at least an adverb, and the operations further comprise:
selecting a position for a particular visual element of the plurality of visual elements based on the adverb; and
including in the layout a positioning of the particular visual element at the selected position.
9 . The non-transitory computer readable medium of claim 1 , wherein the input includes at least an adjective, and wherein a relative position between two visual elements of the plurality of visual elements in the layout is based on the adjective.
10 . The non-transitory computer readable medium of claim 1 , wherein the layout includes a movement of a particular element of the plurality of visual elements, the input includes at least an adverb, and the operations further comprise selecting a pace for the movement based on the adverb.
11 . The non-transitory computer readable medium of claim 1 , wherein the word is a conjunction.
12 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise generating a source code in a style sheet language based on the layout.
13 . The non-transitory computer readable medium of claim 1 , wherein the layout is associated with a first presentation manner, and the operations further comprise:
obtaining an indication of a second presentation manner, the second presentation manner differs from the first presentation manner; and
analyzing the input to obtain, based on the second presentation manner, a second layout for the plurality of visual elements, wherein the second layout differs from the first layout.
14 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise obtaining the layout for the plurality of visual elements based on a characteristic of a target audience, and wherein the operations further comprise analyzing historic activities of individuals included in the target audience to determine the characteristic of the target audience.
15 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise obtaining the layout for the plurality of visual elements based on a particular design style, and wherein the operations further comprise analyzing at least one other layout to determine the particular design style.
16 . The non-transitory computer readable medium of claim 15 , wherein the at least one other layout is at least one layout associated with a specific designer.
17 . The non-transitory computer readable medium of claim 15 , wherein the at least one other layout is at least one layout associated with a specific brand.
18 . The non-transitory computer readable medium of claim 15 , wherein the at least one other layout is at least one layout associated with a specific subject matter.
19 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
identifying a first mathematical vector in a mathematical space, the first mathematical vector corresponds to a first word of the input, wherein the dimension of the first mathematical vector is more than three;
identifying a second mathematical vector in the mathematical space, the second mathematical vector corresponds to a second word of the input, wherein the dimension of the second mathematical vector is more than three;
calculating a function of the first mathematical vector and the second mathematical vector to obtain a third mathematical vector in the mathematical space; and
obtaining the layout for the plurality of visual elements based on the third mathematical vector.
20 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
receiving a sketch from the individual, the sketch is associated with the desire of the individual to arrange the plurality of visual elements; and
analyzing the textual input and the sketch to obtain the layout for the plurality of visual elements.
21 . The non-transitory computer readable medium of claim 20 , wherein the operations further comprise using a multimodal machine learning model to analyze the input in the natural language and the sketch to obtain the layout for the plurality of visual elements, wherein the multimodal machine learning model is a multimodal machine learning model trained using a plurality of training examples including a particular training example, the particular training example includes particular textual data, particular sketch data, and an indication of a particular layout.
22 . The non-transitory computer readable medium of claim 20 , wherein the sketch includes a plurality of pixel values, and the operations further comprise:
calculating a convolution of at least part of the pixel values of the sketch to obtain a result value;
identifying a first mathematical function, the first mathematical function corresponds to a word of the input;
calculating a particular function of the result value and the first mathematical function to obtain a mathematical object in a mathematical space; and
obtaining the layout for the plurality of visual elements based on the mathematical object.
23 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
after generating the visual presentation, receiving a second input in the natural language, the second input is associated with a desire of the individual to change the layout;
analyzing the second input to determine at least one change to the layout; and
implementing the determined at least one change to obtain a second layout for the plurality of visual elements.
24 . The non-transitory computer readable medium of claim 1 , wherein the operations further comprise:
selecting a type for the layout, the selected type is at least one of a static layout, fluid layout, responsive layout, adaptive layout, grid layout, or masonry layout; and
basing the obtaining of the layout on the selected type.
25 . The non-transitory computer readable medium of claim 24 , wherein the operations further comprise analyzing the input in the natural language to select the type for the layout.
26 . A system for selecting layouts using natural language inputs, the system comprising:
at least one processing unit configured to perform operations, the operations comprise:
receiving an indication of a plurality of visual elements;
receiving an input in a natural language, the input is associated with a desire of an individual to arrange the plurality of visual elements, the input includes at least a word;
analyzing the input to obtain a layout for the plurality of visual elements, the layout includes a positioning of a first visual element of the plurality of visual elements closer to a second visual element of the plurality of visual elements than to a third visual element of the plurality of visual elements based on the word; and
generating a visual presentation of the plurality of visual elements based on the layout.
27 . A method for selecting layouts using natural language inputs, the method comprising:
receiving an indication of a plurality of visual elements;
receiving an input in a natural language, the input is associated with a desire of an individual to arrange the plurality of visual elements, the input includes at least a word;
analyzing the input to obtain a layout for the plurality of visual elements, the layout includes a positioning of a first visual element of the plurality of visual elements closer to a second visual element of the plurality of visual elements than to a third visual element of the plurality of visual elements based on the word; and
generating a visual presentation of the plurality of visual elements based on the layout.