FRAGMENT-BASED DESIGN SEARCH

1 . A computer system comprising:

one or more processors; and

a memory to store a set of instructions, wherein the one or more processors execute instructions stored in the memory to perform operations comprising:

dividing one or more hierarchical structures representing one or more design interfaces into a plurality of design fragments;

updating an index with the plurality of design fragments and a plurality of embeddings for the plurality of design fragments; and

processing a search query that specifies one or more design elements using the index.

2 . The computer system of claim 1 , wherein the operations further comprise:

generating the plurality of embeddings using one or more machine learning models.

3 . The computer system of claim 2 , wherein generating the plurality of embeddings comprises:

generating a text-based representation of a design fragment from the plurality of design fragments; and

converting the text-based representation into an embedding for the design fragment.

4 . The computer system of claim 3 , wherein the text-based representation comprises at least one of a text-based description of the design fragment or a text-based definition of the design fragment.

5 . The computer system of claim 2 , wherein the one or more machine learning models comprise at least one of a text embedding model, an image embedding model, or a multimodal embedding model.

6 . The computer system of claim 1 , wherein dividing the one or more hierarchical structures into the plurality of design fragments comprises generating a design fragment from a node within the one or more hierarchical structures based on one or more attributes associated with the node.

7 . The computer system of claim 6 , wherein the one or more attributes comprise at least one of a node type or a dimension.

8 . The computer system of claim 6 , wherein the design fragment is generated based on a classification score outputted by a machine learning model from a representation of the node.

9 . The computer system of claim 1 , wherein processing the search query comprises:

converting a representation of the one or more design elements into one or more embeddings;

matching the one or more embeddings to a set of embeddings in the index; and

adding a set of design fragments corresponding to the set of embeddings to a set of search results for the search query.

10 . The computer system of claim 9 , wherein matching the one or more embeddings to the set of embeddings comprises:

matching the one or more embeddings to a set of clusters of embeddings in the index;

determining a set of representative design fragments associated with the set of clusters of embeddings; and

adding the set of representative design fragments to the set of search results.

11 . The computer system of claim 10 , wherein processing the search query further comprises:

matching a user selection of a design fragment in the set of representative design fragments to a cluster in the set of clusters; and

updating the set of search results with one or more additional design fragments in the cluster.

12 . The computer system of claim 10 , wherein matching the one or more embeddings to the set of clusters of embeddings comprises determining that a similarity between the one or more embeddings and a set of representative embeddings for the set of clusters of embeddings meets or exceeds a threshold.

13 . The computer system of claim 9 , wherein processing the search query further comprises:

determining an embedding for a design fragment that is selected by a user from the set of search results; and

generating an additional set of search results based on the embedding.

14 . The computer system of claim 1 , wherein the operations further comprise detecting a change to the one or more hierarchical structures prior to dividing the one or more hierarchical structures into the plurality of design fragments.

15 . The computer system of claim 1 , wherein the search query specifies the one or more design elements using at least one of an image, a text-based description, a sketch, or at least a portion of a design interface.

16 . A non-transitory computer-readable medium that stores instructions, executable by one or more processors, to cause the one or more processors to perform operations comprising:

receiving a search query specifying a representation of one or more design elements;

matching one or more embeddings associated with the representation to a set of embeddings for a set of design fragments, wherein each design fragment in the set of design fragments corresponds to a subset of one or more hierarchical structures representing one or more design interfaces;

generating a set of search results using the set of embeddings; and

outputting the set of search results in a response to the search query.

17 . The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:

generating the one or more embeddings using one or more machine learning models.

18 . The non-transitory computer-readable medium of claim 17 , wherein the one or more machine learning models comprise at least one of a text embedding model, an image embedding model, or a multimodal embedding model.

19 . The non-transitory computer-readable medium of claim 16 , wherein the one or more embeddings are matched to the set of embeddings based on one or more distances computed between the one or more embeddings and the set of embeddings.

20 . The non-transitory computer-readable medium of claim 16 , wherein generating the set of search results comprises:

determining that the set of embeddings corresponds to representative embeddings for a set of clusters of embeddings;

determining a set of representative design fragments associated with the set of clusters of embeddings; and

adding the set of representative design fragments to the set of search results.

21 . The non-transitory computer-readable medium of claim 20 , wherein determining the set of representative design fragments comprises retrieving, from an index, the set of representative design fragments from a set of mappings that include the representative embeddings.

22 . The non-transitory computer-readable medium of claim 21 , wherein each of the representative embeddings comprises a centroid of a corresponding cluster in the set of clusters.

23 . The non-transitory computer-readable medium of claim 20 , wherein generating the set of search results further comprises:

matching a user selection of a design fragment in the set of representative design fragments to a cluster in the set of clusters; and

updating the set of search results with one or more additional design fragments from the cluster.

24 . The non-transitory computer-readable medium of claim 16 , wherein generating the set of search results comprises:

retrieving, from an index, the set of design fragments from a set of mappings that include the set of embeddings; and

adding the set of design fragments to the set of search results.

25 . The non-transitory computer-readable medium of claim 16 , wherein generating the set of search results comprises:

determining an embedding for a design fragment that is selected by a user from the set of search results; and

generating an additional set of search results based on the embedding.

26 . The non-transitory computer-readable medium of claim 25 , wherein generating the additional set of search results comprises:

matching the embedding to an additional set of embeddings in an index; and

adding, to the additional set of search results, an additional set of design fragments mapped to the additional set of embeddings within the index.

27 . The non-transitory computer-readable medium of claim 25 , wherein generating the additional set of search results comprises:

matching an aggregation of the one or more embeddings with the embedding to an additional set of embeddings in an index; and

adding, to the additional set of search results, an additional set of design fragments mapped to the additional set of embeddings within the index.

28 . A computer-implemented method comprising:

dividing one or more hierarchical structures representing one or more design interfaces into a plurality of design fragments;

updating an index with the plurality of design fragments and a plurality of embeddings for the plurality of design fragments; and

processing a search query that specifies a representation of one or more design elements using the index.