Providing diverse and representative swatches extracted from an input image

1. A method of determining a set of representative swatches of an image, wherein the method includes one or more processing devices performing operations comprising:

determining a salient object in an image, where the salient object includes pixels that depict a pattern or color visible on the salient object;

extracting multiple swatches from the image, wherein each of the multiple swatches includes pixels that depict the pattern or color visible on the salient object;

computing selection scores for the multiple swatches, respectively, wherein, for each particular swatch of the multiple swatches, a respective selection score is computed by:

determining, for the particular swatch, (i) a respective likelihood score indicating a representativeness of the pattern or color depicted by the pixels included in the particular swatch, wherein the respective likelihood score is calculated based on a combination of an inner region similarity score of the particular swatch and an outer region similarity score of the particular swatch, and (ii) a respective dominance value indicating a dominance of the pattern or color depicted by the pixels included in the particular swatch, and

determining, for the particular swatch, a respective selection score based on a combination of the respective likelihood score and the respective dominance value; and

generating, based on the selection scores of the multiple swatches, a subset of the multiple swatches extracted from the image.

2. The method of claim 1 , wherein the inner region similarity score of the particular swatch is determined by additional operations comprising:

extracting features from a first region associated with the particular swatch, wherein the first region includes pixels that are also included in the particular swatch;

determining a first feature vector for the first region, the first feature vector based on the extracted features; and

defining the inner region similarity score based on the first feature vector for the first region.

3. The method of claim 1 , wherein the outer region similarity score of the particular swatch further is determined by additional operations comprising:

extracting additional features from a second region associated with the particular swatch, wherein the second region includes pixels that are not included in the particular swatch;

determining a second feature vector for the second region, the second feature vector based on the extracted additional features; and

defining the outer region similarity score based on the second feature vector for the second region.

4. The method of claim 1 , wherein the respective likelihood score of the particular swatch is based on an output from a deep learning neural network.

5. The method of claim 1 , wherein determining the respective dominance value of the particular swatch comprises:

defining a matrix, the matrix having a dimension based on a quantity of the multiple swatches;

determining multiple values in the matrix that are related to the particular swatch, wherein each respective value of the multiple values is based on (i) a respective similarity metric between the particular swatch and a respective additional swatch of the multiple swatches, and (ii) the respective likelihood score of the respective additional swatch; and

determining an aggregation of the multiple values related to the particular swatch.

6. The method of claim 1 , wherein determining the respective selection score of the particular swatch comprises:

determining, for each of the multiple swatches, a respective product based on the respective likelihood score of each of the multiple swatches and the respective dominance value of each of the multiple swatches; and

determining a maximum product of the respective products.

7. The method of claim 1 , wherein determining the salient object in the image comprises generating a binary mask for a localized foreground region, wherein a mask pixel included in the binary mask indicates that a corresponding image pixel in the image is included in the salient object.

8. A non-transitory computer-readable medium embodying program code for determining a set of representative swatches of an image, the program code comprising instructions which, when executed by a processor, cause the processor to perform operations comprising:

determining a salient object in an image, where the salient object includes pixels that depict a pattern or color visible on the salient object;

extracting multiple swatches from the image, wherein each of the multiple swatches includes pixels that depict the pattern or color visible on the salient object;

computing selection scores for the multiple swatches, respectively, wherein, for each particular swatch of the multiple swatches, a respective selection score is computed by:

determining, for the particular swatch, (i) a respective likelihood score indicating a representativeness of the pattern or color depicted by the pixels included in the particular swatch, wherein the respective likelihood score is calculated based on a combination of an inner region similarity score of the particular swatch and an outer region similarity score of the particular swatch, and (ii) a respective dominance value indicating a dominance of the pattern or color depicted by the pixels included in the particular swatch, and

determining, for the particular swatch, a respective selection score based on a combination of the respective likelihood score and the respective dominance value; and

generating, based on the selection scores of the multiple swatches, a subset of the multiple swatches extracted from the image.

9. The non-transitory computer-readable medium of claim 8 , wherein the inner region similarity score of the particular swatch is determined by:

extracting features from a first region associated with the particular swatch, wherein the first region includes pixels that are also included in the particular swatch;

determining a first feature vector for the first region, the first feature vector based on the extracted features; and

defining the inner region similarity score based on the first feature vector for the first region.

10. The non-transitory computer-readable medium of claim 8 , wherein the outer region similarity score of the particular swatch is determined by:

extracting additional features from a second region associated with the particular swatch, wherein the second region includes pixels that are not included in the particular swatch;

determining a second feature vector for the second region, the second feature vector based on the extracted additional features; and

defining the outer region similarity score based on the second feature vector for the second region.

11. The non-transitory computer-readable medium of claim 8 , wherein the respective likelihood score of the particular swatch is based on an output from a deep learning neural network.

12. The non-transitory computer-readable medium of claim 8 , wherein determining the respective dominance value of the particular swatch comprises:

defining a matrix, the matrix having a dimension based on a quantity of the multiple swatches;

determining multiple values in the matrix that are related to the particular swatch, wherein each respective value of the multiple values is based on (i) a respective similarity metric between the particular swatch and a respective additional swatch of the multiple swatches, and (ii) the respective likelihood score of the respective additional swatch; and

determining an aggregation of the multiple values related to the particular swatch.

13. The non-transitory computer-readable medium of claim 8 , wherein determining the respective selection score of the particular swatch comprises:

determining, for each of the multiple swatches, a respective product based on the respective likelihood score of each of the multiple swatches and the respective dominance value of each of the multiple swatches; and

determining a maximum product of the respective products.

14. A system for determining a set of representative swatches of an image, the system comprising:

a means for determining a salient object in an image, where the salient object includes pixels that depict a pattern or color visible on the salient object;

a means for extracting multiple swatches from the image, wherein each of the multiple swatches includes pixels that depict the pattern or color visible on the salient object;

a means for computing selection scores for the multiple swatches, respectively, wherein, for each particular swatch of the multiple swatches, a respective selection score is computed by:

determining, for the particular swatch, (i) a respective likelihood score indicating a representativeness of the pattern or color depicted by the pixels included in the particular swatch, wherein the respective likelihood score is calculated based on a combination of an inner region similarity score of the particular swatch and an outer region similarity score of the particular swatch, and (ii) a respective dominance value indicating a dominance of the pattern or color depicted by the pixels included in the particular swatch, and

determining, for the particular swatch, a respective selection score based on a combination of the respective likelihood score and the respective dominance value; and

a means for generating, based on the selection scores of the multiple swatches, a subset of the multiple swatches extracted from the image.

15. The system of claim 14 , wherein the inner region similarity score of the particular swatch is determined by:

a means for extracting features from a first region associated with the particular swatch, wherein the first region includes pixels that are also included in the particular swatch;

a means for determining a first feature vector for the first region, the first feature vector based on the extracted features; and

a means for defining the inner region similarity score based on the first feature vector for the first region.

16. The system of claim 14 , wherein the outer region similarity score of the particular swatch is determined by:

a means for extracting additional features from a second region associated with the particular swatch, wherein the second region includes pixels that are not included in the particular swatch;

a means for determining a second feature vector for the second region, the second feature vector based on the extracted additional features; and

a means for defining the outer region similarity score based on the second feature vector for the second region.

17. The system of claim 14 , wherein the respective likelihood score of the particular swatch is based on an output from a deep learning neural network.

18. The system of claim 14 , wherein determining the dominance value of the particular swatch comprises:

a means for defining a matrix, the matrix having a dimension based on a quantity of the multiple swatches;

a means for determining multiple values in the matrix that are related to the particular swatch, wherein each respective value of the multiple values is based on (i) a respective similarity metric between the particular swatch and a respective additional swatch of the multiple swatches, and (ii) the respective likelihood score of the respective additional swatch; and

a means for determining an aggregation of the multiple values related to the particular swatch.

19. The system of claim 14 , wherein determining the selection score of the particular swatch comprises:

a means for determining, for each of the multiple swatches, a respective product based on the respective likelihood score of each of the multiple swatches and the respective dominance value of each of the multiple swatches; and

a means for determining a maximum product of the respective products.

20. The system of claim 14 , wherein determining the salient object in the image comprises generating a binary mask for a localized foreground region, wherein a mask pixel included in the binary mask indicates that a corresponding image pixel in the image is included in the salient object.