Augmentation of digital images with simulated surface coatings
A coating product selection system and method. Recognized objects in an input image can be used to determine one or more dominant colors for determining recommended coating products. An image augmentation system and method for simulating the application of a coating to a surface of the image in a scene. A scene record can store data records related to visualization of a scene such that multiple scene visualization clients can present painted images augmented based on assigned coatings.
1. A coating product selection system comprising:
one or more memories and one or more processors configured to:
receive an input image,
perform searches to detect a plurality of recognized objects depicted in the input image;
generate a plurality of first image segments, each respective first image segment of the plurality of first image segments comprising pixels of the input image that depict a respective recognized object of the plurality of recognized objects and not pixels of the input image that do not depict the respective recognized object;
generate a plurality of eroded image segmentations for paintable surfaces corresponding to respective ones of the recognized objects, wherein the eroded image segmentations for the paintable surfaces comprise reduced size copies of the first image segments;
replace the plurality of first image segments with a plurality of second image segments, wherein to replace the plurality of first image segments with the plurality of second image segments, the one or more processors, for each respective recognized object of the plurality of recognized objects:
apply a graph cut algorithm to generate a second image segment, of the plurality of second image segments, for a respective paintable surface, wherein:
the respective paintable surface corresponds to the respective recognized object,
the graph cut algorithm uses a plurality of foreground seed points and a plurality of background seed points,
an eroded image segmentation, of the plurality of eroded image segmentations, for the respective paintable surface is used as the foreground seed points, and
eroded image segmentations, of the plurality of eroded image segmentations, for paintable surfaces corresponding to other ones of the recognized objects are used as the background seed points; and
replace a first image segment, of the plurality of first image segments, for the respective paintable surface with the second image segment for the respective paintable surface;
for at least a first recognized object of the plurality of recognized objects, determine one or more dominant colors of the first recognized object from a second image segment, of the plurality of second image segments, for the first recognized object;
determine one or more recommended coating products, each of the one or more recommended coating products having a color selected to coordinate with at least one of the one or more dominant colors of the first recognized object;
present, on a display, a user interface containing elements that show colors of a plurality of coating products, wherein the colors of the plurality of coating products include the colors of the one or more recommended coating products; and
receive a user selection of a selected coating product from among the plurality of coating products.
2. The coating product selection system of claim 1 , wherein the one or more processors are configured to apply a classification model trained to identify a room type of the input image and wherein each of the one or more recommended coating products has a product type selected based on the identified room type.
3. The coating product selection system of claim 2 , wherein the classification model is trained to identify the room type selected from a group of room types consisting of: kitchen, living room, dining room, bedroom, bathroom, laundry room, mud room, office, nursery, and recreation room.
4. The coating product selection system of claim 1 , wherein the one or more processors are configured to perform the searches by providing the input image to an image segmentation model trained to identify pixels corresponding to at least one class of focus objects.
5. The coating product selection system of claim 1 , wherein the one or more processors are configured to determine the one or more dominant colors from the plurality of recognized objects.
6. The coating product selection system of claim 5 , wherein the one or more processors are configured to determine the one or more dominant colors from a subset of the plurality of recognized objects.
7. The coating product selection system of claim 1 , wherein the one or more dominant colors are determined based on colors within a pre-existing color library stored in the one or more memories.
8. The coating product selection system of claim 1 , wherein each of the one or more recommended coating products has a color corresponding to a color within a pre-existing color library stored in the one or more memories.
9. The coating product selection system of claim 1 , wherein:
wherein the user interface is configured to receive a user selection of one or more coating assignments, each of the one or more coating assignments comprising a selected coating product and a selected paintable image segment corresponding to a paintable image segment of the plurality of recognized objects; and
the one or more processors are further configured to generate a painted image, each pixel of the painted image having a painted color determined to be the same color as a corresponding pixel of the input image if the pixel is not within a paintable image segment of at least one of the one or more coating assignments and determined based on the selected coating product of a coating assignment of the one or more coating assignments if the corresponding pixel of the input image is within the paintable image segment of the coating assignment.
10. The coating product selection system of claim 9 , wherein the one or more processors are configured to detect the plurality of recognized objects by applying an image segmentation model trained to the input image to identify classes of surfaces selected from a group consisting of: wall surfaces, ceiling surfaces, and trim surfaces.
11. A computer-implemented method for selecting and displaying a coating product, the computer-implemented method comprising:
receiving, by one or more processors, an input image;
performing, by the one or more processors, searches to detect a plurality of recognized objects depicted in the input image;
generating, by the one or more processors, a plurality of first image segments, each respective image segment of the plurality of first image segments comprising pixels of the input image that depict a respective recognized object of the plurality of recognized objects and not pixels of the input image that do not depict the respective recognized object;
generating, by the one or more processors, a plurality of eroded image segmentations for paintable surfaces corresponding to respective ones of the recognized objects, wherein the eroded image segmentations for the paintable surfaces comprise reduced size copies of the first image segments;
replacing, by the one or more processors, the plurality of first image segments with a plurality of second image segments, wherein replacing the plurality of first image segments with the plurality of second image segments comprises, for each respective recognized object of the plurality of recognized objects:
applying a graph cut algorithm to generate a second image segment, of the plurality of second image segments, for a respective paintable surface, wherein:
the respective paintable surface corresponds to the respective recognized object,
the graph cut algorithm uses a plurality of foreground seed points and a plurality of background seed points,
an eroded image segmentation, of the plurality of eroded image segmentations, for the respective paintable surface is used as the foreground seed points, and
eroded image segmentations, of the plurality of eroded image segmentations, for paintable surfaces corresponding to other ones of the recognized objects are used as the background seed points; and
replacing the first image segment for the respective paintable surface with the second image segment for the respective paintable surface;
for at least a first recognized object of the plurality of recognized objects, determining, by the one or more processors, one or more dominant colors of the first recognized object from a second image segment, of the plurality of second image segments, for the first recognized object;
determining, by the one or more processors, one or more recommended coating products, each of the one or more recommended coating products having a color selected to coordinate with at least one of the one or more dominant colors of the first recognized object;
presenting, by the one or more processors, on a display, a user interface containing elements that show colors of a plurality of coating products, wherein the colors of the plurality of coating products include the colors of the one or more recommended coating products; and
receiving, by the one or more processors, a user selection of a selected coating product from among the plurality of coating products.
12. The computer-implemented method of claim 11 , wherein performing the searches comprises providing the input image to a classification model trained to identify a room type of the input image and wherein each of the one or more recommended coating products has a product type selected based on the identified room type.
13. The computer-implemented method of claim 12 , wherein the classification model is trained to identify the room type selected from a group of room types consisting of: kitchen, living room, dining room, bedroom, bathroom, laundry room, mud room, office, nursery, and recreation room.
14. The computer-implemented method of claim 11 , wherein performing the searches comprises providing the input image to an image segmentation model trained to identify pixels corresponding to at least one class of focus objects.
15. The computer-implemented method of claim 11 , wherein determining the one or more dominant colors of the first recognized object comprises determining colors within a pre-existing color library stored in one or more memories that correspond to color data of pixels in the second image segment.
16. The computer-implemented method of claim 11 , wherein each of the one or more recommended coating products has a color corresponding to a color within a pre-existing color library stored in one or more memories.
17. The computer-implemented method of claim 11 , further comprising:
receiving, by the one or more processors, at the user interface, a user selection of one or more coating assignments, each of the one or more coating assignments comprising a selected coating product and a selected paintable image segment corresponding to a paintable image segment of the plurality of recognized objects; and
generating, by the one or more processors, a painted image for display at the user interface, each pixel of the painted image having a painted color determined to be the same color as a corresponding pixel of the input image if the pixel is not within a paintable image segment of at least one of the one or more coating assignments and determined based on the selected coating product of a coating assignment of the one or more coating assignments if the corresponding pixel of the input image is within the paintable image segment of the coating assignment.
18. The computer-implemented method of claim 17 , wherein detecting the plurality of recognized objects comprises providing the input image to an image segmentation model trained to identify classes of surfaces selected from a group consisting of: wall surfaces, ceiling surfaces, and trim surfaces.