Generating semantic scene graphs utilizing template graphs for digital image modification
The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate and implement semantic scene graph for digital image editing. For instance, in some embodiments, the disclosed systems receive a digital image from a client device. The disclosed systems determine, utilizing one or more neural networks, characteristics of the digital image by determining a plurality of objects portrayed in the digital image and a plurality of relationships associated with the plurality of objects. Further, the disclosed systems generate a semantic scene graph for the digital image based on its characteristics. For example, in some cases, the disclosed systems generate a structure of nodes and edges representing the characteristics of the digital image utilizing an image analysis graph and assign behaviors to the plurality of objects based on the plurality of relationships using a behavioral policy graph. The disclosed systems modify the digital image using the semantic scene graph.
1 . A computer-implemented method comprising:
receiving a digital image from a client device;
determining, utilizing one or more neural networks, characteristics of the digital image by determining a plurality of objects portrayed in the digital image and a plurality of relationships associated with the plurality of objects;
generating, based on the characteristics of the digital image, a semantic scene graph for the digital image using a combination of pre-generated template graphs by:
generating, utilizing a first pre-generated template graph comprising an image analysis graph, a structure of nodes and edges representing the characteristics of the digital image; and
assigning, utilizing a second pre-generated template graph comprising a behavioral policy graph, behaviors to the plurality of objects based on the plurality of relationships;
detecting, via the client device, a user interaction with the digital image; and
modifying, based on the user interaction, the digital image utilizing the semantic scene graph by:
determining pixels corresponding to an object portrayed within the digital image;
identifying at least one of an object attribute or an object relationship that is associated with the object within the digital image via the semantic scene graph; and
modifying the pixels corresponding to the object based on at least one of the object attribute or the object relationship associated with the object via the semantic scene graph.
2 . The computer-implemented method of claim 1 , further comprising generating the semantic scene graph for the digital image by associating, utilizing a real-world class description graph, object attributes with the plurality of objects portrayed in the digital image.
3 . The computer-implemented method of claim 2 , further comprising associating, utilizing the real-world class description graph, hierarchies of object classifications with the plurality of objects portrayed in the digital image.
4 . The computer-implemented method of claim 2 , wherein associating, utilizing the real-world class description graph, the object attributes with the plurality of objects portrayed in the digital image comprises:
determining an object class for an object portrayed in the digital image;
identifying one or more object attributes associated with the object class within the real-world class description graph; and
associating the one or more object attributes with the object within the semantic scene graph.
5 . The computer-implemented method of claim 4 , wherein:
determining the plurality of objects portrayed in the digital image comprises identifying at least one object that supports one or more additional objects within the digital image; and
associating the one or more object attributes with the object within the semantic scene graph comprises associating a supporting weight with the at least one object within the semantic scene graph.
6 . The computer-implemented method of claim 1 , wherein generating, utilizing the image analysis graph, the structure of nodes and edges representing the characteristics of the digital image comprises:
generating, within the semantic scene graph, a plurality of nodes representing the plurality of objects portrayed in the digital image; and
generating, within the semantic scene graph, a plurality of edges connecting the plurality of nodes to represent the plurality of relationships.
7 . The computer-implemented method of claim 6 , wherein generating, utilizing the image analysis graph, the structure of nodes and edges representing the characteristics of the digital image comprises:
generating, within the semantic scene graph, an additional plurality of nodes representing object classes of the plurality of objects portrayed in the digital image; and
generating, for each node from the additional plurality of nodes, an edge connecting the node to another node from the plurality of nodes that represents an object associated with an object class represented by the node.
8 . The computer-implemented method of claim 1 , wherein modifying the digital image utilizing the semantic scene graph comprises:
receiving one or more user interactions targeting an object from the plurality of objects for modification;
identifying one or more characteristics of the digital image that are associated with the object via the semantic scene graph; and
modifying the digital image by modifying the object targeted by the one or more user interactions in accordance with the one or more characteristics that are associated with the object.
9 . The computer-implemented method of claim 1 , wherein determining the characteristics of the digital image comprises determining the characteristics of the digital image by further determining one or more of a scene of the digital image, a lighting source for the digital image, a setting of the digital image, or a location of the digital image.
10 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
generating an image analysis graph providing a structural template of nodes and edges corresponding to characteristics potentially represented in one or more digital images;
generating a real-world class description graph providing contextual information with respect to semantic areas potentially represented in the one or more digital images;
generating a behavioral policy graph that assigns behaviors to object classes based on object relationships potentially represented in the one or more digital images;
generating, for a digital image via pre-processing before receiving one or more user interactions for modifying the digital image, a semantic scene graph portraying a plurality of objects utilizing the image analysis graph, the real-world class description graph, and the behavioral policy graph; and
modifying the digital image using the semantic scene graph in response to detecting one or more user interactions with the digital image by:
determining pixels corresponding to an object portrayed within the digital image;
identifying at least one of an object attribute or an object relationship that is associated with the object within the digital image via the semantic scene graph; and
modifying the pixels corresponding to the object based on at least one of the object attribute or the object relationship associated with the object via the semantic scene graph.
11 . The non-transitory computer-readable medium of claim 10 , wherein generating the real-world class description graph providing the contextual information with respect to the semantic areas potentially represented in the one or more digital images comprises generating hierarchies of object classifications for objects potentially represented within the one or more digital images.
12 . The non-transitory computer-readable medium of claim 11 , wherein generating the semantic scene graph for the digital image portraying the plurality of objects utilizing the real-world class description graph comprises:
determining an object class for an object portrayed in the digital image; and
associating a hierarchy of object classifications that corresponds to the object class with the object within the semantic scene graph for the digital image.
13 . The non-transitory computer-readable medium of claim 11 , wherein generating the hierarchies of object classifications for the objects potentially represented within the one or more digital images comprises:
generating a first node representing an object class associated with an object potentially represented in the one or more digital images;
generating a second node representing a subclass of the object class; and
generating an edge connecting the second node to the first node.
14 . The non-transitory computer-readable medium of claim 13 , wherein generating the hierarchies of object classifications for the objects potentially represented within the one or more digital images comprises:
generating a third node representing a subclass of the subclass of the object class; and
generating an additional edge connecting the third node to the second node.
15 . The non-transitory computer-readable medium of claim 10 , wherein generating the real-world class description graph providing the contextual information with respect to the semantic areas potentially represented in the one or more digital images comprises generating anatomies for objects potentially represented within the one or more digital images, an anatomy for an object representing one or more components of the object.
16 . The non-transitory computer-readable medium of claim 10 , wherein generating the image analysis graph providing the structural template of nodes and edges corresponding to characteristics potentially represented in the one or more digital images comprises:
generating a first set of template nodes corresponding to scene components potentially portrayed in the one or more digital images;
generating a second set of template nodes corresponding to one or more objects potentially portrayed in the one or more digital images; and
generating edges connecting template nodes from the first set of template nodes and the second set of template nodes in a template structure.
17 . A system comprising:
at least one memory device; and
at least one processor configured to cause the system to:
determine a plurality of objects portrayed in a digital image;
determine an object class for each object from the plurality of objects;
determine relationships between pairs of objects from the plurality of objects;
generate a semantic scene graph for the digital image using a combination of pre-generated template graphs by:
generating a structure having a plurality of nodes representing the plurality of objects and a plurality of edges connecting the plurality of nodes in accordance with a first pre-generated template graph comprising an image analysis graph having a structural template of nodes and edges;
associating one or more object attributes with each object based on the object class of each object in accordance with a second pre-generated template graph comprising a real-world class description graph having object attributes for a plurality of object classes; and
assigning each object one or more behaviors based on the relationships between the pairs of objects in accordance with a third pre-generated template graph comprising a behavioral policy graph that assigns behaviors to object classes based on object relationships; and
modify the digital image using the semantic scene graph in response to detecting one or more user interactions with the digital image by:
determining pixels corresponding to an object portrayed within the digital image;
identifying at least one of an object attribute or an object relationship that is associated with the object within the digital image via the semantic scene graph; and
modifying the pixels corresponding to the object based on at least one of the object attribute or the object relationship associated with the object via the semantic scene graph.
18 . The system of claim 17 , wherein the at least one processor is further configured to cause the system to modify the digital image utilizing the semantic scene graph by modifying one or more objects portrayed in the digital image based on at least one object attribute and at least one behavior associated with the one or more objects within the semantic scene graph.
19 . The system of claim 17 , wherein the at least one processor is further configured to cause the system to generate the semantic scene graph for the digital image by associating at least one of an object mask or a bounding box with each object from the plurality of objects.
20 . The system of claim 17 , wherein the at least one processor is further configured to cause the system to generate the semantic scene graph for the digital image by associating, utilizing the real-world class description graph, a hierarchy of object classifications and an anatomy of components with each object from the plurality of objects.