Using game state data for semantic understanding by an AI image generation model
A method is provided, including: receiving a game image, said game image being captured from gameplay of a video game and said game image depicting a scene of the video game; receiving game state data describing attributes of the scene of the video game that is depicted in the game image; receiving, over a network from a client device, modification data describing changes to the game image, the modification data defined from user input received at the client device; applying the game image, the game state data, and the user input, by an image generation artificial intelligence (AI) to generate an AI generated image; transmitting the AI generated image over the network to the client device for rendering to a display.
1 . A method, comprising:
receiving a game image, said game image being captured from gameplay of a video game and said game image depicting a scene of the video game;
receiving game state data describing attributes of the scene of the video game that is depicted in the game image;
generating, based in part on at least one feature shown in the game image, a user interface suggesting at least one modification to the at least one feature;
receiving, over a network from a client device, at least one modification data describing changes to the game image, the at least one modification data defined from a user input received at the client device;
applying the game image, the game state data, and the user input, by an image generation tool to generate a first image; and
transmitting the first image over the network to the client device for rendering to a display.
2 . The method of claim 1 , wherein the applying of the game state data enables a semantic understanding of the scene depicted in the game image by the image generation tool.
3 . The method of claim 2 , wherein the semantic understanding of the scene is applied by the image generation tool to execute the changes described in the at least one modification data.
4 . The method of claim 1 , wherein the game state data identifies one or more elements in the scene that is depicted in the game image.
5 . The method of claim 1 , wherein the game state data describes depths of one or more virtual objects in the scene.
6 . The method of claim 5 , wherein the at least one modification data describes a change in a location of a given virtual object within the scene, and wherein the first image is generated based in part on the depths of the one or more virtual objects to determine the change in location of the given virtual object described by the at least one modification data.
7 . The method of claim 5 , wherein the at least one modification data describes a placement of a given virtual object within the scene, and wherein the first image is generated based in part on the depths of the one or more virtual objects to determine the placement of the given virtual object described by the at least one modification data.
8 . The method of claim 5 , wherein the depths of the one or more virtual objects is configured to enable proper occlusion of, or by, the one or more virtual objects when executing the changes described by the at least one modification data.
9 . The method of claim 1 , wherein the game state data describes three-dimensional structures of one or more virtual objects in the scene.
10 . A non-transitory computer-readable medium having program instructions embodied thereon, said program instructions being configured, when executed by at least one server computer, to cause said at least one server computer to perform a method including:
receiving a game image, said game image being captured from gameplay of a video game and said game image depicting a scene of the video game;
receiving game state data describing attributes of the scene of the video game that is depicted in the game image;
generating, based in part on at least one feature shown in the game image, a user interface suggesting at least one modification to the at least one feature;
receiving, over a network from a client device, at least one modification data describing changes to the game image, the at least one modification data defined from a user input received at the client device;
applying the game image, the game state data, and the user input, by an image generation tool to generate a first image; and
transmitting the first image over the network to the client device for rendering to a display.
11 . The non-transitory computer-readable medium of claim 10 , wherein the applying of the game state data enables a semantic understanding of the scene depicted in the game image by the image generation tool.
12 . The non-transitory computer-readable medium of claim 11 , wherein the semantic understanding of the scene is applied by the image generation tool to execute the changes described in the at least one modification data.
13 . The non-transitory computer-readable medium of claim 10 , wherein the game state data identifies one or more elements in the scene that is depicted in the game image.
14 . The non-transitory computer-readable medium of claim 10 , wherein the game state data describes depths of one or more virtual objects in the scene.
15 . The non-transitory computer-readable medium of claim 14 , wherein the at least one modification data describes a change in a location of a given virtual object within the scene, and wherein the first image is generated based in part on the depths of the one or more virtual objects to determine the change in location of the given virtual object described by the at least one modification data.
16 . The non-transitory computer-readable medium of claim 14 , wherein the at least one modification data describes a placement of a given virtual object within the scene, and wherein the first image is based in part on the depths of the one or more virtual objects to determine the placement of the given virtual object described by the at least one modification data.
17 . The non-transitory computer-readable medium of claim 14 , wherein the depths of the one or more virtual objects is configured to enable proper occlusion of, or by, the one or more virtual objects when executing the changes described by the at least one modification data.
18 . The non-transitory computer-readable medium of claim 10 , wherein the game state data describes three-dimensional structures of one or more virtual objects in the scene.
19 . The non-transitory computer-readable medium of claim 10 , wherein the at least one modification data is defined by words or phrases generated by the user input received at the client device.
20 . The method of claim 1 , further comprising:
identifying, from a plurality of modification data, a modification preference;
storing the modification preference in a user profile; and
generating at least one modification based in part on the modification preference.