Artificial intelligence (AI) player modeling and training
An artificial intelligence (AI) model generates a control input for a virtual player of a video game based on input state data. A video stream of the video game is generated based on the control input for the virtual player of the video game. The video stream is transmitted to a client device. Training feedback data is received from the client device and includes a time-dependent series of human-generated feedback elements correlated to the video stream. Each feedback element indicates that the control input for the virtual player of the video game, as generated by the AI model, is either a player-approved control input or a player-disapproved control input. AI model training includes reinforcing use of the player-approved control input for the virtual player in response to the input state data, and diminishing use of the player-disapproved control input for the virtual player in response to the input state data.
1 . A method for training an artificial intelligence (AI) model for play of a video game, comprising:
receiving game state data of the video game as input state data for the AI model;
using the AI model to generate a control input for a virtual player of the video game based on the input state data;
generating a video stream of play of the video game by executing game logic of the video game in accordance with both the control input for the virtual player of the video game as generated by the AI model and the game state data of the video game;
transmitting the video stream of play of the video game to a client device of a human player of the video game;
receiving sensor data representing a time-varying response of the human player to the video stream;
determining, using the sensor data, whether the response of the human player to the video stream indicates approval or disapproval by the human player at each of multiple different times during the video stream;
generating, using a result of the determination, training feedback data including a time-dependent series of feedback elements correlated to the video stream of play of the video game, wherein each feedback element indicates that the control input for the virtual player of the video game as generated by the AI model is either a player-approved control input for the virtual player or a player-disapproved control input for the virtual player;
training the AI model by reinforcing use of the player-approved control input for the virtual player in response to the input state data; and
training the AI model by diminishing use of the player-disapproved control input for the virtual player in response to the input state data.
2 . The method as recited in claim 1 , wherein the game state data of the video game defines a state of game play of the video game by a particular player at a particular point in time.
3 . The method as recited in claim 1 , wherein the control input for the virtual player of the video game is one or more of a number of control inputs available for use by the human player of the video game in response to a state of play of the video game corresponding to the input state data.
4 . The method as recited in claim 1 , wherein the control input for the virtual player of the video game is one or more control input signals generatable by a game controller device.
5 . The method as recited in claim 1 , wherein the sensor data includes an indication of at least one interaction by the human player with a game controller device.
6 . The method as recited in claim 1 , wherein;
the sensor data includes one or more audio signals generated by a microphone configured to capture sounds of the human player of the video game; and
determining whether the response of the human player to the video stream indicates approval or disapproval by the human player comprises analyzing the one or more audio signals to identify speech of the human player.
7 . The method as recited in claim 1 , wherein the sensor data includes image data generated by a camera configured to capture video of the human player of the video game; and
determining whether the response of the human player to the video stream indicates approval or disapproval by the human player comprises analyzing the image data to identify one or more gestures made by the human player.
8 . The method as recited in claim 1 , comprising correlating the time-dependent series of feedback elements to the video stream of play of the video game by associating a given feedback element with an immediately preceding control input for the virtual player of the video game as generated by the AI model.
9 . The method as recited in claim 1 , further comprising:
time-slicing the game state data of the video game and the time-dependent series of feedback elements into a series of time slices;
identifying a given feedback element within a given time slice;
searching the game state data over a set of time slices in reverse temporal order for an immediately preceding control input for the virtual player of the video game as generated by the AI model relative to the given feedback element, wherein the set of time slices includes the given time slice;
identifying the immediately preceding control input for the virtual player of the video game as generated by the AI model; and
associating the given feedback element with the immediately preceding control input for the virtual player of the video game as generated by the AI model.
10 . The method as recited in claim 1 , wherein use of the player-approved control input for the virtual player by the AI model in response to the input state data is reinforced by increasing a weight of an interconnection between neural nodes within a neural network of the AI model, and wherein use of the player-disapproved control input for the virtual player by the AI model in response to the input state data is diminished by decreasing the weight of the interconnection between neural nodes within the neural network of the AI model.
11 . A computer system, comprising:
a processor; and
a memory coupled to the processor and having stored therein program instructions that when executed by the computer system cause the computer system to execute a method for training an artificial intelligence (AI) model for play of a video game, comprising:
receiving a game state data of the video game as input state data for the AI model;
using the AI model to generate a control input for a virtual player of the video game based on the input state data;
generating a video stream of play of the video game by executing game logic of the video game in accordance with both the control input for the virtual player of the video game as generated by the AI model and the game state data of the video game;
transmitting the video stream of play of the video game to a client device of a human player of the video game;
receiving sensor data representing a time-varying response of the human player to the video stream;
determining, using the sensor data, whether the response of the human player to the video stream indicates approval or disapproval by the human player at each of multiple different times during the video stream
generating, using a result of the determination, training feedback data including a time-dependent series of feedback elements correlated to the video stream of play of the video game, wherein each feedback element indicates that the control input for the virtual player of the video game as generated by the AI model is either a player-approved control input for the virtual player or a player-disapproved control input for the virtual player;
training the AI model by reinforcing use of the player-approved control input for the virtual player in response to the input state data; and
training the AI model by diminishing use of the player-disapproved control input for the virtual player in response to the input state data.
12 . The computer system as recited in claim 11 , wherein the game state data of the video game defines a state of game play of the video game by a particular player at a particular point in time.
13 . The computer system as recited in claim 11 , wherein the control input for the virtual player of the video game is one or more of a number of control inputs available for use by the human player of the video game in response to a state of play of the video game corresponding to the input state data.
14 . The computer system as recited in claim 11 , wherein the control input for the virtual player of the video game is one or more control input signals generatable by a game controller device.
15 . The computer system as recited in claim 11 , wherein the sensor data includes an indication of at least one interaction by the human player with a game controller device.
16 . The computer system as recited in claim 11 , wherein the sensor data includes one or more audio signals generated by a microphone configured to capture sounds of the human player of the video game; and
determining whether the response of the human player to the video stream indicates approval or disapproval by the human player comprises analyzing the one or more audio signals to identify speech of the human player.
17 . The computer system as recited in claim 11 , wherein the sensor data includes image data generated by a camera configured to capture video of the human player of the video game; and
determining whether the response of the human player to the video stream indicates approval or disapproval by the human player comprises analyzing the image data to identify one or more gestures made by the human player.
18 . The computer system as recited in claim 11 , wherein the memory includes program instructions for associating a given feedback element with an immediately preceding control input for the virtual player of the video game as generated by the AI model in order to correlate the time-dependent series of feedback elements to the video stream of play of the video game.
19 . The computer system as recited in claim 11 , wherein the memory includes program instructions for:
time-slicing the game state data of the video game and the time-dependent series of feedback elements into a series of time slices,
identifying a given feedback element within a given time slice,
searching the game state data over a set of time slices in reverse temporal order for an immediately preceding control input for the virtual player of the video game as generated by the AI model relative to the given feedback element, wherein the set of time slices includes the given time slice,
identifying the immediately preceding control input for the virtual player of the video game as generated by the AI model, and
associating the given feedback element with the immediately preceding control input for the virtual player of the video game as generated by the AI model.
20 . The computer system as recited in claim 11 , wherein the memory includes program instructions for reinforcing use of the player-approved control input for the virtual player by the AI model in response to the input state data by increasing a weight of an interconnection between neural nodes within a neural network of the AI model, and wherein memory includes program instructions for diminishing use of the player-disapproved control input for the virtual player by the AI model in response to the input state data by decreasing the weight of the interconnection between neural nodes within the neural network of the AI model.