Indirect video game context determination based on game I/O
A system for generating gameplay context information for a game may include a game screen classification module trained to classify contextually relevant data from gameplay data, one or more game object recognition modules trained to detect game icons from gameplay data, and a multimodal context generation neural network module trained to generate structured gameplay context information from the contextually relevant data and icons within the gameplay data. The multimodal context generation neural network module at least partially generates structured gameplay context information. The modules may include neural networks trained by suitable machine learning algorithms using suitable masked data and labeled data.
1 . A method for training a multi-modal system to generate a gameplay context information from unstructured data, comprising:
training a game screen classification neural network module to classify contextually relevant data from masked gameplay data using labeled gameplay data;
training one or more game object recognition modules with the masked gameplay data and labeled gameplay data to detect game icons from the masked gameplay data;
labeling contextually relevant images and game icon data with context information; and
training a multimodal context generation neural network module with unlabeled contextually relevant images and unlabeled game icon data using the contextually relevant images and the game icon data labeled with context information to generate structured gameplay context information from the unlabeled contextually relevant images and the unlabeled game icon data.
2 . The method of claim 1 further comprising:
training one or more audio classification neural network modules to classify gameplay events within gameplay audio; and
receiving at the multimodal context generation neural network module gameplay events with masked context information during training.
3 . The method of claim 1 wherein the labeled gameplay data includes labeled contextually relevant data corresponding to one or more map screens and labeled game icons that include a player location indicator and map location indicator, and wherein the structured gameplay context information includes at least one of absolute location of a player or relative map location of the player.
4 . The method of claim 1 wherein the labeled gameplay data includes contextually relevant data corresponding to one or more inventory screens within the gameplay data and labeled game icons that include item indicators and wherein the structured gameplay context information includes at least inventory contents information.
5 . The method of claim 1 further comprising:
extracting machine-readable text from gameplay data; and
labeling the machine-readable text with context information,
wherein training the multimodal context generation neural network module includes training with unlabeled machine-readable text using the machine-readable text labeled with contextual information.
6 . The method of claim 1 wherein the labeled gameplay data includes contextually relevant images corresponding to one or more character status screens within the gameplay data and labeled game icons that include status indicators and wherein the structured gameplay context information includes at least character status information.
7 . The method of claim 1 wherein the labeled gameplay data includes contextually relevant data corresponding to one or more user interface images and labeled game icons that include a user interface element and wherein the structured gameplay context information includes game character status information.
8 . The method of claim 1 further comprising:
training one or more peripheral input classification modules to classify gameplay commands using peripheral inputs labeled with gameplay commands; and
receiving at the multi-modal context generation neural network module the gameplay commands and masked context information during training.
9 . The method of claim 1 further comprising:
training one or more motion input classification modules with a fourth machine learning algorithm to classify gameplay commands using one or more motion inputs labeled with gameplay commands; and
receiving at the multi-modal context generation neural network module the gameplay commands and masked context information during training.
10 . A system for generating gameplay context information for a game, the system comprising:
memory storing instructions; and
one or more processors coupled to the memory, that upon execution of the instructions, configured to execute one or more operations comprising:
classifying contextually relevant data from gameplay data;
detecting one or more game icons from gameplay data; and
at least partially generating structured gameplay context information from the contextually relevant data and the one or more icons within the gameplay data.
11 . The system of claim 10 , wherein the one or more operations further comprise:
classifying one or more gameplay events within the gameplay audio into corresponding classified gameplay events; and
generating structured gameplay context information from the classified gameplay events.
12 . The system of claim 10 , wherein the one or more operations further comprise:
extracting machine-readable text from gameplay data; and
generating structured gameplay context information from the machine-readable text.
13 . The system of claim 10 , wherein the one or more operations further comprise:
classifying gameplay commands from gameplay data; and
generating structured gameplay context information from the classified gameplay commands.
14 . The system of claim 10 , wherein the one or more operations further comprise:
classifying gameplay commands from gameplay data into corresponding classified gameplay commands; and
generating structured gameplay context information from the classified gameplay commands.
15 . The system of claim 10 wherein the structured gameplay context information includes at least one of absolute location of a player or relative map location of the player.
16 . The system of claim 10 wherein the structured gameplay context information includes game character status information.
17 . The system of claim 10 wherein the structured gameplay context information includes game inventory information.
18 . The system of claim 10 , wherein the one or more operations further comprise:
classifying a contextually relevant sound from the gameplay data into corresponding classified contextually relevant sound;
sending an event flag to enable classifying contextually relevant data from gameplay data and detecting one or more game icons from gameplay data; and
generating structured gameplay context information from the classified contextually relevant sound.
19 . The system of claim 18 wherein the contextually relevant sound is a sound based on a menu in the game.
20 . The system of claim 10 wherein the one or more operations further comprise:
classifying a contextually relevant command from the gameplay data into corresponding classified contextually relevant command;
sending an event flag to enable classifying contextually relevant data from gameplay data and detecting one or more game icons from gameplay data; and
generating structured gameplay context information from the classified contextually relevant command.