Modality-expanding gaming experience
Systems and methods are disclosed for expanding the output modalities of a gameplay session. Gameplay input data, such as audio data, video data, haptics data, and Gameplay context data, can be processed using machine learning (ML) models to generate output state vectors, which can be used to drive state changes in remote devices, such as smarthome devices (e.g., smart lightbulbs, smart light switches, smart fans, smart thermostats, smart blinds, smart speakers, etc.). To generate the output state vectors, first gameplay features are extracted from the gameplay input data, then the gameplay features are processed by a state vector generation model to generate the state vectors. Each modality of gameplay input data can be processed by a corresponding feature extractor. Gameplay sessions can be evaluated to train the ML models.
1 . A computer-implemented method comprising:
receiving gameplay input data associated with a first electronic device engaging in a virtual gameplay session;
identifying one or more gameplay features from the gameplay input data;
determining, for a remote device in a physical environment that includes the first electronic device, a latency value;
generating, based at least in part on the one or more gameplay features, a state change instruction to change a physical state for a remote device, wherein the state change instruction compensate for corresponding latency, based at least in part on the latency value, when facilitating the change of the physical state; and
causing, based at least in part on the state change instruction, the remote device to change from a first physical state to a second physical state.
2 . The computer-implemented method of claim 1 , wherein the gameplay input data includes: i) gameplay audio data; ii) gameplay video data; iii) gameplay haptics data; iv) gameplay context data; or v) any combination of i-iv.
3 . The computer-implemented method of claim 1 , wherein identifying the one or more gameplay features includes applying the gameplay input data to one or more machine learning models trained according to a unique input modality.
4 . The computer-implemented method of claim 3 , wherein:
the gameplay input data includes a plurality of gameplay data streams;
each game play data stream of the plurality of gameplay data streams is associated with a different input modality of a plurality of input modalities;
the one or more machine learning models being a plurality of machine learning models;
each machine learning model of the plurality of machine learning models being associated with the different input modality; and
identifying the one or more gameplay features from the gameplay input data includes applying each gameplay data stream to a respective machine learning model of the machine learning models according to a respective input modality of the plurality of input modalities that corresponds with each gameplay data stream.
5 . The computer-implemented method of claim 1 , wherein generating the state change instruction includes applying the one or more gameplay features to a physical state control model that controls the physical state of the remote device.
6 . The computer-implemented method of claim 5 , wherein the physical state control model is a probabilistic generative model.
7 . The computer-implemented method of claim 5 , wherein:
applying the one or more gameplay features to the physical state control model includes applying a weighting value for each of the one or more gameplay features to generate one or more weighted gameplay features; and
the computer-implemented method further comprises determining one or more state change instructions based at least in part on the one or more weighted gameplay features.
8 . The computer-implemented method of claim 7 , further comprising:
determining a gameplay experience quality score associated with the virtual gameplay session; and
adjusting the weighting value, based at least in part on the gameplay experience quality score, to improve the gameplay experience quality score.
9 . The computer-implemented method of claim 8 , wherein determining the gameplay experience quality score includes:
receiving user feedback associated with the virtual gameplay session; and
processing the user feedback to determine the gameplay experience quality score.
10 . The computer-implemented method of claim 5 , wherein the physical state control model is pretrained according to a baseline model and the method further comprises:
receiving user feedback associated with the virtual gameplay session; and
updating the physical state control model based at least in part on the user feedback.
11 . The computer-implemented method of claim 1 ,
wherein the remote device is one remote device of a plurality of remote devices and the state change instruction is one of a plurality of state change instructions, and
the computer-implemented method further comprises determining a set of changeable states associated with the plurality of remote devices, wherein generating the plurality of state change instructions include generating, for each changeable state of the set of changeable states, a corresponding state change instruction of the plurality of state change instructions.
12 . The computer-implemented method of claim 1 , wherein causing the change from the first physical state includes transmitting a command signal via a wired connection or a wireless connection to change the first physical state of the remote device to the second physical state.
13 . The computer-implemented method of claim 1 , wherein:
wherein the remote device is one remote device of a plurality of remote devices, the state change instruction is one state change instruction of a plurality of state change instructions, and the latency value is one latency value of a plurality of latency values; and
identifying the one or more gameplay features from the gameplay input data includes predicting values for at least one gameplay feature of the one or more gameplay features up to a future time equal to a current time plus a largest latency value of the plurality of latency values, wherein generating the state change instruction is based at least in part on the plurality of latency values such that, for a given remote device of the plurality of remote devices, a respective generated state change instruction accounts for a respective latency value of the plurality of latency values.
14 . The computer-implemented method of claim 1 , wherein:
generating the state change instruction occurs on a server that is communicatively coupled to a gaming device being used by a user to engage in the virtual gameplay session; and
causing the change from the first physical state includes transmitting the state change instruction to the gaming device.
15 . The computer-implemented method of claim 14 , wherein causing the change to the second physical state includes transmitting, from the gaming device, a command signal, via a wired connection or a wireless connection, to change the first physical state of the remote device to the second physical state.
16 . The computer-implemented method of claim 14 , wherein:
the state change instruction is one state change instruction of a plurality of state change instructions corresponding to a plurality of available physical state changes of the physical environment; and
the plurality of state change instructions, when received by the gaming device, are processed to exclude each state change instruction of the plurality of state change instructions associated with a first available physical state changes of the plurality of available physical state changes.
17 . The computer-implemented method of claim 1 , further comprising receiving output modality capability information indicative of a plurality of available physical state changes of the physical environment, wherein generating the state change instruction is based at least in part on the output modality capability information such that the state change instruction is associated with one available physical state change of the plurality of available physical state changes.
18 . The computer-implemented method of claim 1 , further comprising receiving gameplay output state data associated with the first electronic device engaging in the virtual gameplay session, wherein:
the gameplay output state data is:
indicative of one or more desired physical states associated with one or more available physical state change of a plurality of available physical state changes; and
associated with the virtual gameplay session; and
causing the change from the first physical state to the second physical state is further based at least in part on the gameplay output state data.
19 . The computer-implemented method of claim 1 , further comprising determining position information of the remote device relative to an expected location of the first electronic device, wherein generating the state change instruction is further based at least in part on the position information.
20 . The computer-implemented method of claim 1 , wherein the remote device can be one of: i) a light source; ii) a fan; iii) a thermostat; iv) a window blinds controller; v) a speaker, the speaker not directly outputting gameplay audio data; vi) a switch; or vii) a dimmer.
21 . The computer-implemented method of claim 1 , wherein causing the change from the first physical state includes employing an application programming interface associated with the remote device.
22 . The computer-implemented method of claim 1 , wherein the physical environment includes a second electronic device outputting a display of the virtual gameplay session and a third electronic device outputting audio of the virtual gameplay.
23 . A system comprising:
a memory having stored thereon computer-executable instructions; and
a control system including one or more processors configured to access the memory and execute the computer-executable instructions to perform operations comprising:
receiving gameplay input data associated with a first electronic device engaging in a virtual gameplay session;
identifying one or more gameplay features from the gameplay input data;
receiving output modality capability information indicative of a plurality of available physical state changes of a physical environment that includes the first electronic device;
generating, based at least in part on the one or more gameplay features and the output modality capability information, a state change instruction to change an available physical state change of the plurality of available physical state changes for a remote device in the physical environment; and
causing, based at least in part on the state change instruction, the remote device to change from a first physical state to a second physical state.
24 . One or more non-transitory computer readable media comprising computer-executable instructions that, when executed by one or more processors of an electronic device, cause the electronic device perform operations comprising:
receiving gameplay input data associated with a first electronic device engaging in a virtual gameplay session;
identifying one or more gameplay features from the gameplay input data;
determining position information of a remote device, relative to an expected location of the first electronic device, in a physical environment that includes the first electronic device;
generating, based at least in part on the one or more gameplay features and the position information, a state change instruction to change a physical state for the remote device; and
causing, based at least in part on the state change instruction, the remote device to change from a first physical state to a second physical state.
25 . The or more non-transitory computer-readable media of claim 24 , further comprising additional computer-executable instructions that, when executed by the one or more processors, cause the electronic device to perform additional operations comprising determining, for the remote device, a latency value, wherein generating the state change instruction is further based at least in part on the latency value such that corresponding latency is compensated for when facilitating the change of the physical state the remote device.