Method and system for generating a recording of video game gameplay
Methods and apparatus provide for: receiving video game data generated during the playing of a video game, the video game including at least a video and corresponding audio signal; and inputting each signal in the received video game data into a respective machine learning model and determining based on the output of each model, whether a highlight event has occurred during the playing of the video game. At least some of the frames in the video and or audio signal are then selected for inclusion in a recording of the video game gameplay, based on a determination that a highlight event has occurred and a recording is generated that includes the selected video and or audio frames. A method for training the models is also provided, along with systems for training and executing the trained models.
1. A method of generating a recording of video game gameplay, the method comprising:
receiving video game data generated during the playing of a video game, the video game data comprising at least a video signal and corresponding audio signal, each signal comprising a plurality of frames, wherein the video game data further comprises a player input signal indicating player inputs received at one or more devices being used to play the video game;
receiving at each of a first, a second and a third module, a respective one of the video signal, the audio signal, and the player input signal included in the received video game data and determining, using a trained model, an event occurring within the video game based on only the respective one of the video signal, the audio signal, and the player input signal input to that module;
detecting that one or more highlight events have occurred during the playing of the video game when an output of each of the first, second and third modules indicates an event occurring between two frames of the plurality of frames; and
generating a recording of the video game gameplay based on the detecting, the recording including at least some of the video frames and/or the audio frames in the video game data that correspond to the one or more highlight events of the detecting step;
receiving, at each of a plurality of feature extractors: (i) a respective one of the video signal and the audio signal in the received video game data and to generate feature representations of the frames in such respective one of the video signal and the audio signal, and (ii) previously generated video game data, each feature extractor being configured to receive a different signal in the previously generated video game data, where at least the first and second modules that receive the video signal and the audio signal, respectively, receive the feature representations generated for a respective one of the video signal and the audio signal;
receiving, at each of a plurality of clustering units, the feature representations output by a different feature extractor and to use unsupervised learning to sort the received feature representations into a plurality of clusters;
generating labels for the clusters output by each clustering unit, the generating labels including generating the labels based on an input from a user, each label indicating an event associated with the frames or corresponding feature representations in a respective cluster; and
training at least the first and second modules, the training including determining a relationship between the frames or feature representations input to the first and second modules and the corresponding labels generated by the labelling unit,
wherein each clustering unit comprises a k-means or mini-batch k-means clustering algorithm configured to sort the feature representations into different respective clusters.
2. A method according to claim 1 , wherein each machine learning model is trained using previously generated video game data generated during a previous playing of the video game, and each machine learning model is trained using semi-supervised learning to determine a relationship between one of the video signal and the audio signal input to that model and corresponding events.
3. A method according to claim 1 , wherein the video game data further comprises one or more of:
i. a haptic signal indicating haptic feedback output at one or more devices being used to play the video game;
ii. a motion signal indicating motion of the player;
iii. a speech signal comprising the player's speech;
iv. a video camera signal comprising a video recording of the player; and
v. a telemetry signal indicating an in-game event, where the method further comprises determining whether a highlight event has occurred based on an output of each machine learning model and received telemetry signal.
4. A method according to claim 1 , comprising inputting at least some of the frames of the video signal into a video machine learning model among the plurality of machine learning models; and
wherein the video machine learning model is trained to identify a type of scene to which each of the at least some frames of the video signal correspond.
5. A method according to claim 4 , wherein the video machine learning model comprises a neural network.
6. A method according to claim 1 , comprising inputting at least some of the audio frames of the audio signal into an audio machine learning model among the plurality of machine learning models; wherein the audio machine learning model is trained to identify an audio event to which each of the at least some frames of the audio signal correspond.
7. A method according to claim 6 , comprising:
inputting at least some of the frames of the video signal into a video machine learning model among the plurality of machine learning models, wherein the video machine learning model is trained to identify a type of scene to which each of the at least some frames of the video signal correspond;
generating respective feature representations of the at least some of the frames of the video signal and the at least some of the frames of the audio signal; and
wherein the inputting the at least some of the frames of the video signal, and the inputting the at least some of the audio frames of the audio signal into, respectively, into the video machine learning model and the audio machine learning model comprises inputting the respective feature representations into the respective video machine learning model and the audio machine learning model.
8. A method according to claim 7 , wherein generating the feature representations of the video signal comprises inputting at least some of the frames of the video signal into a pre-trained model.
9. A method according to claim 7 , wherein generating the feature representations of the audio signal comprises generating a mel-spectrogram of at least some of the frames of the audio signal.
10. A method according to claim 6 , wherein the audio machine learning model comprises a logistic regression model or a binary classification model.
11. A method according to claim 1 , wherein each trained one of the machine learning models is executed locally at a computing device being used to play the video game.
12. A method according to claim 1 , comprising:
receiving historical data generating during the playing of the video game by one or more other players;
determining a correspondence between at least some of the historical data and the at least some of the frames of the video signal and/or the audio signal selected; and
displaying the historical data that corresponds with the at least some of the frames of the video signal and/or the audio signal selected, when playing back the recording of video game gameplay.
13. A non-transitory, computer readable medium having computer executable instructions stored thereon, which when executed by a computer, cause the computer system to generate a recording of video game gameplay, by carrying out actions, comprising:
receiving video game data generated during the playing of a video game, the video game data comprising at least a video signal and corresponding audio signal, each signal comprising a plurality of frames, wherein the video game data further comprises a player input signal indicating player inputs received at one or more devices being used to play the video game;
receiving at each of a first, a second and a third module, a respective one of the video signal, the audio signal, and the player input signal included in the received video game data and determining, using a trained model, an event occurring within the video game based on only the respective one of the video signal, the audio signal, and the player input signal input to that module;
detecting that one or more highlight events have occurred during the playing of the video game when an output of each of the first, second and third modules indicates an event occurring between two frames of the plurality of frames; and
generating a recording of the video game gameplay based on the detecting, the recording including at least some of the video frames and/or the audio frames in the video game data that correspond to the one or more highlight events of the detecting step;
receiving, at each of a plurality of feature extractors: (i) a respective one of the video signal and the audio signal in the received video game data and to generate feature representations of the frames in such respective one of the video signal and the audio signal, and (ii) previously generated video game data, each feature extractor being configured to receive a different signal in the previously generated video game data, where at least the first and second modules that receive the video signal and the audio signal, respectively, receive the feature representations generated for a respective one of the video signal and the audio signal;
receiving, at each of a plurality of clustering units, the feature representations output by a different feature extractor and to use unsupervised learning to sort the received feature representations into a plurality of clusters;
generating labels for the clusters output by each clustering unit, the generating labels including generating the labels based on an input from a user, each label indicating an event associated with the frames or corresponding feature representations in a respective cluster; and
training at least the first and second modules, the training including determining a relationship between the frames or feature representations input to the first and second modules and the corresponding labels generated by the labelling unit,
wherein each clustering unit comprises a k-means or mini-batch k-means clustering algorithm configured to sort the feature representations into different respective clusters.
14. A system for generating a recording of video game gameplay, the system comprising:
a receiving unit configured to receive video game data generated during the playing of a video game, the video game data comprising at least a video signal and corresponding audio signal, each signal comprising a plurality of frames, wherein the video game data further comprises a player input signal indicating player inputs received at one or more devices being used to play the video game;
a first, second and third module, each module being configured to receive a respective one of the video signal, the audio signal, and the player input signal included in the video game data and to determine, using a trained model, an event occurring within the video game based on only the respective one of the video signal, the audio signal, and the player input signal input to that module;
a highlight detector configured to detect that a highlight event has occurred during the playing of the video game when an output of each of the first, second and third modules indicates an event occurring between two frames of the plurality of frames;
a recording unit configured to generate a recording of the video game gameplay based on an output of the highlight detector, the recording including at least some of the video and or audio frames in the video game data that corresponds to the one or more detected highlight events;
a plurality of feature extractors, each feature extractor being configured to: (i) receive a respective one of the video signal and the audio signal in the received video game data and to generate feature representations of the frames in such respective one of the video signal and the audio signal, and (ii) receive previously generated video game data, each feature extractor being configured to receive a different signal in the previously generated video game data, where at least the first and second modules that receive the video signal and the audio signal, respectively, are configured to receive the feature representations generated for a respective one of the video signal and the audio signal;
a plurality of clustering units, each clustering unit being configured to receive the feature representations output by a different feature extractor and to use unsupervised learning to sort the received feature representations into a plurality of clusters;
a labelling unit operable to generate labels for the clusters output by each clustering unit, the labelling unit being configured to generate the labels based on an input from a user, each label indicating an event associated with the frames or corresponding feature representations in a respective cluster; and
a training unit configured to train at least the first and second modules, the training unit being configured to determine a relationship between the frames or feature representations input to the first and second modules and the corresponding labels generated by the labelling unit,
wherein each clustering unit comprises a k-means or mini-batch k-means clustering algorithm configured to sort the feature representations into different respective clusters.
15. A system according to claim 14 , wherein at least the first and second modules that receive the video signal and the audio signal, respectively, are trained using semi-supervised learning so as to determine a relationship between the respective one of the video signal and the audio signal input to that model and corresponding events.
16. A system according to claim 14 , wherein
the video game data further comprises a telemetry signal indicating in-game events; and
wherein the highlight detector is configured to determine an event occurring within the video game based on an output of the first, second, and third modules and a received telemetry signal.
17. A system according to 14 , wherein a first feature extractor is configured to receive video frames and a second feature extractor is configured to receive audio frames; and
wherein the first module is configured to receive feature representations of video frames and the second module is configured to receive feature representations of audio frames.