Video tools for mobile rendered augmented reality game
A video is generated based on a user's interactions with an augmented reality (AR) client in an AR system. The AR system uses machine learning models to determine content within the images of the user's interactions (e.g., gameplay of an AR game) and the quality of the images (e.g., based on features of the images such as brightness, contrast, particular AR objects, behaviors of AR objects, etc.). A first machine learning model is applied to identify content within the images (e.g., the presence of an AR object). The AR system selects a first subset of the images to which the system applies a second machine learning model that classifies a quality score for each image. The AR system generates a video using a second subset of the image frames classified by the second machine learning model (e.g., a gameplay highlight video showing the user interacting with AR objects).
1. A non-transitory computer readable storage medium comprising stored instructions, the instructions when executed by a processor cause the processor to:
receive a set of image frames, each image frame comprising image data displayed on a mobile client;
apply a first machine learning model to the received set of image frames, the first machine learning model trained on training image data representative of a plurality real-world objects and a plurality of augmented reality (AR) engine generated objects, the first machine learning model configured to classify a real-world object and an AR engine generated object in the set of image frames;
determine, based on classifications by the first machine learning model, a first subset of the image frames that comprise image data associated with the real-world object and the AR engine generated object;
apply a second machine learning model to the first subset of the image frames, the second machine learning model configured to classify each image frame of the first subset of the image frames based on a plurality of quality scores;
select a second subset of the image frames based on classifications by the second machine learning model;
for each image frame of the second subset of the image frames:
access a pixel buffer,
generate an image from the pixel buffer, and
modify the image generated from the pixel buffer, the modified image appearing as the image data displayed on the mobile client; and
generate a video comprising the modified images associated with the second subset of the image frames.
2. The non-transitory computer readable storage medium of claim 1 , wherein the instructions further comprise instructions that when executed by the processor cause the processor to train the first machine learning model using the training image data representative of the plurality real-world objects and the plurality of AR engine generated objects.
3. The non-transitory computer readable storage medium of claim 1 , wherein the set of image frames is a first set of image frames, and wherein the instructions further comprise instructions that when executed by the processor cause the processor to:
receive a second set of image frames;
apply respective labels to the second set of image frames, each of the respective labels indicating a respective quality score of the plurality of quality scores; and
train the second machine learning model using the labeled second set of image frames.
4. The non-transitory computer readable storage medium of claim 3 , wherein the second set of image frames are received from an AR client during user operation of the AR client.
5. The non-transitory computer readable storage medium of claim 3 , wherein the instructions further comprise instructions that when executed by the processor cause, for each image frame of the second set of image frames, the processor to:
receive a user-specified quality score of the plurality of quality scores; and
determine a corresponding label based on the user-specified quality score.
6. The non-transitory computer readable storage medium of claim 1 , wherein the plurality of quality scores are associated with at least one of a presence of the real-world object in a given image frame of the set of image frames, a presence of the AR engine generated object in the given image frame, the presence of the AR engine generated object in consecutive image frames of the set of image frames, or a distance between the AR engine generated object and the real-world object in the given image frame.
7. The non-transitory computer readable storage medium of claim 1 , wherein the set of image frames are received from an AR client during user operation of the AR client.
8. The non-transitory computer readable storage medium of claim 1 , wherein instructions to modify the image generated from the pixel buffer further comprise instructions that when executed by the processor cause the processor to overlay the AR engine generated object on to the image generated from the pixel buffer.
9. The non-transitory computer readable storage medium of claim 1 , wherein the instructions further comprise instructions that when executed by the processor cause the processor to transmit the generated video to a mobile client.
10. The non-transitory computer readable storage medium of claim 1 , wherein the AR engine rendered object is rendered by a game engine.
11. A computer-implemented method comprising:
receiving a set of image frames, each image frame comprising image data displayed on a mobile client;
applying a first machine learning model to the received set of image frames, the first machine learning model trained on training image data representative of a plurality real-world objects and a plurality of augmented reality (AR) engine generated objects, the first machine learning model configured to classify a real-world object and an AR engine generated object in the set of image frames;
determining, based on classifications by the first machine learning model, a first subset of the image frames that comprise image data associated with the real-world object and the AR engine generated object;
applying a second machine learning model to the first subset of the image frames, the second machine learning model configured to classify each image frame of the first subset of the image frames based on a plurality of quality scores;
selecting a second subset of the image frames based on classifications by the second machine learning model;
for each image frame of the second subset of the image frames:
accessing a pixel buffer,
generating an image from the pixel buffer, and
modifying the image generated from the pixel buffer, the modified image appearing as the image data displayed on the mobile client; and
generating a video comprising the modified images associated with the second subset of the image frames.
12. The computer-implemented method of claim 11 , wherein the set of image frames is a first set of image frames, further comprising:
receiving a second set of image frames;
applying respective labels to the second set of image frames, each of the respective labels indicating a respective quality score of the plurality of quality scores; and
training the second machine learning model using the labeled second set of image frames.
13. The computer-implemented method of claim 12 , further comprising, for each image frame of the second set of image frames:
receiving a user-specified quality score of the plurality of quality scores; and
determining a corresponding label based on the user-specified quality score.
14. The computer-implemented method of claim 11 , wherein the plurality of quality scores are associated with at least one of a presence of the real-world object in a given image frame of the set of image frames, a presence of the AR engine generated object in the given image frame, the presence of the AR engine generated object in consecutive image frames of the set of image frames, or a distance between the AR engine generated object and the real-world object in the given image frame.
15. The computer-implemented method of claim 11 , wherein the set of image frames are received from an AR client during user operation of the AR client.
16. A system comprising:
a video classifier configured to:
receive a set of image frames, each image frame comprising image data displayed on a mobile client;
apply a first machine learning model to the received set of image frames, the first machine learning model trained on training image data representative of a plurality real-world objects and a plurality of augmented reality (AR) engine generated objects, the first machine learning model configured to classify a real-world object and an AR engine generated object in the set of image frames;
determine, based on classifications by the first machine learning model, a first subset of the image frames that comprise image data associated with the real-world object and the AR engine generated object;
apply a second machine learning model to the first subset of the image frames, the second machine learning model configured to classify each image frame of the first subset of the image frames based on a plurality of quality scores; and
select a second subset of the image frames based on classifications by the second machine learning model; and
a video generation module configured to:
for each image frame of the second subset of the image frames:
access a pixel buffer,
generate an image from the pixel buffer, and
modify the image generated from the pixel buffer, the modified image appearing as the image data displayed on the mobile client; and
generate a video comprising the modified images associated with the second subset of the image frames.
17. The system of claim 16 , wherein the set of image frames is a first set of image frames, further comprising:
receiving a second set of image frames;
applying respective labels to the second set of image frames, each of the respective labels indicating a respective quality score of the plurality of quality scores; and
training the second machine learning model using the labeled second set of image frames.
18. The system of claim 17 , further comprising, for each image frame of the second set of image frames:
receiving a user-specified quality score of the plurality of quality scores; and
determining a corresponding label based on the user-specified quality score.
19. The system of claim 16 , wherein the plurality of quality scores are associated with at least one of a presence of the real-world object in a given image frame of the set of image frames, a presence of the AR engine generated object in the given image frame, the presence of the AR engine generated object in consecutive image frames of the set of image frames, or a distance between the AR engine generated object and the real-world object in the given image frame.
20. A computer-implemented method comprising:
receiving a set of image frames, each image frame comprising image data displayed on a mobile client;
applying a first machine learning model to the received set of image frames, the first machine learning model configured to classify a real-world object and an AR engine generated object in the set of image frames;
determining, based on classifications by the first machine learning model, a first subset of the image frames that comprise image data associated with the real-world object and the AR engine generated object;
applying a second machine learning model to the first subset of the image frames, the second machine learning model configured to classify each image frame of the first subset of the image frames based on a plurality of quality scores;
selecting a second subset of the image frames based on classifications by the second machine learning model; and
generating a video based on the second subset of the image frames.