LLM-BASED ANIMATION SELECTION FOR AN AR CHARACTER
A client device selects animations for an AR character by prompting a large language model (LLM) to select from a set of possible animations for the AR character. The client device captures an image of its environment using a camera and identifies objects that are depicted in the image. The client device generates a prompt for an LLM that instructs the LLM to select from a set of candidate actions for an AR character to perform based on the identified objects. The LLM returns a response to the client device and the client device extracts a set of selected actions from the LLM's response. The client device identifies a set of animations that correspond to the actions selected by the LLM and renders AR content that depicts the AR character performing those actions. The client device augments the captured image to include the AR content and displays the augmented image.
1 . A computer-implemented method comprising:
accessing an image captured by a camera of a client device, wherein the image depicts an area of a physical world around the client device;
identifying a set of objects depicted in the image;
generating an LLM prompt based on the accessed image and the set of objects, wherein the LLM prompt comprises a set of candidate actions performable by an AR character and instructions for an LLM to select a subset of the candidate actions for the AR character to perform based on the accessed image and the set of objects;
transmitting the LLM prompt to the LLM;
receiving a response to the LLM, wherein the response comprises a series of selected actions, wherein the series of selected actions comprises a selected subset of the candidate actions;
generating augmented reality content based on the series of selected actions, wherein the augmented reality content comprises a series of animations of the AR character corresponding to the series of selected actions; and
displaying the augmented reality content on the client device.
2 . The method of claim 1 , further comprising:
identifying a target object of the set of objects; and
generating the LLM prompt with instructions for the LLM to select the subset of the candidate actions based on the identified target object.
3 . The method of claim 1 , wherein the LLM prompt further comprises instructions to generate computer-executable code for animating the AR character.
4 . The method of claim 3 , wherein the LLM prompt further comprises instructions to generate computer-executable code in a markup language.
5 . The method of claim 1 , wherein the LLM prompt comprises a tag for each of the set of candidate actions.
6 . The method of claim 5 , wherein the LLM prompt comprises a text description corresponding to each of the set of candidate actions.
7 . The method of claim 1 , wherein generating the augmented reality content comprises:
identifying an animation for the AR character corresponding to each of the series of selected actions.
8 . The method of claim 1 , wherein displaying the augmented reality content comprises:
transmitting the AR content to the client device.
9 . The method of claim 1 , wherein displaying the augmented reality content on the client device comprises:
augmenting the accessed image to include the augmented reality content.
10 . The method of claim 1 , wherein displaying the augmented reality content on the client device comprises:
augmenting a set of images captured after the accessed image to include the AR content.
11 . A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
accessing an image captured by a camera of a client device, wherein the image depicts an area of a physical world around the client device;
identifying a set of objects depicted in the image;
generating an LLM prompt based on the accessed image and the set of objects, wherein the LLM prompt comprises a set of candidate actions performable by an AR character and instructions for an LLM to select a subset of the candidate actions for the AR character to perform based on the accessed image and the set of objects;
transmitting the LLM prompt to the LLM;
receiving a response to the LLM, wherein the response comprises a series of selected actions, wherein the series of selected actions comprises a selected subset of the candidate actions;
generating augmented reality content based on the series of selected actions, wherein the augmented reality content comprises a series of animations of the AR character corresponding to the series of selected actions; and
displaying the augmented reality content on the client device.
12 . The computer-readable medium of claim 11 , the operations further comprising:
identifying a target object of the set of objects; and
generating the LLM prompt with instructions for the LLM to select the subset of the candidate actions based on the identified target object.
13 . The computer-readable medium of claim 11 , wherein the LLM prompt further comprises instructions to generate computer-executable code for animating the AR character.
14 . The computer-readable medium of claim 13 , wherein the LLM prompt further comprises instructions to generate computer-executable code in a markup language.
15 . The computer-readable medium of claim 11 , wherein the LLM prompt comprises a tag for each of the set of candidate actions.
16 . The computer-readable medium of claim 15 , wherein the LLM prompt comprises a text description corresponding to each of the set of candidate actions.
17 . The computer-readable medium of claim 11 , wherein generating the augmented reality content comprises:
identifying an animation for the AR character corresponding to each of the series of selected actions.
18 . The computer-readable medium of claim 11 , wherein displaying the augmented reality content comprises:
transmitting the AR content to the client device.
19 . The computer-readable medium of claim 11 , wherein displaying the augmented reality content on the client device comprises:
augmenting the accessed image to include the augmented reality content.
20 . The computer-readable medium of claim 11 , wherein displaying the augmented reality content on the client device comprises:
augmenting a set of images captured after the accessed image to include the AR content.