Artificial intelligence device for a digital avatar with 3D interaction capabilities and control method thereof

1 . A method for controlling an artificial intelligence (AI) device for implementing a digital avatar, the method comprising:

receiving, by a processor in the AI device, an audio signal corresponding to a user query;

converting, by a speech-to-text neural network model, the audio signal into a text query;

inputting the text query into a large language gesture instruction model to generate high level movement instructions for controlling a digital avatar to move or manipulate a 3D object;

inputting the text query and the high level movement instructions into an information retrieval model to generate a text response including at least one sentence and digital avatar control information based on the high level movement instructions;

inputting the text response into a text-to-speech neural network model to generate an audio response corresponding to the at least one sentence;

inputting the audio response into an audio-to-facial animation model and an audio-to-conversational gesture model to generate updated digital avatar control information including gesture information; and

outputting the audio response and the updated digital avatar control information for controlling the digital avatar.

2 . The method of claim 1 , wherein the updated digital avatar control information includes the high level movement instructions, hand gestures, and blendshape weights for animating a face of the digital avatar.

3 . The method of claim 1 , further comprising:

displaying the digital avatar on a display; and

moving the digital avatar to carry out an animation sequence based on the updated digital avatar control information in synchronization with audio playback of the audio response.

4 . The method of claim 1 , further comprising:

retrieving relevant documents based on the text query;

generating an input prompt based on the text query, a prior conversation history between the user and the digital avatar, and the relevant documents; and

inputting the input prompt into a retrieval-augmented large language model and outputting the text response by the retrieval-augmented large language model based on the input prompt.

5 . The method of claim 1 , further comprising:

inputting the text query and a prior conversation history between the user and the digital avatar into a contextualizer large language model, to generate an updated standalone question by replacing at least one word in the text query with at least one word extracted from the prior conversation history.

6 . The method of claim 5 , further comprising:

performing a similarity search between the updated standalone question and a vector store to retrieve relevant documents from the vector store.

7 . The method of claim 6 , further comprising:

generating an input prompt based on the text query, the prior conversation history, and the relevant documents from the vector store; and

inputting the input prompt into a retrieval-augmented large language model and outputting the text response by the retrieval-augmented large language model based on the input prompt.

8 . The method of claim 6 , further comprising:

receiving one or more documents;

dividing the one or more documents into a plurality of segments;

generating embeddings for the plurality of segments using an encoder; and

storing the one or more documents and the embeddings in the vector store for creating a searchable index.

9 . The method of claim 8 , wherein the one or more documents include a product manual, an instruction manual or a user manual.

10 . The method of claim 1 , further comprising:

receiving a video including a face of the user;

generating a 3D face mesh based on the video and a learned 3D morphable face model; and

animating a face of the digital avatar based on the 3D face mesh.

11 . The method of claim 10 , wherein the generating the 3D face mesh includes aligning parts of the learned 3D morphable face model with parts of frame images from the video.

12 . An artificial intelligence (AI) device for implementing a digital avatar, the AI device comprising:

a memory configured to store information for controlling a digital avatar; and

a controller configured to:

receive an audio signal corresponding to a user query,

convert, by a speech-to-text neural network model, the audio signal into a text query,

input the text query into a large language gesture instruction model to generate high level movement instructions for controlling a digital avatar to move or manipulate a 3D object,

input the text query and the high level movement instructions into an information retrieval model to generate a text response including at least one sentence and digital avatar control information based on the high level movement instructions,

input the text response into a text-to-speech neural network model to generate an audio response corresponding to the at least one sentence,

input the audio response into an audio-to-facial animation model and an audio-to-conversational gesture model to generate updated digital avatar control information including gesture information, and

output the audio response and the updated digital avatar control information for controlling the digital avatar.

13 . The AI device of claim 12 , wherein the updated digital avatar control information includes the high level movement instructions, hand gestures, and blendshape weights for animating a face of the digital avatar.

14 . The AI device of claim 12 , wherein the controller is further configured to:

display the digital avatar on a display, and

move the digital avatar to carry out an animation sequence based on the updated digital avatar control information in synchronization with audio playback of the audio response.

15 . The AI device of claim 12 , wherein the controller is further configured to:

retrieve relevant documents based on the text query,

generate an input prompt based on the text query, a prior conversation history between the user and the digital avatar, and the relevant documents, and

input the input prompt into a retrieval-augmented large language model and output the text response by the retrieval-augmented large language model based on the input prompt.

16 . The AI device of claim 12 , wherein the controller is further configured to:

perform a similarity search between an updated standalone question and a vector store to retrieve relevant documents from the vector store.

17 . The AI device of claim 16 , wherein the controller is further configured to:

generate an input prompt based on the text query, the prior conversation history, and the relevant documents from the vector store, and

input the input prompt into a retrieval-augmented large language model and output the text response by the retrieval-augmented large language model based on the input prompt.

18 . The AI device of claim 16 , wherein the controller is further configured to:

receive one or more documents,

divide the one or more documents into a plurality of segments,

generate embeddings for the plurality of segments using an encoder, and

store the one or more documents and the embeddings in the vector store for creating a searchable index.

19 . The AI device of claim 18 , wherein the one or more documents include a product manual, an instruction manual or a user manual.

20 . The AI device of claim 12 , wherein the controller is further configured to:

receive a video including a face of the user,

generate a 3D face mesh based on the video and a learned 3D morphable face model, and

animate a face of the digital avatar based on the 3D face mesh.