AI avatar-based interaction service method and apparatus

1. An artificial intelligence (AI) avatar-based interaction service method performed in a system including an unmanned information terminal and an interaction service device, the method comprising:

transmitting a sound signal collected from a microphone array mounted in the unmanned information terminal and an image signal collected from a vision sensor to the interaction service device;

setting a sensing area based on a received sound signal and image signal by the interaction service device;

recognizing an active speaker based on a voice signal of a user and an image signal of the user collected in the sensing area, by the interaction service device;

determining a voice information based on the voice signal in response to recognizing the active speaker;

determining a non-verbal information based on the image signal in response to recognizing the active speaker;

generating a response for the recognized active speaker, 3D rendering an artificial intelligence avatar, said artificial intelligence avatar reflecting a desired response, wherein generating the response includes applying a first weight to the voice information and a second weight to the non-verbal information in response to the voice information and the non-verbal information having consistent results, and applying a third weight different than the first weight to the voice information and a fourth weight different than the second weight to the non-verbal information in response to the voice information and the non-verbal information having inconsistent results; and

using the interaction service device to provide the rendered artificial intelligence avatar to the unmanned information terminal.

2. The artificial intelligence avatar-based interaction service method according to claim 1 , further comprising:

using the interaction service device to estimate a sound source direction based on the received sound signal by sound source direction estimation;

limiting an input of a sound from a side by sidelobe signal cancellation; and

limiting image input after an object recognized by applying background separation to the received image signal.

3. The artificial intelligence avatar-based interaction service method according to claim 1 , further comprising:

when recognizing the active speaker, using the interaction service device to determine a number of people from the image signal of the user in the sensing area by facial recognition; and

in the case of recognizing a plurality of people in the sensing area, selecting a person as a speaker as an active speaker using any one or more of sound source position estimation, voice recognition, and mouth-shape recognition.

4. The artificial intelligence avatar-based interaction service method according to claim 1 , further comprising:

determining the non-verbal information by analyzing information including at least one of the group consisting of a facial expression, a pose, a gesture, or a voice tone of a speaker from the received image or audio signal of the user.

5. The artificial intelligence avatar-based interaction service method according to claim 4 , further comprising:

determining the voice information by any one or more of voice recognition, natural language understanding, or speech-to-text.

6. The artificial intelligence avatar-based interaction service method according to claim 4 , further comprising, in the providing of the artificial intelligence avatar to the unmanned information terminal, analyzing information including at least one of the group consisting of a facial expression, a gesture, and a voice tone from the audio or image of the user to recognize an emotional state of the user to change an expression, a gesture, or a voice tone of the AI avatar in response to the recognized emotional state or add an effect.

7. The artificial intelligence avatar-based interaction service method according to claim 1 , wherein the third weight is lower than the first weight, and the fourth weight is lower than the second weight.

8. The artificial intelligence avatar-based interaction service method according to claim 1 , wherein the voice information and the non-verbal information have consistent results based on the voice information being positive, and the non-verbal information being positive.

9. The artificial intelligence avatar-based interaction service method according to claim 1 , wherein the voice information and the non-verbal information have consistent results based on the voice information being negative, and the non-verbal information being negative.

10. The artificial intelligence avatar-based interaction service method according to claim 1 , wherein the voice information and the non-verbal information have inconsistent results based on the voice information being positive, and the non-verbal information being negative.

11. An artificial intelligence (AI) avatar-based interaction service apparatus, comprising:

an unmanned information terminal which includes a microphone array and a vision sensor, configured to:

collect a sound signal from the microphone array, and

collect an image signal from the vision sensor; and

an interaction service device, configured to:

receive the sound signal and the image signal,

set a sensing area based on the sound signal and the image signal,

recognize an active speaker based on a voice signal of a user and an image signal of the user collected in the sensing area,

determine a voice information based on the voice signal in response to recognizing the active speaker,

determine a non-verbal information based on the image signal in response to recognizing the active speaker,

generate a response for the recognized active speaker, including applying a first weight to the voice information and a second weight to the non-verbal information in response to the voice information and the non-verbal information having consistent results, and applying a third weight different than the first weight to the voice information and a fourth weight different than the second weight to the non-verbal information in response to the voice information and the non-verbal information having inconsistent results,

3D render the artificial intelligence avatar that reflects the response, and

provide the rendered artificial intelligence avatar to the unmanned information terminal.

12. The artificial intelligence avatar-based interaction service apparatus according to claim 11 , wherein the interaction service device estimates a sound source direction based on the received sound signal by sound source direction estimation, limits an input of a sound from a side by sidelobe signal cancellation, and limits image input after an object recognized by applying a background separation to the received image signal.

13. The artificial intelligence avatar-based interaction service apparatus according to claim 11 , wherein the interaction service device checks a number of people from the image signal of the user in the sensing area by face recognition and in the case of recognizing a plurality of people in the sensing area, the interaction service device selects a person as an active speaker using any one or more of sound source position estimation, voice recognition, and mouth-shape recognition.

14. The artificial intelligence avatar-based interaction service apparatus according to claim 11 , wherein the interaction service device is configured to determine the non-verbal information based on any one or more of a facial expression, a pose, a gesture, or a voice tone of a speaker from the received image signal of the user.

15. The artificial intelligence avatar-based interaction service apparatus according to claim 14 , wherein the interaction service device is configured to determine the voice information based on any one or more of voice recognition, natural language understanding, and speech-to-text.

16. The artificial intelligence avatar-based interaction service apparatus according to claim 14 , wherein the interaction service device analyzes a facial expression, a gesture, and a voice tone from the image of the user to recognize an emotional state of the user to change an expression, a gesture, or a voice tone of the AI avatar in response to the recognized emotional state or add an effect.

17. The artificial intelligence avatar-based interaction service apparatus according to claim 11 , wherein the third weight is lower than the first weight, and the fourth weight is lower than the second weight.

18. The artificial intelligence avatar-based interaction service apparatus according to claim 11 , wherein the voice information and the non-verbal information have consistent results based on the voice information being positive, and the non-verbal information being positive.

19. The artificial intelligence avatar-based interaction service apparatus according to claim 11 , wherein the voice information and the non-verbal information have consistent results based on the voice information being negative, and the non-verbal information being negative.

20. The artificial intelligence avatar-based interaction service apparatus according to claim 11 , wherein the voice information and the non-verbal information have inconsistent results based on the voice information being positive, and the non-verbal information being negative.