System and method for an audio-visual avatar evaluation
The present disclosure relates to a system to evaluate an avatar generated by an avatar generator. The system comprises an evaluation module including an audio evaluation module for evaluating audio features and a video evaluation module for evaluating video features. Evaluation of the avatar includes extracting audio and video features from the avatar and applying a set of evaluation metrics for generating audio and video evaluation scores. The scores are combined to generate a final score. For avatar generator evaluation, audio clip and video clip are provided to the audio evaluation module and video evaluation module, respectively. A set of evaluation metrics is applied for evaluation. Each metric can generate a score. All scores are combined to generate a final evaluation score.
1. A method for automated evaluation of an avatar generated by an avatar generator comprising the steps of:
obtaining, by an audio evaluator, a speech generated by a text-to-speech module;
obtaining, by a video evaluator, a video clip generated by a video generator;
obtaining, by the audio evaluator, the audio features of the target person;
obtaining, by the video evaluator, the video features of the target person;
comparing the speech with the audio features of the target person using a set of audio metrics, and generating an audio evaluation score for the speech;
wherein generating the audio evaluation score comprises evaluating speech intelligibility using automatic-speech-recognition (ASR) based evaluation metrics, evaluating audio noise level using voice-activity-detection (VAD) based evaluation metrics, evaluating naturalness of speech intonation using pitch-based metrics, evaluating voice similarities using equal-error-rate (EER) and cosine (COS) metrics, and evaluating speech pronunciation statistics;
comparing the video clip with the video features of the target person using a set of video metrics, and generating a video evaluation score for the video clip;
combining the audio evaluation score and the video evaluation score; and
generating a combined naturalness score for the avatar generator based on the combined score of the audio evaluation score and the video evaluation score.
2. The method of claim 1 , wherein evaluation scores generated by each of the set of audio metrics are combined to generate the audio evaluation score.
3. The method of claim 1 , wherein the step of evaluating the video clip further comprises:
evaluating a video quality with a reference image using a peak signal-to-noise ratio (PSNR), a multi-scale structured similarity indexing method (MS-SSIM), a feature similarity indexing method (FSIM), a learned perceptual image patch similarity (LPIPS), a video multimethod assessment fusion (VMAF), visual information fidelity, (VIF), or natural language processing (NLP) metrics;
evaluating a video quality with no reference images using a deep neural network-based IQA model (WaDIQaM), a deep bilinear convolutional neural network (DUBCNN), a transformer, relative ranking, and self consistency (TReS) model, or a chip quality assurance (ChipQA) model;
evaluating distribution using distribution-based metrics;
evaluating lip synchronization using lip synchronization metrics; and
evaluating an identity of the target using identity metrics.
4. The method of claim 3 , wherein the evaluation scores generated by each of the set of video metrics are combined to generate a video evaluation score.
5. The method of claim 1 , wherein the step of generating a combined naturalness score includes generating one or more human-interpretable scores of an avatar.
6. The method of claim 1 , wherein the step of combining the audio evaluation score and the video evaluation score comprises combining using at least one of a weighted average with fixed weights method and a trainable combination method.
7. The method of claim 6 , wherein the weighted average with fixed weights method comprises scaling all evaluation scores to a predefined range of weights and determining an average of the weights.
8. The method of claim 6 , wherein the trainable combination method comprises: using a dataset containing pairs of a video of the target person and corresponding mean opinion scores to train a regression module to predict the final score.
9. An evaluation system for automated evaluation of an avatar generated by an avatar generator comprising:
a processor configured to:
obtain a speech generated by a text-to-speech module of the avatar generator;
obtain the audio features of the target person;
compare the speech and the audio features using a set of audio metrics; wherein the audio metrics comprise automatic-speech-recognition (ASR) based evaluation metrics voice-activity-detection (VAD) based evaluation metrics, gross-pitch-error (GPE), or F0-frame-error (FFE) metrics, and equal-error-rate (EER) and cosine (COS) metrics, and generate an audio evaluation score based on the comparison;
the processor further configured to:
obtain a video clip generated by a video generator of the avatar generator;
obtain the video features of the target person;
compare the video clip with the video features using a set of video metrics; and
generate a video evaluation score based on the comparison; and
a score combination module configured to:
combine the audio evaluation score and the video evaluation score;
generate a combined naturalness score for the avatar generator based on the combination; and
generate an overall naturalness score based on the naturalness score and the combined naturalness score.
10. The system of claim 9 , wherein evaluation scores generated by each of the set of audio metrics are combined to generate the audio evaluation score.
11. The system of claim 9 , wherein the set of video metric includes:
a peak signal-to-noise ratio PSNR, a multi-scale structured similarity indexing method MS-SSIM, a feature similarity indexing method FSIM, a learned perceptual image patch similarity LPIPS, a video multimethod assessment fusion VMAF, visual information fidelity VIF, or natural language processing NLP metrics;
a deep neural network-based IQA model (WaDIQaM), a deep bilinear convolutional neural network (DUBCNN), a transformer, relative ranking, and self consistency (TReS), or chip quality assurance (ChipQA) model;
distribution-based metrics;
lip synchronization metrics; or
identity metrics.
12. The system of claim 9 , wherein the evaluation scores generated by each of the set of video metrics are combined to generate a video evaluation score.
13. The system of claim 9 , wherein the score combination module generates one or more human-interpretable scores of an avatar.
14. The system of claim 9 , wherein the score combination module is configured to combine the audio evaluation score and the video evaluation score using at least one of a weighted average with fixed weights method and a trainable combination method.
15. The system of claim 14 , wherein the score combination module using the weighted average with fixed weights method is configured to scale all evaluation scores to a predefined range of weights and determine an average of the weights.
16. The system of claim 14 , wherein the score combination module using the trainable combination method is configured to use a dataset containing pairs of a video of the target person and corresponding mean opinion scores to train a regression module to predict the final score.