Server device, terminal device, and display method for controlling facial expressions of a virtual character
A non-transitory computer readable medium storing computer executable instructions which, when executed by processing circuitry, cause the processing circuitry to acquire, from a first sensor, data related to a face of a performer; provide first data, that is generated based on the data, to a classifier; receive, from the classifier, specific facial expression data indicating one specific facial expression among a plurality of predetermined specific facial expressions on the basis of the first data; and select, for display output, a specific facial expression corresponding to the specific facial expression data received from the classifier.
1. A non-transitory computer readable medium storing computer executable instructions which, when executed by processing circuitry, cause the processing circuitry to:
acquire, from a first sensor, data related to a face of a performer;
provide first data to a classifier, the first data being generated based on the data;
receive, from the classifier, specific facial expression data based on the first data, wherein the specific facial expression data indicates one specific facial expression among a plurality of specific facial expressions, the classifier having selected the one specific facial expression based on any of processing of the first data and a manual input indicating the one specific facial expression, and the manual input is a direction of a swipe operation performed by the performer over a touch panel and an amount of movement of the swipe operation;
select, for display output, a specific facial expression corresponding to the specific facial expression data received from the classifier; and
operate in accordance with an algorithm in which, in a case where the specific facial expression data is received from the classifier in response to the first data, when specific facial expression designation data that designates the one specific facial expression among the plurality of specific facial expressions is acquired from the performer via a user interface in response to the first data, a specific facial expression corresponding to the specific facial expression designation data is selected as the specific facial expression for the display output.
2. The non-transitory computer readable medium according to claim 1 , wherein the first data includes data related to an amount of movement of a specific point in the face of the performer.
3. The non-transitory computer readable medium according to claim 2 , wherein
the processing circuitry is further caused to acquire, from a second sensor, audio data related to sound of the performer,
the processing circuitry provides second data, that is generated based on the audio data, to the classifier together with the first data, and
the specific facial expression data received by the processing circuitry from the classifier is based on the first data and the second data.
4. The non-transitory computer readable medium according to claim 3 , wherein the second data includes data related to volume, sound pressure, speech rate, and/or formant of the audio data of the performer.
5. The non-transitory computer readable medium according to claim 3 , wherein the second data includes data related to a word, a word ending, and/or an exclamation, and the data is obtained by the processing circuitry through natural language processing performed on the audio data.
6. The non-transitory computer readable medium according to claim 1 , wherein the processing circuitry provides the specific facial expression designation data to the classifier as teacher data for the first data and/or the second data.
7. The non-transitory computer readable medium according to claim 1 , wherein the processing circuitry is further caused to receive the manual input, indicating the one specific facial expression, and provide the one specific facial expression to the classifier.
8. The non-transitory computer readable medium according to claim 3 , wherein the classifier executes a principal component analysis on the first data and/or the second data to generate a learning model.
9. The non-transitory computer readable medium according to claim 8 , wherein when specific facial expression designation data that designates one specific facial expression among the plurality of specific facial expressions is input from the performer to the processing circuitry via a user interface in response to the first data and/or the second data, the classifier executes principal component analysis on the specific facial expression designation data, in addition to the first data and/or the second data, to generate a learning model.
10. The non-transitory computer readable medium according to claim 2 , wherein the processing circuitry is further caused to:
control a display to display a script that instructs the performer to show a facial expression related to any one specific facial expression among the plurality of specific facial expressions; and
provide third data, indicating the one specific facial expression, to the classifier as teacher data for the first data and/or the second data in association with the script.
11. The non-transitory computer readable medium according to claim 1 , wherein the processing circuitry is further caused to:
receive a learning model from a server via a network; and
provide the learning model to the classifier.
12. The non-transitory computer readable medium according to claim 1 , wherein the processing circuitry is further caused to control storage of a learning model generated by the classifier in a memory in association with the performer.
13. The non-transitory computer readable medium according to claim 1 , wherein the processing circuitry is further caused to control transmission of the data to a server via a network.
14. The non-transitory computer readable medium according to claim 1 , wherein the plurality of specific facial expressions include:
first facial expressions that express emotion,
second facial expressions in which a face shape is unrealistically deformed, and/or
third facial expressions in which a face is given a symbol, a shape, and/or a color.
15. The non-transitory computer readable medium according to claim 14 , wherein the first facial expressions that are shown based on a user interface mapped into a language- and culture-independent psychological space, the user interface including a Wheel of Emotions.
16. A display method, comprising:
acquiring, by processing circuitry from a first sensor, data related to a face of a performer;
providing first data to a classifier, the first data being generated based on the data;
receiving, from the classifier, specific facial expression data based on the first data, wherein the specific facial expression data indicates one specific facial expression among a plurality of specific facial expressions, the classifier having selected the one specific facial expression based on any processing of the first data and a manual input indicating the one specific facial expression, and the manual input is a direction of a swipe operation performed by the performer over a touch panel and an amount of movement of the swipe operation;
selecting, by the processing circuitry for display output, a specific facial expression corresponding to the specific facial expression data received from the classifier; and
operating by the processing circuitry in accordance with an algorithm in which, in a case where the specific facial expression data is received from the classifier in response to the first data, when specific facial expression designation data that designates the one specific facial expression among the plurality of specific facial expressions is acquired from the performer via a user interface in response to the first data, a specific facial expression corresponding to the specific facial expression designation data is selected as the specific facial expression for the display output.
17. A device, comprising:
processing circuitry configured to:
acquire, from a first sensor, data related to a face of a performer;
provide first data to a classifier, the first data being generated based on the data;
receive, from the classifier, specific facial expression data based on the first data, wherein the specific facial expression data indicates one specific facial expression among a plurality of specific facial expressions, the classifier having selected the one specific facial expression based on any of processing of the first data and a manual input indicating the one specific facial expression, and the manual input is a direction of a swipe operation performed by the performer over a touch panel and an amount of movement of the swipe operation;
select, for display output, a specific facial expression corresponding to the specific facial expression data received, from the classifier; and
operate in accordance with an algorithm in which in a case where the specific facial expression data is received from the classifier in response to the first data, when specific facial expression designation data that designates the one specific facial expression among the plurality of specific facial expressions is acquired from the performer via a user interface in response to the first data, a specific facial expression corresponding to the specific facial expression designation data is selected as the specific facial expression for the display output.