Nonverbal information generation apparatus, method, and program

1. A nonverbal information generation apparatus comprising:

a display device; and

a hardware processor that partitions text into predetermined units and causes the display device to display the text partitioned into the predetermined units and make nonverbal information that represents first information about behavior of a verbal output agent or second information about behavior of a receiver of verbal information of the verbal output agent, the nonverbal information corresponding to the predetermined units of the text when the verbal output agent outputs the verbal information visible in association with the predetermined units of the text,

wherein each of the predetermined units of the text and the corresponding nonverbal information are displayed concurrently, by the hardware processor, on the display device, such that each of the predetermined units of text are aligned with the corresponding nonverbal information on a time axis,

an expression device that outputs voice corresponding to the text and expresses behavior indicated by the corresponding nonverbal information is displayed, by the hardware processor, together with the text and the corresponding nonverbal information, wherein the expression device, the text corresponding to the voice, and text and label corresponding to the nonverbal information are each displayed in different areas on the display device,

the output of the verbal information is outputting the text by voice or outputting as character strings, and

the predetermined units are obtained by dividing the text by the number of moras or the number of accents.

2. The nonverbal information generation apparatus according to claim 1 ,

wherein the hardware processor controls the display device so as to generate the nonverbal information on a basis of feature quantities of the text or feature quantities of voice corresponding to the text and a learned nonverbal information generation model, and cause the display device to display the nonverbal information and the text.

3. The nonverbal information generation apparatus according to claim 2 ,

wherein the display device displays in a state in which an instruction to start, stop, fast-forward, or rewind expression of the behavior by the expression device is receivable, and

the hardware processor, upon receiving the instruction, controls the expression of the behavior by the expression device in accordance with the instruction.

4. The nonverbal information generation apparatus according to claim 2 , wherein time information representing times of the predetermined units of the text is assigned to the text, and

the hardware processor controls the display device so as to cause the display device to display the nonverbal information in association with the predetermined units of the text, on a basis of time-information-stamped nonverbal information generated on a basis of time-information-stamped feature quantities of the text or the voice corresponding to the text, the learned nonverbal information generation model, and the time information representing the times assigned to the predetermined units of the text.

5. The nonverbal information generation apparatus according to claim 4 , wherein the time information is assigned on a basis of a result of partitioning a range of time when the text is output in accordance with the number of partitions when the text has been partitioned in the predetermined units.

6. The nonverbal information generation apparatus according to claim 2 , wherein the display device displays in a state in which a setting of additional information is receivable, and

the hardware processor, upon receiving the setting of the additional information, controls the display so as to cause the display device to display nonverbal information generated on a basis of the feature quantities of the text or the feature quantities of the voice corresponding to the text, the additional information, and the learned nonverbal information generation model, and the text.

7. The nonverbal information generation apparatus according to claim 2 , wherein the hardware processor, upon receiving a change instruction of the nonverbal information, generates a combination of the feature quantities of the text or the feature quantities of the voice corresponding to the text and the nonverbal information changed in accordance with the change instruction as learning data for learning the nonverbal information generation model.

8. The nonverbal information generation apparatus according to claim 7 , wherein the display device displays in a state in which a relearning instruction of the nonverbal information generation model and a setting of a weight for the generated learning data are receivable, and

the hardware processor, upon receiving the relearning instruction and the setting of the weight, uses the generated learning data and the weight that has been set to cause a nonverbal information generation model learning apparatus to learn the nonverbal information generation model.

9. The nonverbal information generation apparatus according to claim 1 , wherein the display device displays in a state in which a change instruction of the nonverbal information is receivable.

10. The nonverbal information generation apparatus according to claim 1 ,

wherein the predetermined units are obtained by dividing the text by the number of moras.

11. The nonverbal information generation apparatus according to claim 1 ,

wherein the predetermined units are obtained by dividing the text by the number of accents.

12. The nonverbal information generation apparatus according to claim 1 , wherein the hardware processor changes the nonverbal information so as to satisfy constraint conditions for deleting unnatural nonverbal information from the nonverbal information.

13. The nonverbal information generation apparatus according to claim 1 , wherein the hardware processor changes the nonverbal information so as to satisfy constraint conditions created, based on actual human behavior, which behavior should be generated and for how long or the number of text units.

14. A nonverbal information generation method in a nonverbal information generation apparatus, the nonverbal information generation method comprising:

partitioning text into predetermined units;

displaying the text partitioned into the predetermined units; and

making nonverbal information that represents first information about behavior of a verbal output agent or second information about behavior of a receiver of verbal information of the verbal output agent the nonverbal information corresponding to the predetermined units of the text when the verbal output agent outputs the verbal information visible in association with the predetermined units of the text,

wherein the predetermined units of the text and the nonverbal information are displayed, by a hardware processor, on a display device at same time,

an expression device that outputs voice corresponding to the text and expresses behavior indicated by the corresponding nonverbal information is displayed together with the text and the corresponding nonverbal information, wherein the expression device, the text corresponding to the voice, and text and label corresponding to the nonverbal information are each displayed in different areas on the display device,

the output of the verbal information is outputting the text by voice or outputting as character strings, and

the predetermined units are obtained by dividing the text by the number of moras or the number of accents.

15. The nonverbal information generation method of claim 14 further comprises a display screen is displayed in a state in which instructions for the expression device to start, stop, fast-forward, or rewind by a predetermined unit can be received.

16. The nonverbal information generation method of claim 14 wherein the display device performs display in a display screen so that it is possible to identify to which part of the text the behavior expressed by the expression device corresponds.

17. The nonverbal information generation method of claim 14 wherein the display device displays a display screen in a state in which a change instruction for a label indicating the nonverbal information can be received.

18. A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to function as the nonverbal information generation apparatus according to claim 1 .