Multi-sensory human-computer interaction system and multi-sensory human-computer interaction method
View Patent ↗A multi-sensory human-computer interaction system includes an interactive device, sensors and a server. The server is coupled to the sensors and is configured to control the interactive device. The multi-sensory human-computer interaction system performs the following operations: each of the plurality of sensors receives a plurality of sensing signals from a real environment; the server generates a plurality of event data corresponding to each of the plurality of sensing signals according to the plurality of sensing signals, wherein a plurality of event data represents the situation of the real environment; the server generates a plurality of multi-sensory correlation data corresponding to the situation according to the plurality of event data; the server determines a sensory feedback result according to the plurality of multi-sensory correlation data; and the interactive device is configured to generate interactive messages according to sensory feedback result. The interactive messages are configured to interact with users.
1 . A multi-sensory human-computer interaction system, comprising:
an interactive device;
a plurality of sensors; and
a server, coupled to the sensors, and configured to control the interactive device;
wherein, the multi-sensory human-computer interaction system executes following operations:
receiving a plurality of sensing signals from a real environment respectively by each of the sensors;
generating a piece of event data corresponding to each of the sensing signals according to sensing signals by the server, wherein the pieces of event data represent a situation of the real environment;
generating pieces of multi-sensory correlation data corresponding to the situation according to the pieces of event data by the server;
determining a sensory feedback result according to the pieces of multi-sensory correlation data by the server;
performing a scoring operation on each of the pieces of multi-sensory correlation data by the server to generate a correlation score by the server;
comparing correlation scores according to a plurality of present values respectively by the server to generate a plurality of comparison results by the server;
determining an action feedback of a plurality of senses of the interactive device as the sensory feedback result according to the comparison results by the server; and
generating an interactive message according to the sensory feedback result by the interactive device, wherein the interactive message is configured to interact with a user.
2 . The multi-sensory human-computer interaction system of claim 1 , wherein the multi-sensory human-computer interaction system executes following operations:
performing a making operation on each of the pieces of event data by the server to generate a plurality of markers, wherein the markers comprise at least one of a semantic marker, a distance marker and a sensory magnitude/intensity marker for the situation.
3 . The multi-sensory human-computer interaction system of claim 2 , wherein the multi-sensory human-computer interaction system executes following operations:
analyzing the markers between the each of the pieces of event data to generate the pieces of multi-sensory correlation data by the server, wherein the pieces of multi-sensory correlation data represent at least one correlation relationship between the pieces of event data.
4 . The multi-sensory human-computer interaction system of claim 3 ,
wherein each of the senses corresponds to one of a visual sense, an auditory sense and a tactile sense.
5 . The multi-sensory human-computer interaction system of claim 1 , wherein the interactive device is a humanoid robot with an ability to understand the situation in the real environment.
6 . The multi-sensory human-computer interaction system of claim 1 , wherein the interactive device is configured to project a virtual robot into a virtual reality, wherein the virtual robot is coupled to a plurality of sensory output devices to generate an action feedback of a plurality of senses through the sensory output devices, wherein each of the senses corresponds to one of a visual sense, an auditory sense and a tactile sense.
7 . The multi-sensory human-computer interaction system of claim 1 , wherein the plurality of sensors comprise at least one of a visual sensor, an auditory sensor and a tactile sensor.
8 . The multi-sensory human-computer interaction system of claim 1 , wherein the multi-sensory human-computer interaction system executes following operations:
generating an action feedback of a plurality of senses according to the sensory feedback result by the interactive device; and
assisting a user to deal with the situation of the real environment through the action feedback by the interactive device.
9 . A multi-sensory human-computer interaction method, executed following steps through a multi-sensory human-computer interaction system, wherein the multi-sensory human-computer interaction method comprises:
receiving a plurality of sensing signals from a real environment;
generating a piece of event data corresponding to each of the sensing signals according to sensing signals, wherein the pieces of event data represent a situation of the real environment;
generating pieces of multi-sensory correlation data corresponding to the situation according to the pieces of event data;
determining a sensory feedback result according to the pieces of multi-sensory correlation data;
performing a scoring operation on each of the pieces of multi-sensory correlation data to generate a correlation score;
comparing correlation scores according to a plurality of present values to generate a plurality of comparison results;
determining an action feedback of a plurality of senses of an interactive device as the sensory feedback result according to the comparison results; and
generating an interactive message according to the sensory feedback result to control an interactive device, wherein the interactive message is configured to interact with a user.
10 . The multi-sensory human-computer interaction method of claim 9 , wherein generating the piece of event data corresponding to each of the sensing signals according to the sensing signals comprises:
performing a making operation on each of the pieces of event data to generate a plurality of markers, wherein the markers comprises at least one of a semantic marker, a distance marker and a sensory magnitude/intensity marker.
11 . The multi-sensory human-computer interaction method of claim 10 , wherein generating the piece of event data corresponding to each of the sensing signals according to the sensing signals further comprises:
analyzing the markers between the each of the pieces of event data to generate the pieces of multi-sensory correlation data, the pieces of multi-sensory correlation data represent at least one correlation relationship between the pieces of event data.
12 . The multi-sensory human-computer interaction method of claim 11 ,
wherein each of the senses corresponds to one of a visual sense, an auditory sense and a tactile sense.
13 . The multi-sensory human-computer interaction method of claim 9 wherein the interactive device is a humanoid robot with an ability to understand the situation in the real environment.
14 . The multi-sensory human-computer interaction method of claim 9 , wherein the interactive device is configured to project a virtual robot into a virtual reality, wherein the virtual robot is coupled to a plurality of sensory output devices to generate an action feedback of a plurality of senses through the sensory output devices, wherein each of the senses corresponds to one of a visual sense, an auditory sense and a tactile sense.
15 . The multi-sensory human-computer interaction method of claim 9 , further comprising:
generating an action feedback of a plurality of senses according to the sensory feedback result; and
assisting a user to deal with the situation of the real environment through the action feedback.