Three-dimensional modeling system and modeling method based on multimodal fusion
Disclosed are a three-dimensional modeling system and modeling method based on multimodal fusion. The method includes: separately collecting feedback data of an electroencephalogram sensor, an electromyography sensor, an eye movement sensor, a gesture sensor, and a voice sensor, conducting multimodal fusion on the feedback data, obtaining multimodal-fused model data, matching the model data with a database instruction, obtaining and analyzing an instruction set, obtaining and identifying a relevant modeling parameter, obtaining a modeling method, automatically conducting modeling according to the modeling method, and obtaining a visual entity model. Through the method of the present disclosure, multi-sense man-machine interaction is combined with traditional geometric modeling, and a complicated modeling command input operation does not require manipulation of interactive devices such as a keyboard and a mouse, such that modeling efficiency is improved, waste of manipulation energy of an operator is reduced, and further modeling experience of the operator is improved.
1 . A three-dimensional modeling method based on multimodal fusion, comprising:
separately collecting feedback data of an electroencephalogram sensor, an electromyography sensor, an eye movement sensor, a gesture sensor, and a voice sensor by arranging the electroencephalogram sensor, the electromyography sensor, the eye movement sensor, the gesture sensor and the voice sensor at different positions of a body of an operator respectively, and collecting brain command data of the operator by the electroencephalogram sensor, collecting facial muscle change data of the operator by the electromyography sensor, collecting eye movement change data of the operator by the eye movement sensor, collecting gesture change data of the operator by the gesture sensor, and collecting voice data transmitted by the operator by the voice sensor;
conducting multimodal fusion on the feedback data by conducting data fusion on the brain command data, the facial muscle change data, the eye movement change data, the gesture change data and the voice data, and obtaining multimodal-fused model data;
matching the model data with a database instruction, and obtaining an instruction set;
analyzing an attribute of the instruction set, and obtaining a relevant modeling parameter;
identifying the relevant modeling parameter, and obtaining a modeling method; and
automatically conducting modeling on the basis of the modeling method, and obtaining a visual entity model, wherein
the model data of the visual entity model is mapped onto the instruction set, a correspondence between the instruction set and the model data is obtained, and the correspondence between the instruction set and the model data is stored in a back-end database for later loading.
2 . The three-dimensional modeling method based on multimodal fusion according to claim 1 , wherein the step of matching the model data with the database instruction, and obtaining the instruction set comprise:
obtaining the instruction set configured to generate different instruction collections on the basis of the model data according to a correspondence between different types of data and different instructions in a database, wherein a model instruction is generated on the basis of the brain command data, a geometric model instruction is generated on the basis of the facial muscle change data, a modeling position determination instruction is generated on the basis of the eye movement change data, rotation and contraction instructions are generated on the basis of the gesture change data, and a revocation or deletion instruction is generated on the basis of the voice data.
3 . The three-dimensional modeling method based on multimodal fusion according to claim 1 , wherein the relevant modeling parameter comprises: a name, a class, a model number, a geometric feature, and a mathematical expression.
4 . The three-dimensional modeling method based on multimodal fusion according to claim 1 , wherein the step of analyzing the attribute of the instruction set, and obtaining the relevant modeling parameter comprise:
transmitting the instruction set to a modeling system, and obtaining the relevant modeling parameter, wherein the modeling system analyzes a mathematical characteristic and a physical characteristic of a modeled object according to the attribute of the instruction set.
5 . The three-dimensional modeling method based on multimodal fusion according to claim 1 , further comprising: basic body modeling, extended body modeling, Boolean operation modeling, stretching modeling, rotation modeling, and complex modeling, wherein
the basic body modeling comprises: using geometric modeling commands for a cuboid, a sphere, a cylinder, a cone, a wedge and a ring body, a geometric sphere, a teapot, a rectangular pyramid, a tubular body, and several irregular bodies;
the extended body modeling comprises: extending geometric modeling command parameters in the basic body modeling;
the Boolean operation modeling comprises: creating a more complex three-dimensional entity model through Boolean operation between all entity elements on the basis of the extended body modeling and the basic body modeling;
the stretching modeling comprises: creating a three-dimensional entity model on the basis of a two-dimensional graphic base surface;
the rotation modeling comprises: conducting rotation around any base line, and generating a three-dimensional entity model; and
the complex modeling comprises: introducing a mathematical function, and creating a surface of a three-dimensional entity model.
6 . A non-transitory readable storage medium, storing a computer program, wherein the computer program implements steps of the method according to claim 1 when executed by a processor.
7 . A three-dimensional modeling system based on multimodal fusion, comprising: a somatosensory sensor module, a communication module, a model building module, and a database module, wherein
the somatosensory sensor module comprises an electroencephalogram sensor, an electromyography sensor, an eye movement sensor, a gesture sensor, and a voice sensor arranged at different positions of a body of an operator respectively,
the electroencephalogram sensor being configured to collect brain command data of the operator,
the electromyography sensor being configured to collect facial muscle change data of the operator,
the eye movement sensor being configured to collect eye movement change data of the operator,
the gesture sensor being configured to collect gesture change data of the operator,
the voice sensor being configured to collect voice data transmitted by the operator, and
the somatosensory sensor module being is configured to conduct multimodal fusion on the brain command data, the facial muscle change data, the eye movement change data, the gesture change data and the voice data, and obtain multimodal-fused model data;
the communication module is configured to upload the model data;
the database module is configured to match the model data with a database instruction, and obtain an instruction set; and store a correspondence between the instruction set and the model data; and
the model building module is configured to analyze an attribute of the instruction set, obtain a relevant modeling parameter, identify the relevant modeling parameter, obtain a modeling method, automatically conduct modeling on the basis of the modeling method, and output a visual entity model.
8 . The three-dimensional modeling system based on multimodal fusion according to claim 7 , wherein
the communication module comprises a transmission terminal and a reception terminal, and the transmission terminal and the reception terminal conduct communication in a message format after establishing a communication relation.