Method and system for converting plant procedures to voice interfacing smart procedures

1 . A system, comprising:

an internal server operated in isolation from an external public communication network; and

at least one smart device communicating with the internal server,

wherein the internal server comprises:

a pre-processing unit including a non-text processing unit configured to separate out images and tables from an input plant procedure; and

a text processing unit configured to extract structural properties and rich text features for each text paragraph of the plant procedure from which the images and tables are separated out;

an extended natural language processing (NLP) unit configured to apply existing NLP techniques by utilizing public NLP tools for each text paragraph returned from the pre-processing unit and to correct any misinterpreted NLP results, the extended NLP unit comprising:

a first NLP unit configured to perform tokenization, sentence splitting, and lemmatization on the text as input;

a second NLP unit configured to perform part-of-speech (POS) tagging for each token and hierarchical structuring of tokens for each sentence; and

a third NLP unit configured to detect and correct any misinterpreted NLP results from the output of the second NLP unit, and to utilize pattern-based built-in rules integrated with a lexical database;

an information extraction unit configured to extract all syntactic and semantic information for each text paragraph of the plant procedure by utilizing the results of the pre-processing unit and the extended NLP unit;

a conversion unit configured to convert the each text paragraph of the plant procedure into one or more individual Business Process Model and Notation (BPMN) elements and their property values by utilizing the syntactic and semantic information extracted; and

a generation unit configured to generate a BPMN process model of the plant procedure in an extensible Markup Language (XML) format by integrating and restructuring the individual BPMN elements,

wherein the syntactic and semantic information includes semantic elements, paragraph types, and step components, and

wherein the smart device is configured such that a mobile application is installed to receive the plant procedure in the XML format, convert the plant procedure expressed in XML into a smart procedure supporting voice interfacing, and provide voice guidance of the plant procedure in response to a user's voice input.

2 . The system of claim 1 ,

wherein the mobile application comprises:

a Voice User Interface (VUI) including a Speech-to-Text (STT) and Text-to-Speech (TTS) components;

a Touch User Interface (TUI); and

a controller configured to control entire operation of the mobile application including user interface functions.

3 . The system of claim 2 ,

wherein the entire operation of the mobile application, including an interface between a user and the mobile application via the VUI or the TUI, follows a predefined interaction scenario for each type of BPMN element in the process model of the plant procedure.

4 . The system of claim 3 ,

wherein proceeding to next BPMN element connected or attached by BPMN elements of a sequence flow or an association is processed in three different ways: a user-oriented way, a system-oriented way, and an automatic way,

wherein a specific notification sound is provided to a user at the end of each interaction scenario for the three ways, being included as a last basic VUI, so that the user clearly recognizes that the smart device is proceeding to the next BPMN element.

5 . The system of claim 2 ,

wherein the controller controls the entire operation of the mobile application including recognizing user speech, detecting user intent, processing required transactions, or providing the processed results in synthesized speech.

6 . The system of claim 2 ,

wherein the VUI activates distinct speech input reception modes of standby, command-detection, and data-entry, switching to a specific speech reception mode according to controls of the controller, converts captured user speech into a text, and forwards the text to the controller by utilizing the STT components.

7 . The system of claim 2 ,

wherein the controller verifies whether a text forwarded from the VUI is a valid command or data type,

if confirmed, the controller processes designated transactions, otherwise the controller requests the user to provide a valid voice input again, and then,

the controller reactivates the STT component of the VUI to a specific speech reception mode, depending on an operating status of the mobile application, and

wherein the controller utilizes no external services.

8 . The system of claim 2 ,

wherein the VUI provides guidance to the user in synthesized speech utilizing the TTS component according to the controls of the controller.

9 . The system of claim 2 ,

wherein the controller composes a text message for a voice output utilizing a pattern-based message template associated with each BPMN element type and property values of current instance of the BPMN element, extends the text message for the voice output using Speech Synthesis Markup Language (SSML) tags, and provides the text message to the user in synthesized speech via the TTS component of the VUI.

10 . The system of claim 2 ,

wherein a display screen for the TUI comprises:

a first area of three rows respectively presenting an ID, title, and section title of a plant procedure currently being performed;

a second area of a single row presenting valid or recognized user speech in text; and

a third area between the first area and the second area presenting a scrollable vertical list of previously completed and subsequent plant procedure steps, ordered as in the plant procedure and centered on a plant procedure step currently being performed.

11 . The system of claim 10 ,

wherein the plant procedure step currently being performed is presented using a generic card layout template including four containers of step description, rows of optionally attached elements in case for an action step, clickable buttons, and a checkbox indicating step execution status,

and wherein a specific card layout customized from the generic card layout template is used according to BPMN element type of the plant procedure step currently being performed.

12 . The system of claim 11 ,

wherein a label of an action button changes in an order of ‘Acknowledge’, ‘Complete’, and ‘Next’,

wherein the action button is one of clickable buttons, and

wherein, when the action button is clicked or corresponding user speech is recognized with the TUI or VUI respectively, different specific transactions are processed according to the label of the action button.

13 . The system of claim 11 ,

wherein a conditional action step, of a kind of the action step, includes a card layout for conditional clause and a card layout for action clause, and

wherein, when the card layout for conditional clause is utilized and then a predetermined condition is satisfied, the card layout for action clause is utilized.

14 . The system of claim 1 ,

wherein the internal server and the smart device are connected through a secure communication network within a plant.

15 . The system of claim 1 ,

wherein the internal server further comprises:

a Web API controller configured to receive and process a request from the mobile application and then return a processed result to the requesting mobile application;

a WebSocket controller configured to broadcast data to a group of designated mobile application clients upon a request from the Web API controller; and

a database configured to store all data related to the operation of the mobile application installed on the smart device.

16 . The system of claim 15 ,

wherein both the request from the mobile application installed on the smart device and a response of the processed result from the internal server are interfaced through an Application Programming Interface (API) that are provided at the internal server and controlled by the Web API controller.

17 . The system of claim 1 ,

wherein the internal server controls download of the plant procedure expressed in the XML format and automatically assigns roles of each mobile application client participating in a workgroup, utilizing work order data issued by a separate plant management system.

18 . The system of claim 15 ,

wherein, when multiple mobile application clients participate in a workgroup,

the internal server grants execution control for any type of input exclusively to a single mobile application of a user assigned to perform a current procedure step, receives an execution status including data input at a mobile application client granted with the execution control via the Web API controller and stores the execution status in the database, and broadcasts the execution result to all other clients in the workgroup for synchronizing operation via the WebSocket controller.

19 . The system of claim 1 ,

wherein the mobile application supports:

previewing plant procedures,

reviewing executed results of the plant procedures;

customizing user or system configurations via a separate screen, including changing a user preference and changing an operation of the mobile application, wherein the user preference comprises a speech speed of synthesized voice output, and the operation includes excluding a basic VUI according to scenario customization; and

re-instructing a current step of the plant procedure and returning to a previous step of the plant procedure, each by a designated voice command.

20 . A method for converting a plant procedure into a smart procedure supporting voice interfacing using a smart device and an internal server, the method comprising:

a first step of pre-processing the plant procedure by the internal server;

a second step of applying, by the internal server, existing natural language processing (NLP) technologies to text paragraphs returned from the first step and correcting any misinterpreted NLP results, of part-of-speech (POS) tags and parse trees of tokens, wherein the second step comprising a first NLP step of performing tokenization, sentence splitting, and lemmatization, a second NLP step of performing POS tagging and syntactic parsing, and a third NLP step of correcting any errors of the second NLP step by using a lexical database and internal rules;

a third step of performing, by the internal server, semantic element extraction, paragraph type classification, and action step component identification for each text paragraph of the plant procedure utilizing the results of the first and the second steps;

a fourth step of converting, by the internal server, each text paragraph of the plant procedure into one or more individual Business Process Model and Notation (BPMN) elements and property values of the individual BPMN elements by using the syntactic and semantic information extracted during the second step; and

a fifth step of generating, by the internal server, a BPMN process model of a procedure by integrating and restructuring the individual BPMN elements in extensible Markup Language (XML);

wherein the smart device is configured to receive the plant procedure in the XML according to a mobile application installed in advance, convert the XML-format plant procedure into the smart procedure supporting voice interfacing, and provide voice guidance of the contents of the plant procedure in response to a user's voice input.

21 . The method of claim 20 ,

wherein the internal server and the smart device are connected via a secure communication network within the plant, and the method is performed according to a predefined scenario for each type of BPMN element in the process model of the plant procedure.

22 . The method of claim 21 ,

wherein each predefined scenario includes:

a speech recognition step of recognizing user speech;

a user intent identification step of identifying user intent;

a processing step of processing designated transactions according to a user intent identified; and

a synthesized speech output step of producing a processed result in synthesized speech.

23 . The method of claim 22 ,

wherein the speech recognition step includes capturing the user speech in an active speech reception mode and then converting the user speech into a text.

24 . The method of claim 23 ,

wherein the user intent identification step includes verifying the text converted from the captured user speech based on determination on whether it is a valid command or data type.

25 . The method of claim 22 ,

wherein the synthesized speech output step including:

a composition step of building a text message for a voice output using a pattern-based message template associated with each BPMN element type;

an extension step of extending the output text message using Speech Synthesis Markup Language (SSML) tags; and

a provision step of providing the extended output text message in synthesized speech via a Text-to-Speech (TTS) component of Voice User Interface (VUI).

26 . The method of claim 20 , further comprising:

in order for multiple smart devices operating within a workgroup to jointly perform a single plant procedure in a harmonized manner,

a sixth step of automatically granting an execution control for any type of input exclusively to a smart device of a worker assigned to perform a target step; and

a seventh step of automatically synchronizing any execution result of a responsible smart device to all other smart devices in the workgroup.