IP Library Granted Patent US 12,412,000
Granted Patent B2
US 12,412,000 · App. 17/442,437 · Granted Sep 9, 2025

Automatic extraction of assets data from engineering data sources for generating an HMI

Inventors: Bhaskar Sinha (Hyderabad, IN); Padmaja Bodanapu (Telangana, IN); Ashish Patil (Maharashtra, IN); Sameer Kondejkar (Maharashtra, IN); Rajkumar Krishnan (Sharon, MA)
Assignee: SCHNEIDER ELECTRIC SYSTEMS USA, INC.
G06F30/12G05B17/02G05B19/0426G05B19/41885G05B23/0221G05B23/0243G05B23/027G05B23/0272G05B23/0281G06F30/27G06N5/022G05B2219/23008G05B2219/23067G05B2219/23272G05B2219/23278
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,412,000
App. No.
17/442,437
Filed
Sep 23, 2021
Granted
Sep 9, 2025
Kind
B2
Art Unit
2146
USPC
706/11
Abstract

Systems and methods for controlling industrial process automation and control systems can automatically, through the use of machine learning (ML) models and algorithms, extract plant assets from engineering diagrams and other plant engineering data sources. The systems and methods can establish asset relationships to create a plant asset registry and build an asset hierarchy from the plant assets. The systems and methods can generate an ontological knowledge base from the plant asset hierarchy, and provide an HMI for controlling the industrial process based on the plant asset hierarchy and the ontological knowledge base.

Claims (39)

1. A control system for an industrial plant, comprising:

one or more processors; and

a storage unit communicatively coupled to the one or more processors and storing processor-executable instructions thereon that, when executed by the one or more processors, cause the control system to:

perform a process that inputs data from a plurality of engineering data sources for the industrial plant;

perform a process that extracts one or more assets of the industrial plant from the data;

perform a process that creates an HMI (human-machine interface) asset model from the assets, the HMI asset model arranging the assets in a hierarchical structure;

perform a process that generates an HMI display from the HMI asset model, the HMI display displaying symbols that represent the one or more assets of the industrial plant and lines that represent processes, connections, and data links between the assets; and

perform a process that displays the HMI display to a user and allows the user to navigate vertically and horizontally along the display, wherein the HMI display dynamically changes which assets are displayed based on a change in position of the user on the HMI display, wherein

asset relationships are established from the one or more assets of the industrial plant to create a plant asset registry and build an asset hierarchy,

an ontological knowledge base is generated from the plant asset hierarchy, and

the process to create the HMI asset model is for an HMI for controlling an industrial process, wherein the model is created from the one or more assets of the industrial plant, the plant asset hierarchy, and the ontological knowledge base.

2. The control system of claim 1 , wherein the processor-executable instructions further cause the control system to perform a process that identifies all alarms currently raised for the industrial plant and assets corresponding to the alarms and displays the alarms on the HMI display.

3. The control system of claim 1 , wherein the processor-executable instructions further cause the control system to perform a process that displays the HMI display at a plant level, the plant level displaying all currently raised alarms identified for the industrial plant and assets corresponding to the alarms.

4. The control system of claim 1 , wherein the processor-executable instructions further cause the control system to perform a process that determines and displays on the HMI display assets that are potential root causes for an alarm upon the user selecting an asset corresponding to one of the alarms.

5. The control system of claim 4 , wherein the control system determines the assets that are potential root causes by finding all assets that are connected to one another that also have an alarm.

6. The control system of claim 5 , wherein the processor-executable instructions further cause the control system to perform a process that allows a user to manually correct a root cause of an alarm on the HMI display, and provides the root cause from the user as feedback to the control system.

7. The control system of claim 1 , wherein the processor-executable instructions further cause the control system to perform a process that automatically zooms in or zooms out on the assets displayed on the HMI display based on a change in a position of the user on the HMI screen.

8. The control system of claim 1 , wherein the processor-executable instructions further cause the control system to perform a process that displays assets in the HMI display as a two-dimensional view, wherein the HMI display allows a user to navigate vertically and horizontally within the two-dimensional view.

9. The control system of claim 1 , wherein the processor-executable instructions further cause the control system to perform a process that dynamically changes the assets displayed on the HMI display based on an alarm occurring on an asset.

10. The control system of claim 1 , wherein the processor-executable instructions further cause the control system to perform a process that displays assets on the HMI display at runtime based on one of: static weight assigned to an asset based on a position of the asset in an asset hierarchy, and dynamic weight assigned to the asset based on an alarm being raised at the asset.

11. A method for controlling for an industrial plant, comprising:

inputting data from a plurality of engineering data sources for the industrial plant;

extracting one or more assets of the industrial plant from the data;

establishing asset relationships to create a plant asset registry and building an asset hierarchy from the assets;

generating an ontological knowledge base from the asset hierarchy;

creating an HMI (human-machine interface) asset model for an HMI controlling an industrial process from the assets, the asset hierarchy, and the ontological knowledge base, the HMI asset model arranging the assets in a hierarchical structure;

generating an HMI display from the HMI asset model, the HMI display displaying symbols that represent the one or more assets of the industrial plant and lines that represent processes, connections, and data links between the assets; and

displaying the HMI display to a user and allowing the user to navigate vertically and horizontally along the display, wherein the HMI display dynamically changes which assets are displayed based on a change in position of the user on the HMI display.

12. The method of claim 11 , further comprising identifying all alarms currently raised for the industrial plant and assets corresponding to the alarms and displays the alarms on the HMI display.

13. The method of claim 11 , further comprising displaying the HMI display at a plant level, the plant level displaying all currently raised alarms identified for the industrial plant and assets corresponding to the alarms.

14. The method of claim 11 , further comprising determining and displaying on the HMI display assets that are potential root causes for an alarm upon the user selecting an asset corresponding to one of the alarms.

15. The method of claim 14 , wherein determining the assets that are potential root causes includes finding all assets that are connected to one another that also have an alarm.

16. The method of claim 11 , further comprising allowing a user to manually correct a root cause of an alarm on the HMI display, and providing the root cause from the user as feedback to a control system.

17. The method of claim 11 , further comprising automatically zooming in or zooming out on the assets displayed on the HMI display based on a change in a position of the user on the HMI screen.

18. The method of claim 11 , further comprising displaying assets in the HMI display as a two-dimensional view, and allowing a user to navigate vertically and horizontally within the two-dimensional view.

19. The method of claim 11 , further comprising dynamically changing the assets displayed on the HMI display based on an alarm occurring on an asset.

20. The method of claim 11 , further comprising displaying assets on the HMI display at runtime based on one of: static weight assigned to an asset based on a position of the asset in an asset hierarchy, and dynamic weight assigned to the asset based on an alarm being raised at the asset.

21. The control system of claim 1 , wherein the process that extracts one or more assets of the industrial plant from the data extracts the one or more assets of the industrial plant from the data using machine learning to identify the one or more assets.

22. The method of claim 11 , wherein the extracting one or more assets of the industrial plant from the data includes extracting the one or more assets of the industrial plant from the data using machine learning to identify the one or more assets.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 24, 2021
From: SINHA, BHASKAR; BODANAPU, PADMAJA; PATIL, ASHISH; KONDEJKAR, SAMEER; KRISHNAN, RAJKUMAR
To: SCHNEIDER ELECTRIC SYSTEMS USA, INC.
Reel/Frame 057602/0001 →
Continuity (4)
Provisional Application 62842929 · May 3, 2019
Provisional Application 62823469 · Mar 25, 2019
Provisional Application 62823377 · Mar 25, 2019
Related Publication 20220197272A1 · Jun 23, 2022
References Cited (55)
US 8365019B2 · Sailer et al. · 2013 [cited by applicant]
US 8370224B2 · Grewal · 2013 [cited by applicant]
US 9177398B2 · Bierweiler et al. · 2015 [cited by applicant]
US 9825818B2 · Tiwari et al. · 2017 [cited by applicant]
US 11756323B2 · Kang et al. · 2023 [cited by applicant]
US 20090328215A1 · Arzi et al. · 2009 [cited by applicant]
US 20100318846A1 · Sailer · 2010 [cited by examiner]
US 20120062574A1 · Dhoolia et al. · 2012 [cited by applicant]
US 20120259436A1 · Resurreccion · 2012 [cited by examiner]
US 20150220076A1 · Uchida · 2015 [cited by applicant]
US 20150242286A1 · Grewal · 2015 [cited by examiner]
US 20150261215A1 · Blevins et al. · 2015 [cited by applicant]
US 20160132538A1 · Bliss et al. · 2016 [cited by applicant]
US 20160292895A1 · Billi et al. · 2016 [cited by applicant]
US 20170192407A1 · Kurella et al. · 2017 [cited by applicant]
US 20170329322A1 · Downor · 2017 [cited by applicant]
US 20170351226A1 · Bliss et al. · 2017 [cited by applicant]
US 20180113442A1 · Nixon et al. · 2018 [cited by applicant]
US 20180137424A1 · Gabaldon Royval · 2018 [cited by examiner]
US 20200175372A1 · Sharma et al. · 2020 [cited by applicant]
CN 104914807A1 · 2015 [cited by applicant]
CN 105589349A · 2016 [cited by applicant]
CN 106020138A · 2016 [cited by applicant]
CN 107976969 · 2018 [cited by applicant]
JP 2017533645A · 2017 [cited by applicant]
JP 2019101514A · 2021 [cited by applicant]
WO 2019023982A1 · 2019 [cited by applicant]
Iyun “Plant-Wide Diagnosis: Cause-and-Effect Analysis Using Process Connectivity and Directionality Information”, retrieved from https://core.ac.uk/download/1592216.pdf and dated Oct. 2011 (Year: 2011). [cited by examiner]
International Search Report for PCT Application No. PCT/US2020/024506 dated Jun. 3, 2020. [cited by applicant]
Written Opinion for PCT Application No. PCT/US2020/024506 dated Jun. 3, 2020. [cited by applicant]
Fleischmann, Hans, et al. “Automated generation of human-machine interfaces in electric drives manufacturing.” 2017 7th International Electric Drives Production Conference (EDPC). IEEE, 2017. [cited by applicant]
Rahul, Rohit, et al. “Automatic information extraction from piping and instrumentation diagrams.” arXiv preprint arXiv:1901.11383 (2019). [cited by applicant]
International Electrotechnical Commission. “Engineering data exchange format for use in industrial automation systems engineering—Automation Markup Language—Part 1: Architecture and general requirements.” International … [cited by applicant]
Examination Report for EP Application No. 20720224.3 dated Mar. 28, 2024. [cited by applicant]
Markus Schleburg et al., A Combined Analysis of Plant Connectivity and Alarm Logs to Reduce the Number of Alerts in an Automation System, Journal of Process Control, Oxford, GB, vol. 23, No. 6, May 31, 2013. [cited by applicant]
EP Extended Search Report for EP Application No. 20720593.1 dated Sep. 28, 2021. [cited by applicant]
EP Extended Search Report for EP Application No. 20719074.5 dated Oct. 6, 2021. [cited by applicant]
EP Extended Search Report for EP Application No. 23210388.7 dated Feb. 27, 2024. [cited by applicant]
PCT Search Report and PCT Written Opinion for PCT/US2020/024501 dated Jun. 17, 2020. [cited by applicant]
PCT Search Report and PCT Written Opinion for PCT/US2020/024502 dated Feb. 3, 2022. [cited by applicant]
Urbas et al., “Towards context adaptive HMIs in process industries”, 2011 9th IEEE International Conference on Industrial Informatics Jul. 26, 2011 (pp. 244-249). IEEE. [cited by applicant]
Pang et al., “IEC 61499 based model-driven process control engineering”, Proceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA) Sep. 16, 2014 (pp. 1-8). IEEE. [cited by applicant]
Steinegger et al., “Automated code generation for programmable logic controllers based on knowledge acquisition from engineering artifacts: Concept and case study”, Proceedings of 2012 IEEE 17th International Conference… [cited by applicant]
Schuhmacher et al., “Knowledge-based graph document modeling.” Proceedings of the 7th ACM international conference on Web search and data mining. 2014 (Year: 2014). [cited by applicant]
Louting Fu, “From engineering diagrams to engineering models: Visual recognition and applications” (Year: 2010). [cited by applicant]
Michal Ondrejcek, “Information Extraction from Scanned Engineering Drawings” (Year: 2009). [cited by applicant]
Koltun, Gennadiy, Moritz Kolter, and Birgit Vogel-Heuser. “Automated generation of modular PLC control software from P&ID diagrams in process industry.” In 2018 IEEE International Systems Engineering Symposium (ISSE), p… [cited by applicant]
Blevins, Terrence L., and Mark Nixon. Control loop foundation: batch and continuous processes. ISA, 2011. (Year: 2011). [cited by applicant]
Romero, David Dorantes, and Nina F. Thornhill. “Integration, navigation and exploration of plant topology networks using the property-graph model.” In 2014 IEEE/SICE International Symposium on System Integration, pp. 74… [cited by applicant]
Rantala, Miia, Hannu Niemist6, Tommi Karhela, Seppo Sierla, and Valeriy Vyatkin. “Applying graph matching techniques to enhance reuse of plant design information.” Computers in Industry 107 (2019): 81-98. (Year: 2019). [cited by applicant]
Arroyo, Esteban, Mario Hoernicke, Pablo Rodriguez, and Alexander Fay. “Automatic derivation of qualitative plant simulation models from legacy piping and instrumentation diagrams.” Computers & Chemical Engineering 92 (2… [cited by applicant]
Binford, T., T. Chen, J. Kunz, and K. H. Law. “Computer Interpretation of Process and Instrument.” (Stanford, 1997). (Year: 1997). [cited by applicant]
Eppstein, D. “ICS 161: Design and analysis of algorithms lecture notes for Feb. 1, 1996.” (UC Irvine, 1996). [Retrieved Online May 14, 2025] <https://ics.uci.edui-eppstein/161/960201.html > (Year: 1996). [cited by applicant]
ISA. “ANSI/ISA-88.01-1995, Batch Control, Part 1: Models and Terminology”, (Instrument Society of America, 1995). ISBN: 1-55617-562-0 (Year: 1995). [cited by applicant]
ISO. “ISO-7200, Technical product documentation—Data fields in title blocks and document headers” (International Organization for Standardization, 2004) (Year: 2004). [cited by applicant]
Cited By (1)
US 12,717,850