IP Library Granted Patent US 10,416,614
Granted Patent B2
US 10,416,614 · App. 15/284,571 · Granted Sep 17, 2019

Human programming interfaces for machine-human interfaces

Inventors: Lingyun Wang (Princeton, NJ); Arquimedes Martinez Canedo (Plainsboro, NJ)
Assignee: SIEMENS AKTIENGESELLSCHAFT
G05B11/01G06Q10/06G06F3/0484
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,416,614
App. No.
15/284,571
Granted
Sep 17, 2019
Kind
B2
Abstract

A system for interacting with humans in an automation environment includes an automation system machine, one or more processors and a non-transitory, computer-readable storage medium in operable communication with the processors. The computer-readable storage medium comprises one or more programming instructions that, when executed, cause the processors to select a human from a plurality of humans working in the automation environment; retrieve information related to the human using a human programming interface (HPI); and perform one or more actions with the automation system machine based on the information related to the human.

Claims (73)

1. A system for interacting with human behavior in an automation environment, the system comprising:

an automation system machine;

one or more processors;

a non-transitory, computer-readable storage medium in operable communication with the processors, wherein the computer-readable storage medium comprises one or more programming instructions that, when executed, cause the processors to:

implement a human programming interface (HPI) comprising a data structure representative of a human, including properties and methods that digitally define ways in which the human may interact with the automation system machine;

define an HPI for each of a plurality of human attributes relating to the human;

store the HPIs in a machine human interface (MHI) configured to control the automation environment;

select a human from a plurality of humans working in the automation environment;

retrieve information related to the human from the HPIs;

implement a digital twin of the selected human;

update information relating to the selected human in the digital twin in real time; and

interact, using the MHI via the digital twin, with the automation system machine such that one or more actions attributable to the selected human are performed on the automation system machine in real time based on the information related to the selected human.

2. The system of claim 1 , wherein the instructions further cause the processors to:

generate a message describing the one or more actions of the automation system based on a property or a method of at least one of the HPIs; and

provide the message describing the one or more actions to the human.

3. The system of claim 2 , wherein the message is provided by:

using the HPI to identify a communication device associated with the human; and

sending the message describing the one or more actions to the communication device using the HPI.

4. The system of claim 3 , wherein the communication device is a smartphone and the message is sent to the human as a text message or email.

5. The system of claim 2 , wherein the message is provided by:

using the HPI to insert an event with the message into a calendar corresponding to the human.

6. The system of claim 1 , where the human is selected from the plurality of humans working in the automation environment by:

using one of the HPIs relating to a human attribute to retrieve calendar information corresponding to each of the plurality of humans; and

using the calendar information to select the human based on the human's availability over a time period during which the one or more actions will be performed.

7. The system of claim 6 , where the human is selected from the plurality of humans working in the automation environment by:

using one of the HPIs relating to a human attribute to retrieve health-related information corresponding to each of the plurality of humans; and using one or more of the calendar information and the health-related information to select the human based on the human's availability over the time period during which the one or more actions will be performed.

8. The system of claim 6 , where the human is selected from the

plurality of humans working in the automation environment by:

using one of the HPIs relating to a human attribute to retrieve skillset information corresponding to each of the plurality of humans; and

using one or more of the calendar information and the skillset information to select the human based on the human's availability over the time period during which the one or more actions will be performed.

9. The system of claim 1 , wherein the one or more actions

require interaction by the human with the automation system machine and wherein the instructions further cause the processors to:

use one or more of one of the HPIs relating to a human attribute to monitor the interaction by the human with the automation system machine while performing the one or more actions.

10. The system of claim 9 , wherein the instructions further cause the processors to:

during the interaction by the human with the automation system machine, retrieve biological information associated with the human using one of the HPIs relating to a human attribute;

determine that the human is incapable of performing the one or more actions based on the biological information; and

provide a notification to the human to stop performing the one or more actions.

11. The system of claim 1 , wherein the HPIs relating to a human attribute is part of a digital twin associated with the human and the instructions cause the processors to perform at least a portion of the one or more actions using the digital twin associated with the human.

12. A system for interacting with humans in an automation environment, the system comprising:

an automation system machine;

one or more processors;

a non-transitory, computer-readable storage medium in operable communication with the processors, wherein the computer-readable storage medium comprises one or more programming instructions that, when executed, cause the processors to:

implement a data structure representative of an interface between the automation system machine and a human, including properties and methods that digitally provide ways in which the human may interact with the automation system machine, in the form of a human programming interface (HPI);

define an HPI for each of a plurality of human attributes relating to the human;

detect one or more workers present in the automation environment;

for each of the one or more workers, instantiate a digital twin for the worker using a process comprising:

generating the digital twin for the worker;

retrieving current data associated with the worker from the HPIs;

using the current data associated with the worker to update the digital twin; and

providing access to the digital twin to external entities via the HPIs;

update data associated with the worker in real time; and

interact, using the MHI via the digital twin, with the automation system machine such that one or more actions attributable to the human are performed on the automation system machine in real time based on the information related to the human.

13. The system of claim 12 , wherein the instructions further cause the processors to:

receive a request for execution of a human action from a machine in the automation environment; and

use the digital twin associated one of the workers to perform the human action.

14. The system of claim 13 , wherein the request for execution of the human action is received via the HPIs associated with the digital twin.

15. The system of claim 12 , wherein the instructions further cause the processors to:

use the digital twin associated with at least one of the workers to perform testing of at least one product.

16. The system of claim 15 , wherein the product is a digital twin of a real product.

17. The system of claim 12 , wherein the instructions further cause the processors to update the digital twin associated with a worker based on information manually input by the worker.

18. A method for using a human programming interface (HPI) to interact with humans in an automation environment, the method comprising:

providing, by a machine-human interface (MH) computer, a programming interface containing data structures including properties and methods relating to human attributes for accessing sensor data and service data corresponding to a plurality of humans working the automation environment;

instantiating a digital twin human for each of the plurality of humans working in the automation environment;

updating the data structures for each of the plurality of humans in real time based on current data associated with each human;

receiving, by the MHI computer, a function call from a machine via the programming interface for information associated with a human, wherein the function call comprises a request for retrieval of human information;

automatically generating, by the machine-human interface computer, a response to the function call comprising the human information;

transmitting, by the MHI computer, the response to the machine; and

utilizing by the machine the human information and the digital twin human to perform one or more actions attributable to the human on the machine in real time.

19. The method of claim 18 , further comprising:

automatically retrieving the human information from a sensor associated with the human in real time; and

updating the data structure to represent the received human information.

20. The method of claim 18 , further comprising:

automatically retrieving the human information from an external database over a computer network.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 12, 2016
From: SIEMENS CORPORATION
To: SIEMENS AKTIENGESELLSCHAFT
Reel/Frame 040706/0750 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 15, 2016
From: CANEDO, ARQUIMEDES MARTINEZ; WANG, LINGYUN
To: SIEMENS CORPORATION
Reel/Frame 040316/0008 →
Continuity (3)
Provisional Application 62241919 · Oct 15, 2015
Provisional Application 62247214 · Oct 28, 2015
Related Publication 20170108834A1 · Apr 20, 2017