IP Library › Granted Patent US 11,360,461
Granted Patent B2
US 11,360,461 · App. 16/932,374 · Granted Jun 14, 2022

Audio recognition-based industrial automation control

Inventors: Thong Nguyen (New Berlin, WI); Paul David Schmirler (Glendale, WI); Alvaro Sanchez (Oak Creek, WI); Darrell Steven Filtz (Cedarburg, WI); Steven James Litzau (Greendale, WI)
Assignee: Rockwell Automation Technologies, Inc.
G05B19/4183G05B19/4184G06F16/634G06F16/683G05B2219/35458G05B2219/37337
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,360,461
App. No.
16/932,374
Granted
Jun 14, 2022
Kind
B2
Abstract

A system for performing industrial automation control may include an audio device that receives audio data from an element in an industrial automation system. The audio device may determine orientation data based on the audio data. In addition, the audio device may determine an automation command to control a machine in the industrial automation system based on the audio data and the orientation data. After determining the automation command, the audio device may implement a first control action for the machine based at least in part on the automation command, where the first control action causes the machine to adjust an operation.

Claims (59)

1. A method, comprising:

detecting, via an audio device, audio data from a user in an industrial automation system;

determining, via the audio device, an automation command based on the audio data;

identifying, via the audio device, the user based on the audio data and a voice signature of the user;

determining, via the audio device, that the user belongs to a subset of users eligible to provide the automation command based on identifying the user;

determining, via the audio device, a confidence value for the automation command based on one or more secondary inputs, wherein the one or more secondary inputs are configured to confirm that the audio data corresponded to the automation command; and

sending, via the audio device, a command to a machine in the industrial automation system, wherein the command is configured to cause the machine to implement a control action, and wherein the control action is selected based on the automation command and the confidence value.

2. The method of claim 1 , comprising determining, via the audio device, orientation data associated with the user relative to the audio device based on detected audio strength data, the one or more secondary inputs, a triangulation scheme, or a combination thereof.

3. The method of claim 2 , comprising identifying, via the audio device, the machine as associated with the automation command based on the orientation data.

4. The method of claim 1 , comprising:

determining, via the audio device, a location of the user based on the one or more secondary inputs and the audio data; and

implementing, via the audio device, the automation command in response to the location being within a safety perimeter.

5. The method of claim 1 , wherein the one or more secondary inputs comprise a manual switch, a button, image data, pressure data detected on a pressure pad, a signal from a light curtain, one or more signals from a wearable device, or any combination thereof.

6. The method of claim 5 , comprising:

determining, via the audio device, whether the confidence value is greater than a threshold;

in response to the confidence value being greater than the threshold, sending, via the audio device, the command to the machine; and

in response to the confidence value being less than the threshold:

sending, via the audio device, a default automation command to the machine;

sending, via the audio device, a notification requesting a manual confirmation;

sending, via the audio device, a second notification indicating a low confidence value;

or any combination thereof.

7. The method of claim 6 , wherein the default automation command comprises a safe-state command or a continuing operation command.

8. The method of claim 1 , comprising:

determining, via the audio device, whether an abnormal operating condition is present in the machine based on the one or more secondary inputs; and

determining, via the audio device, the automation command to correct the machine based on whether the abnormal operating condition is present.

9. The method of claim 1 , wherein determining that the user belongs to the subset of users eligible to provide the automation command comprises:

determining, via the audio device, that the user belongs to the subset of users from a plurality of subsets of users; and

determining that users associated with the subset of users are eligible to provide a first subset of automation commands that includes the automation command, wherein one or more subsets of the plurality of subsets of users are eligible to provide a second subset of automation commands different from the first subset of automation commands.

10. A device, comprising:

a memory comprising one or more instructions; and

a processor configured to execute the one or more instructions, which cause the processor to perform operations comprising:

detecting audio data associated with a user in an industrial automation system from one or more sensors;

determining an automation command based on the audio data;

identifying the user based on the audio data and a voice signature of the user;

determining that the user belongs to a subset of users eligible to provide the automation command based on identifying the user;

determining a confidence value for the automation command based on one or more secondary inputs, wherein the one or more secondary inputs are configured to confirm that the audio data corresponded to the automation command; and

sending a command to a machine in the industrial automation system, wherein the command is configured to cause the machine to implement a control action, wherein the control action is selected based on the automation command and the confidence value.

11. The device of claim 10 , wherein the processor is configured to perform the operations comprising determining orientation data associated with the user based on strength data associated with the audio data, one or more secondary inputs, a triangulation scheme, or any combination thereof.

12. The device of claim 10 , wherein the processor is configured to perform the operations comprising determining the automation command based on a plurality of automation commands stored in a database and a plurality of audio datasets associated with the plurality of automation commands stored in the database.

13. The device of claim 10 , the processor is configured to perform the operations comprising:

determining that the audio data comprises noise;

receiving additional audio data from an additional sensor in response to determining that the audio data comprises noise; and

determining the automation command based on the additional audio data.

14. The device of claim 13 , wherein the additional audio data corresponds to the one or more secondary inputs.

15. The device of claim 10 , wherein the audio data comprises a redundant command repeated at least two times.

16. The device of claim 15 , wherein the processor is configured to perform the operations comprising determining the confidence value based on the redundant command.

17. A tangible, non-transitory computer-readable medium configured to store instructions executable by a processor of an electronic device that, when executed by the processor, cause the processor to perform operations comprising:

detecting audio data associated with a user in an industrial automation system from one or more sensors;

determining an automation command that corresponds to a control action configured to control one or more operations of a machine in the industrial automation system based on the audio data;

identifying the user based on the audio data and a voice signature of the user;

determining that the user belongs to a subset of users eligible to provide the automation command based on identifying the user;

determining a confidence value for the automation command based on one or more secondary inputs, wherein the one or more secondary inputs are configured to confirm that the audio data corresponded to the automation command; and

sending the automation command to the machine in the industrial automation system based on the confidence value.

18. The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the processor, cause the processor to perform the operations comprising in response to the confidence value being less than a threshold, sending a notification to a computing device, and wherein the notification comprises a request for a manual confirmation.

19. The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the processor, cause the processor to perform the operations comprising:

determining one or more characteristics associated with the audio data;

matching the one or more characteristics to a plurality of automation commands stored in a database and a plurality of audio datasets associated with the plurality of automation commands stored in the database; and

determining the automation command based on one or more relationships between the one or more characteristics, the plurality of automation commands, and the plurality of audio datasets.

20. The non-transitory computer-readable medium of claim 19 , wherein the one or more characteristics comprise an amplitude, a frequency, or both.

Continuity (2)
Continuation 16116605 · Aug 29, 2018
Related Publication 20200348651A1 · Nov 5, 2020