Granted Patent B2

US 12,051,317 · App. 17/740,826 · Granted Jul 30, 2024

Inventors: Guelord Kamitula Ngala-Ngala (Simpang Ampat, MY); Baskar Santhrasegran (Ipoh, MY); Charles Paul Manianglung Alfonso (Bayan Lepas, MY)

Assignee: SanDisk Technologies, Inc.

G08B21/187G08B21/18G08B23/00G08B25/00G08B25/002G05B23/0272G06T7/73G06T2207/20081

Abstract

Claims (62)

1. An apparatus comprising:

a central control circuit configured to remotely connect to a plurality of machines over a network, each machine having a respective local user interface to indicate a machine state of the machine and enable local user input to the machine and having a control switch to enable remote replication of local user interface images and remote control of the machine, the central control circuit configured to:

receive a plurality of alarm codes including a first alarm code and a second alarm code from a local user interface of a machine;

determine whether the plurality of alarm codes correspond to machine states for which a machine learning application has been trained;

obtain an image from the local user interface in response to a determination that the machine learning application has been trained for a first machine state corresponding to the first alarm code, the image corresponding to a screenshot displayed on the monitor of the local user interface;

analyze the image to identify one or more features and compare the one or more features of the image with one or more features of images of a teaching set to determine whether the image correlates with the images of the teaching set;

generate one or more commands in the machine learning application according to correlation of the image and the images of the teaching set;

send the one or more commands to the local user interface according to the features to change the machine state to a non-alarm state; and

assign a machine error associated with the second alarm code to a human operator without assistance of the machine learning application in response to determining that the machine learning application has not been trained for the second machine state corresponding to the second alarm code.

2. The apparatus of claim 1 wherein the central control circuit is further configured to:

receive an additional alarm code from a local user interface of an additional machine;

determine whether the additional alarm code corresponds to a condition for which the machine learning application has been trained; and

send a digital alert to one or more human recipients to indicate an alarm state of the additional machine in response to a determination that the additional alarm code does not correspond to a condition for which the machine learning application has been trained.

3. The apparatus of claim 1 wherein the central control circuit is further configured to determine that the alarm code corresponds to a condition for which the machine learning application has been trained by searching a list that includes a plurality of alarm codes corresponding to conditions for which the machine learning application has been trained.

4. The apparatus of claim 1 wherein the central control circuit is further configured to monitor the machine state to determine whether the machine state changes from an alarm state to a non-alarm state in response to sending the one or more commands to the local user interface.

5. The apparatus of claim 4 wherein the central control circuit is further configured to:

obtain one or more additional images from the local user interface in response to a determination that the machine state has not changed from the alarm state;

analyze the one or more additional images to identify one or more additional features;

generate one or more additional commands in the machine learning application; and

send the one or more additional commands to the local user interface according to the additional features to change the machine state.

6. The apparatus of claim 5 wherein the central control circuit is further configured to send a digital alert to one or more human recipients to indicate the alarm state of the machine in response to a determination that the machine state has not changed from the alarm state to the non-alarm state in response to the one or more additional commands.

7. The apparatus of claim 1 wherein the image corresponds to a screenshot associated with the alarm code including information regarding an alarm state of the machine.

8. The apparatus of claim 1 wherein the teaching set includes recorded human input in response to the images of the teaching set that resulted in machine state changes from alarm states to non-alarm states.

9. The apparatus of claim 8 wherein the one or more commands correspond to human input of the teaching set including at least one of pointer movement, feature selection, selection from drop-down or pop-up menu, and text entry.

10. A method comprising:

monitoring a plurality of machines that each has a user interface that includes a monitor for monitoring of the machine by a local user and a local input device for control of the machine by the local user, the user interface of each machine is connected to a central control circuit by a network;

acquiring a training screenshot at the central control circuit from a user interface of a machine in an alarm state;

subsequently, recording a sequence of human input to resolve the alarm state;

adding the training screenshot and the sequence of human input to a training set that includes additional training screenshots and additional sequences of human input;

subsequently, generating a machine learning model for a machine learning application from the training image, the additional training images, the sequence and the additional sequences of human input;

subsequently, receiving an alarm code from a user interface of a machine;

determining that the alarm code corresponds to the alarm state that was used to generate the machine learning model for the machine learning application has been trained;

subsequently, obtaining a screenshot from the user interface of the machine in response to determining that the machine learning application has been trained for the alarm state;

analyzing the screenshot to identify one or more features;

generating one or more commands in the machine learning application according to the machine learning model that was generated using the training image and the sequence of human input; and

sending the one or more commands to the user interface of the machine to resolve the alarm state.

11. The method of claim 10 further comprising:

receiving an additional alarm code from an additional machine;

determining whether the additional alarm code corresponds to a condition for which the machine learning application has been trained; and

sending a digital alert to one or more human recipients to indicate an alarm state of the additional machine in response to determining that the additional alarm code does not correspond to a condition for which the machine learning application has been trained.

12. The method of claim 10 further comprising determining that the alarm code corresponds to a condition for which the machine learning application has been trained by searching a list that includes a plurality of alarm codes corresponding to conditions for which the machine learning application has been trained.

13. The method of claim 10 further comprising:

determining whether the machine state changes from an alarm state to a non-alarm state in response to sending the one or more commands to the user interface;

in response to determining that the machine state has not changed from the alarm state to the non-alarm state:

obtaining one or more additional images from the user interface;

analyzing the one or more additional images to identify one or more additional features;

generating one or more additional commands in the machine learning application;

sending the one or more additional commands to the user interface according to the additional features to change the machine state; and

subsequently, in response to determining that the machine state has not changed from the alarm state to the non-alarm state, sending a digital alert to one or more human recipients to indicate the alarm state of the machine.

14. The method of claim 10 further comprising:

comparing the one or more features of the image with one or more features of images of a teaching set to determine whether the image correlates with the images of the teaching set.

15. The method of claim 10 wherein analyzing the screenshot includes finding the brightest areas of the image and updating pointer position and generating the one or more commands includes generating a click event at the updated pointer position.

16. The method of claim 10 wherein the one or more commands correspond to human input of the teaching set including at least one of pointer movement, feature selection, selection from drop-down or pop-up menu, and text entry.

17. An apparatus comprising

a plurality of machines including at least one of manufacturing machines and testing machines, each machine having a respective user interface to indicate a machine state of the machine and enable user input to the machine and each machine having a control switch to enable remote replication of local user interface images and remote control of the machine;

a network connecting user interfaces of the plurality of machines to enable remote access to user interfaces of the plurality of machines through respective control switches;

means for monitoring and controlling the plurality of machines, the means for monitoring and controlling is connected to the network to receive a plurality of alarm codes including a first alarm code and a second from the plurality of machines and to obtain screenshots from respective user interfaces of the plurality of machines;

means for determining that the first alarm code corresponds to a first machine state for which a machine learning application has been trained, obtaining a screenshot corresponding to the first machine state, determining that the second alarm code corresponds to a second machine state for which the machine learning application has not been trained and in response assigning resolution of a machine error associated with the second alarm code to a human operator; and

means for analyzing the screenshots and for generating one or more commands using the machine learning application and sending the one or more commands to the user interface to change the machine state from an alarm state to a non-alarm state.

18. The apparatus of claim 17 wherein the plurality of machines includes one or more of a backgrind machine, a dicing machine, a surface mount machine, a die attach machine, a test machine, and a wire bonding machine.

19. The apparatus of claim 18 wherein the alarm codes include one or more of temperature-related error codes, pressure-related error codes, and alignment-related error codes.

20. The apparatus of claim 17 further comprising means for updating a machine learning model by recording successful human and machine responses.

Assignments (9)

Continuity (1)