Granted Patent B2

US 12,387,317 · App. 18/354,357 · Granted Aug 12, 2025

Inventors: Jonathan Lee (Brooklyn, NY); Anuj Doshi (Long Island City, NY)

Assignee: Nanotronics Imaging, Inc.

G06T7/001G06F3/011G06T7/0008G06V20/41G06T2200/24G06T2207/30164

Abstract

Claims (74)

1. A manufacturing system comprising:

a monitoring platform configured to capture data of an operator during assembly of an article of manufacture, the monitoring platform comprising one or more cameras and one or more microphones; and

an analytics platform in communication with the monitoring platform, the analytics platform configured to analyze the data captured by the monitoring platform, the analytics platform configured to perform operations comprising:

receiving, from the monitoring platform, an indication of the operator performing a step in an assembly process for generating the article of manufacture;

identifying components associated with the step in the assembly process;

prompting the operator to place the components in a field of view of the one or more cameras of the monitoring platform;

receiving, from the monitoring platform, image data corresponding to the components in the field of view of the one or more cameras;

analyzing the image data to determine that the operator has selected all the components required for performing the step in the assembly process;

responsive to determining that the operator has selected all the components required for the step in the assembly process, providing nominal assembly instructions to the operator;

receiving real-time or near real-time video and audio data of the operator performing the step in the assembly process in accordance with the nominal assembly instructions;

detecting an error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process; and

based on the detecting, prompting the operator to repair the error.

2. The manufacturing system of claim 1 , wherein the operations further comprise:

responsive to determining that the operator has not selected all the components required for the step in the assembly process, instructing the operator to select a missing component and place the missing component in the field of view of the one or more cameras.

3. The manufacturing system of claim 1 , wherein the operations further comprise:

responsive to determining that the operator has selected an incorrect component, flagging the incorrect component and instructing the operator to remove the incorrect component and replace the incorrect component with a correct component.

4. The manufacturing system of claim 3 , wherein flagging the incorrect component comprises:

virtually annotating the incorrect component on a computing device accessible to the operator to flag the incorrect component.

5. The manufacturing system of claim 1 , wherein detecting the error in the assembly process comprises:

comparing the real-time or near real-time video or audio data of the operator to the nominal assembly instructions.

6. The manufacturing system of claim 1 , wherein detecting the error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process comprises:

determining whether the error is a critical error; and

responsive to determining that the error is not a critical error, allowing the assembly process to continue.

7. The manufacturing system of claim 1 , wherein detecting the error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process comprises:

determining whether the error is a critical error;

responsive to determining that the error is not a critical error, determining whether the error is repairable; and

responsive to determining that the error is repairable, generating updated instructions for repairing the article of manufacture.

8. A method of performing an assembly process on an article of manufacture comprising:

receiving, by a computing system from a monitoring platform, an indication of an operator performing a step in an assembly process for generating an article of manufacture;

identifying, by the computing system, components associated with the step in the assembly process;

prompting, by the computing system, the operator to place the components in a field of view of one or more cameras of the monitoring platform;

receiving, by the computing system from the monitoring platform, image data corresponding to the components in the field of view of the one or more cameras;

analyzing, by the computing system, the image data to determine that the operator has selected the components required for performing the step in the assembly process;

responsive to determining that the operator has selected the components required for performing the step in the assembly process, providing, by the computing system, nominal assembly instructions to the operator;

receiving, by the computing system, real-time or near real-time video and audio data of the operator performing the step in the assembly process in accordance with the nominal assembly instructions;

detecting, by the computing system, an error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process; and

based on the detecting, prompting, by the computing system, the operator to repair the error.

9. The method of claim 8 , further comprising:

responsive to determining that the operator has not selected the components required for performing the step in the assembly process, instructing, by the computing system, the operator to select a missing component and place the missing component in the field of view of the one or more cameras.

10. The method of claim 8 , further comprising:

responsive to determining that the operator has selected an incorrect component, flagging, by the computing system, the incorrect component and instructing the operator to remove the incorrect component and replace the incorrect component with a correct component.

11. The method of claim 10 , further comprising, wherein flagging, by the computing system, the incorrect component comprises:

virtually annotating the incorrect component on a computing device accessible to the operator to flag the incorrect component.

12. The method of claim 8 , wherein detecting, by the computing system, the error in the assembly process comprises:

comparing the real-time or near real-time video or audio data of the operator to the nominal assembly instructions.

13. The method of claim 8 , wherein detecting, by the computing system, the error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process comprises:

determining whether the error is a critical error; and

responsive to determining that the error is not a critical error, allowing the assembly process to continue.

14. The method of claim 8 , wherein detecting, by the computing system, the error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process comprises:

determining whether the error is a critical error;

responsive to determining that the error is not a critical error, determining whether the error is repairable; and

responsive to determining that the error is repairable, generating updated instructions for repairing the article of manufacture.

15. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by a processor, causes a computing system to perform operations comprising:

identifying, by the computing system, components associated with a step in an assembly process for generating an article of manufacture;

prompting, by the computing system, an operator to place the components in a field of view of one or more cameras of a monitoring platform;

receiving, by the computing system from the monitoring platform, image data corresponding to the components in the field of view of the one or more cameras;

analyzing, by the computing system, the image data to determine that the operator has selected the components required for performing the step in the assembly process;

responsive to determining that the operator has selected the components required for performing the step in the assembly process, providing, by the computing system, nominal assembly instructions to the operator;

receiving, by the computing system, real-time or near real-time video and audio data of the operator performing the step in the assembly process in accordance with the nominal assembly instructions;

detecting, by the computing system, an error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process; and

based on the detecting, prompting, by the computing system, the operator to repair the error.

16. The non-transitory computer readable medium of claim 15 , further comprising:

responsive to determining that the operator has not selected the components required for performing the step in the assembly process, instructing, by the computing system, the operator to select a missing component and place the missing component in the field of view of the one or more cameras.

17. The non-transitory computer readable medium of claim 15 , further comprising:

responsive to determining that the operator has selected an incorrect component, flagging, by the computing system, the incorrect component and instructing the operator to remove the incorrect component and replace the incorrect component with a correct component.

18. The non-transitory computer readable medium of claim 17 , further comprising, wherein flagging, by the computing system, the incorrect component comprises:

virtually annotating the incorrect component on a computing device accessible to the operator to flag the incorrect component.

19. The non-transitory computer readable medium of claim 15 , wherein detecting, by the computing system, the error in the assembly process comprises:

comparing the real-time or near real-time video or audio data of the operator to the nominal assembly instructions.

20. The non-transitory computer readable medium of claim 15 , wherein detecting, by the computing system, the error in the assembly process based on the real-time or near real-time video and audio data of the operator performing the step of the assembly process comprises:

determining whether the error is a critical error;

responsive to determining that the error is not a critical error, allowing the assembly process to continue;

responsive to determining that the error is not a critical error, determining whether the error is repairable; and

responsive to determining that the error is repairable, generating updated instructions for repairing the article of manufacture.

Assignments (2)

Continuity (2)

References Cited (19)