Determining process deviations through video analysis
A computer-implemented method, according to one embodiment, includes identifying, in a reference video of a production process of a product, a discrete and non-overlapping set of first tasks. The first tasks define at least a first sub-process. The method further includes analyzing a live video of the production process to identify frames of the live video that include second tasks that define a second sub-process, and analyzing the frames of the live video for determining whether a match exists between the first tasks and the second tasks. In response to a determination that the match does not exist, an alert that a deviation is present in the production process depicted in the live video is output.
1 . A computer-implemented method, comprising:
identifying, in a reference video of a production process of a product, a discrete and non-overlapping set of first tasks, wherein the first tasks define at least a first sub-process;
analyzing a live video of the production process to identify frames of the live video that include second tasks that define a second sub-process;
analyzing the frames of the live video for determining whether a match exists between the first tasks and the second tasks, wherein analyzing the live video of the production process to identify the frames of the live video that include the second tasks includes: converting the reference video into a plurality of frames of the production process of the product, comparing the frames to find overlapping tasks in the first tasks of the reference video, and filtering redundant frames having the overlapping tasks comprising at least one of cutting the product, folding the product, or coloring the product in the first tasks from the plurality of frames of the reference video, wherein non-redundant frames remaining of the plurality of frames each depict one of the second tasks; and
in response to a determination that the match does not exist, outputting an alert that a deviation is present in the production process depicted in the live video, the alert being a first notification, wherein the first notification is at least one of output to a user device, output to a controller configured to display a light of a predetermined color on an assembly line, or output to the controller configured to sound an audio alarm on the assembly line.
2 . The computer-implemented method of claim 1 , wherein:
a predetermined fix to the deviation is output in a separate notification from the alert;
a given sub-process on the product is performed by a robot; and
the analyzing of the live video of the production process excludes the given sub-process performed by the robot because the given sub-process is routinely performed with an acceptable amount of the deviation.
3 . The computer-implemented method of claim 1 , wherein the first tasks define a third sub-process of the production process, and comprising: training an object detection algorithm to identify the first tasks and the second tasks; analyzing the live video of the production process to identify second frames of the live video that include fourth tasks that define a fourth sub-process; and causing the trained object detection algorithm to determine whether a match exists between the first tasks that define the third sub-process and the fourth tasks.
4 . The computer-implemented method of claim 3 , wherein the object detection algorithm is implemented in a neural network, wherein the first and second tasks are used as objects for the training of the neural network.
5 . The computer-implemented method of claim 1 , wherein the match between the first tasks and the second tasks is defined by each of the first tasks matching with a different one of the second tasks.
6 . The computer-implemented method of claim 1 , wherein the match is based on a group of properties consisting of: product shape, an order that the second tasks are detected to occur in, and product color.
7 . A computer program product, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
identify, by the computer, in a reference video of a production process of a product, a discrete and non-overlapping set of first tasks, wherein the first tasks define at least a first sub-process;
analyze, by the computer, a live video of the production process to identify frames of the live video that include second tasks that define a second sub-process;
analyze, by the computer, the frames of the live video for determining whether a match exists between the first tasks and the second tasks, wherein analyzing the live video of the production process to identify the frames of the live video that include the second tasks includes: converting the reference video into a plurality of frames of the production process of the product, comparing the frames to find overlapping tasks in the first tasks of the reference video, and filtering redundant frames having the overlapping tasks comprising at least one of cutting the product, folding the product, or coloring the product in the first tasks from the plurality of frames of the reference video, wherein non-redundant frames remaining of the plurality of frames each depict one of the second tasks; and
in response to a determination that the match does not exist, output, by the computer, an alert that a deviation is present in the production process depicted in the live video, the alert being a first notification, wherein the first notification is at least one of output to a user device, output to a controller configured to display a light of a predetermined color on an assembly line, or output to the controller configured to sound an audio alarm on the assembly line.
8 . The computer program product of claim 7 , wherein:
a predetermined fix to the deviation is output in a separate notification from the alert;
a given sub-process on the product is performed by a robot; and
the analyzing of the live video of the production process excludes the given sub-process performed by the robot because the given sub-process is routinely performed with an acceptable amount of the deviation.
9 . The computer program product of claim 7 , wherein the first tasks define a third sub-process of the production process, and the program instructions executable by the computer to cause the computer to: train, by the computer, an object detection algorithm to identify the first tasks and the second tasks; analyze, by the computer, the live video of the production process to identify second frames of the live video that include fourth tasks that define a fourth sub-process; and cause, by the computer, the trained object detection algorithm to determine whether a match exists between the first tasks that define the third sub-process and the fourth tasks.
10 . The computer program product of claim 9 , wherein the object detection algorithm is implemented in a neural network, wherein the first and second tasks are used as objects for the training of the neural network.
11 . The computer program product of claim 7 , wherein the match between the first tasks and the second tasks is defined by each of the first tasks matching with a different one of the second tasks.
12 . The computer program product of claim 7 , wherein the first sub-process comprises drilling the product.
13 . A system, comprising:
a hardware processor; and
logic integrated with the processor, executable by the processor, or integrated with and executable by the processor, the logic being configured to:
identify, in a reference video of a production process of a product, a discrete and non-overlapping set of first tasks, wherein the first tasks define at least a first sub-process;
analyze a live video of the production process to identify frames of the live video that include second tasks that define a second sub-process;
analyze the frames of the live video for determining whether a match exists between the first tasks and the second tasks, wherein analyzing the live video of the production process to identify the frames of the live video that include the second tasks includes: converting the reference video into a plurality of frames of the production process of the product, comparing the frames to find overlapping tasks in the first tasks of the reference video, and filtering redundant frames having the overlapping tasks comprising at least one of cutting the product, folding the product, or coloring the product in the first tasks from the plurality of frames of the reference video, wherein non-redundant frames remaining of the plurality of frames each depict one of the second tasks; and
in response to a determination that the match does not exist, output an alert that a deviation is present in the production process depicted in the live video, the alert being a first notification, wherein the first notification is at least one of output to a user device, output to a controller configured to display a light of a predetermined color on an assembly line, or output to the controller configured to sound an audio alarm on the assembly line.
14 . The system of claim 13 , wherein:
a predetermined fix to the deviation is output in a separate notification from the alert;
a given sub-process on the product is performed by a robot; and
the analyzing of the live video of the production process excludes the given sub-process performed by the robot because the given sub-process is routinely performed with an acceptable amount of the deviation.
15 . The system of claim 13 , wherein the first tasks define a third sub-process of the production process, and the logic being configured to: train an object detection algorithm to identify the first tasks and the second tasks; analyze the live video of the production process to identify second frames of the live video that include fourth tasks that define a fourth sub-process; and cause the trained object detection algorithm to determine whether a match exists between the first tasks that define the third sub-process and the fourth tasks.
16 . The system of claim 13 , wherein the match between the first tasks and the second tasks is defined by each of the first tasks matching with a different one of the second tasks.
17 . The system of claim 13 , wherein the match is based on a group of properties consisting of: product shape, an order that the second tasks are detected to occur in, and product color.
18 . The system of claim 13 , wherein the first sub-process comprises drilling the product.