Method and apparatus for diagnosing error of object placement using artificial neural network
There is provided a method and an apparatus for diagnosing an object placement error by using an artificial neural network by which feature data of the (N+1)-th stage among the feature data of a plurality of stages is generated by using any one feature data among a plurality of feature data generated in an operation process of a main network and feature data of the plurality of stages, and a placement error of at least one object is diagnosed by using an artificial neural network of a new structure, and thus, the diagnosis accuracy of a placement error of an object that is difficult to detect an edge, such as a transparent vial, may be greatly increased.
1. An object placement error diagnosing method comprising:
inputting, to an artificial neural network, at least one input image representing a space in which at least one object is placed; and
acquiring data representing a diagnosis result of a placement error of the at least one object from an output of the artificial neural network,
wherein the artificial neural network comprises:
a main network configured to perform a convolution operation and a pooling operation on the at least one input image multiple times; and
a pyramid network configured to generate feature data of a plurality of stages for generating data representing the diagnosis result of the placement error of the at least one object by using a plurality of feature data generated in an operation process of the main network, and
feature data of an (N+1)-th stage among the feature data of the plurality of stages is generated by using any one feature data among the plurality of feature data generated in the operation process of the main network and feature data of an N-th stage among the feature data of the plurality of stages.
2. The object placement error diagnosing method of claim 1 , wherein the pyramid network comprises:
an (N+1)-th feature block configured to generate the feature data of the (N+1)-th stage by using any one feature data among the plurality of feature data generated in the operation process of the main network and the feature data of the N-th stage; and
an (N+2)-th feature block configured to generate feature data of an (N+2)-th stage by using any other feature data among the plurality of feature data generated in the operation process of the main network and the feature data of the (N+1)-th stage.
3. The object placement error diagnosing method of claim 2 , wherein the pyramid network further comprises:
an N-th reduction layer configured to perform a pooling operation and a convolution operation on the feature data of the N-th stage; and
the (N+1)-th feature block generates the feature data of the (N+1)-th stage by performing a convolution operation and a pooling operation on any one feature data among the plurality of feature data generated in the operation process of the main network and feature data output as an operation result of the N-th reduction layer.
4. The object placement error diagnosing method of claim 3 , wherein the pyramid network further comprises:
an N-th pyramidal convolution layer for performing a convolutional operation on the feature data of the (N+2)-th stage;
an (N+2)-th reduction layer configured to perform a pooling operation and a convolution operation on the feature data of the (N+2)-th stage; and
an (N+3)-th feature block configured to generate feature data of an (N+3)-th stage by using feature data output as an operation result of the N-th pyramidal convolution layer and feature data output as an operation result of the (N+2)-th reduction layer.
5. The object placement error diagnosing method of claim 1 , wherein
the main network includes a plurality of transition layers for performing a convolution operation and a pooling operation on any one feature data among the plurality of feature data generated in the operation process of the main network such that a size of a feature map is reduced, and
the feature data of the (N+1)-th stage is generated by using feature data output as an operation result of any one of the plurality of transition layers and the feature data of the N-th stage among the feature data of the plurality of stages.
6. The object placement error diagnosing method of claim 2 , wherein the main network further includes a plurality of dense block layers that are inserted between the plurality of transition layers and perform a convolution operation on feature data output as an operation result of a previous transition layer according to a DenseNet model such that sizes of feature maps are equal to each other.
7. The object placement error diagnosing method of claim 1 , wherein the pyramid network comprises:
a first feature block configured to generate feature data of a first stage by performing a convolution operation and a pooling operation on any one feature data among the plurality of feature data generated in the operation process of the main network; and
a second feature block configured to generate feature data of a second stage by using any other feature data among the plurality of feature data generated in the operation process of the main network and the feature data of the first stage output as an operation result of the first feature block.
8. The object placement error diagnosing method of claim 1 , wherein
the at least one object is at least one transparent vial, and
the object placement error diagnosing method further includes generating the at least one input image from a captured image of a tray on which the at least one transparent vial is placed.
9. The object placement error diagnosing method of claim 1 , wherein the artificial neural network further includes a detection layer for generating each class score representing a diagnosis result of the placement error of the at least one object for each stage by using feature data of each stage.
10. The object placement error diagnosing method of claim 9 , wherein
each class score includes a value representing success or failure of placement of each object, a probability value of success or failure of the placement of each object, a value representing a position of a bounding box of each object, and a value representing a size of the bounding box of each object, and
the artificial neural network further includes a non-maximum suppression (NMS) layer for selecting any one class score from among a plurality of class scores generated in the detection layer by using a non-maximum suppression (NMS) algorithm and outputting the selected one class score as data representing a diagnosis result of the placement error of the at least one object.
11. A non-transitory computer-readable recording medium comprising: a program that is recorded thereon and causes a computer to perform the object placement error diagnosing method of claim 1 .
12. An object placement error diagnosing apparatus comprising:
a processor configured to input, to an artificial neural network, at least one input image representing a space in which at least one object is placed, and to acquire data representing a diagnosis result of a placement error of the at least one object from an output of the artificial neural network,
wherein the artificial neural network comprises:
a main network configured to perform a convolution operation and a pooling operation on the at least one input image multiple times; and
a pyramid network configured to generate feature data of a plurality of stages for generating data representing the diagnosis result of the placement error of the at least one object by using a plurality of feature data generated in an operation process of the main network, and
feature data of an (N+1)-th stage among the feature data of the plurality of stages is generated by using any one feature data among the plurality of feature data generated in the operation process of the main network and feature data of an N-th stage among the feature data of the plurality of stages.