Device and method for detecting on product in automatic checkout system, storage medium storing instructions to perform method for detecting on product in automatic checkout system

1 . A method for detecting a product in an automatic checkout system, the method comprising:

receiving frame images of an automatic checkout counter and preprocessing the received frame images;

detecting one or more products in each of the preprocessed frame images using a pre-trained detection model, wherein the detecting one or more products includes detecting a bounding box corresponding to each product in each of the preprocessed frame images;

tracking the one or more products detected in the preprocessed frame images, wherein the tracking the one or more products includes:

determining whether at least a part of the bounding box is included in a region of interest set in the preprocessed frame images and assigning a product identifier corresponding to the bounding box wherein the determining includes determining that the bounding box is included in the region of interest when a center of the bounding box is included in the region of interest and a distance between the center of the bounding box and a center of the region of interest is equal to or less than a predetermined first value; and

checking whether the bounding box is included in the region of interest during preset consecutive frames among the preprocessed frame images based on the product identifier; and

counting the one or more detected products based on tracking results for the one or more products.

2 . The method of claim 1 , wherein the preprocessing the received frame images includes:

removing at least one of noise and motion blur from each frame image; and

predicting a human hand region in each frame image and removing the predicted human hand region in each frame image by masking.

3 . The method of claim 2 , wherein the removing at least one of noise and motion blur includes removing a residual region excluding an overlapping region in which the predicted human hand region is overlapped with each product.

4 . The method of claim 1 , wherein the pre-trained detection model is pre-trained to receive a plurality of training images generated by manipulating at least one of a rotation angle, an enlargement or reduction ratio, or a gamma adjustment value of each of a plurality of sample images, and detect at least one of training products in the plurality of training images.

5 . A device for detecting a product in an automatic checkout system, the device comprising:

a memory configured to store one or more instructions; and

a processor configured to execute the one or more instructions stored in the memory, wherein the instructions, when executed by the processor, cause the processor to:

receive frame images of an automatic checkout counter and preprocess the received frame images;

detect one or more products in each of the preprocessed frame images using a pre-trained detection model, wherein detecting one or more products includes detecting a bounding box corresponding to each product in each of the preprocessed frame images;

track the one or more products detected in the preprocessed frame images, wherein to track the one or more products includes:

determine whether at least a part of the bounding box is included in a region of interest set in the preprocessed frame images and assigning a product identifier corresponding to the bounding box wherein the determining includes determining that the bounding box is included in the region of interest when a center of the bounding box is included in the region of interest and a distance between the center of the bounding box and a center of the region of interest is equal to or less than a predetermined first value; and

check whether the bounding box is included in the region of interest during preset consecutive frames among the preprocessed frame images based on the product identifier; and

count the one or more detected products based on tracking results for the one or more products.

6 . The device of claim 5 , wherein the processor is configured to:

remove at least one of noise and motion blur from each frame image; and

predict a human hand region in each frame image and removing the predicted human hand region in each frame image by masking.

7 . The device of claim 6 , wherein the processor is configured to remove a residual region excluding an overlapping region in which the predicted human hand region is overlapped with each product.

8 . The device of claim 5 , wherein the pre-trained detection model is pre-trained to receive a plurality of training images generated by manipulating at least one of a rotation angle, an enlargement or reduction ratio, or a gamma adjustment value of each of a plurality of sample images, and detect at least one of training products in the plurality of training images.

9 . A non-transitory computer readable storage medium storing computer executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform a method for detecting a product in an automatic checkout system, the method comprising:

receiving frame images of an automatic checkout counter and preprocessing the received frame images;

detecting one or more products in each of the preprocessed frame images using a pre-trained detection model, wherein detecting one or more products includes detecting a bounding box corresponding to each product in each of the preprocessed frame images;

tracking the one or more products detected in the preprocessed frame images, wherein the tracking the one or more products includes:

determining whether at least a part of the bounding box is included in a region of interest set in the preprocessed frame images and assigning a product identifier corresponding to the bounding box wherein the determining includes determining that the bounding box is included in the region of interest when a center of the bounding box is included in the region of interest and a distance between the center of the bounding box and a center of the region of interest is equal to or less than a predetermined first value; and

checking whether the bounding box is included in the region of interest during preset consecutive frames among the preprocessed frame images based on the product identifier; and

counting the one or more detected products based on tracking results for the one or more products.

10 . The method of claim 9 , wherein the preprocessing the received frame images includes:

removing at least one of noise and motion blur from each frame image; and

predicting a human hand region in each frame image and removing the predicted human hand region in each frame image by masking.

11 . The method of claim 10 , wherein the removing at least one of noise and motion blur includes removing a residual region excluding an overlapping region in which the predicted human hand region is overlapped with each product.

12 . The method of claim 9 , wherein the pre-trained detection model is pre-trained to receive a plurality of training images generated by manipulating at least one of a rotation angle, an enlargement or reduction ratio, or a gamma adjustment value of each of a plurality of sample images, and detect at least one of training products in the plurality of training images.

13 . A method for detecting a product in an automatic checkout system, the method comprising:

receiving frame images of an automatic checkout counter and preprocessing the received frame images;

detecting one or more products in each of the preprocessed frame images using a pre-trained detection model, wherein the detecting one or more products includes detecting a bounding box corresponding to each product in each of the preprocessed frame images;

tracking the one or more products detected in the preprocessed frame images, wherein the tracking the one or more products includes:

determining whether at least a part of the bounding box is included in a region of interest set in the preprocessed frame images and assigning a product identifier corresponding to the bounding box wherein the determining includes determining that the bounding box is included in the region of interest when the center of the bounding box is included in the region of interest and an intersection over unit (IoU) between the bounding box and the region of interest is equal to or greater than a predetermined second value; and

checking whether the bounding box is included in the region of interest during preset consecutive frames among the preprocessed frame images based on the product identifier; and

counting the one or more detected products based on tracking results for the one or more products.

14 . The method of claim 13 , wherein the preprocessing the received frame images includes:

removing at least one of noise and motion blur from each frame image; and

predicting a human hand region in each frame image and removing the predicted human hand region in each frame image by masking.

15 . The method of claim 14 , wherein the removing at least one of noise and motion blur includes removing a residual region excluding an overlapping region in which the predicted human hand region is overlapped with each product.

16 . The method of claim 13 , wherein the pre-trained detection model is pre-trained to receive a plurality of training images generated by manipulating at least one of a rotation angle, an enlargement or reduction ratio, or a gamma adjustment value of each of a plurality of sample images, and detect at least one of training products in the plurality of training images.

17 . A method for detecting a product in an automatic checkout system, the method comprising:

receiving frame images of an automatic checkout counter and preprocessing the received frame images;

detecting one or more products in each of the preprocessed frame images using a pre-trained detection model, wherein the detecting one or more products includes detecting a bounding box corresponding to each product in each of the preprocessed frame images;

tracking the one or more products detected in the preprocessed frame images, wherein the tracking the one or more products includes:

determining whether at least a part of the bounding box is included in a region of interest set in the preprocessed frame images and assigning a product identifier corresponding to the bounding box wherein the determining includes determining that the bounding box is included in the region of interest when the center of the bounding box is included in the region of interest and a distance between at least one side of the bounding box and a vertex of the region of interest adjacent to the one side is equal to or greater than a predetermined third value; and

checking whether the bounding box is included in the region of interest during preset consecutive frames among the preprocessed frame images based on the product identifier; and

counting the one or more detected products based on tracking results for the one or more products.

18 . The method of claim 17 , wherein the preprocessing the received frame images includes:

removing at least one of noise and motion blur from each frame image; and

predicting a human hand region in each frame image and removing the predicted human hand region in each frame image by masking.

19 . The method of claim 18 , wherein the removing at least one of noise and motion blur includes removing a residual region excluding an overlapping region in which the predicted human hand region is overlapped with each product.

20 . The method of claim 17 , wherein the pre-trained detection model is pre-trained to receive a plurality of training images generated by manipulating at least one of a rotation angle, an enlargement or reduction ratio, or a gamma adjustment value of each of a plurality of sample images, and detect at least one of training products in the plurality of training images.