System and method for anti-shoplifting in self-checkout

1. A system for self-checkout, comprising a scanner, an imaging device, and a computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein the computer executable code, when executed at the processor, is configured to:

initiate a self-checkout event;

instruct the imaging device to capture video frames of a region of interest (ROI), wherein the ROI comprises a table region for placing a product, and the table region comprises a scanning region for scanning the product;

track the product in the video frames;

record scanning status and location status of the product, wherein the scanning status comprises scanned and unscanned, and the location status comprises in the table region, and out of the table region but within the ROI;

in response to receiving a scanning signal from the scanner when the product is placed in the scanning region, record scanning status of the product as scanned;

calculate a shoplifting risk score based on a number of the product having the scanning status of unscanned and disappear from the table region or the ROI; and

provide a shoplifting warning when the shoplifting risk score is greater than a predetermined value.

2. The system of claim 1 , wherein the computer executable code is configured to, before track the product:

segment the video frames such that each pixel of the video frames is labeled with hand of a customer, product in hand, product on table, and background; and

detect the product in hand and the product on table based on the labels of the pixels.

3. The system of claim 2 , wherein the computer executable code is configured to segment and detect using a deep convolutional neural network (CNN).

4. The system of claim 1 , wherein the computer executable code is configured to track the product using tracking-by-detection and greedy search when the product is the product in hand, and track the product using appearance based tracker when the product is the product on table.

5. The system of claim 1 , wherein the shoplifting risk score R is calculated by:

R=r 1 ×N 1 +r 2 ×N 2 ,

wherein N 1 is a number of the product that has the scanning status of unscanned and disappears from the table region, N 2 is a number of the product that has the scanning status of unscanned and disappears from the ROI, r 1 is a medium risk factor, and r 2 is a high risk factor.

6. The system of claim 5 , wherein the medium risk factor is in a range of 0.5-1.0, and the high risk factor is in a range of 1.0-2.0.

7. The system of claim 5 , the computer executable code is further configured to:

provide the shoplifting warning that shoplifting happened when R≥β 1 ×(N 3 ) α ; and

providing the shoplifting warning that shoplifting might happened when β 1 ×(N 3 ) α >R≥β 2 ×(N 3 ) α ,

wherein N 3 is a number of the product that has the scanning status of scanned, α is a predetermined number in a range of 0.29-1.0, β 1 is a predetermined number in a range of 0.5-1.0, and β 2 is a predetermined number in a range of 1.0-2.0.

8. The system of claim 1 , wherein the computer executable code is further configured to:

provide a scan difficulty message when the product is placed in the scanning region and has the scanning status of unscanned, and the computing device has not received a scanning signal of the product from the scanner for an accumulated time greater than a predetermined time.

9. The system of claim 8 wherein the predetermined time is three seconds.

10. The system of claim 1 , wherein the imaging device is a red green blue-depth (RGBD) camera.

11. The system of claim 1 , wherein the computing device is an embedded device.

12. A method for self-checkout, comprising:

initiating, by a computing device, a self-checkout event;

instructing, by the computing device, an imaging device to capture video frames of a region of interest (ROI), wherein the ROI comprises a table region for placing a product, and the table region comprises a scanning region for scanning the product;

tracking, by the computing device, the product in the video frames;

recording, by the computing device, scanning status and location status of the product, wherein the scanning status comprises scanned and unscanned, and the location status comprises in the table region, and out of the table region but within the ROI;

in response to receiving a scanning signal from a scanner when the product is placed in the scanning region, recording, by the computing device, scanning status of the product as scanned;

calculating a shoplifting risk score based on a number of the product having the scanning status of unscanned and disappears from the table region or the ROI; and

providing a shoplifting warning when the shoplifting risk score is greater than a predetermined value.

13. The method of claim 12 , further comprising, before the step of tracking the product:

segmenting, by the computing device, the video frames such that each pixel of the video frames is labeled with hand of a customer, product in hand, product on table, and background; and

detecting, by the computing device, the product in hand and the product on table based on the labels of the pixels.

14. The method of claim 13 , wherein at least one of the steps of segmenting and detecting is performed using a deep convolutional neural network (CNN).

15. The method of claim 12 , wherein the step of tracking the product is performed using tracking-by-detection and greedy search when the product is the product in hand, and performed using appearance based tracker when the product is the product on table.

16. The method of claim 12 , wherein the shoplifting risk score R is calculated by:

R=r 1 ×N 1 +r 2 ×N 2 ,

wherein N 1 is a number of the product that has the scanning status of unscanned and disappears from the table region, N 2 is a number of the product that has the scanning status of unscanned and disappears from the ROI, r 1 is a medium risk factor in a range of 0.5-1.0, and r 2 is a high risk factor in a range of 1.0-2.0.

17. The method of claim 16 , further comprising:

providing the shoplifting warning that shoplifting happened when R≥β 1 ×(N 3 ) α ; and

providing the shoplifting warning that shoplifting might happened when β 1 ×(N 3 ) α >R≥β 2 ×(N 3 ) α ,

wherein N 3 is a number of the product that has the scanning status of scanned, α is a predetermined number in a range of 0.2-1.0, β 1 is a predetermined number in a range of 0.5-1.0, and β 2 is a predetermined number in a range of 1.0-2.0.

18. The method of claim 12 , further comprising:

providing a scan difficulty message when the product is placed in the scanning region and has the scanning status of unscanned, and the computing device has not received a scanning signal of the product from the scanner for an accumulated time greater than a predetermined time.

19. The method of claim 12 , wherein the imaging device is a red green blue-depth (RGBD) camera, and the computing device is an embedded device.

20. A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of a computing device, is configured to:

initiate a self-checkout event;

instruct an imaging device to capture video frames of a region of interest (ROI), wherein the ROI comprises a table region for placing a product, and the table region comprises a scanning region for scanning the product;

track the product in the video frames;

record scanning status and location status of the product, wherein the scanning status comprises scanned and unscanned, and the location status comprises in the table region, and out of the table region but within the ROI;

in response to receiving a scanning signal from a scanner when the product is placed in the scanning region, record scanning status of the product as scanned;

calculate a shoplifting risk score based on a number of the product having the scanning status of unscanned and disappear from the table region or the ROI; and

provide a shoplifting warning when the shoplifting score is greater than a predetermined value.