System and method for vision-assisted checkout
This present disclosure proposes a system and method for providing a 3D computer vision-assisted frictionless self-checkout experience in a traditional retail store by using two RGB-D camera sensors and a conveyer belt. The disclosure provides a self-checkout counter ( 106 ) that enables a customer ( 102 ) to go through a self-service checkout process by simply placing the collected products one by one on a conveyer belt. One vertical and another horizontal RGB-D sensor ( 108 ) mounted in a housing frame attached towards the end of the conveyer belt capture RGB and depth image of each product passing through the housing and pass it to a product recognition engine ( 216 ). The engine ( 216 ) identifies the unique product along with its volumetric attributes processing the RGB-D data that is further compared with a master product database and processed for invoicing. The customer wallet and payment may be integrated with the customer phone number at the self-checkout counter ( 106 ) for providing a completely automated checkout experience.
1 . A system for providing a computer vision assisted self-service checkout, the system comprising:
a processor;
a memory coupled to the processor, wherein the memory comprises processor-executable instructions, which on execution, causes the processor to:
capture RGB and depth images of one or more products at a first timestamp using one or more Red Green Blue Depth (RGB-D) sensors mounted in an RGB-D housing frame, wherein the RGB-D housing frame comprises a rectangular framework walled at three sides and provided with diffused lighting forming an upside-down U-shaped tunnel, and wherein the rectangular framework is a 2×2 meter rectangular framework;
combine the captured RGB and depth images to create a unified payload for the one or more products at the first timestamp;
determine a count of the one or more products based on the unified payload of the one or more products;
generate a unique product ID of each of the one or more products;
identify a Stock Keeping Unit (SKU) of the one or more products based on the unique product ID of each of the one or more products; and
generate an invoice bill for the one or more products based on the identified SKU of the one or more products.
2 . The system as claimed in claim 1 , wherein the one or more RGB-D sensors comprise a phone camera or a hand-held camera mounted in the RGB-D housing frame.
3 . The system as claimed in claim 1 , wherein the RGB-D housing frame is configured proximate to a conveyor belt.
4 . The system as claimed in claim 1 , wherein the processor is configured to perform a volumetric analysis to distinguish the one or more products that are visually similar but volumetrically different.
5 . The system as claimed in claim 4 , wherein the volumetric analysis is performed using a two-stream Convolutional Neural Network (CNN) approach.
6 . The system as claimed in claim 1 , wherein the unique product ID comprises a brand logo, a net weight, a Maximum Retail Price (MRP), a vegetation logo, a Food Safety and Standards Authority of India (FSSAI) mark, a Food Process Order (FPO) certification mark, and a license number of the one or more products.
7 . The system as claimed in claim 1 , wherein the unique product ID is generated by aggregating a brand logo, a net weight, a Maximum Retail Price (MRP), a vegetation logo, a Food Safety and Standards Authority of India (FSSAI) mark, a Food Process Order (FPO) certification mark, and a license number of the one or more products by a decision heuristics approach.
8 . The system as claimed in claim 1 , wherein the SKU of the one or more products is identified using a product registration database.
9 . The system as claimed in claim 8 , wherein the product registration database comprises the unique product ID mapped with the SKU of the one or more products created during product registration.
10 . The system as claimed in claim 8 , wherein the SKU of the one or more products is identified by performing a reverse search in the product registration database.
11 . A method for providing a computer vision assisted self-service checkout, the method comprising:
capturing, by a processor, RGB and depth images of one or more products at a first timestamp using one or more Red Green Blue Depth (RGB-D) sensors mounted in an RGB-D housing frame, wherein the RGB-D housing frame comprises a rectangular framework walled at three sides and provided with diffused lighting forming an upside-down U-shaped tunnel, and wherein the rectangular framework is a 2×2 meter rectangular framework;
combining, by the processor, the captured RGB and depth images to create a unified payload for the one or more products at the first timestamp;
determining, by the processor, a count of the one or more products based on the unified payload of the one or more products;
generating, by the processor, a unique product ID of each of the one or more products;
identifying, by the processor, a Stock Keeping Unit (SKU) of the one or more products based on the unique product ID of each of the one or more products; and
generating, by the processor, an invoice bill for the one or more products based on the identified SKU of the one or more products.
12 . The method as claimed in claim 11 , wherein the one or more RGB-D sensors comprise a phone camera or a hand-held camera mounted in RGB-D housing frame.
13 . The method as claimed in claim 11 , wherein the RGB-D housing frame is configured proximate to a conveyor belt.
14 . The method as claimed in claim 11 , wherein the method comprises performing a volumetric analysis to distinguish the one or more products that are visually similar but volumetrically different.
15 . The method as claimed in claim 14 , wherein the volumetric analysis is performed using a two-stream Convolutional Neural Network (CNN) approach.
16 . The method as claimed in claim 11 , wherein the unique product ID comprises a brand logo, a net weight, a Maximum Retail Price (MRP), a vegetation logo, a Food Safety and Standards Authority of India (FSSAI) mark, a Food Process Order (FPO) certification mark, and a license number of the one or more products.
17 . The method as claimed in claim 11 , wherein the unique product ID is generated by aggregating a brand logo, a net weight, a Maximum Retail Price (MRP), a vegetation logo, a Food Safety and Standards Authority of India (FSSAI) mark, a Food Process Order (FPO) certification mark, and a license number of the one or more products by a decision heuristics approach.
18 . The method as claimed in claim 11 , wherein the SKU of the one or more products is identified using a product registration database.
19 . The method as claimed in claim 18 , wherein the product registration database comprises the unique product ID mapped with the SKU of the one or more products created during product registration.
20 . The method as claimed in claim 18 , wherein the SKU of the one or more products is identified by performing a reverse search in the product registration database.
21 . A user equipment (UE) for providing a computer vision assisted self-service checkout, the UE comprising:
a processor; and
a memory coupled to the processor, wherein the memory comprises processor-executable instructions, which on execution, causes the processor to:
capture RGB and depth images of one or more products at a first timestamp using one or more Red Green Blue Depth (RGB-D) sensors mounted in an RGB-D housing frame, wherein the RGB-D housing frame comprises a rectangular framework walled at three sides and provided with diffused lighting forming an upside-down U-shaped tunnel, and wherein the rectangular framework is a 2×2 meter rectangular framework;
send the captured RGB and depth images to a system,
wherein the system is configured to;
combine the captured RGB and depth images to create a unified payload for the one or more products at the first timestamp;
determine a count of the one or more products based on the unified payload of the one or more products;
generate a unique product ID of each of the one or more products;
identify a Stock Keeping Unit (SKU) of the one or more products based on the unique product ID of each of the one or more products;
generate an invoice bill for the one or more products based on the identified SKU of the one or more products; and
display the invoice bill for the one or more products based on the identified SKU for the self-service checkout.
22 . The UE as claimed in claim 21 , wherein the one or more RGB-D sensors comprise a phone camera or a hand-held camera mounted in the RGB-D housing frame.