Systems and methods for identifying exceptions in feature detection analytics
Systems and methods for identifying exceptions in feature detection analytics include systems and methods configured to identify exceptions in product displays. In one exemplary embodiment, price tag mismatches are reported to a customer if certain criteria are met based on analytics from optical character recognition, image object detection, and reference catalogs/planograms. These price tag mismatches, for example, provide actionable insights for humans working alongside robots which can be quickly identified and resolved.
1 . A system configured to identify exceptions in product displays, comprising:
a robot comprising at least one sensor configured to take images of objects as the robot travels in an environment, each image acquired by the robot is localized by a controller thereon;
at least one user device; and
a server in communication with the robot and at least one user device, the server comprises at least one processor configured to execute computer readable instructions to:
receive the images of objects and associated localization data for the images from the robot, the images depict at least one feature to be identified, the images corresponded to a location of the robot measured by the robot during acquisition of the images;
generate a first data set comprising image object detection predictions for each of the at least one features to be identified within the image based on visual characteristics of the at least one features using a first detection model, the image object detection predictions include at least a product identification and location for each of the at least one features;
generate a second data set comprising optical character recognition predictions for each of the at least one features to be identified within the image based on text depicted on price labels within the image using a second detection model, the predictions include at least a price value and a product identification for each of the at least one features;
receive a third data set corresponding to a catalog, the catalog indicates an expected arrangement, location, and price values of features within the environment;
identify at least one exception, the at least one exception being any one or more of: (i) a discrepancy between the product identifications in the first and second data sets, (ii) a discrepancy between the product identifications in the first and second data sets and products in the catalog; (iii) a discrepancy between the price values in the second data set with the price values in the catalog;
generate a report comprising a list of identified features and the at least one exception; and
communicate the report to a user device.
2 . The system of claim 1 , wherein the at least one processor of the server is further configured to execute the computer readable instructions to:
identify exclusions if at least one of following fields corresponding to each of the at least one features is missing or invalid: (i) site location; (ii) robot location; (iii) annotation information associated with the object being scanned; (iv) bin information; (v) UPC, SKU, or GTIN values for the product identifications; or (vi) description of an item.
3 . The system of claim 1 , wherein the at least one processor of the server is further configured to execute the computer readable instructions to:
receive annotations to a computer readable map of an environment of the robot, the annotations define on the computer readable map are encompassed by objects to be scanned for features, each area includes at least one face on its perimeter, each face is assigned a face identifier (“ID”) value;
receive configurations for each face ID of each of the objects to be scanned, wherein the configurations denote semantic information, functional information, and exception information associated with the face ID;
wherein the annotations are received via a user interface coupled to the device, server, or robot.
4 . The system of claim 3 , wherein
the exception information contains a reserve storage field; and
an exception is generated for an identified feature if the feature is a reserve storage item which is detected where the exception information indicates reserve storage should not be present, or vice versa, based on either the first or second data sets.
5 . The system of claim 3 , wherein
the configurations includes a department information field; and
the at least one processor of the server produces an exception for a detected feature if the detected feature cannot be stored or displayed in the department, the departments in which certain features can or cannot be present are denoted by a catalog provided to the server via the annotations.
6 . The system of claim 1 , wherein the at least one processor of the server is further configured to execute the computer readable instructions to:
provide at least one image captured by the robot to the device upon the device requesting to view one or more noted exceptions in more detail via a user interface of the device.
7 . The system of claim 1 , wherein the at least one processor of the server is further configured to execute the computer readable instructions to:
compare a location of the at least one feature to a reference planogram, the reference planogram being retrieved based in part on the location of the robot during acquisition of the images; and
generate an exception for any of the at least one features comprise locations different from their denoted location in the reference planogram.
8 . A method for identifying exceptions in product displays, comprising:
a server comprising at least one processor configured to execute computer readable instructions to perform steps of:
receiving an image of objects and associated localization data for the image from a robot comprising at least one sensor configured to take images of objects as the robot travels in an environment and a controller configured to localize the image as it is taken, wherein the image depicts at least one feature to be identified, the images corresponded to a location of the robot measured by the robot during acquisition of the images;
generating a first data set comprising image object detection predictions for each of the at least one features to be identified within the image based on visual characteristics of the at least one features using a first detection model, the image object detection predictions include at least a product identification and location for each of the at least one features;
generating a second data set comprising optical character recognition predictions for each of the at least one features to be identified within the image based on text depicted on price labels within the image using a second detection model, the predictions include at least a price value and a product identification for each of the at least one features;
receiving a third data set corresponding to a catalog, the catalog indicates an expected arrangement, location, and price values of features within the environment;
identifying at least one exception being any one or more of: (i) a discrepancy between the product identifications in the first and second data sets, (ii) a discrepancy between the product identifications in the first and second data sets with products in the catalog; (iii) a discrepancy between the price values in the second data set with the price values in the catalog;
generating a report comprising a list of identified features and the at least one exception; and
communicating the report to a user device.
9 . The method of claim 8 , further comprising the at least one processor to execute the computer readable instructions to perform the step of,
identifying exclusions if at least one of following fields corresponding to each of the at least one features is missing or invalid: (i) site location; (ii) robot location; (iii) annotation information associated with the object being scanned; (iv) bin information; (v) UPC, SKU, or GTIN values for the product identifications; or (vi) description of an item.
10 . The method of claim 8 , further comprising the at least one processor to perform the steps of,
receiving annotations to a computer readable map of an environment of the robot, the annotations defining on the computer readable map are encompassed by objects to be scanned for features, each area includes at least one face on its perimeter, and each face is assigned a face identifier (“ID”) value; and
receiving face configurations for each face ID of each of the objects to be scanned, the configurations denoting semantic information, functional information, and exception information associated with the face ID; wherein the annotations are received via a user interface coupled to the device, server, or robot.
11 . The method of claim 10 , wherein,
the exception information contains a reserve storage field; and
the method comprises generating an exception for an identified feature if the feature is a reserve storage item which is detected where the exception information indicates reserve storage should not be present, or vice versa based on either the first or second data sets.
12 . The method of claim 10 , wherein,
the configurations includes a department information field; and
the method further comprises producing an exception for a detected feature if the detected feature cannot be stored or displayed in the department, the departments in which certain features can or cannot be present are denoted by a catalog provided to the server via the annotations.
13 . The method of claim 8 , further comprising the at least one processor of the server providing at least one image captured by the robot to the device upon the device requesting to view one or more noted exceptions in more detail via a user interface of the device.
14 . The method of claim 8 , further comprising the at least one processor of the server comparing a location of the at least one feature to a reference planogram, the reference planogram being retrieved based in part on the location of the robot during acquisition of the images; and
generating an exception for any of the at least one features comprise locations different from their denoted location in the reference planogram.
15 . A non-transitory computer readable storage medium having a plurality of computer readable instructions stored thereon which, when executed by at least one processor in communication with a robot, configure the at least one processor to identify exceptions in product displays by executing instructions to,
receive at least one image and associated localization data from the robot, wherein the robot comprises at least one sensor configured to take images of objects as the robot travels in an environment, wherein each image acquired by the robot is localized by a controller thereon and each image depicts at least one feature to be identified and corresponds to a location of the robot measured by the robot during acquisition of the image;
generate a first data set comprising image object detection predictions for each of the at least one features to be identified within the image based on visual characteristics of the at least one features using a first detection model, the image object detection predictions include at least a product identification and location for each of the at least one features;
generate a second data set comprising optical character recognition predictions for each of the at least one features to be identified within the image based on text depicted on price labels within the image using a second detection model, the predictions include at least a price value and a product identification for each of the at least one features;
receive a third data set corresponding to a catalog, the catalog indicates an expected arrangement, location, and price values of features within the environment;
identify at least one exception being any one or more of: (i) a discrepancy between the product identifications in the first and second data sets, (ii) a discrepancy between the product identifications in the first and second data sets with products in the catalog; (iii) a discrepancy between the price values in the second data set with the price values in the catalog;
generate a report comprising a list of identified features and the at least one exception; and
communicate the report to a user device.
16 . The non-transitory computer readable storage medium of claim 15 , wherein the at least one processor is further configured to execute the computer readable instructions to:
identifying exclusions if at least one of following fields corresponding to each of the at least one features is missing or invalid: (i) a robot location; (ii) annotation information associated with the object being scanned; (iii) bin information; (iv) UPC, SKU, or GTIN values for ethe product identifications; (v) site location; or (vi) description of an item.
17 . The non-transitory computer readable storage medium of claim 15 , wherein the at least one processor is further configured to execute the computer readable instructions to:
receive annotations to a computer readable map of an environment of the robot, the annotations defined on the computer readable map are encompassed by objects to be scanned for features, each area includes at least one face on its perimeter, each face is assigned a face identifier (“ID”) value;
receive face configurations for each face ID of each of the objects to be scanned, wherein the configurations denote semantic information, functional information, and exception information associated with the face ID; wherein the annotations are received via a user interface in communication with the device, server, or robot.
18 . The non-transitory computer readable storage medium of claim 17 ,
wherein the exception information contains a reserve storage field; and
the at least one processor is further configured to execute the computer readable instructions to generate an exception for an identified feature if the feature is a reserve storage item which is detected where the exception information indicates reserve storage should not be present, or vice versa based on either the first or second data sets.
19 . The non-transitory computer readable storage medium of claim 17 ,
wherein the configurations includes a department information field; and
the at least one processor of the server produces an exception for a detected feature if the detected feature cannot be stored or displayed in the department, the departments in which certain features can or cannot be present are denoted by a catalog provided to the server via the annotations.
20 . The non-transitory computer readable storage medium of claim 15 , wherein the at least one processor is further configured to execute the computer readable instructions to:
provide at least one image captured by the robot to the device upon the device requesting to view one or more noted exceptions in more detail via a user interface of the device.
21 . The non-transitory computer readable storage medium of claim 15 , wherein the at least one processor is further configured to execute the computer readable instructions to:
compare a location of the at least one feature to a reference planogram, the reference planogram being retrieved based in part on the location of the robot during acquisition of the images; and
generate an exception for any of the at least one features comprise locations different from their denoted location in the reference planogram.
22 . The non-transitory computer readable storage medium of claim 15 , wherein the at least one processor is further configured to execute the computer readable instructions to:
receive a first feedback signal from the device, the feedback signal indicates one or more exceptions of the at least one exception is valid or invalid;
remove the one or more exceptions which are invalid from the report;
receive a second feedback signal from the device, the second feedback signal indicates if the one or more valid exceptions has been resolved;
remove the valid exceptions which have been resolved from the report.
23 . The non-transitory computer readable storage medium of claim 15 , wherein,
the identified exceptions are based on exception criteria provided to the at least one processor, the exception criteria including a list of exceptions to include in the report.