Edge-based machine learning pipeline for determining and assessing in-store conditions

1 . A method for edge-based assessment of conditions associated with a location, the method comprising:

executing, by an edge computing device, instructions to run an image processing pipeline at the edge computing device;

capturing, by the edge computing device, one or more images of products and fixtures associated with the location; and

for each image of the one or more images:

determining, by the edge computing device and based on processing the image, conditions associated with the location;

determining, by the edge computing device, whether the conditions associated with the location satisfy one or more KPI analyses or predetermined objectives for the location;

generating, by the edge computing device, one or more recommendations for improving the conditions associated with the location based on a determination that the conditions associated with the location do not satisfy the one or more KPI analyses or predetermined objectives for the location; and

returning, in one or more graphical user interface (GUI) displays at the edge computing device, at least the one or more recommendations for improving the conditions associated with the location;

applying, by the edge computing device, a product recognition model of a plurality of compressed machine learning models to the image;

receiving, at the edge computing device, user input indicating a user identification of at least one product amongst the products; and

transmitting, by the edge computing device to a backend computer system, at least the user input and a model output,

wherein the backend computer system is configured to perform a retraining process of the product recognition model, the retraining process comprising:

performing a quality assessment of the model output based on the user input, one or more model accuracy criteria and a data drift detection technique;

determining whether the quality assessment satisfies one or more quality criteria:

determining, based on a determination that the quality assessment does not satisfy the one or more quality criteria, whether sufficient image samples are available for retraining the product recognition model on the at least one product for which accuracy of the model output was below a predetermined threshold level;

invoking, based on a determination that insufficient image samples are available, a synthetic data module to generate synthetic image samples of the at least one product; and

retraining the product recognition model based on at least the synthetic image samples of the at least one product until the accuracy of the model output meets or exceeds the predetermined threshold level.

2 . The method of claim 1 , wherein executing, by the edge computing device, the instructions to run the image processing pipeline at the edge computing device comprises running the image processing pipeline in an existing mobile application that is launched at the edge computing device.

3 . The method of claim 1 , wherein executing, by the edge computing device, the instructions to run the image processing pipeline at the edge computing device comprises running the image processing pipeline in a mobile application that is provided by the backend computer system.

4 . The method of claim 1 , wherein executing, by the edge computing device, the instructions to run the image processing pipeline at the edge computing device comprises deep linking an existing mobile application at the edge computing device to the image processing pipeline.

5 . The method of claim 1 , further comprising, for each image of the one or more images:

receiving, by the edge computing device, the model output based on applying the product recognition model, wherein the model output comprises indications of the products that were recognized in the image by the product recognition model.

6 . The method of claim 5 , further comprising, for each image of the one or more images:

applying, by the edge computing device, at least a portion of the plurality of compressed machine learning models to the model output; and

determining, by the edge computing device and based on applying the at least a portion of the plurality of compressed machine learning models to the model output, the conditions associated with the location.

7 . The method of claim 6 , wherein:

applying, by the edge computing devices, at least a portion of the plurality of compressed machine learning models to the model output comprises applying a product measurements model to the model output to identify measurements of the products relative to measurements of the fixtures associated with the location, and

determining, by the edge computing device and based on applying the at least portion of the plurality of compressed machine learning models to the model output, conditions associated the location comprises identifying placement of the products associated with the location based on the identified measurements of the products relative to the identified measurements of the fixtures associated with the location.

8 . The method of claim 6 , wherein the at least portion of the plurality of compressed machine learning models are applied, by the edge computing device, in parallel.

9 . The method of claim 6 , wherein the at least portion of the plurality of compressed machine learning models are applied, by the edge computing device, in series.

10 . The method of claim 6 , wherein at least one of the capturing, the applying, the receiving, the applying, the determining, the determining, the generating, or the returning operations is performed on the edge when the edge computing device is disconnected from a network.

11 . The method of claim 5 , wherein the product recognition model was trained in a process that comprises:

receiving, by the backend computer system, at least images of the products associated with the location;

generating, by the backend computer system, a bounding box around one or more products in the images using object detection techniques;

annotating, by the backend computer system and based on user input from the edge computing device, the one or more products in the respective bounding box of the images to identify the one or more products;

providing, by the backend computer system, the annotated images as training data to the product recognition model;

training, by the backend computer system, the product recognition model based on the training data to identify the one or more products that were annotated by the backend computer system in other images;

assessing, by the backend computer system, accuracy of the trained product recognition model;

identifying, by the backend computer system and based on determining that the accuracy of the trained product recognition model exceeds a predetermined threshold level of accuracy, edge device deployment information;

compressing, by the backend computer system, the trained product recognition model for edge deployment based on the identified information; and

returning, by the backend computer system, the compressed product recognition model, wherein returning the compressed product recognition model comprises transmitting the compressed product recognition model with instructions to one or more network-connected edge computing devices to execute the compressed product recognition model on the edge.

12 . The method of claim 5 , wherein the product recognition model was trained in a process that comprises:

receiving, by the backend computer system, at least images of a product in one or more dimensions;

identifying, by the backend computer system, packaging artwork of the product in a first dimension using object detection techniques;

annotating, by the backend computer system, the packaging artwork in the first dimension in the images;

reconstructing, by the backend computer system, the packaging artwork in N dimensions based on applying generative artificial intelligence (AI) techniques to the annotated images of the packaging artwork in the first dimension;

providing, by the backend computer system, the reconstructed packaging artwork in the N dimensions as training data to the product recognition model;

training, by the backend computer system, the product recognition model to identify products in the N dimensions in other images; and

returning, by the backend computer system, the trained product recognition model.

13 . The method of claim 1 , wherein determining, by the edge computing device, whether the conditions associated with the location satisfy one or more KPI analyses or predetermined objectives for the a physical space comprises determining whether the products are arranged in the fixtures according to a planogram for the location.

14 . The method of claim 1 , wherein generating, by the edge computing device one or more recommendations for improving the conditions associated with a physical space comprises: generating instructions to add or remove a predetermined quantity of at least one product associated with the location.

15 . The method of claim 1 , where generating, by the edge computing device one or more recommendations for improving the conditions associated with the location comprises: generating instructions to move a predetermined quantity of at least one product associated with a physical space from a current fixture to another fixture associated with the physical space.

16 . The method of claim 1 , wherein the fixtures associated with the location comprise racks or shelves.

17 . The method of claim 1 , wherein the products comprise a food product.

18 . The method of claim 1 , wherein the fixtures comprise at least one of shelves, racks, freezers, or coolers.

19 . A system for edge-based assessment of conditions associated with a location, the system comprising:

a backend computer system configured to (i) train machine learning models, and (ii) provide an image processing pipeline for edge deployment, and perform a retraining process of a product recognition model; and

an edge computing device configured to:

receive, over a network connection with the backend computer system, the machine learning models and instructions to execute the image processing pipeline at the edge computing device;

execute the instructions to run the image processing pipeline at the edge computing device;

capture one or more images of products and fixtures associated with the location; and

for each image of the one or more images:

identify, based on applying at least a portion of the machine learning models to the image, conditions associated with the location in the image;

generate output based on the identified conditions associated with the location; and

return, the output for presentation in one or more graphical user interface (GUI) displays at the edge computing device;

apply the product recognition model of a plurality of compressed machine learning models to the image;

receive user input indicating a user identification of at least one product amongst the products; and

transmit to the backend computer system, at least the user input and a model output, wherein the backend computer system performing the retraining process comprises:

performing a quality assessment of the model output based on the user input, one or more model accuracy criteria and a data drift detection technique;

determining whether the quality assessment satisfies one or more quality criteria;

determining, based on a determination that the quality assessment does not satisfy the one or more quality criteria, whether sufficient image samples are available for retraining the product recognition model on the at least one product for which accuracy of the model output was below a predetermined threshold level;

invoking, based on a determination that insufficient image samples are available, a synthetic data module to generate synthetic image samples of the at least one product; and

retraining the product recognition model based on at least the synthetic image samples of the at least one product until the accuracy of the model output meets or exceeds the predetermined threshold level.

20 . The system of claim 19 , wherein, for each image of the one or more images, the edge computing device is further configured to:

apply the product recognition model of the machine learning models to the image;

receive the model output based on applying the product recognition model, wherein the model output comprises indications of the products that were recognized in the image by the product recognition model;

apply at least a portion of the machine learning models to the model output; and

identify, based on applying at least the portion of the machine learning models to the model output, the conditions associated with the location in the image.

21 . The system of claim 20 , wherein the backend computer system is configured to train the product recognition model using a process that comprises:

receiving images of the products associated with the location, wherein the products comprise at least one of food products or beverages;

generating a bounding box around one or more products in the images using object detection techniques;

annotating the one or more products in the respective bounding box of the images to identify the one or more products;

providing the annotated images as training data to the product recognition model;

training the product recognition model based on the training data to identify the one or more products that were annotated in other images;

compressing the trained product recognition model for edge deployment; and

returning the compressed product recognition model, wherein returning the compressed product recognition model comprises transmitting the compressed product recognition model with instructions to the edge computing device when the edge computing device is connected to a network via the network connect.

22 . The system of claim 19 , wherein:

applying at least the portion of the machine learning models to the image comprises applying a product measurements model to the image to identify measurements of products represented in the image relative to measurements of the fixtures associated with the location, and

wherein identifying, based on applying at least the portion of the machine learning models to the image, the conditions associated the location comprises identifying placement of the products associated with the location based on the identified measurements of the products relative to the identified measurements of the fixtures associated with the location.

23 . The system of claim 19 , wherein the edge computing device is further configured to: determine whether the conditions associated with the location satisfy one or more KPI analyses or predetermined objectives for the location.

24 . The system of claim 23 , wherein the edge computing device is further configured to:

based on a determination that the conditions associated with the location do not satisfy the one or more KPI analyses or predetermined objectives for the location, generate one or more recommendations for improving the conditions associated with the location; and

return the one or more recommendations for presentation in the GUI displays at the edge computing device.

25 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations for edge-based assessment of conditions associated with a location comprising:

executing instructions to run an image processing pipeline;

capturing one or more images of products and fixtures associated with the location; and

for each image of the one or more images:

determining, based on processing the image, conditions associated with the location;

determining whether the conditions associated with the location satisfy one or more KPI analyses or predetermined objectives for the location;

generating one or more recommendations for improving the conditions associated with the location based on a determination that the conditions associated with the location do not satisfy the one or more KPI analyses or predetermined objectives for the location;

returning, in one or more graphical user interface (GUI) displays, at least the one or more recommendations for improving the conditions associated with the location;

applying a product recognition model of a plurality of compressed machine learning models to the image;

receiving user input indicating a user identification of at least one product amongst the products; and

performing a retraining process of the product recognition model, the retraining process comprising:

performing a quality assessment of a model output based on the user input, one or more model accuracy criteria and a data drift detection technique;

determining whether the quality assessment satisfies one or more quality criteria;

determining, based on a determination that the quality assessment does not satisfy the one or more quality criteria, whether sufficient image samples are available for retraining the product recognition model on the at least one product for which accuracy of the model output was below a predetermined threshold level;

invoking, based on a determination that insufficient image samples are available, a synthetic data module to generate synthetic image samples of the at least one product; and

retraining the product recognition model based on at least the synthetic image samples of the at least one product until the accuracy of the model output meets or exceeds the predetermined threshold level.