Item location tracking for display racks using digital image processing

1 . An image processing device, comprising:

a memory operable to store a master template that is associated with a rack, wherein:

the master template identifies a plurality of rack positions that each identify a shelf of the rack and a position on the shelf; and

each rack position is associated with an item; and

a processor operably coupled to the memory, and configured to:

receive a rack identifier that identifies a rack configured to hold a plurality of items;

identify the master template that is associated with the rack based on the rack identifier;

receive a set of images of the plurality of items on the rack;

combine the set of images into a composite image;

generate a plurality of bounding boxes, wherein each bounding box corresponds with an item on the rack in the composite image;

execute a machine-learning algorithm associated with a machine-learning model to associate each bounding box from the plurality of bounding boxes with an item identifier;

identify a plurality of shelves on the rack within the composite image by:

identifying reference markers located on each shelf of the rack from the composite image; and

identifying a range of pixels in the composite image that corresponds to each shelf of the rack based on the reference markers, wherein a plurality of pixels of the composite image that are positioned between two reference markers is identified as the range of pixels corresponding to a particular shelf of the rack;

associate each bounding box from the plurality of bounding boxes with an item location based at least on the range of pixels identified for each shelf of the rack, wherein each item location identifies a shelf from among the plurality of shelves on the rack and a position on the shelf, wherein the associating comprises:

for each bounding box:

comparing pixel positions of those pixels in the composite image that are included in the bounding box with the range of pixels associated with each shelf;

determining based on the comparing that at least a portion of the pixels included in the bounding box are within the range of pixels associated with a particular shelf;

determining that a particular item identified within the bounding box is placed on the particular shelf;

for each shelf of the rack:

generate a cluster of bounding boxes, wherein the cluster of bounding boxes corresponding to that shelf comprises all bounding boxes having at least a portion of pixels that are within the range of pixels associated with that shelf, wherein to generate the cluster of bounding boxes, the processor is configured to:

set a pixel column value in pixel locations, corresponding to items on the shelf, to a common value; and

generate a plot of the pixel locations, wherein pixel row values in the pixel locations are grouped to form the cluster;

sort the bounding boxes from the cluster of bounding boxes based on column values of pixels associated with the bounding boxes, wherein a bounding box associated with lower column values of pixels is determined to be placed nearer to one end of the shelf as compared to another bounding box associated with higher column values of pixels; and

compare item locations for each bounding box to the plurality of rack positions from the master template;

generate a rack analysis message based on the comparison of the item locations for each bounding box and the plurality of rack positions from the master template, wherein the rack analysis message indicates whether the plurality of items are in correct locations on the rack; and

output the rack analysis message.

2 . The device of claim 1 , wherein identifying the plurality of shelves on the rack within the composite image comprises:

determining a pixel location in the composite image for each item from among the plurality of items within the composite image, wherein the pixel location identifies a pixel row value and a pixel column value within the composite image;

identifying a plurality of clusters based on pixel rows values for the plurality of items; and

associating each cluster from the plurality of clusters with a shelf.

3 . The device of claim 1 , wherein generating the plurality of bounding boxes comprises:

determining an overlap percentage between a first bounding box from the plurality of bounding boxes and a second bounding box from the plurality of bounding boxes;

determining the overlap percentage is greater than a predetermined threshold value; and

removing one of the first bounding box or the second bounding box in response to determining that the overlap percentage is greater than the predetermined threshold value.

4 . The device of claim 1 , wherein associating each bounding box from the plurality of bounding boxes with an item identifier comprises:

extracting a portion of the composite image within a first bounding box from among the plurality of bounding boxes;

inputting the portion of the composite image into a machine learning model that is configured to output an item identifier based on features of a first item that are present in the portion of the composite image;

receiving a first item identifier for the first item in response to inputting the portion of the composite image into the machine learning model; and

associating the first bounding box with the first item identifier for the first item.

5 . The device of claim 1 , wherein associating each bounding box from the plurality of bounding boxes with an item identifier comprises:

extracting a portion of the composite image within a first bounding box from among the plurality of bounding boxes;

comparing the portion of the composite image to a plurality of images of items;

identifying a first image from among the plurality of images of items that at least partially matches the portion of the composite image;

identifying a first item identifier that corresponds with the first image; and

associating the first bounding box with the first item identifier for the first item.

6 . The device of claim 1 , wherein generating the rack analysis message comprises:

identifying a mismatch between a first item location and a first rack position from the master template;

identifying a first item associated with the first rack position from the master template; and

generating the rack analysis message that identifies the first item and the first rack position from the master template.

7 . The device of claim 1 , wherein comparing the item locations for each bounding box to the plurality of rack positions from the master template comprises:

selecting a first shelf from among the plurality of shelves;

identifying a set of item identifiers that are associated with the first shelf;

assigning an alphanumeric character to each item identifier from among the set of item identifiers;

combining the alphanumeric characters to form a first word for the first shelf;

identifying a second word from the master template that corresponds with the first shelf; and

comparing the first word to the second word.

8 . The device of claim 1 , wherein generating the rack analysis message comprises:

identifying a mismatch between a first item location and a first rack position from the master template; and

generating the rack analysis message that comprises a recommendation for resolving the mismatch.

9 . An item location tracking method, comprising:

receiving a rack identifier that identifies a rack configured to hold a plurality of items;

identifying a master template that is associated with the rack based on the rack identifier, wherein:

the master template identifies a plurality of rack positions that each identify a shelf of the rack and a position on the shelf; and

each rack position is associated with an item;

receiving a set of images of the plurality of items on the rack;

combining the set of images into a composite image;

generating a plurality of bounding boxes, wherein each bounding box corresponds with an item on the rack in the composite image;

executing a machine-learning algorithm associated with a machine-learning model to associate each bounding box from the plurality of bounding boxes with an item identifier;

identifying a plurality of shelves on the rack within the composite image by:

identifying reference markers located on each shelf of the rack from the composite image; and

identifying a range of pixels in the composite image that corresponds to each shelf of the rack based on the reference markers, wherein a plurality of pixels of the composite image that are positioned between two reference markers is identified as the range of pixels corresponding to a particular shelf of the rack;

associating each bounding box from the plurality of bounding boxes with an item location based at least on the range of pixels identified for each shelf of the rack, wherein each item location identifies a shelf from among the plurality of shelves on the rack and a position on the shelf, wherein the associating comprises:

for each bounding box:

comparing pixel positions of those pixels in the composite image that are included in the bounding box with the range of pixels associated with each shelf;

determining based on the comparing that at least a portion of the pixels included in the bounding box are within the range of pixels associated with a particular shelf;

determining that a particular item identified within the bounding box is placed on the particular shelf;

for each shelf of the rack:

generating a cluster of bounding boxes, wherein the cluster of bounding boxes corresponding to that shelf comprises all bounding boxes having at least a portion of pixels that are within the range of pixels associated with that shelf, wherein to generate the cluster of bounding boxes, the processor is configured to:

set a pixel column value in pixel locations, corresponding to items on the shelf, to a common value; and

generate a plot of the pixel locations, wherein pixel row values in the pixel locations are grouped to form the cluster;

sorting the bounding boxes from the cluster of bounding boxes based on column values of pixels associated with the bounding boxes, wherein a bounding box associated with lower column values of pixels is determined to be placed nearer to one end of the shelf as compared to another bounding box associated with higher column values of pixels;

comparing item locations for each bounding box to the plurality of rack positions from the master template;

generating a rack analysis message based on the comparison of the item locations for each bounding box and the plurality of rack positions from the master template, wherein the rack analysis message indicates whether the plurality of items are in correct locations on the rack; and

outputting the rack analysis message.

10 . The method of claim 9 , wherein identifying the plurality of shelves on the rack within the composite image comprises:

determining a pixel location in the composite image for each item from among the plurality of items within the composite image, wherein the pixel location identifies a pixel row value and a pixel column value within the composite image;

identifying a plurality of clusters based on pixel rows values for the plurality of items; and

associating each cluster from the plurality of clusters with a shelf.

11 . The method of claim 9 , wherein generating the plurality of bounding boxes comprises:

determining an overlap percentage between a first bounding box from the plurality of bounding boxes and a second bounding box from the plurality of bounding boxes;

determining the overlap percentage is greater than a predetermined threshold value; and

removing one of the first bounding box or the second bounding box in response to determining that the overlap percentage is greater than the predetermined threshold value.

12 . The method of claim 9 , wherein associating each bounding box from the plurality of bounding boxes with an item identifier comprises:

extracting a portion of the composite image within a first bounding box from among the plurality of bounding boxes;

inputting the portion of the composite image into a machine learning model that is configured to output an item identifier based on features of a first item that are present in the portion of the composite image;

receiving a first item identifier for the first item in response to inputting the portion of the composite image into the machine learning model; and

associating the first bounding box with the first item identifier for the first item.

13 . The method of claim 9 , wherein associating each bounding box from the plurality of bounding boxes with an item identifier comprises:

extracting a portion of the composite image within a first bounding box from among the plurality of bounding boxes;

comparing the portion of the composite image to a plurality of images of items;

identifying a first image from among the plurality of images of items that at least partially matches the portion of the composite image;

identifying a first item identifier that corresponds with the first image; and

associating the first bounding box with the first item identifier for the first item.

14 . The method of claim 9 , wherein generating the rack analysis message comprises:

identifying a mismatch between a first item location and a first rack position from the master template;

identifying a first item associated with the first rack position from the master template; and

generating the rack analysis message that identifies the first item and the first rack position from the master template.

15 . The method of claim 9 , wherein comparing the item locations for each bounding box to the plurality of rack positions from the master template comprises:

selecting a first shelf from among the plurality of shelves;

identifying a set of item identifiers that are associated with the first shelf;

assigning an alphanumeric character to each item identifier from among the set of item identifiers;

combining the alphanumeric characters to form a first word for the first shelf;

identifying a second word from the master template that corresponds with the first shelf; and

comparing the first word to the second word.

16 . The method of claim 9 , wherein generating the rack analysis message comprises:

identifying a mismatch between a first item location and a first rack position from the master template; and

generating the rack analysis message that comprises a recommendation for resolving the mismatch.

17 . A non-transitory computer-readable medium storing instructions that when executed by a processor cause the processor to:

receive a rack identifier that identifies a rack configured to hold a plurality of items;

identify a master template that is associated with the rack based on the rack identifier, wherein:

the master template identifies a plurality of rack positions that each identify a shelf of the rack and a position on the shelf; and

each rack position is associated with an item;

receive a set of images of the plurality of items on the rack;

combine the set of images into a composite image;

generate a plurality of bounding boxes, wherein each bounding box corresponds with an item on the rack in the composite image;

execute a machine-learning algorithm associated with a machine-learning model to associate each bounding box from the plurality of bounding boxes with an item identifier;

identify a plurality of shelves on the rack within the composite image by:

identifying reference markers located on each shelf of the rack from the composite image; and

identifying a range of pixels in the composite image that corresponds to each shelf of the rack based on the reference markers, wherein a plurality of pixels of the composite image that are positioned between two reference markers is identified as the range of pixels corresponding to a particular shelf of the rack;

associate each bounding box from the plurality of bounding boxes with an item location based at least on the range of pixels identified for each shelf of the rack, wherein each item location identifies a shelf from among the plurality of shelves on the rack and a position on the shelf, wherein the associating comprises:

for each bounding box:

comparing pixel positions of those pixels in the composite image that are included in the bounding box with the range of pixels associated with each shelf;

determining based on the comparing that at least a portion of the pixels included in the bounding box are within the range of pixels associated with a particular shelf;

determining that a particular item identified within the bounding box is placed on the particular shelf;

for each shelf of the rack:

generating a cluster of bounding boxes, wherein the cluster of bounding boxes corresponding to that shelf comprises all bounding boxes having at least a portion of pixels that are within the range of pixels associated with that shelf, wherein to generate the cluster of bounding boxes, the processor is configured to:

set a pixel column value in pixel locations, corresponding to items on the shelf, to a common value; and

generate a plot of the pixel locations, wherein pixel row values in the pixel locations are grouped to form the cluster;

sorting the bounding boxes from the cluster of bounding boxes based on column values of pixels associated with the bounding boxes, wherein a bounding box associated with lower column values of pixels is determined to be placed nearer to one end of the shelf as compared to another bounding box associated with higher column values of pixels;

compare item locations for each bounding box to the plurality of rack positions from the master template;

generate a rack analysis message based on the comparison of the item locations for each bounding box and the plurality of rack positions from the master template, wherein the rack analysis message indicates whether the plurality of items are in correct locations on the rack; and

output the rack analysis message.

18 . The non-transitory computer-readable medium of claim 17 , wherein identifying the plurality of shelves on the rack within the composite image comprises:

determining a pixel location in the composite image for each item from among the plurality of items within the composite image, wherein the pixel location identifies a pixel row value and a pixel column value within the composite image;

identifying a plurality of clusters based on pixel rows values for the plurality of items; and

associating each cluster from the plurality of clusters with a shelf.

19 . The non-transitory computer-readable medium of claim 17 , wherein generating the plurality of bounding boxes comprises:

determining an overlap percentage between a first bounding box from the plurality of bounding boxes and a second bounding box from the plurality of bounding boxes;

determining the overlap percentage is greater than a predetermined threshold value; and

removing one of the first bounding box or the second bounding box in response to determining that the overlap percentage is greater than the predetermined threshold value.

20 . The non-transitory computer-readable medium of claim 17 , wherein associating each bounding box from the plurality of bounding boxes with an item identifier comprises:

extracting a portion of the composite image within a first bounding box from among the plurality of bounding boxes;

inputting the portion of the composite image into a machine learning model that is configured to output an item identifier based on features of a first item that are present in the portion of the composite image;

receiving a first item identifier for the first item in response to inputting the portion of the composite image into the machine learning model; and

associating the first bounding box with the first item identifier for the first item.

21 . The non-transitory computer-readable medium of claim 17 , wherein associating each bounding box from the plurality of bounding boxes with an item identifier comprises:

extracting a portion of the composite image within a first bounding box from among the plurality of bounding boxes;

comparing the portion of the composite image to a plurality of images of items;

identifying a first image from among the plurality of images of items that at least partially matches the portion of the composite image;

identifying a first item identifier that corresponds with the first image; and

associating the first bounding box with the first item identifier for the first item.

22 . The non-transitory computer-readable medium of claim 17 , wherein generating the rack analysis message comprises:

identifying a mismatch between a first item location and a first rack position from the master template;

identifying a first item associated with the first rack position from the master template; and

generating the rack analysis message that identifies the first item and the first rack position from the master template.

23 . The non-transitory computer-readable medium of claim 17 , wherein comparing the item locations for each bounding box to the plurality of rack positions from the master template comprises:

selecting a first shelf from among the plurality of shelves;

identifying a set of item identifiers that are associated with the first shelf;

assigning an alphanumeric character to each item identifier from among the set of item identifiers;

combining the alphanumeric characters to form a first word for the first shelf;

identifying a second word from the master template that corresponds with the first shelf; and

comparing the first word to the second word.

24 . The non-transitory computer-readable medium of claim 17 , wherein generating the rack analysis message comprises:

identifying a mismatch between a first item location and a first rack position from the master template; and

generating the rack analysis message that comprises a recommendation for resolving the mismatch.