Identification of out-of-stock products
The present disclosure relates to a fixture overlay algorithm that identifies the out-of-stock product by relying on shelf fixture start and end coordinate data points on an image to map the captured image to pre-existing planogram data. The present fixture overlay disclosure detects the start and stop shelf fixture positions for each of the shelves within the shelf image and uses the detected shelf fixture points to map the shelf image to the planogram coordinates in order to determine the identity of the out-of-stock product.
1 . A method for determining an identity of an out-of-stock product within a shelf image, the method comprising:
receiving the shelf image from an imaging device;
retrieving, from a data store, a planogram image that corresponds to the shelf image;
retrieving, from the data store, a homography transformation matrix that maps the shelf image to the planogram image, wherein the homography transformation matrix is calculated based on a reference shelf image that includes a plurality of annotations that each marks one or more shelf fixtures within the reference shelf image;
detecting one or more empty shelf spaces within the shelf image;
generating a bounding box surrounding each of the one or more empty shelf spaces;
determining whether the imaging device has shifted in position since capturing the reference shelf image by comparing a vertical position of the bounding box of at least one of the one or more empty shelf spaces to a vertical position of at least one of the plurality of annotations;
upon determining that the imaging device has not shifted in position, for each empty shelf space of the one or more empty shelf spaces, determining an image coordinate corresponding to the empty shelf space, wherein the image coordinate is an x-y coordinate corresponding to the position of the empty shelf space within the shelf image;
determining a planogram coordinate within the planogram image that corresponds to the image coordinate from the shelf image by applying the homography transformation matrix to the image coordinate; and
determining the identity of the out-of-stock product associated with the empty shelf space by determining a product identifier associated with the planogram coordinate.
2 . The method of claim 1 , further comprising:
receiving the reference shelf image from the imaging device;
receiving the plurality of annotations;
for each of the plurality of annotations, identifying a corresponding location within the planogram image;
calculating the homography transformation matrix by mapping the reference shelf image with the plurality of annotations to the planogram image; and
storing the homography transformation matrix in the data store.
3 . The method of claim 2 , wherein the homography transformation matrix is calculated using the following equation:
[
x
2
y
2
1
]
=
[
h
0
0
h
0
1
h
0
2
h
1
0
h
1
1
h
1
2
h
2
0
h
2
1
h
2
2
]
[
x
1
y
1
1
]
,
wherein, (x 1 ,y 1 ) corresponds to a location of one of the plurality of annotations within the reference shelf image, (x 2 ,y 2 ) corresponds to a location within the planogram image that corresponds to the location of the one of the plurality of annotations within the reference shelf image and
[
h
0
0
h
01
h
0
2
h
1
0
h
1
1
h
1
2
h
2
0
h
2
1
h
2
2
]
is the homography transformation matrix that when applied to (x 1 ,y 1 ) results in (x 2 ,y 2 ).
4 . The method of claim 2 , wherein the plurality of annotations includes at least 4 annotations marking at least 4 different shelf fixture locations within the reference shelf image.
5 . The method of claim 1 , further comprising:
sending a notification to a user to re-stock the out-of-stock product, wherein the notification includes the product identifier.
6 . The method of claim 1 , further comprising:
upon determining that the imaging device has shifted in position, re-generating a new homography transformation matrix; and
replacing the homography transformation matrix in the data store with the new homography transformation matrix.
7 . The method of claim 1 , wherein receiving the shelf image from the imaging device includes:
receiving a plurality of shelf images from a plurality of imaging devices;
de-warping each of the plurality of shelf images; and
stitching together the plurality of shelf images to create the shelf image.
8 . The method of claim 1 , wherein detecting the one or more empty shelf spaces within the shelf image includes:
detecting the one or more empty shelf spaces using an object detection algorithm that is trained to detect empty shelf spaces.
9 . The method of claim 1 , wherein the imaging device is camera with a fixed position within a retail store.
10 . A system for determining an identity of an out-of-stock product within a shelf image, the system comprising:
an imaging device;
a computing system comprising:
a processor; and
a memory communicatively connected to the processor which stores program instructions executable by the processor, wherein, when executed, the program instructions cause the system to:
receive the shelf image from the imaging device;
retrieve, from a data store, a planogram image that corresponds to the shelf image;
retrieve, from the data store, a homography transformation matrix that maps the shelf image to the planogram image, wherein the homography transformation matrix is calculated based on a reference shelf image that includes a plurality of annotations that each mark one or more shelf fixtures within the reference shelf image;
detect one or more empty shelf spaces within the shelf image;
generate a bounding box surrounding each of the one or more empty shelf spaces;
determine whether the imaging device has shifted in position since capturing the reference shelf image by comparing a vertical position of the bounding box of at least one of the one or more empty shelf spaces to a vertical position of at least one of the plurality of annotations;
upon determining that the imaging device has not shifted in position,
for each empty shelf space of the one or more empty shelf spaces, determine an image coordinate corresponding to the empty shelf space, wherein the image coordinate is an x-y coordinate corresponding to the position of the empty shelf space within the shelf image;
determine a planogram coordinate within the planogram image that corresponds to the image coordinate from the shelf image by applying the homography transformation matrix to the image coordinate; and
determine the identity of the out-of-stock product associated with the empty shelf space by determining a product identifier associated with the planogram coordinate.
11 . The system of claim 10 , wherein the program instructions further cause the computing system to:
receive the reference shelf image from the imaging device during initial setup of the imaging device;
receive the plurality of annotations;
for each of the plurality of annotations, identify a corresponding location within the planogram image;
calculate the homography transformation matrix by mapping the reference shelf image with the plurality of annotations to the planogram image; and
store the homography transformation matrix in the data store.
12 . The system of claim 11 , wherein the homography transformation matrix is calculated using the following equation:
[
x
2
y
2
1
]
=
[
h
0
0
h
01
h
0
2
h
1
0
h
1
1
h
1
2
h
2
0
h
2
1
h
2
2
]
[
x
1
y
1
1
]
,
wherein, (x 1 ,y 1 ) corresponds to a location of one of the plurality of annotations within the reference shelf image, (x 2 ,y 2 ) corresponds to a location within the planogram image that corresponds to the location of the one of the plurality of annotations within the reference shelf image and
[
h
0
0
h
01
h
0
2
h
1
0
h
1
1
h
1
2
h
2
0
h
2
1
h
2
2
]
is the homography transformation matrix that when applied to (x 1 ,y 1 ) results in (x 2 ,y 2 ).
13 . The system of claim 11 , wherein the plurality of annotations includes at least 4 annotations marking at least 4 different shelf fixture locations within the reference shelf image.
14 . The system of claim 10 , further comprising:
a user electronic computing device; and
wherein the program instructions further cause the computing system to:
send a notification to the user electronic computing device to alert a user to re-stock the out-of-stock product, wherein the notification includes the product identifier.
15 . The system of claim 10 , wherein the program instructions further cause the computing system to:
upon determining that the imaging device has shifted in position, re-generate a new homography transformation matrix; and
replace the homography transformation matrix in the data store with the new homography transformation matrix.
16 . The system of claim 10 , wherein to receive the shelf image from the imaging device includes to:
receive a plurality of shelf images from a plurality of imaging devices;
de-warp each of the plurality of shelf images; and
stitch together the plurality of shelf images to create the shelf image.
17 . The system of claim 10 , wherein to detect the one or more empty shelf spaces within the shelf image includes to:
detect the one or more empty shelf spaces using an object detection algorithm that is trained to detect empty shelf spaces.
18 . The system of claim 10 , wherein the imaging device is camera with a fixed position within a retail store.
19 . A system to alert a user to re-stock an out-of-stock product within a retail store shelf, the system comprising:
an imaging device;
a user electronic computing device;
a computing system comprising:
a processor; and
a memory communicatively connected to the processor which stores program instructions executable by the processor, wherein, when executed, the program instructions cause the system to:
receive a reference shelf image from the imaging device during initial setup of the imaging device;
cause the display of the reference shelf image on the user electronic computing device;
receive, at the user electronic computing device, a plurality of manual annotations, wherein each annotation marks a shelf fixture location within the reference shelf image;
for each of the plurality of annotations, identify a corresponding location within a planogram image;
calculate a homography transformation matrix by mapping the reference shelf image with the plurality of annotations that each mark one or more shelf fixtures within the reference shelf image to the planogram image; and
receive a shelf image from the imaging device;
detect one or more empty shelf spaces within the shelf image;
generate a bounding box surrounding each of the one or more empty shelf spaces;
determine whether the imaging device has shifted in position since capturing the reference shelf image by comparing a vertical position of the bounding box of at least one of the one or more empty shelf spaces to a vertical position of at least one of the plurality of annotations;
upon determining that the imaging device has not shifted in position, for each empty shelf space of the one or more empty shelf spaces, determine an image coordinate corresponding to the empty shelf space, wherein the image coordinate is an x-y coordinate corresponding to the position of the empty shelf space within the shelf image;
determine a planogram coordinate within the planogram image that corresponds to the image coordinate from the shelf image by applying the homography transformation matrix to the image coordinate;
determine the identity of the out-of-stock product associated with the empty shelf space by determining a product identifier associated with the planogram coordinate; and
send a notification to the user electronic computing device to alert the user to re-stock the out-of-stock product, wherein the notification includes the product identifier.
20 . The system of claim 19 , wherein the homography transformation matrix is calculated using the following equation:
[
x
2
y
2
1
]
=
[
h
0
0
h
01
h
0
2
h
1
0
h
1
1
h
1
2
h
2
0
h
2
1
h
2
2
]
[
x
1
y
1
1
]
,
wherein, (x 1 ,y 1 ) corresponds to a location of one of the plurality of annotations within the reference shelf image, (x 2 ,y 2 ) corresponds to a location within the planogram image that corresponds to the location of the one of the plurality of annotations within the reference shelf image and
[
h
0
0
h
01
h
0
2
h
1
0
h
1
1
h
1
2
h
2
0
h
2
1
h
2
2
]
is the homography transformation matrix that when applied to (x 1 ,y 1 ) results in (x 2 ,y 2 ).