Granted Patent B2

US 12,536,769 · App. 18/587,200 · Granted Jan 27, 2026

Inventors: Marios Savvides (Pittsburgh, PA); Chenchen Zhu (Pittsburgh, PA); Fangyi Chen (Pittsburgh, PA); Uzair Ahmed (Pittsburgh, PA); Ran Tao (Pittsburgh, PA)

Assignee: CARNEGIE MELLON UNIVERSITY

G06V10/443G06F18/214G06T7/73G06V10/25G06V20/20

Abstract

Claims (34)

1 . A computer-implemented method comprising:

obtaining an image of a plurality of objects on a shelf using one of more cameras;

identifying a first and second object from the image;

performing a size match between the first and second objects using a number of pixels in one or more dimensions of the first and second objects, wherein a depth pixel from a depth sensor has been applied to each pixel in the image comprising the first and second objects to adjust for different depths of the first and second objects;

performing a color match between the first and the second objects using average pixel values from corresponding portions of the first and second objects; and

when the size and color of the first and second objects match, using a deep learning convolution neural network to:

extract features from portions of the image representing the first and second objects using a trained feature extractor; and

determine, using a classifier, that the first and second objects match if the differences between the features extracted from the first object and features extracted from the second object fall within a predetermined distance threshold;

wherein the method is implemented in software executing on a processor;

wherein the trained feature extractor is a deep neural network trained to on a dataset comprising multiple views of each object and associated identifying information of each object; and

wherein the classifier is a deep neural network trained to compare the extracted features from the first and second objects to determine if the objects match.

2 . The method of claim 1 wherein obtaining an image of the plurality of objects comprises:

obtaining an image of a shelf; and

detecting regions of interest in the image of the shelf, each region of interest bounding an image of an object.

3 . The method of claim 2 wherein the first and second object are adjacent on the shelf.

4 . The method of claim 3 further comprising:

detecting one or more shelf labels in the image of the shelf;

associating a region of interest with each of the one of the one or more shelf labels.

5 . The method of claim 4 further comprising:

determining that the first and second objects match; and

determining that the first and second objects are in a same region of interest.

6 . The method of claim 4 further comprising:

determining that the first and second objects match;

determining that the first and second objects are not in a same region of interest; and

reporting a spread situation.

7 . The method of claim 4 further comprising:

determining that the first and second objects do not match;

determining that the first and second objects are in a same region of interest; and

reporting a plug situation.

8 . The method of claim 1 wherein the deep feature matching is performed by a trained CNN.

9 . The method of claim 1 wherein performing deep feature matching further comprises:

performing optical character recognition of writing detected within the portions of the image representing the first and second objects; and

determining of the writings associated with the first and second objects match.

10 . The method of claim 9 wherein the optical character recognition is performed by a trained CNN.

Assignments (1)

Continuity (4)

References Cited (50)