Granted Patent B1

US 11,498,776 · App. 16/530,864 · Granted Nov 15, 2022

Inventors: Sandeep Sikka (Natick, MA); Nitin Verma (Natick, MA); Michael Bhaskaran (Sherborn, MA)

Assignee: Staples, Inc.

B65G47/905B07C5/342G06N5/048

Abstract

Claims (75)

1. A method comprising:

determining, by a computer, a first candidate item type and a second candidate item type;

determining one or more appearance features of the first candidate item type and one or more appearance features of the second candidate item type;

determining, using a dissimilarity model, a distinguishability score between the first candidate item type and the second candidate item type based on the one or more appearance features of the first candidate item type and the one or more appearance features of the second candidate item type;

determining to store the first candidate item type and the second candidate item type together in a container based on the distinguishability score between the first candidate item type and the second candidate item type; and

placing one or more items of the first candidate item type and one or more items of the second candidate item type in the container.

2. The method of claim 1 , wherein placing the one or more items of the first candidate item type and the one or more items of the second candidate item type in the container includes:

placing the one or more items of the first candidate item type and the one or more items of the second candidate item type in a same compartment of the container.

3. The method of claim 1 , wherein determining the first candidate item type and the second candidate item type for the container includes:

determining an item affinity between the first candidate item type and the second candidate item type based on a number of orders including the first candidate item type and the second candidate item type; and

determining that the item affinity between the first candidate item type and the second candidate item type satisfies an item affinity threshold.

4. The method of claim 1 , further comprising:

training the dissimilarity model using one or more item images of a first item type and one or more item images of a second item type.

5. The method of claim 4 , wherein training the dissimilarity model includes:

generating a synthesized image using the one or more item images of the first item type and the one or more item images of the second item type;

detecting, using an object detection model, one or more items in the synthesized image;

computing a detection accuracy score associated with the synthesized image;

determining a first appearance feature associated with a feature type of the first item type and a second appearance feature associated with the feature type of the second item type; and

adjusting one or more parameters of the dissimilarity model based on the first appearance feature of the first item type, the second appearance feature of the second item type, and the detection accuracy score associated with the synthesized image.

6. The method of claim 5 , wherein:

the first appearance feature of the first item type specifying the feature type and a value of the feature type associated with the first item type; and

the feature type includes one or more of a color, a geometric shape, a dimension, and a packaging material.

7. The method of claim 5 , wherein adjusting the one or more parameters of the dissimilarity model includes:

adjusting a dissimilarity metric between the first appearance feature and the second appearance feature in the dissimilarity model based on the detection accuracy score associated with the synthesized image; and

adjusting a discrimination weight of the feature type associated with the first appearance feature and the second appearance feature in the dissimilarity model based on the detection accuracy score associated with the synthesized image.

8. The method of claim 5 , wherein generating the synthesized image includes:

retrieving a clutter image depicting a clutter structure; and

generating the synthesized image using the one or more item images of the first item type, the one or more item images of the second item type, and the clutter image, the synthesized image depicting one or more items of the first item type and one or more items of the second item type positioned based on the clutter structure.

9. The method of claim 5 , wherein computing the detection accuracy score associated with the synthesized image includes:

determining a target detection result associated with the synthesized image; and

computing the detection accuracy score associated with the synthesized image based on a detection result of the synthesized image generated by the object detection model and the target detection result of the synthesized image.

10. The method of claim 5 , wherein:

the object detection model has an algorithm complexity metric satisfying an algorithm complexity metric threshold.

11. A system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the system to:

determine a first candidate item type and a second candidate item type for a container;

determine one or more appearance features of the first candidate item type and one or more appearance features of the second candidate item type;

determine, using a dissimilarity model, a distinguishability score between the first candidate item type and the second candidate item type based on the one or more appearance features of the first candidate item type and the one or more appearance features of the second candidate item type;

determine to store the first candidate item type and the second candidate item type together in the container based on the distinguishability score between the first candidate item type and the second candidate item type; and

place one or more items of the first candidate item type and one or more items of the second candidate item type in the container.

12. The system of claim 11 , wherein placing the one or more items of the first candidate item type and the one or more items of the second candidate item type in the container includes:

placing the one or more items of the first candidate item type and the one or more items of the second candidate item type in a first compartment of the container, wherein the container includes two or more compartments, and an item having a third item type that is different from the first candidate item type and from the second candidate item type is assigned to and placed in a second compartment of the two or more compartments.

13. The system of claim 11 , wherein determining the first candidate item type and the second candidate item type for the container includes:

determining an item affinity between the first candidate item type and the second candidate item type based on a number of orders including the first candidate item type and the second candidate item type; and

determining that the item affinity between the first candidate item type and the second candidate item type satisfies an item affinity threshold.

14. The system of claim 11 , wherein the instructions, when executed by the one or more processors, further cause the system to:

train the dissimilarity model using one or more item images of a first item type and one or more item images of a second item type.

15. The system of claim 14 , wherein training the dissimilarity model includes:

generating a synthesized image using the one or more item images of the first item type and the one or more item images of the second item type;

detecting, using an object detection model, one or more items in the synthesized image;

computing a detection accuracy score associated with the synthesized image;

determining a first appearance feature associated with a feature type of the first item type and a second appearance feature associated with the feature type of the second item type; and

adjusting one or more parameters of the dissimilarity model based on the first appearance feature of the first item type, the second appearance feature of the second item type, and the detection accuracy score associated with the synthesized image.

16. The system of claim 15 , wherein:

the first appearance feature of the first item type specifying the feature type and a value of the feature type associated with the first item type; and

the feature type includes one or more of a color, a geometric shape, a dimension, and a packaging material.

17. The system of claim 15 , wherein adjusting the one or more parameters of the dissimilarity model includes:

adjusting a dissimilarity metric between the first appearance feature and the second appearance feature in the dissimilarity model based on the detection accuracy score associated with the synthesized image; and

adjusting a discrimination weight of the feature type associated with the first appearance feature and the second appearance feature in the dissimilarity model based on the detection accuracy score associated with the synthesized image.

18. The system of claim 15 , wherein generating the synthesized image includes:

retrieving a clutter image depicting a clutter structure; and

generating the synthesized image using the one or more item images of the first item type, the one or more item images of the second item type, and the clutter image, the synthesized image depicting one or more items of the first item type and one or more items of the second item type positioned based on the clutter structure.

19. The system of claim 15 , wherein:

computing the detection accuracy score associated with the synthesized image includes:

determining a target detection result associated with the synthesized image; and

computing the detection accuracy score associated with the synthesized image based on a detection result of the synthesized image generated by the object detection model and the target detection result of the synthesized image; and

the object detection model has an algorithm complexity metric satisfying an algorithm complexity metric threshold.

20. A method comprising:

training a dissimilarity model using one or more item images of a plurality of item types;

determining, from the plurality of item types, a first candidate item type and a second candidate item type;

determining one or more appearance features of the first candidate item type and one or more appearance features of the second candidate item type;

determining, using the trained dissimilarity model, a distinguishability score between the first candidate item type and the second candidate item type based on the one or more appearance features of the first candidate item type and the one or more appearance features of the second candidate item type;

determining to store the first candidate item type and the second candidate item type together in a container based on the distinguishability score between the first candidate item type and the second candidate item type; and

instructing a robotic device to place one or more items of the first candidate item type and one or more items of the second candidate item type in the container.

Assignments (8)