Granted Patent B1

US 12,361,366 · App. 17/943,633 · Granted Jul 15, 2025

Inventors: Badrinath Gurappa Srinivas (Bellevue, WA); Karthik Ram Srinivasan (Bellevue, WA); Brian Michael Rock (Austin, TX)

Assignee: Amazon Technologies, Inc.

G06Q10/083G06N20/20G06T19/006G06V10/40

Abstract

Claims (75)

1. A device comprising:

a camera;

a display;

one or more processors; and

one or more memories storing computer-readable instructions that, upon execution by the one or more processors, configure the device to:

receive image data corresponding to a portion of a delivery location;

determine an indication of at least one delivery object in the portion, wherein the at least one delivery object comprises at least one item being delivered at the delivery location;

input the indication into a first model trained for detecting entity-associated packaging associated with the at least one delivery object, wherein the first model is an extreme gradient boosting model being run on the device, and wherein the entity-associated packaging comprises at least one indicator of shipping packaging including (i) an entity logo or (ii) a marking associated with the shipping packaging;

receive, from the first model, an output of whether the at least one delivery object includes the entity-associated packaging;

cause a first presentation about the output to be provided at the device, wherein the first presentation simultaneously includes the image data and a text box indicating the output;

input, based at least in part on the output indicating the at least one delivery object is unassociated with the entity-associated packaging, the indication of the at least one delivery object in the portion and information associated with the at least one delivery object into a second model trained for detecting counts of delivery objects;

receive, from the second model, a result indicating a count of the at least one delivery object in the portion;

determine whether there is a match between the count and a manifest of items to be delivered at the delivery location; and

cause a second presentation about the result to be provided at the device based at least in part on whether there is the match.

2. The device of claim 1 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the device to determine whether there is the match between the count and the manifest by:

determining a mismatch between the count and the manifest of items to be delivered at the delivery location; and

causing the second presentation about the result to be provided at the device based at least in part on the mismatch, wherein the second presentation indicates an incorrect delivery.

3. The device of claim 1 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the device to determine whether there is the match between the count and the manifest by:

determining the match between the count and the manifest of items to be delivered at the delivery location; and

causing the second presentation about the result to be provided at the device based at least in part on the match, wherein the second presentation indicates a correct delivery.

4. The device of claim 1 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the device to:

prior to determining the indication of the at least one delivery object in the portion:

determine, based on user account information associated with the delivery location, that the delivery location is associated with a delivery-object action; and

output a notification of the delivery-object action to at least one of: (i) a user associated with the device or (ii) a delivery verification engine for determining correct deliveries and incorrect deliveries.

5. A method implemented on a computer system, the method comprising:

receiving image data corresponding to a portion of a delivery location;

determining an indication of at least one delivery object in the portion, wherein the at least one delivery object comprises at least one item being delivered at the delivery location;

inputting the indication into a first model trained for detecting entity-associated packaging associated with the at least one delivery object, wherein the first model is an extreme gradient boosting model being run on a device, and wherein the entity-associated packaging comprises at least one indicator of shipping packaging including (i) an entity logo or (ii) a marking associated with the shipping packaging;

receiving, from the first model, an output of whether the at least one delivery object includes the entity-associated packaging;

causing a first presentation about the output to be provided at the device, wherein the first presentation simultaneously includes the image data and a text box indicating the output;

inputting, based at least in part on the output indicating the at least one delivery object is unassociated with the entity-associated packaging, the indication of the at least one delivery object in the portion and information associated with the at least one delivery object into a second model trained for detecting counts of delivery objects;

receiving, from the second model, a result indicating a count of the at least one delivery object in the portion;

determining, whether there is a match between the count and a manifest of items to be delivered at the delivery location; and

causing a second presentation about the result to be provided at the device based at least in part on whether there is the match.

6. The method of claim 5 , wherein the computer system comprises the device, and further comprising:

generating, based at least in part on a camera of the device, the image data; and

presenting, on a display of the device and in real time relative to the image data being generated, the image data and the first presentation in an overlay over the image data.

7. The method of claim 5 , wherein determining the indication comprises:

inputting the image data into a computer vision model trained for detecting delivery objects in images; and

receiving, from the computer vision model based at least in part on the image data, the indication of the at least one delivery object in the portion.

8. The method of claim 5 , wherein the first model is trained based at least in part on at least one of: (i) images showing entity-associated packaging or (ii) additional images showing non-entity-associated packaging.

9. The method of claim 8 , wherein the indication of the at least one delivery object comprises a list of the at least one delivery object and a probability of the image data including the at least one delivery object for each delivery object of the at least one delivery object.

10. The method of claim 5 , wherein determining whether there is the match between the count and the manifest comprises:

determining a mismatch between the count and the manifest of items to be delivered at the delivery location; and

causing the second presentation about the result to be provided at the device based at least in part on the mismatch, wherein the second presentation indicates an incorrect delivery.

11. The method of claim 10 , wherein the information comprises the manifest of items to be delivered at the delivery location and the result indicates a first likelihood of the image data including the count of the at least one delivery object and a second likelihood of the image data including a second count of the at least one delivery object, wherein the first likelihood is greater than the second likelihood.

12. The method of claim 5 , wherein the output indicates the at least one delivery object is associated with the entity-associated packaging, and the method further comprising:

causing the first presentation about the output to be provided at the device, wherein the first presentation includes a notification indicating that the entity-associated packaging is to be removed.

13. The method of claim 5 , wherein determining whether there is the match between the count and the manifest comprises:

determining the match between the count and the manifest of items to be delivered at the delivery location; and

causing the second presentation about the result to be provided at the device based at least in part on the match, wherein the second presentation indicates an incorrect delivery.

14. A computer system comprising:

one or more processors; and

one or more memories storing computer-readable instructions that, upon execution by the one or more processors, configure the computer system to:

receive image data corresponding to a portion of a delivery location;

determine an indication of at least one delivery object in the portion, wherein the at least one delivery object comprises at least one item being delivered at the delivery location;

input the indication into a first model trained for detecting entity-associated packaging associated with the at least one delivery object, wherein the first model is an extreme gradient boosting model being run on a device, and wherein the entity-associated packaging comprises at least one indicator of shipping packaging including (i) an entity logo or (ii) a marking associated with the shipping packaging;

receive, from the first model, an output of whether the at least one delivery object includes the entity-associated packaging;

cause a first presentation about the output to be provided at the device, wherein the first presentation simultaneously includes the image data and a text box indicating the output;

input, based at least in part on the output indicating the at least one delivery object is unassociated with the entity-associated packaging, the indication of the at least one delivery object in the portion and information associated with the at least one delivery object into a second model trained for detecting counts of delivery objects;

receive, from the second model, a result indicating a count of the at least one delivery object in the portion;

determine whether there is a match between the count and a manifest of items to be delivered at the delivery location; and

cause a second presentation about the result to be provided at the device based at least in part on whether there is the match.

15. The computer system of claim 14 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the computer system to:

generate, based at least in part on a camera of the device, the image data; and

generate, based at least in part on a depth sensor of the device, dimensional information of the portion of the delivery location.

16. The computer system of claim 15 , wherein determining the indication comprises:

inputting the image data and the dimensional information into a computer vision model trained for detecting delivery objects in images; and

receiving, from the computer vision model based at least in part on the image data and the dimensional information, the indication of the at least one delivery object in the portion.

17. The computer system of claim 14 , wherein the first presentation comprises a notification indicating the entity-associated packaging is to be removed or an additional packaging is to be added.

18. The computer system of claim 14 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the computer system to determine whether there is the match between the court and the manifest by:

determining a mismatch between the count and the manifest of items to be delivered at the delivery location; and

causing the second presentation about the result to be provided at the device based at least in part on the mismatch, wherein the second presentation indicates an incorrect delivery.

19. The computer system of claim 18 , wherein the second presentation comprises a notification indicating additional image data is to be generated for the portion of the delivery location based at least in part on the mismatch.

20. The computer system of claim 14 , wherein the image data comprises one or more images of one or more portions of the delivery location or video data of the delivery location.

Assignments (1)

References Cited (15)