IP Library Granted Patent US 11,699,287
Granted Patent B1
US 11,699,287 · App. 17/089,189 · Granted Jul 11, 2023

Artificial intelligence (AI) models to improve image processing related to item deliveries

Inventors: Zheshen Wang (Bellevue, WA); Dimitris Papadimitriou (Berkeley, CA); Mehran Kafai (Bellevue, WA); Jarrod Sherwin (Kent, WA); Anthony Sharma (Kenmore, WA); Leah Autumn Thompkins (Seattle, WA)
Assignee: Amazon Technologies, Inc.
G06V20/20G06F18/21G06F18/2431G06N20/00G06Q10/083G06T7/0002G06T2207/20081G06T2207/30168
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,699,287
App. No.
17/089,189
Granted
Jul 11, 2023
Kind
B1
Abstract

Techniques for improving image processing related to item deliveries are described. In an example, a computer system receives image data showing a portion of a delivery location. The computer system determines an artificial intelligence (AI) model associated with the delivery location. The computer system inputs the image data to the AI model. The computer system receives an indication of whether the portion corresponds to a correct drop-off location and causes a presentation about the indication to be provided at a device.

Claims (80)

1. A device comprising:

a camera;

a display;

one or more processors; and

one or more memories storing computer-readable instructions that, upon execution by the one or more processors, configure the device to:

receive, from a server storing a plurality of artificial intelligence (AI) models, an AI model associated with a delivery location, the AI model trained based at least in part on images showing drop-offs at the delivery location;

generate, by the camera, image data showing a portion of the delivery location;

input the image data to the AI model;

receive, from the AI model based at least in part on the image data, an indication of whether the portion corresponds to a correct drop-off location at the delivery location; and

present, on the display and in real-time relative to the image data being generated, the image data and the indication in an overlay over the image data.

2. The device of claim 1 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the device to:

receive, from the server, data about a delivery route that comprises a plurality of delivery locations; and

receive, from the server, the plurality of AI models, each one of the plurality of AI models associated specifically with one of the plurality of delivery locations.

3. The device of claim 1 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the device to:

generate an image that includes second image data showing a drop-off of an item, wherein the item is associated with a delivery to the delivery location;

input the second image data to the AI model;

receive, from the AI model based at least in part on the second image data, an indication of an incorrect drop-off location; and

present, on the display, the image and the indication of the incorrect drop-off location.

4. The device of claim 1 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the device to:

generate an image that includes second image data showing a drop-off of an item, wherein the item is associated with a delivery to the delivery location;

send the image to the server; and

receive, from the server, data indicating an incorrect drop-off of the item and identifying at least one of: a delivery address at which the item was dropped or an incorrect delivery at an unknown location, wherein:

the plurality of AI models stored at the server comprises a multi-class classifier,

the multi-class classifier is associated with the delivery location and is trained based at least in part on (i) the images showing drop-offs at the delivery location and (ii) additional images showing at least one of: drop-offs at an additional delivery location that corresponds to the delivery address or other drop-offs at random delivery locations, and

the data is received based at least in part on the image being input to the multi-class classifier.

5. A method implemented on a computer system, the method comprising:

receiving image data showing a portion of a delivery location;

determining an artificial intelligence (AI) model associated with the delivery location, the AI model trained based at least in part on images showing drop-offs at the delivery location;

inputting the image data to the AI model;

receiving, from the AI model, an indication of whether the portion corresponds to a correct drop-off location; and

causing a presentation about the indication to be provided at a device.

6. The method of claim 5 , wherein the computer system comprises the device, and further comprising:

generating, based at least in part on a camera of the device, the image data;

presenting, on a display of the device and in real time relative to the image data being generated, the image data and the indication in an overlay over the image data.

7. The method of claim 5 , wherein receiving the image data comprises receiving, from the device and based at least in part on a camera of the device, an image that includes the image data, and further comprising:

determining that the device is associated with the delivery location; and

sending, to the device and in real-time relative to the image being received, the indication, wherein the presentation comprises an overlay of the indication over the image data on a display of the device.

8. The method of claim 5 , further comprising:

receiving, from the device, second image data showing a drop-off an item, the item associated with a delivery to the delivery location;

inputting the second image data to the AI model;

receiving, from the AI model and based at least in part on the second image data, a second indication about an incorrect drop-off of the item; and

sending, to at least one of the device or another device, the second indication.

9. The method of claim 8 , further comprising:

inputting, based at least in part of the second indication, the second image data to a second AI model, wherein the second AI model is trained based at least in part on at least one of: (i) the images showing the drop-offs at the delivery location or (ii) additional images showing drop-offs at an additional delivery location;

receiving, from the second AI model, data identifying the additional delivery location that corresponds to the incorrect drop-off; and

sending the data to at least one of the device or the other device, the other device associated with the delivery location or with the additional delivery location.

10. The method of claim 9 , further comprising:

determining the second AI model based at least in part on the delivery location, wherein the second AI model comprises a multi-class classifier that is associated with the delivery location.

11. The method of claim 9 , further comprising:

determining that the AI model is associated with the delivery location, wherein the AI model comprises a first binary classifier that is associated with the delivery location;

determining that the additional delivery location is a neighboring location of the delivery location; and

determining that the second AI model is associated with the additional delivery location, wherein the second AI model comprises a second binary classifier that is associated with the additional delivery location, wherein receiving the data identifying the additional delivery location comprises receiving an output of the second binary classifier indicating a drop-off of the item.

12. The method of claim 5 , further comprising:

storing an image that comprises the image data showing a drop-off of an item based at least in part on the indication that the portion corresponds to the correct drop-off location, wherein the AI model is further trained based at least in part on the image.

13. The method of claim 12 , further comprising:

determining a trigger event associated with further training the AI model, the trigger event comprising at least one of a delivery of another item to the delivery location or a user input indicating a change to the delivery location; and

inputting the image to the AI model as a part of the further training.

14. A computer system comprising:

one or more processors; and

one or more memories storing computer-readable instructions that, upon execution by the one or more processors, configure the computer system to:

receive image data showing a portion of a delivery location;

determine an artificial intelligence (AI) model associated with the delivery location, the AI model trained based at least in part on images showing drop-offs at the delivery location;

input the image data to the AI model;

receive, from the AI model, an indication of whether the portion corresponds to a correct drop-off location; and

cause a presentation about the indication to be provided at a device.

15. The computer system of claim 14 , wherein the AI model comprises a machine learning (ML) model that is trained for feature extraction based at least in part on training images showing historical item drop-offs at a plurality of delivery locations, wherein the AI model further comprises a classifier that is trained for drop-off classification based at least in part on the the training images.

16. The computer system of claim 15 , wherein inputting the image data comprises:

inputting the image data to the ML model;

receiving features extracted by the ML model based at least in part on the image data;

inputting the features to the classifier; and

receiving the indication from the classifier based at least in part on the features.

17. The computer system of claim 15 , wherein the ML model is trained with a larger number of images than the classifier.

18. The computer system of claim 14 , wherein the AI model comprises a multi-class classifier that is trained for delivery location prediction based at least in part on (i) the images showing the drop-offs at the delivery location and (ii) additional images showing at least one of other drop-offs at one or more other delivery locations within a distance range of the delivery location or other drop-offs at random delivery locations.

19. The computer system of claim 14 , wherein the one or more memories store additional computer-readable instructions that, upon execution by the one or more processors, further configure the computer system to:

receive second image data showing a drop-off an item;

input the second image data to a second AI model, the second AI model trained for drop-off image quality evaluation based at least in part on training images showing drop-offs of items at a plurality of delivery locations;

receive, from the second AI model and based at least in part on the second image data, an image quality evaluation; and

send, to the device, data that comprises the image quality evaluation.

20. The computer system of claim 19 , wherein the second AI model is further trained for image quality improvement based at least in part on an image attribute, the image attribute comprising at least one of: blurriness, brightness, angle relative to an item drop-off, relative location of the item drop-off in an image, or contextual background around the item drop-off, and wherein the one or more memories store further computer-readable instructions that, upon execution by the one or more processors, additionally configure the computer system to:

receive, from the second AI model and based at least in part on the second image data, a recommendation about an image quality improvement, wherein the data comprises the recommendation.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 4, 2020
From: WANG, ZHESHEN; PAPADIMITRIOU, DIMITRIS; KAFAI, MEHRAN; SHERWIN, JARROD; SHARMA, ANTHONY; THOMPKINS, LEAH AUTUMN
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 054273/0233 →
Cited By (5)
US 12,260,643 US 12,367,682 US 12,609,011 US 12,646,051 US 12,670,640