IP Library › Granted Patent US 12675106

Granted Patent B2

US 12675106 · App. 17/727,207 · Granted Jul 7, 2026

Multi-sensor perception for resource tracking and quantification

Inventors: Ying Feng (The Colony, TX); Jingting Hui (Chicago, IL); Matthew Torres (Plano, TX)

Assignee: PepsiCo, Inc.

G05D1/0022 G05D1/0027 G05D1/0094

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12675106
App. No.: 17/727,207
Granted: Jul 7, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Disclosed herein are system, method, and computer program product embodiments for multi-sensor perception for resource tracking and quantification. An embodiment operates by receiving resource information indicating a resource identifier, a resource location, and a resource amount. Based on sensor data received from a first sensing device and the resource identifier, a resource removed from the resource location and placed at a predefined location may be tracked. An amount of the resource placed at the predefined location may be determined based on depth information from sensor data received from a second sensing device. A notification that indicates that additional resources that match the resource should stop being removed from the resource location and placed at the predefined location may be generated based on a match between the resource amount and the amount of the resource placed at the predefined location.

Claims (72)

1 . A computer-implemented method for multi-sensor perception for resource tracking and quantification, the method comprising:

receiving resource information indicating a resource identifier, a resource location, and a resource amount;

tracking at the resource location, based on sensor data received from a first sensing device and the resource identifier, a resource removed from the resource location and placed at a predefined location,

wherein the tracking comprises:

extracting, by a machine learning based classifier, a feature from a first image within the sensor data received from the first sensing device;

identifying, by the machine learning based classifier, a type of resource based on the feature;

identifying, by the machine learning based classifier, a resource based on a correspondence between the type of resource and the resource identifier;

applying a bound around the resource in the first image based on the identifying of the resource;

tracking the resource by adjusting a location of the bound around the resource in a second image in the sensor data received from the first sensing device based on the identifying of the resource; and

determining, based on at least a difference between a position of the bound around the resource in the first image and a position of the bound around the resource in the second image, that the resource is placed in an area of the predefined location;

determining an amount of the resource removed from the resource location and placed at the predefined location by determining the amount of the resource in the bound around the resource in the second image using depth information indicated by sensor data received from a second sensing device;

determining a match between the resource amount and the amount of the resource removed from the resource location and placed at the predefined location; and

generating, based on the match between the resource amount and the amount of the resource removed from the resource location and placed at the predefined location, a notification signal that indicates that additional resources that match the resource should stop being removed from the resource location and placed at the predefined location.

2 . The method of claim 1 , further comprising:

causing display of an indication of the resource placed in the area of the predefined location.

3 . The method of claim 2 , wherein the indication of the resource placed in the area of the predefined location comprises at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource.

4 . The method of claim 1 , wherein the determining the amount of the resource removed from the resource location and placed at the predefined location further comprises:

causing display of an indication of the resource placed in an area of the predefined location that is associated with a resource type of the resource; and

determining, based on a match between a depth value indicated by the depth information and a depth value associated with an amount of the resource type, the amount of the resource.

5 . The method of claim 1 , wherein the notification signal that causes additional resources that match the resource to stop being removed from the resource location and placed at the predefined location comprises at least one of an audible notification, a text notification, or a visual notification.

6 . The method of claim 1 , further comprising:

generating, based on a mismatch between the resource amount and the amount of the resource removed from the resource location and placed at the predefined location, a notification signal that indicates the mismatch.

7 . The method of claim 6 , wherein the notification signal that indicates the mismatch comprises at least one of an audible notification, a text notification, or a visual notification.

8 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations for multi-sensor perception to track locations and quantities of resources, the operations comprising:

receiving resource information indicating a resource identifier, a resource location, and a resource amount;

wherein the tracking comprises:

extracting, by a machine learning based classifier, a feature from a first image within the sensor data received from the first sensing device;

identifying, by the machine learning based classifier, a type of resource based on the feature;

identifying, by the machine learning based classifier, a resource based on a correspondence between the type of resource and the resource identifier;

applying a bound around the resource in the first image based on the identifying of the resource;

determining a match between the resource amount and the amount of the resource removed from the resource location and placed at the predefined location; and

9 . The non-transitory computer-readable medium of claim 8 , wherein the tracking the resource removed from the resource location and placed at the predefined location further comprises:

causing display of an indication of the resource placed in the area of the predefined location.

10 . The non-transitory computer-readable medium of claim 9 , wherein the indication of the resource placed in the area of the predefined location comprises at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource.

11 . The non-transitory computer-readable medium of claim 8 , wherein the determining the amount of the resource removed from the resource location and placed at the predefined location further comprises:

causing display of an indication of the resource placed in an area of the predefined location that is associated with a resource type of the resource; and

determining, based on a match between a depth value indicated by the depth information and a depth value associated with an amount of the resource type, the amount of the resource.

12 . The non-transitory computer-readable medium of claim 8 , wherein the notification signal that causes additional resources that match the resource to stop being removed from the resource location and placed at the predefined location comprises at least one of an audible notification, a text notification, or a visual notification.

13 . The non-transitory computer-readable medium of claim 8 , the operations further comprising:

14 . The non-transitory computer-readable medium of claim 13 , wherein the notification signal that indicates the mismatch comprises at least one of an audible notification, a text notification, or a visual notification.

15 . A system, comprising:

a memory; and

at least one processor coupled to the memory and configured to perform operations for multi-sensor perception to track locations and quantities of resources, the operations comprising:

receiving resource information indicating a resource identifier, a resource location, and a resource amount;

wherein the tracking comprises:

extracting, by a machine learning based classifier, a feature from a first image within the sensor data received from the first sensing device;

identifying, by the machine learning based classifier, a type of resource based on the feature;

identifying, by the machine learning based classifier, a resource based on a correspondence between the type of resource and the resource identifier;

applying a bound around the resource in the first image based on the identifying of the resource;

determining a match between the resource amount and the amount of the resource removed from the resource location and placed at the predefined location; and

16 . The system of claim 15 , further comprising:

causing display of an indication of the resource placed in the area of the predefined location.

17 . The system of claim 16 , wherein the indication of the resource placed in the area of the predefined location comprises at least one of a value associated with the resource, an image of the resource, or a graphical representation of the resource.

18 . The system of claim 15 , wherein the determining the amount of the resource removed from the resource location and placed at the predefined location further comprises:

causing display of an indication of the resource placed in an area of the predefined location that is associated with a resource type of the resource; and

determining, based on a match between a depth value indicated by the depth information and a depth value associated with an amount of the resource type, the amount of the resource.

19 . The system of claim 15 , wherein the notification signal that causes additional resources that match the resource to stop being removed from the resource location and placed at the predefined location comprises at least one of an audible notification, a text notification, or a visual notification.

20 . The system of claim 15 , the operations further comprising:

21 . The method of claim 1 , further comprising:

automatically sending, based on the notification signal, a control signal to an autonomous device to stop removing additional resources that match the resource from the resource location.