On-shelf commodity detection method and system

1. A method for on-shelf merchandise detection, comprising:

obtaining a plurality of depth images associated with a shelf from a first camera, wherein the shelf comprises a plurality of tiers;

obtaining a plurality of images from one or more second cameras associated with each of the plurality of tiers of the shelf, each of the plurality of tiers of the shelf holding merchandise, and the one or more second cameras associated with each of the plurality of tiers being disposed on each of the plurality of tiers;

detecting, based on one or more of the plurality of depth images, motions of a user's hand reaching into and moving away from the shelf;

determining, based on the one or more of the plurality of depth images and in response to detecting the motions of the user's hand reaching into and moving away from the shelf, a tier of the plurality of tiers of the shelf associated with the detected motions, a first point in time associated with the motion of the user's hand reaching into the shelf, and a second point in time associated with the motion of the user's hand moving away from the shelf;

in response to determining the tier of the shelf associated with the detected motions, identifying, from images obtained from one or more second cameras associated with the determined tier, one or more first images captured before the first point in time and one or more second images captured after the second point in time; and

determining one or more changes to merchandise on the determined tier of the shelf associated with the detected motions by comparing the one or more first images with the one or more second images.

2. The method of claim 1 , wherein the one or more changes to the merchandise on the determined tier of the shelf comprises:

a change in quantity of a type of the merchandise on the determined tier.

3. The method of claim 1 , wherein the one or more second cameras comprise:

one or more wide-angle cameras; or

one or more fisheye cameras.

4. The method of claim 1 ,

wherein the one or more second cameras associated with each of the plurality of tiers of the shelf comprise at least two cameras, and

wherein the obtaining a plurality of images from one or more second cameras associated with each of the plurality of tiers of the shelf comprises:

receiving a plurality of images from each of the at least two cameras; and

combining the received plurality of images from each of the at least two cameras to obtain a plurality of images that cover positions of all merchandise on each of the plurality of the tiers.

5. The method of claim 1 , further comprising, before the detecting motions of a user's hand reaching into and moving away from the shelf:

associating each pixel of the plurality of depth images with one of the plurality of tiers of the shelf.

6. The method of claim 1 , wherein the determining one or more changes to merchandise on the determined tier of the shelf associated with the detected motions by comparing the one or more first images with the one or more second images comprises:

determining a first merchandise information set based on the one or more first images;

determining a second merchandise information set based on the one or more second images; and

determining the one or more changes to the merchandise on the determined tier of the shelf based on a difference between the first merchandise information set and the second merchandise information set.

7. The method of claim 1 ,

wherein the obtained plurality of depth images comprise a third image showing the user's hand and captured after the second point in time, and

wherein the method further comprises:

identifying merchandise in the user's hand based on the third image; and

determining whether the identified merchandise is consistent with the determined one or more changes to the merchandise on the determined tier of the shelf.

8. A system for on-shelf merchandise detection, comprising a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations comprising:

obtaining a plurality of depth images associated with a shelf from a first camera, wherein the shelf comprises a plurality of tiers;

obtaining a plurality of images from one or more second cameras associated with each of the plurality of tiers of the shelf, each of the plurality of tiers of the shelf holding merchandise, and the one or more second cameras associated with each of the plurality of tiers being disposed on each of the plurality of tiers;

detecting, based on one or more of the plurality of depth images, motions of a user's hand reaching into and moving away from the shelf;

determining, based on the one or more of the plurality of depth images and in response to detecting the motions of the user's hand reaching into and moving away from the shelf, a tier of the plurality of tiers of the shelf associated with the detected motions, a first point in time associated with the motion of the user's hand reaching into the shelf, and a second point in time associated with the motion of the user's hand moving away from the shelf;

in response to determining the tier of the shelf associated with the detected motions, identifying, from images obtained from one or more second cameras associated with the determined tier, one or more first images captured before the first point in time and one or more second images captured after the second point in time; and

determining one or more changes to merchandise on the determined tier of the shelf associated with the detected motions by comparing the one or more first images with the one or more second images.

9. The system of claim 8 , wherein the one or more changes to the merchandise on the determined tier of the shelf comprises:

a change in quantity of a type of the merchandise on the determined tier.

10. The system of claim 8 , wherein the one or more second cameras comprise:

one or more wide-angle cameras; or

one or more fisheye cameras.

11. The system of claim 8 ,

wherein the one or more second cameras associated with each of the plurality of tiers of the shelf comprise at least two cameras, and

wherein the obtaining a plurality of images from one or more second cameras associated with each of the plurality of tiers of the shelf comprises:

receiving a plurality of images from each of the at least two cameras; and

combining the received plurality of images from each of the at least two cameras to obtain a plurality of images that cover positions of all merchandise on each of the plurality of the tiers.

12. The system of claim 8 , wherein the operations further comprise, before the detecting motions of a user's hand reaching into and moving away from the shelf:

associating each pixel of the plurality of depth images with one of the plurality of tiers of the shelf.

13. The system of claim 8 , wherein the determining one or more changes to merchandise on the determined tier of the shelf associated with the detected motions by comparing the one or more first images with the one or more second images comprises:

determining a first merchandise information set based on the one or more first images;

determining a second merchandise information set based on the one or more second images; and

determining the one or more changes to the merchandise on the determined tier of the shelf based on a difference between the first merchandise information set and the second merchandise information set.

14. The system of claim 8 ,

wherein the obtained plurality of depth images comprise a third image showing the user's hand and captured after the second point in time, and

wherein the operations further comprise:

identifying merchandise in the user's hand based on the third image; and

determining whether the identified merchandise is consistent with the determined one or more changes to the merchandise on the determined tier of the shelf.

15. A non-transitory computer-readable storage medium for on-shelf merchandise detection, configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

obtaining a plurality of depth images associated with a shelf from a first camera, wherein the shelf comprises a plurality of tiers;

obtaining a plurality of images from one or more second cameras associated with each of the plurality of tiers of the shelf, each of the plurality of tiers of the shelf holding merchandise, and the one or more second cameras associated with each of the plurality of tiers being disposed on each of the plurality of tiers;

detecting, based on one or more of the plurality of depth images, motions of a user's hand reaching into and moving away from the shelf;

determining, based on the one or more of the plurality of depth images and in response to detecting the motions of the user's hand reaching into and moving away from the shelf, a tier of the tiers of the shelf associated with the detected motions, a first point in time associated with the motion of the user's hand reaching into the shelf, and a second point in time associated with the motion of the user's hand moving away from the shelf;

in response to determining the tier of the shelf associated with the detected motions, identifying, from images obtained from one or more second cameras associated with the determined tier, one or more first images captured before the first point in time and one or more second images captured after the second point in time; and

determining one or more changes to merchandise on the determined tier of the shelf associated with the detected motions by comparing the one or more first images with the one or more second images.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the one or more changes to the merchandise on the determined tier of the shelf comprises:

a change in quantity of a type of the merchandise on the determined tier.

17. The non-transitory computer-readable storage medium of claim 15 , wherein the one or more second cameras comprise:

one or more wide-angle cameras; or

one or more fisheye cameras.

18. The non-transitory computer-readable storage medium of claim 15 ,

wherein the one or more second cameras associated with each of the plurality of tiers of the shelf comprise at least two cameras, and

wherein the obtaining a plurality of images from one or more second cameras associated with each of the plurality of tiers of the shelf comprises:

receiving a plurality of images from each of the at least two cameras; and

combining the received plurality of images from each of the at least two cameras to obtain a plurality of images that cover positions of all merchandise on each of the plurality of the tiers.

19. The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise, before the detecting motions of a user's hand reaching into and moving away from the shelf:

associating each pixel of the plurality of depth images with one of the plurality of tiers of the shelf.

20. The non-transitory computer-readable storage medium of claim 15 , wherein the determining one or more changes to merchandise on the determined tier of the shelf associated with the detected motions by comparing the one or more first images with the one or more second images comprises:

determining a first merchandise information set based on the one or more first images;

determining a second merchandise information set based on the one or more second images; and

determining the one or more changes to the merchandise on the determined tier of the shelf based on a difference between the first merchandise information set and the second merchandise information set.