Interaction behavior detection method, apparatus, system, and device

1. A computer-implemented method for interaction behavior detection, comprising:

obtaining a depth image from a depth camera, wherein the depth camera is disposed above a rack, and a filming range of the depth camera covers an outer edge of each shelf in the rack and an aisle;

extracting a foreground image associated with a moving object from the depth image;

obtaining spatial coordinates of the moving object based on the foreground image; and

determining an object on a shelf in the rack that is physically contacted by the moving object based on comparing the spatial coordinates of the moving object with spatial coordinates of the shelf, wherein the spatial coordinates of the shelf are determined by:

obtaining a background depth image set photographed by the depth camera at a predetermined angle unobstructed by the moving object;

determining a shelf edge of the shelf based on the background depth image set, wherein determining the shelf edge of the shelf comprises:

preprocessing the background depth image set to generate preprocessed images, wherein the preprocessing is performed based on image averaging and image denoising;

performing edge detection on the preprocessed images to obtain edge detected images;

performing straight line detection from the predetermined angle on the edge detected images to obtain one or more straight lines, wherein the predetermined angle is determined based on a relative position between the depth camera and the rack; and

filtering out straight lines of the one or more straight lines that are shorter than a predetermined length threshold to obtain the shelf edge;

converting pixel coordinates of pixels corresponding to the shelf edge in the background depth image set to first-dimension coordinates and second-dimension coordinates of the spatial coordinates of the shelf based on calibrating the depth camera; and obtaining third-dimension coordinates of the spatial coordinates of the shelf based on depths of the pixels corresponding to the shelf edge.

2. The computer-implemented method of claim 1 , wherein the depth image set is a first background depth image set, the predetermined angle is a first predetermined angle, and the extracting the foreground image comprises:

obtaining a background image by performing background modeling based on a second background depth image set, and the second background depth image set is a depth image set photographed by the depth camera at a second predetermined angle unobstructed by the moving object; and

extracting the foreground image from the depth image based on comparing the depth image with the background image.

3. The computer-implemented method of claim 2 , wherein the depth image is one of a set of adjacent depth images obtained by the depth camera, and the method further comprising:

determining a set of foreground objects from the set of adjacent depth images, wherein each of the set of foreground objects has a smallest depth among foreground objects of a corresponding depth image of the set of adjacent depth images;

extracting a set of foreground images corresponding to the set of foreground objects from the set of adjacent depth images; and

obtaining a moving path of the moving object based on the set of foreground images.

4. The computer-implemented method of claim 1 , wherein the spatial coordinates of the moving object are spatial coordinates of a position on the moving object that is closest to the rack, and the obtaining the spatial coordinates of the moving object comprises:

converting pixel coordinates of pixels corresponding to a position in the foreground image to first-dimension coordinates and second-dimension coordinates of the position based on calibrating the depth camera; and

obtaining third-dimension coordinates of the spatial coordinates of the shelf based on depths of the pixels corresponding to the position.

5. The computer-implemented method of claim 4 , wherein objects on the shelf in the rack are same objects, and the determining the object that is physically contacted by the moving object comprises:

determining, based on the spatial coordinates of the moving object and the spatial coordinates of the shelf, that the moving object extends into the rack based on the first-dimension coordinates and the second-dimension coordinates;

determining that the shelf of the rack is extended into by the moving object based on the third-dimension coordinates; and

determining the object based on a predetermined correspondence between the shelf and the objects on the shelf.

6. A computer-implemented system for interaction behavior detection, comprising:

one or more computers, and

one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform operations comprising:

obtaining a depth image from a depth camera, wherein the depth camera is disposed above a rack, and a filming range of the depth camera covers an outer edge of each shelf in the rack and an aisle;

extracting a foreground image associated with a moving object from the depth image;

obtaining spatial coordinates of the moving object based on the foreground image; and

determining an object on a shelf in the rack that is physically contacted by the moving object based on comparing the spatial coordinates of the moving object with spatial coordinates of the shelf, wherein the spatial coordinates of the shelf are determined by:

obtaining a background depth image set photographed by the depth camera at a predetermined angle unobstructed by the moving object;

determining a shelf edge of the shelf based on the background depth image set, wherein determining the shelf edge of the shelf comprises:

preprocessing the background depth image set to generate preprocessed images, wherein the preprocessing is performed based on image averaging and image denoising;

performing edge detection on the preprocessed images to obtain edge detected images;

performing straight line detection from the predetermined angle on the edge detected images to obtain one or more straight lines, wherein the predetermined angle is determined based on a relative position between the depth camera and the rack; and

filtering out straight lines of the one or more straight lines that are shorter than a predetermined length threshold to obtain the shelf edge;

converting pixel coordinates of pixels corresponding to the shelf edge in the background depth image set to first-dimension coordinates and second-dimension coordinates of the spatial coordinates of the shelf based on calibrating the depth camera; and obtaining third-dimension coordinates of the spatial coordinates of the shelf based on depths of the pixels corresponding to the shelf edge.

7. The computer-implemented system of claim 6 , wherein the depth image set is a first background depth image set, the predetermined angle is a first predetermined angle, and the extracting the foreground image comprises:

obtaining a background image by performing background modeling based on a second background depth image set, and the second background depth image set is a depth image set photographed by the depth camera at a second predetermined angle unobstructed by the moving object; and

extracting the foreground image from the depth image based on comparing the depth image with the background image.

8. The computer-implemented system of claim 7 , wherein the depth image is one of a set of adjacent depth images obtained by the depth camera, and the operations further comprising:

determining a set of foreground objects from the set of adjacent depth images, wherein each of the set of foreground objects has a smallest depth among foreground objects of a corresponding depth image of the set of adjacent depth images;

extracting a set of foreground images corresponding to the set of foreground objects from the set of adjacent depth images; and

obtaining a moving path of the moving object based on the set of foreground images.

9. The computer-implemented system of claim 6 , wherein the spatial coordinates of the moving object are spatial coordinates of a position on the moving object that is closest to the rack, and the obtaining the spatial coordinates of the moving object comprises:

converting pixel coordinates of pixels corresponding to a position in the foreground image to first-dimension coordinates and second-dimension coordinates of the position based on calibrating the depth camera; and

obtaining third-dimension coordinates of the spatial coordinates of the shelf based on depths of the pixels corresponding to the position.

10. The computer-implemented system of claim 9 , wherein objects on the shelf in the rack are same objects, and the determining the object that is physically contacted by the moving object comprises:

determining, based on the spatial coordinates of the moving object and the spatial coordinates of the shelf, that the moving object extends into the rack based on the first-dimension coordinates and the second-dimension coordinates;

determining that the shelf of the rack is extended into by the moving object based on the third-dimension coordinates; and

determining the object based on a predetermined correspondence between the shelf and the objects on the shelf.

11. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations for interaction behavior detection, comprising:

obtaining a depth image from a depth camera, wherein the depth camera is disposed above a rack, and a filming range of the depth camera covers an outer edge of each shelf in the rack and an aisle;

extracting a foreground image associated with a moving object from the depth image;

obtaining spatial coordinates of the moving object based on the foreground image; and

determining an object on a shelf in the rack that is physically contacted by the moving object based on comparing the spatial coordinates of the moving object with spatial coordinates of the shelf, wherein the spatial coordinates of the shelf are determined by:

obtaining a background depth image set photographed by the depth camera at a predetermined angle unobstructed by the moving object;

determining a shelf edge of the shelf based on the background depth image set, wherein determining the shelf edge of the shelf comprises:

preprocessing the background depth image set to generate preprocessed images, wherein the preprocessing is performed based on image averaging and image denoising;

performing edge detection on the preprocessed images to obtain edge detected images;

performing straight line detection from the predetermined angle on the edge detected images to obtain one or more straight lines, wherein the predetermined angle is determined based on a relative position between the depth camera and the rack; and

filtering out straight lines of the one or more straight lines that are shorter than a predetermined length threshold to obtain the shelf edge;

converting pixel coordinates of pixels corresponding to the shelf edge in the background depth image set to first-dimension coordinates and second-dimension coordinates of the spatial coordinates of the shelf based on calibrating the depth camera; and obtaining third-dimension coordinates of the spatial coordinates of the shelf based on depths of the pixels corresponding to the shelf edge.

12. The non-transitory, computer-readable medium of claim 11 , wherein the depth image set is a first background depth image set, the predetermined angle is a first predetermined angle, and extracting the foreground image comprises:

obtaining a background image by performing background modeling based on a second background depth image set, and the second background depth image set is a depth image set photographed by the depth camera at a second predetermined angle unobstructed by the moving object; and

extracting the foreground image from the depth image based on comparing the depth image with the background image.

13. The non-transitory, computer-readable medium of claim 12 , wherein:

the depth image is one of a set of adjacent depth images obtained by the depth camera; and

the operations for interaction behavior detection further comprise:

determining a set of foreground objects from the set of adjacent depth images, wherein each of the set of foreground objects has a smallest depth among foreground objects of a corresponding depth image of the set of adjacent depth images;

extracting a set of foreground images corresponding to the set of foreground objects from the set of adjacent depth images; and

obtaining a moving path of the moving object based on the set of foreground images.

14. The non-transitory, computer-readable medium of claim 11 , wherein the spatial coordinates of the moving object are spatial coordinates of a position on the moving object that is closest to the rack, and the obtaining the spatial coordinates of the moving object comprises:

converting pixel coordinates of pixels corresponding to a position in the foreground image to first-dimension coordinates and second-dimension coordinates of the position based on calibrating the depth camera; and

obtaining third-dimension coordinates of the spatial coordinates of the shelf based on depths of the pixels corresponding to the position.

15. The non-transitory, computer-readable medium of claim 14 , wherein objects on the shelf in the rack are same objects, and the determining the object that is physically contacted by the moving object comprises:

determining, based on the spatial coordinates of the moving object and the spatial coordinates of the shelf, that the moving object extends into the rack based on the first-dimension coordinates and the second-dimension coordinates;

determining that the shelf of the rack is extended into by the moving object based on the third-dimension coordinates; and

determining the object based on a predetermined correspondence between the shelf and the objects on the shelf.