VIBRATION DATA ANALYSIS AND IMAGE ANALYSIS FOR ROBUST ACTION RECOGNITION IN RETAIL ENVIRONMENT

1 .- 80 . (canceled)

81 . A non-transitory computer-readable medium including instructions that when executed by a processor cause the processor to perform a method for using vibration data analysis and image analysis for robust action recognition in retail environment, the method comprising:

receiving vibration data captured using one or more vibration sensors mounted to a shelving unit including at least one retail shelf;

receiving at least one image captured using at least one image sensor from a retail environment including the shelving unit;

analyzing the vibration data and the at least one image to detect an action performed in the retail environment; and

providing information based on the detected action.

82 . The non-transitory computer-readable medium of claim 81 , wherein the action includes at least one of picking a product from a retail shelf, placing a product on a retail shelf and moving a product on a retail shelf.

83 . The non-transitory computer-readable medium of claim 81 , wherein detecting the action performed in the retail environment includes recognizing a type of the action.

84 . The non-transitory computer-readable medium of claim 81 , wherein detecting the action performed in the retail environment includes at least one of identifying a product type associated with the action and determining a quantity of products associated with the action.

85 . The non-transitory computer-readable medium of claim 81 , wherein the method further comprises:

calculating a convolution of at least part of the at least one image to obtain a value of the calculated convolution; and

using the value of the calculated convolution to analyze the vibration data to detect the action performed in the retail environment.

86 . The non-transitory computer-readable medium of claim 81 , wherein the method further comprises:

calculating a convolution of at least part of the vibration data to obtain a value of the calculated convolution; and

using the value of the calculated convolution to analyze the at least one image to detect the action performed in the retail environment.

87 . The non-transitory computer-readable medium of claim 81 , wherein the method further comprises:

calculating a convolution of at least part of the at least one image to obtain a value of the calculated convolution;

analyzing the vibration data to determine a frequency associated with the vibration data;

in response to a first combination of the value of the calculated convolution and the frequency associated with the vibration data, detecting the action performed in the retail environment; and

in response to a second combination of the value of the calculated convolution and the frequency associated with the vibration data, forgoing the detection of the action performed in the retail environment.

88 . The non-transitory computer-readable medium of claim 81 , wherein the vibration data includes a time series of samples captured using the one or more vibration sensors at different points in time.

89 . The non-transitory computer-readable medium of claim 88 , wherein the method further comprises analyzing the time series of samples to select the at least one image of a plurality of images.

90 . The non-transitory computer-readable medium of claim 88 , wherein the method further comprises:

comparing two samples of the time series of samples; and

using a result of the comparison to analyze the at least one image to detect the action performed in the retail environment.

91 . The non-transitory computer-readable medium of claim 81 , wherein the at least one image includes a plurality of frames of a video captured using the at least one image sensor.

92 . The non-transitory computer-readable medium of claim 91 , wherein the method further comprises:

comparing two frames of the plurality of frames; and

using a result of the comparison to analyze the vibration data to detect the action performed in the retail environment.

93 . The non-transitory computer-readable medium of claim 81 , wherein the at least one image includes at least one three-dimensional image.

94 . The non-transitory computer-readable medium of claim 81 , wherein the method further comprises:

analyzing the vibration data to select a portion of the at least one image; and

analyzing the selected portion of the at least one image to detect the action performed in the retail environment.

95 . The non-transitory computer-readable medium of claim 81 , wherein the method further comprises:

analyzing the vibration data to attempt to detect the action performed in the retail environment; and

in response to a failure of the attempt to successfully detect the action, analyzing the at least one image to detect the action performed in the retail environment.

96 . The non-transitory computer-readable medium of claim 95 , wherein the failure to successfully detect the action is a failure to successfully detect the action at a confidence level higher than a selected threshold.

97 . The non-transitory computer-readable medium of claim 95 , wherein the failure to successfully detect the action is a failure to determine at least one aspect of the action.

98 . The non-transitory computer-readable medium of claim 95 , wherein the method further comprises, in response to a failure to successfully detect the action, triggering the capturing of the at least one image using the at least one image sensor.

99 . A system for using vibration data analysis and image analysis for robust action recognition in retail environment, the system comprising:

at least one processing unit configured to:

receive vibration data captured using one or more vibration sensors mounted to a shelving unit including at least one retail shelf;

receive at least one image captured using at least one image sensor from a retail environment including the shelving unit;

analyze the vibration data and the at least one image to detect an action performed in the retail environment; and

provide information based on the detected action.

100 . A method for using vibration data analysis and image analysis for robust action recognition in retail environment, the method comprising:

receiving vibration data captured using one or more vibration sensors mounted to a shelving unit including at least one retail shelf;

receiving at least one image captured using at least one image sensor from a retail environment including the shelving unit;

analyzing the vibration data and the at least one image to detect an action performed in the retail environment; and

providing information based on the detected action.