Infrared data analysis and image analysis for robust action recognition in retail environment

1 .- 60 . (canceled)

61 . A non-transitory computer-readable medium including instructions that when executed by a processor cause the processor to perform a method for using infrared data analysis and image analysis for robust action recognition in retail environment, the method comprising:

receiving infrared data captured using one or more infrared sensors from a retail environment;

receiving at least one image captured using at least one image sensor from the retail environment;

analyzing the infrared data and the at least one image to detect an action performed in the retail environment; and

providing information based on the detected action.

62 . The non-transitory computer-readable medium of claim 61 , wherein the action includes at least one of picking a product from a retail shelf, placing a product on a retail shelf and moving a product on a retail shelf.

63 . The non-transitory computer-readable medium of claim 61 , wherein detecting the action performed in the retail environment includes recognizing a type of the action.

64 . The non-transitory computer-readable medium of claim 61 , wherein detecting the action performed in the retail environment includes at least one of identifying a product type associated with the action and determining a quantity of products associated with the action.

65 . The non-transitory computer-readable medium of claim 61 , wherein the method further comprises:

calculating a convolution of at least part of the at least one image to obtain a value of the calculated convolution; and

using the value of the calculated convolution to analyze the infrared data to detect the action performed in the retail environment.

66 . The non-transitory computer-readable medium of claim 61 , wherein the method further comprises:

calculating a convolution of at least part of the infrared data to obtain a value of the calculated convolution; and

using the value of the calculated convolution to analyze the at least one image to detect the action performed in the retail environment.

67 . The non-transitory computer-readable medium of claim 61 , wherein the method further comprises:

calculating a convolution of at least part of the at least one image to obtain a value of the calculated convolution;

analyzing the infrared data to determine a wavelength associated with the infrared data;

in response to a first combination of the value of the calculated convolution and the wavelength associated with the infrared data, detecting the action performed in the retail environment; and

in response to a second combination of the value of the calculated convolution and the wavelength associated with the infrared data, forgoing the detection of the action performed in the retail environment.

68 . The non-transitory computer-readable medium of claim 61 , wherein the infrared data includes a time series of samples captured using the one or more infrared sensors at different points in time.

69 . The non-transitory computer-readable medium of claim 68 , wherein the method further comprises analyzing the time series of samples to select the at least one image of a plurality of images.

70 . The non-transitory computer-readable medium of claim 68 , wherein the method further comprises:

comparing two samples of the time series of samples; and

using a result of the comparison to analyze the at least one image to detect the action performed in the retail environment.

71 . The non-transitory computer-readable medium of claim 61 , wherein the at least one image includes a plurality of frames of a video captured using the at least one image sensor.

72 . The non-transitory computer-readable medium of claim 71 , wherein the method further comprises:

comparing two frames of the plurality of frames; and

using a result of the comparison to analyze the infrared data to detect the action performed in the retail environment.

73 . The non-transitory computer-readable medium of claim 61 , wherein the at least one image includes at least one three-dimensional image.

74 . The non-transitory computer-readable medium of claim 61 , wherein the method further comprises:

analyzing the infrared data to select a portion of the at least one image; and

analyzing the selected portion of the at least one image to detect the action performed in the retail environment.

75 . The non-transitory computer-readable medium of claim 61 , wherein the method further comprises:

analyzing the infrared data to attempt to detect the action performed in the retail environment; and

in response to a failure of the attempt to successfully detect the action, analyzing the at least one image to detect the action performed in the retail environment.

76 . The non-transitory computer-readable medium of claim 75 , wherein the failure to successfully detect the action is a failure to successfully detect the action at a confidence level higher than a selected threshold.

77 . The non-transitory computer-readable medium of claim 75 , wherein the failure to successfully detect the action is a failure to determine at least one aspect of the action.

78 . The non-transitory computer-readable medium of claim 75 , wherein the method further comprises, in response to a failure to successfully detect the action, triggering the capturing of the at least one image using the at least one image sensor.

79 . A system for using infrared data analysis and image analysis for robust action recognition in retail environment, the system comprising:

at least one processing unit configured to:

receive infrared data captured using one or more infrared sensors from a retail environment;

receive at least one image captured using at least one image sensor from the retail environment;

analyze the infrared data and the at least one image to detect an action performed in the retail environment; and

provide information based on the detected action.

80 . A method for using infrared data analysis and image analysis for robust action recognition in retail environment, the method comprising:

receiving infrared data captured using one or more infrared sensors from a retail environment;

receiving at least one image captured using at least one image sensor from the retail environment;

analyzing the infrared data and the at least one image to detect an action performed in the retail environment; and

providing information based on the detected action.

81 .- 100 . (canceled)