Systems, devices and methods for micro-vibration data extraction using a time of flight (ToF) imaging device

1 . A system for detecting vibrations, of one or more objects in a scene, the system comprising:

at least one external illumination source configured to project coherent light in a structured light pattern with a plurality of light spots on the scene;

a Time-of-Flight (ToF) imaging device spaced from the at least one external illumination source comprising:

an illumination source configured to project modulated light into the scene;

a TOF sensor, said TOF sensor is configured to:

capture a plurality of images, said plurality of images comprising reflections of said modulated light and said structured light pattern from the one or more objects in the scene, and ambient light images of one or more objects in the scene; and

at least one processor configured to:

map each of one or more speckle patterns of the plurality of light spots with a plurality of pixels;

extract depth data of said one or more objects by analyzing reflections of the modulated light in the plurality of images;

analyze one or more changes in one or more speckle patterns of at least one of the reflections of said structured light pattern in at least some consecutive images of the plurality of images by averaging the one or more changes in at least two of the plurality of pixels; and

identify the vibrations of the one or more objects based on said speckle pattern analysis and said depth data.

2 . The system of claim 1 , wherein said depth data is extracted by measuring the phase difference between the projected and reflected modulated light and further analyze the plurality of images to extract said ambient illumination images of said scene and wherein said ambient illumination images are used to analyze one or more changes in the speckle pattern of at least one of the plurality of reflected diffused light elements in at least some consecutive images of the plurality of images.

3 . The system of claim 1 , wherein said depth data, said ambient light images and said detected vibrations are obtained and detected simultaneously in the same images by the TOF sensor, resulting in said depth data, said ambient light images and said detected vibrations being in the same coordinate system.

4 . The system of claim 3 , further including utilizing the coordinate system to correlate the vibrations with the one or more objects.

5 . The system of claim 1 , wherein the at least one processor is configured to:

detect and classify one of the one or more objects based on at least one of the extracted depth data or the ambient light images; and

based on the classification, map each of the one or more speckle patterns of the plurality of light spots reflected from the classified object with the plurality of pixels to assign an attribute to the classified object.

6 . The system of claim 1 , wherein the one or more changes to the speckle patterns includes a lateral translation which is indicative of a tilt of the speckle pattern with respect to the TOF sensor.

7 . The system of claim 6 , wherein an angular velocity is derived from analysis of the lateral translation from at least some of the plurality of images that are consecutively captured.

8 . The system of claim 1 , wherein the processor is further configured to: classify or identify attributes of the one or more objects based on said identified at least one micro-vibration.

9 . The system of claim 1 , wherein the processor is further configured to: derive, from the depth data, a classification of at least one or more objects present in the scene by assigning a probability score to one or more class labels.

10 . The system of claim 9 , wherein the visual analysis comprises using edge detection.

11 . The system of claim 1 , wherein the processor is configured to:

detect and classify a human based on at least one of the extracted depth data or the ambient light images; and

based on the classification, map each of the one or more speckle patterns of the plurality of light spots reflected from the human with the plurality of pixels to assign a physical attribute to the human, wherein the physical attribute is associated with safely operating a vehicle.

12 . The system of claim 1 , wherein the at least one processor is configured to apply an offset coefficient due to ambient illumination during the speckle content analysis.

13 . The system of claim 1 , wherein said ToF sensor includes a photodiode or photodiode array and each speckle pattern is mapped to 1-100 pixels of said ToF sensor and the averaging of the one or more changes in at least two of the plurality of pixels, includes averaging at least two adjacent pixels.

14 . The system of claim 1 , configured for mounting inside a vehicle.

15 . A system for detecting vibrations of one or more objects in a scene, the system comprising:

at least one external illumination source configured to project light in a structured light pattern with a plurality of light spots on the scene;

a Time-of-Flight (ToF) imaging device spaced from the at least one external illumination source comprising:

an illumination source configured to project modulated light into the scene;

a ToF sensor, said ToF sensor is configured to:

capture a plurality of images, said plurality of images comprising reflections of said modulated and structured light pattern from one or more objects in the scene; and

at least one processor configured to:

extract depth data of said one or more objects by analyzing the plurality of images;

analyze one or more changes in two or more adjacent speckle patterns of the reflections of at least two of the light spots in at least some conservative images of the plurality of images by averaging illumination intensity of the adjacent speckle patterns in the at least two of the light spots;

applying temporal filtering over the averaged illumination intensity; and

identify the vibrations of the at least one object based on said speckle pattern analysis and said depth data.

16 . The system of claim 15 , wherein said depth data is extracted by measuring the phase difference between the projected modulated and structured light pattern and further analyze the plurality of images to extract an ambient illumination image of said scene and wherein said ambient illumination image is used to determine an offset coefficient due to the ambient illumination to analyze one or more changes in the speckle pattern of at least one of the plurality of reflected diffused light elements in at least some consecutive images of the plurality of images.

17 . The system of claim 16 , wherein the at least one processor is configured to:

measure the phase and amplitude of the reflected modulated and structured light pattern to yield the illumination intensity of said two or more speckle patterns;

analyze the phase to extract depth information data of the scene; and

analyze, on said measured illumination intensity, the one or more changes in the speckle pattern of at least one of the plurality of reflected diffused light elements in at least some consecutive images of the plurality of images;

identify at least one micro-vibration of the one or more objects based on said speckle pattern analysis and said measured illumination intensity.

18 . The system of claim 17 , wherein the processor is further configured to: classify or identify attributes of the one or more objects based on said identified at least one micro-vibration.

19 . The system of claim 17 , wherein the processor is further configured to: classify the least one or more objects presented in the scene by visually analyzing at least one image of the plurality of images.

20 . The system of claim 19 , wherein the visual analysis comprises using at least one trained machine learning model for classifying the one or more at least one objects, wherein the machine learning model includes a neural network that includes an input layer to receive the plurality of images, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a first fully connected layer, a second fully connected layer, a SOFTMAX loss layer, and an output layer outputting the classification.