Visual, depth and micro-vibration data extraction using a unified imaging device
A unified imaging device used for detecting and classifying objects in a scene including motion and micro-vibrations by receiving a plurality of images of the scene captured by an imaging sensor of the unified imaging device comprising a light source adapted to project on the scene a predefined structured light pattern constructed of a plurality of diffused light elements, classifying object(s) present in the scene by visually analyzing the image(s), extracting depth data of the object(s) by analyzing position of diffused light element(s) reflected from the object(s), identifying micro-vibration(s) of the object(s) by analyzing a change in a speckle pattern of the reflected diffused light element(s) in at least some consecutive images and outputting the classification, the depth data and data of the one or more micro-vibrations which are derived from the analyses of images captured by the imaging sensor and are hence inherently registered in a common coordinate system.
1. A computer implemented method of detecting objects, extracting depth information of the objects and identifying micro-vibrations of the objects using a unified imaging device, comprising:
using at least one processor for:
receiving a plurality of images of a scene captured by an imaging sensor of a unified imaging device comprising a light source adapted to project on the scene a predefined structured light pattern constructed of a plurality of diffused light elements;
classifying at least one object present in the scene by visually analyzing at least one of the plurality of images;
extracting depth data of the at least one object by analyzing a positon of at least one of the plurality of diffused light elements reflected from the at least one object;
identifying at least one micro-vibration of the at least one object by analyzing a change in a speckle pattern of at least one of the plurality of reflected diffused light elements in at least some consecutive images of the plurality of images; and
outputting the classification, the depth data and data of the at least one micro-vibration which are derived from the analyses of images captured by the imaging sensor and are hence inherently registered in a common coordinate system;
wherein the change in the speckle pattern is identified by applying a speckle pattern translation to the speckle pattern to determine an angular velocity of at least one point in the speckle pattern with respect to at least one adjacent point in the speckle pattern, the angular velocity is indicative of the at least one micro-vibration.
2. The computer implemented method of claim 1 , wherein the light source and the imaging sensor operate in a common light wavelength spectrum which is a member of a group consisting of: visible light, infrared light and ultraviolet light.
3. The computer implemented method of claim 1 , wherein the classification, the depth data and the at least one micro-vibration derived from the analyses of images captured by the imaging sensor are time synchronized.
4. The computer implemented method of claim 1 , wherein each of the plurality of diffused light elements is a member of a group consisting of: a dot, a spot, a line and a combination thereof.
5. The computer implemented method of claim 1 , wherein the light source is further adapted to periodically emit the structured light pattern such that the reflected structured light pattern is depicted in a subset of the plurality of images.
6. The computer implemented method of claim 1 , further comprising averaging the intensity of the at least one point by applying a temporal sliding window over the at least one point in multiple consecutive images of the plurality of images.
7. The computer implemented method of claim 1 , further comprising averaging the intensity of the at least one point by applying an infinite response filter to the intensity of the at least one point and dividing a result of the filter by an averaged intensity calculated based on the intensity measured in multiple consecutive images of the plurality of images.
8. The computer implemented method of claim 1 , further comprising averaging the intensity of the at least one point by calculating the intensity for multiple adjacent points in the speckle pattern.
9. The computer implemented method of claim 1 , wherein the change in the speckle pattern is identified by analyzing a temporal standard deviation in an intensity of the at least one reflected diffused light element in the at least some consecutive images compared to a threshold value.
10. The computer implemented method of claim 9 , further comprising averaging the temporal standard deviation by calculating the temporal standard deviation for multiple reflected diffused light elements reflected from a surface of the at least one object.
11. The computer implemented method of claim 1 , wherein the scene is cabin of a vehicle.
12. The computer implemented method of claim 11 , wherein the at least one micro-vibration is analyzed to derive at least one of: a vehicle occupants count, a position and movement of at least one vehicle occupant, a posture of at least one vehicle occupant, a head and body parts position of at least one vehicle occupant, an activity of at least one vehicle occupant and a presence of at least one item associated with at least one vehicle occupant.
13. The computer implemented method of claim 1 , wherein the visual analysis comprising using at least one trained machine learning model for classifying the at least one object.
14. The computer implemented method of claim 13 , wherein the at least one machine learning model is trained for classifying the at least one object based on the visual data depicted in the at least one image, the visual data comprising visual data of the at least one object and at least one of the reflected diffused light elements.
15. The computer implemented method of claim 13 , wherein the at least one machine learning model is trained for classifying the at least one object according to visual data of the at least one object depicted in the at least one image combined with at least one depth map generated based on the extracted depth data.
16. A unified imaging device, comprising:
a light source adapted to project a predefined structured light pattern on a scene, the predefined structured light pattern comprising a plurality of diffused light elements; and
an imaging sensor adapted to capture a plurality of images of the scene;
wherein at least some of the plurality of images are analyzed for classifying at least one object present in the scene, for extracting depth data of the at least one object and for identifying at least one micro-vibration of the at least one object by analyzing a change in a speckle pattern of at least one of the plurality of diffused light elements in at least some consecutive images of the plurality of images, the classification, the depth data and the at least one micro-vibration are derived from the simultaneous analyses of the at least some images and are hence inherently registered in a common coordinate system;
wherein the change in the speckle pattern is identified by applying a speckle pattern translation to the speckle pattern to determine an angular velocity of at least one point in the speckle pattern with respect to at least one adjacent point in the speckle pattern, the angular velocity is indicative of the at least one micro-vibration.