Non-contact breathing activity monitoring and analyzing through thermal and CO2 imaging
Various embodiments of the disclosed technology present a structural foundation for volumetric flow reconstructions for expiratory modeling enabled through multi-modal imaging for pulmonology. In some embodiments, this integrated multi-modal system includes infrared (IR) imaging, thermal imaging of carbon dioxide (CO 2 ), depth imaging (D), and visible spectrum imaging. These multiple image modalities can be integrated into flow models of exhale behaviors enable the creation of three-dimensional volume reconstructions based on visualized CO 2 distributions over time, formulating a four-dimensional exhale model which can be used to estimate various pulmonological traits (e.g., breathing rate, flow rate, exhale velocity, nose/mouth distribution, tidal volume estimation, and CO 2 density distributions). Various embodiments also enable the accurate acquisition of numerous pulmonary metrics that are then stored within distributed systems for respiratory data analytics and feature extraction through deep learning embodiments.
1. A system for exhale gas imaging comprising:
an infrared thermal imaging device to perform a spectral band-pass filter tuned to limit incoming wavelengths to that relevant to the visualization of exhale gaseous flow;
an infrared imaging device that provides depth imaging;
a visible spectrum (RGB) imaging device; and
a computing device communicably coupled to the infrared thermal imaging device, the infrared imaging device and the visible spectrum (RGB) imaging device to perform fusion, respiratory analysis, and metric extraction, wherein the computing device receives pixel intensity values from the infrared thermal imaging device and estimates fluid flow characteristics of the exhale behavior from the pixel intensity values, wherein the fluid flow characteristics are identified independently of exact pixel values and characterized by changes of apparent flow measurements to reconstruct 4D models of exhale episodes.
2. The system of claim 1 , wherein the infrared thermal imaging device is tuned to a sub-interval of the infrared portion of the electromagnetic spectrum to identify a target infrared wavelength corresponding to a desired gas.
3. The system of claim 1 , wherein the computing device generates a 4D model composed of three-dimensional flow and volume information extracted through relative pixel measurements over time.
4. The system of claim 1 , wherein the computing device generates a representation of exhale behavior and measurements obtained independent of image content through the isolation and extraction of flow behaviors.
5. The system of claim 1 , wherein the computing device generates a localization and reconstruction of exhale flow behaviors by separating the exhale flow behaviors from identifiable objects within background images.
6. The system of claim 1 , wherein the computing device generates flow and volume reconstructions by processing a sequence of images that compose a video stream of thermal CO2, depth, and RGB images.
7. The system of claim 1 , wherein the thermal imaging device, infrared imaging device, and visible spectrum (RGB) imaging device are housed within a single housing that provides a targetable vision system.
8. The system of claim 7 , wherein an orientation and position of the single housing is defined through inverse kinematics within adjustable segments or through a ceiling mounted system that contains rotational freedom and an attached arm with adjustable linear movement.
9. The system of claim 8 , wherein the single housing includes a mobile or wearable form factor.
10. The system of claim 1 , wherein the computing device constructs a 4D model of the exhale volume comprising a sequence of 3D voxel grids representative of the exhale density and volumetric distribution at points in time.
11. The system of claim 10 , wherein one or more voxels of the 3D voxel grids comprise a density value and directional flow data.
12. The system of claim 10 , wherein the computing device constructs a 3D voxel grid of the 3D voxel grids by seeding a voxel volume with a vector field central to the voxel volume and extrapolating outwardly from the central vector field based at least on a diffusion process applied to the 3D voxel grid and predicts 3D fluid movement from the 3D voxel grid using a prediction model that produces an estimated flow of the observed 2D behavior to 3D space.
13. The system of claim 12 , wherein the computing device generates a vector field representative of apparent flow velocities based at least on differences in pixel intensity values between sequential images received from the infrared thermal imaging device, paired with flow metrics obtained from the depth imaging device.
14. The system of claim 1 , wherein the computing device integrates pixel intensity values of exhale behaviors with depth measurements from a surrounding region to incorporate unit measurements in a 3D exhale reconstruction.
15. The system of claim 1 , wherein the computing device performs a correlation between thermal flow and depth images as captured by the thermal infrared imaging device and the infrared imaging device providing depth imaging to estimate depth measurements.
16. A method for analyzing pulmonological function, the method comprising, by a computing device:
obtaining, using a first imaging device, a thermal image of an exhale flow of a subject;
isolating pulmonological behavior of the subject by identifying and tracking exhale fluid dynamics exhibited by pixel intensity values corresponding to CO2 gas flow behaviors collected by the first imaging device; and
estimating fluid flow characteristics of the exhale behavior from the pixel intensity values, wherein the fluid flow characteristics are identified independently of exact pixel intensity values and characterized by changes of apparent flow measurements to reconstruct 4D models of exhale episodes.
17. The method of claim 16 , further comprising:
identifying and tracking exhale behaviors from thermal objects surrounding the subject; and
constructing a three-dimensional (3D) representation of the exhale flow based on data collected via a depth camera that provides units to the volumetric measurements of the exhale flow.
18. The method of claim 16 , wherein the pixel intensity values from the first imaging device represent a coupled relationship between thermal energy and CO2 emissivity, and wherein the method further comprises localizing and extracting the exhale flow from a mixed signal that represents IR emission from thermal heat sources interfering with the exhale signal due to the combined infrared wavelengths detected from the background heat source and the energy of the exhaled CO2 region.
19. A system comprising:
a thermal camera to take an image of an exhale of a patient;
a communications interface communicably coupled to the thermal camera to receive the image;
a processor;
a memory having instructions stored thereon that when executed by the processor cause the processor to generate a representation of the exhale of the patient and extract metrics for respiratory analysis, wherein the metrics include the reconstruction of exhale flows to obtain tidal volume estimates, the identification of the separation between nose and mouth exhale flows to measure nose/mouth distribution, and both the velocity and strength of exhale flows.
20. The system of claim 19 , wherein the thermal camera is a CO2 thermal imaging camera and wherein the instructions when executed by the processor cause the processor to create a 3D reconstruction of exhale flows through the use of the CO2 thermal imaging camera.
21. The system of claim 20 , wherein the instructions when executed by the processor create the 3D reconstruction by recording an exhale flow and extract flow characteristics and spatial metrics required to provide a direct measurement of the physiological phenomena, wherein the spatial metrics comprise depth measurements based on data captured by a depth imaging device.
22. The system of claim 19 , wherein the instructions when executed by the processor further generate an inverse thermal segmentation that separates and extracts CO2 exhale signatures observed in front of other thermal sources.
23. The system of claim 19 , wherein the instructions further cause the processor to generate flow vector fields based on the velocity and strength of exhale flows derived from pixel intensity values from images produced by the thermal camera.
24. The system of claim 19 , wherein the instructions further cause the processor to estimate breathing strength based on a 4D model of an exhale episode derived from the flow and intensity characteristics of the measured exhale.