System and method for the 3D thermal imaging capturing and visualization

1 . A wearable imaging system comprising:

a plurality of four or more Long Wave Infrared (LWIRI imaging modules (sensor-lens assemblies) arranged in a non-collinear way maximizing horizontal and vertical parallax with nominally parallel optical axes;

a virtual reality (VR) headset with an ability to present a high-resolution stereoscopic view of the processed image data; and

an image processing module device to

input continuous video stream from the sensor modules;

extract depth information, consolidating data from multiple simultaneously captured image pairs (intrascene) and previously registered image pairs for the same static scene objects (interscene);

build and maintain a three dimensional (3D) scene model independent of a current field of view (FoV);

generate a synthetic binocular video stream with enhanced contrast and added texturing for the horizontal features and low-textured objects; and

modify the vantage point and stereo base under user control;

the system further comprising:

two wide-Fo V thermal cameras covering the immediate surroundings of the user; and

additional functionality of the image processing device to

display wide-FoV view in the peripheral areas of the VR headset;

compare the predicted (from the maintained 3D scene model) and the actual camera views and notify user when the discrepancy exceeds the threshold; and

use sideways-pointed wide-Fo V cameras to augment the 3D scene model with Structure-from-Motion (SfM) algorithms.

2 . A wearable imaging system comprising:

a plurality of four or more Long Wave Infrared (LWIR) imaging modules (sensor-lens assemblies) arranged in a non-collinear way maximizing horizontal and vertical parallax with nominally parallel optical axes;

a virtual reality (VR) headset with an ability to present a high-resolution stereoscopic view of the processed image data; and

an image processing device to

input continuous video stream from the sensor modules;

extract depth information, consolidating data from multiple simultaneously captured image pairs (intrascene) and previously registered image pairs for the same static scene objects (interscene);

build and maintain a three dimensional (3D) scene model independent of a current field of view (FoV);

generate a synthetic binocular video stream with enhanced contrast and added texturing for the horizontal features and low-textured objects; and

modify the vantage point and stereo base under user control;

the system further comprising:

a ground or an aerial vehicle-mounted plurality of four or more LWIR imaging modules arranged in the non-collinear way, maximizing horizontal and vertical parallax; and

head-mounted inertial measurement unit (IMU) already available in most VR headsets to control the synthetic binocular video stream as the user changes head orientation relative to the ground or the aerial vehicle.

3 . A method of three dimensional 3D thermal imaging and visualization comprising of:

synchronously capturing Long Wave Infrared (LWIR) images with a plurality of the nominally identical sensor modules with their optical axes oriented in the same direction and arranged in a non-collinear configuration; and

processing the images:

synchronously acquiring LWIR images;

convert overlapping image tiles to the frequency domain, perform differential rectification and optical aberration correction using factory and field calibration data;

calculate pairwise two dimensional (2D) correlations for all or selected image pairs, keeping results in the frequency domain;

combine low-contrast data for static scene features from multiple frames:

current one and several previous ones applying appropriate geometric transformations dependent on the camera poses;

normalize frequency domain correlations with a “fat zero” off set defined by the number of scenes used;

perform inverse transformation from the frequency domain to the space (pixel) domain resulting in per-tile, per-pair 2D phase correlations;

generate a depth map from the 2D phase correlation using one of the following methods or a combination thereof:

rotating and scaling individual pairs' phase correlations matching relative positions of their sensors, accumulating the results and obtaining depth;

using Levenberg-Marquardt algorithm (LMA) or similar methods of simultaneous fitting the parametrized model to all available 2D phase correlations; or

training and inferring a deep neural network (DNN) to output depth and depth confidence from an array of the 2D correlation tiles;

grow and maintain a view-independent 3D model of the static environment objects by fusing depth maps built for individual scenes;

prepare parameters for binocular stereo generation-select automatically or under user control optimal vantage point and stereo base not limited to physical sensor modules layout;

augment objects' textures with synthetic modulation to enable the 3D perception of the horizontal features such as most negative obstacles and to enhance the 3D perception of other low-textured objects; and

synthesize binocular video stream using the 3D model and texture elements generated from a plurality of simultaneous views (intrascene) and previously acquired images (interscene) when intrascene-only contrast alone is insufficient.

4 . The method of claim 3 further comprising:

image acquisition from two wide-FoV LWIR imaging modules pointed sideways (right and left, respectively);

generating synthetic views from the current state of the 3D model matching the physical views of the side-view imaging modules;

comparing synthetic and acquired images and notifying the user by blinking corresponding visualization elements in the peripheral areas of a VR headset or by other means when the difference exceeds a specified threshold; and

augmenting the 3D model by processing data from the side-view imaging modules with Structure-from-Motion (SfM) algorithms.

5 . The method of claim 3 further comprising:

image acquisition from a plurality of the vehicle-mounted (ground or aerial) nominally identical LWIR sensor modules with their optical axes oriented in the same direction and arranged in a non-collinear configuration used either simultaneously with the head-mounted ones or alone; and

using inertial measurement unit (IMU) capability of the VR headset and the vehicle-mounted IMU to control binocular video stream generation matching user head orientation relative to that of the vehicle.

6 . The method of claim 5 further comprising:

image acquisition from additional vehicle-mounted LWIR sensor modules or groups of sensor modules with the optical axes not parallel to the vehicle's forward direction; and

augmenting the 3D model by processing data from the side-view imaging modules with Structure-from-Motion (SfM) algorithms.