Generation of thermal images from visible images
Techniques for generating image data of a modality different from a visible light modality are discussed herein. For example, visible light images may be readily available for training and testing vehicle systems, but corresponding image data of a thermal modality may not be available. In examples, one or more trained machine learning (ML) models may be used to generate synthetic thermal images based on an input visible light image. Such ML models may be trained, based on a training dataset including metadata and semantic segmentation associated with visible light images, to determine an appearance of the output synthetic thermal images. In some examples, an emittance level of regions of the synthetic thermal image may be further adjusted based on information associated with the respective region. The synthetic thermal images may be used for simulation and training of vehicle systems to interpret thermal imagery.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving a visible light image representing an environment;
receiving an input prompt indicative of a characteristic of the environment;
determining, based on segmenting the visible light image, a segmented image including at least one segment characterized by an object type;
inputting, to a machine-learning (ML) model, the visible light image, the input prompt, and the segmented image, wherein the ML model is a generative ML model trained to output a thermal image corresponding to an input visible light image; and
receiving, as output from the ML model and based at least in part on the input prompt, a thermal image representing the environment, wherein;
an intensity value of a portion of the thermal image corresponding to the segment is adjusted based on the object type, and
an appearance of the thermal image is in accordance with the characteristic of the environment.
2 . The system of claim 1 , wherein the ML model is a first ML model, the operations further comprising:
training, based at least in part on the visible light image and the thermal image, a second ML model configured to:
receive, from thermal sensors associated with an autonomous vehicle, an input thermal image, and
output, to a component of the autonomous vehicle, an indication of the object type.
3 . The system of claim 1 , wherein the object type comprises one of: a human, an animal, a vehicle, or a road surface.
4 . The system of claim 1 , wherein the characteristic of the environment comprises at least one of: temperature, humidity, precipitation, or an angle of the sun.
5 . The system of claim 1 , wherein the object type is a vehicle, the operations further comprising:
receiving, from a component of an autonomous vehicle, motion information corresponding to the segment;
determining, based on the motion information, an adjustment to be applied to the thermal image, wherein the adjustment comprises:
a first adjustment corresponding to a moving vehicle,
a second adjustment corresponding to a recently moving vehicle, or
a third adjustment corresponding to a stationary vehicle; and
determining, based on applying the adjustment to the thermal image, a refined thermal image.
6 . A method comprising:
receiving a first image of a first modality, the first image illustrating a scene and including metadata associated with the scene;
receiving an input prompt indicative of an environmental condition associated with the scene;
determining, by segmenting the first image, a segmented image including at least one segment;
inputting, to a machine-learning (ML) model, the first image, the input prompt, and the segmented image, wherein the ML model is a generative ML model trained to receive an input image of the first modality and generate an output image of a second modality different from the first modality; and
receiving, as output from the ML model, a second image of the scene, the second image being of the second modality, wherein:
intensity values of the second image are based at least in part on the metadata, and
an appearance of the second image is based at least in part on the input prompt.
7 . The method of claim 6 , wherein the metadata comprises one or more of:
date and time information associated with the first image,
location information associated with the first image, or
an environmental condition associated with the first image.
8 . The method of claim 6 , further comprising:
determining an object type associated with the at least one segment;
determining a portion of the second image corresponding to the at least one segment;
determining, based on the object type, an adjustment to be applied to the portion of the second image; and
generating, based on applying the adjustment, a refined second image of the second modality,
wherein an intensity value of the portion of the second image is based at least in part on the object type.
9 . The method of claim 6 , further comprising:
receiving, a representation of an object including a target location of the object in the first image; and
inputting, to the ML model, the representation of the object,
wherein the second image includes a region representing the object at a location in the second image corresponding to the target location.
10 . The method of claim 6 , wherein the first modality corresponds to visible light, and the second modality corresponds to an infrared modality.
11 . The method of claim 6 , further comprising:
generating, by applying an adjustment to the second image, a third image of the second modality; and
adding, to a training dataset, the second image and the third image,
wherein, in the training dataset, the second image and the third image include an indication of an association with the first image.
12 . The method of claim 11 , wherein the ML model is a first ML model, the method further comprising:
training, based at least in part on the training dataset, a second ML model configured to:
receive, from sensors of the second modality associated with an autonomous vehicle, an input image, and
output, to a component of the autonomous vehicle, a segmentation associated with the input image.
13 . The method of claim 6 , wherein the ML model:
implements a neural style transfer algorithm, and
is trained using a training dataset comprising images of the first modality and images of the second modality.
14 . A non-transitory computer-readable medium storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving first data captured by a first sensor of a first modality;
receiving an input prompt indicative of an environmental condition;
generating, based on the first data, a segmentation of the first data indicating at least a first region of the first data and an object type associated with the first region; and
generating, based on inputting the first data, the input prompt, and the segmentation to a machine-learning (ML) model, second data of a second modality, different from the first modality,
wherein an appearance of a second region in the second data corresponding to the first region is based at least in part on the object type and the input prompt.
15 . The non-transitory computer-readable medium of claim 14 , wherein the ML model is a first ML model, the operations further comprising:
training, based at least in part on the second data, a second ML model configured to detect a region of the object type in an input image of the second modality.
16 . The non-transitory computer-readable medium of claim 14 , wherein the object type comprises one of: a human, an animal, a vehicle, or a road surface.
17 . A method comprising:
receiving a first image of a first modality illustrating a scene and including metadata associated with the scene;
receiving, a representation of an object including a target location of the object in the first image;
inputting, to a machine-learning (ML) model trained to receive an input image of the first modality and generate an output image of a second modality different from the first modality, the first image and the representation of the object; and
receiving, as output from the ML model, a second image of the scene, the second image being of the second modality, wherein:
the second image includes a region representing the object at a location in the second image corresponding to the target location, and
intensity values of the second image are based at least in part on the metadata.
18 . The method of claim 17 , wherein the ML model is a generative ML model, the method further comprising:
receiving an input prompt indicative of an environmental condition; and
inputting, to the ML model, the input prompt, wherein an appearance of the second image is based at least in part on the input prompt.
19 . The method of claim 17 , further comprising:
determining, by segmenting the first image, a segmented image including at least one segment;
determining an object type associated with the at least one segment;
determining a portion of the second image corresponding to the at least one segment;
determining, based on the object type, an adjustment to be applied to the portion of the second image; and
determining, based on applying the adjustment, a refined second image of the second modality,
wherein an intensity value of the portion of the second image is based at least in part on the object type.
20 . The method of claim 19 , wherein the object type is a vehicle, the method further comprising:
receiving, from a component of an autonomous vehicle, motion information corresponding to the segment; and
determining, based on the motion information, an additional adjustment to be applied to the second image, wherein the additional adjustment comprises:
a first adjustment corresponding to a moving vehicle,
a second adjustment corresponding to a recently moving vehicle, or
a third adjustment corresponding to a stationary vehicle,
wherein the refined second image is further determined based on applying the additional adjustment.