Illuminated multi-view sensing using reflectance for in-cabin applications
Optical sensors (e.g., cameras) and (e.g., IR) illumination sources may be distributed in an environment (e.g., an interior space such as a cabin or cockpit of an ego-machine) and synchronized to generate frames of sensor data. By positioning the optical sensors and assigning them corresponding frequency ranges, the resulting sensor data (e.g., images from different perspectives and with different illumination patterns) may be used to extract reflectance data, the reflectance data may be used to generate more accurate sensor data (e.g., HDR images, images re-rendered using an extracted bidirectional reflectance distribution function), and the resulting sensor data may be used in one or more downstream tasks, such as operator or occupant monitoring or detection tasks (e.g., gaze detection, pose detection, attentiveness or fatigue assessment, facial recognition, gesture recognition, occupant presence detection, child presence detection, seat belt detection, hands-on-wheel detection, etc.), generating visualizations (e.g., video conference calls), and/or otherwise.
1 . One or more processors comprising processing circuitry to:
generate, using a plurality of optical sensors synchronized with light emitters distributed within an interior space of an ego-machine, image data representing at least a portion of the interior space;
extract, based at least on the image data, reflectance data encoding one or more reflectance values representative of at least the portion of the interior space; and
execute one or more operations of the ego-machine based at least on the reflectance data.
2 . The one or more processors of claim 1 , wherein the one or more operations of the ego-machine comprise generating one or more high dynamic range (HDR) images of at least the portion of the interior space based at least on the reflectance data.
3 . The one or more processors of claim 1 , wherein the reflectance data encodes an extracted bidirectional reflectance distribution function (BRDF) representative of at least the portion of the interior space.
4 . The one or more processors of claim 1 , wherein the one or more operations of the ego-machine comprise re-rendering initial image data of at least the portion of the interior space using an extracted BRDF encoded by the reflectance data.
5 . The one or more processors of claim 1 , wherein the reflectance data comprises one or more reflectance maps encoding the one or more reflectance values representative of at least the portion of the interior space.
6 . The one or more processors of claim 1 , wherein the one or more operations of the ego-machine comprise one or more operator or occupant monitoring or detection tasks that evaluate rendered image data generated using the reflectance data.
7 . The one or more processors of claim 1 , wherein the light emitters distributed within the interior space are associated with corresponding filters, wherein at least two filters of the corresponding filters pass different frequency bands.
8 . The one or more processors of claim 1 , wherein the image data comprises a plurality of frames representing multiple views of at least the portion of the interior space and multiple illumination patterns in substantially the same time slice.
9 . The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
10 . A system comprising one or more processors to execute one or more operations of an ego-machine based at least on reflectance data representative of at least a portion of an interior space of the ego-machine, the reflectance data extracted using a plurality of optical sensors synchronized with light emitters distributed within the interior space.
11 . The system of claim 10 , wherein the one or more operations of the ego-machine comprise generating one or more high dynamic range (HDR) images of at least the portion of the interior space based at least on the reflectance data.
12 . The system of claim 10 , wherein the reflectance data encodes an extracted bidirectional reflectance distribution function (BRDF) representative of the interior space.
13 . The system of claim 10 , wherein the one or more operations of the ego-machine comprise re-rendering initial image data of at least the portion of the interior space using an extracted BRDF encoded by the reflectance data.
14 . The system of claim 10 , wherein the reflectance data comprises one or more reflectance maps encoding one or more reflectance values representative of at least the portion of the interior space.
15 . The system of claim 10 , wherein the one or more operations of the ego-machine comprise one or more operator or occupant monitoring or detection tasks that evaluate rendered image data generated using the reflectance data.
16 . The system of claim 10 , wherein the light emitters distributed within the interior space are associated with corresponding filters, wherein at least two filters of the corresponding filters pass different frequency bands.
17 . The system of claim 10 , the reflectance data generated based at least on image data comprising a plurality of frames representing multiple views of at least the portion of the interior space and multiple illumination patterns in substantially the same time slice.
18 . The system of claim 11 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
19 . A method comprising:
generating, using a plurality of optical sensors synchronized with light emitters distributed within an interior space of an ego-machine, reflectance data encoding one or more reflectance values representative of at least a portion of the interior space; and
executing one or more operations of the ego-machine based at least on the reflectance data.
20 . The method of claim 19 , wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.