Illuminated multi-view sensing using 3D reconstruction for in-cabin applications
Optical sensors (e.g., cameras) and (e.g., IR) illumination sources may be distributed in an environment (e.g., an interior space such as a cabin or cockpit of an ego-machine) and synchronized to generate frames of sensor data, which may be used to reconstruct 3D geometry and/or 3D pose of an occupant, operator, or other object in the environment. For example, stereo vision may be used to generate one or more depth maps from image data generated using different cameras, the depth map(s) may be transformed into a 3D point cloud, and surface reconstruction may be applied to reconstruct the 3D geometry of surface(s) in the environment. A 3D pose, one or more keypoints (e.g., facial landmarks), or some other representation of the shape of the reconstructed surface(s) may be extracted from the reconstructed surface and used in one or more downstream tasks, such as driver and/or occupant monitoring tasks.
1 . One or more processors comprising processing circuitry to:
generate, using a plurality of optical sensors and light emitters distributed within an interior space of an ego-machine, image data representing at least a portion of the interior space;
generate, based at least on the image data, a reconstructed three-dimensional (3D) surface corresponding to at least the portion of the interior space; and
control, by an occupant or operator monitoring system, one or more operations of the ego-machine based at least on the reconstructed 3D surface.
2 . The one or more processors of claim 1 , wherein the one or more operations comprise reconstructing a 3D pose of an occupant or operator of the ego-machine based at least on the reconstructed 3D surface.
3 . The one or more processors of claim 1 , wherein the one or more operations of the ego-machine comprise one or more operator or occupant monitoring or detection tasks that evaluate a 3D pose generated using the reconstructed 3D surface.
4 . The one or more processors of claim 1 , the light emitters comprising one or more infrared (IR) emitters disposed behind one or more displays of the ego-machine.
5 . The one or more processors of claim 1 , the light emitters comprising one or more infrared (IR) emitters disposed inside one or more light fixtures of the ego-machine.
6 . The one or more processors of claim 1 , the light emitters comprising one or more infrared (IR) emitters positioned inside one or more footwell light fixtures of the ego-machine.
7 . The one or more processors of claim 1 , wherein the processing circuitry is further to adapt an intensity of light output by one or more of the light emitters based at least on a detected gradient of ambient light in the interior space.
8 . The one or more processors of claim 1 , wherein the processing circuitry is further to control an intensity of light output by a first set of the light emitters illuminating a first target region of the interior space independently of a second set of the light emitters illuminating a second target region of the interior space.
9 . The one or more processors of claim 1 , wherein the image data comprises a plurality of frames of sensor data representing multiple views of the interior space and multiple illumination patterns in substantially the same time slice.
10 . The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
11 . A system comprising one or more processors to execute one or more operations of an ego-machine based at least on a three-dimensional (3D) surface of at least a portion of an interior space reconstructed using a plurality of optical sensors and light emitters distributed within the interior space.
12 . The system of claim 11 , wherein the one or more operations comprise reconstructing a 3D pose of an occupant or operator of the ego-machine based at least on the 3D surface.
13 . The system of claim 11 , wherein the one or more operations of the ego-machine comprise one or more operator or occupant monitoring or detection tasks that evaluate a 3D pose generated using the 3D surface.
14 . The system of claim 11 , the light emitters comprising one or more infrared (IR) emitters disposed behind one or more displays of the ego-machine.
15 . The system of claim 11 , the light emitters comprising one or more infrared (IR) emitters disposed inside one or more light fixtures of the ego-machine.
16 . The system of claim 11 , the light emitters comprising one or more infrared (IR) emitters positioned inside one or more footwell light fixtures of the ego-machine.
17 . The system of claim 11 , wherein the one or more processors are further to adapt an intensity of light output by one or more of the light emitters based at least on a detected gradient of ambient light in the interior space.
18 . The system of claim 11 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
19 . A method comprising:
generating, using a plurality of optical sensors and light emitters distributed within an interior space of an ego-machine, a reconstructed three-dimensional (3D) surface of at least a portion of the interior space; and
executing one or more operations of the ego-machine based at least on the reconstructed 3D surface.
20 . The method of claim 19 , wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system implementing one or more vision language models (VLMs);
a system implementing one or more multi-modal language models;
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.