Selective operating mode switching for visible and infrared imaging
In various examples, an image processing pipeline may switch between different operating or switching modes based on speed of ego-motion and/or the active gear (e.g., park vs. drive) of a vehicle or other ego-machine in which an RGB/IR camera is being used. For example, a first operating or switching mode that toggles between IR and RGB imaging modes at a fixed frame rate or interval may be used when the vehicle is in motion, in a particular gear (e.g., drive), and/or traveling above a threshold speed. In another example, a second operating or switching mode that toggles between IR and RGB imaging modes based on detected light intensity may be used when the vehicle is in stationary, in park (or out of gear), and/or traveling below a threshold speed.
1 . One or more processors comprising processing circuitry to:
operate an image processing pipeline of an ego-machine in a first imaging switching mode that toggles between an RGB imaging mode and an infrared (IR) imaging mode at a fixed rate; and
based at least on a speed or a state of the ego-machine, switch to operating the image processing pipeline in a second imaging switching mode that toggles between the RGB imaging mode and the IR imaging mode based at least on an amount of detected light intensity;
select, while operating in at least one individual imaging switching mode of the first imaging switching mode or the second imaging switching mode, one or more machine learning models that are applicable to the at least one individual imaging switching mode from a plurality of supported machine learning models; and
perform one or more detection tasks using the one or more machine learning models.
2 . The one or more processors of claim 1 , wherein the processing circuitry is further to switch, based at least on detecting that the speed of the ego-machine is above a threshold, to operating the image processing pipeline in the first imaging switching mode that toggles between the RGB imaging mode and the IR imaging mode at the fixed rate.
3 . The one or more processors of claim 1 , wherein the processing circuitry is further to switch, based at least on determining that the state engaged by the ego-machine is a drive gear, to operating the image processing pipeline in the first imaging switching mode that toggles between the RGB imaging mode and the IR imaging mode at the fixed rate.
4 . The one or more processors of claim 1 , wherein the processing circuitry is further to operate the image processing pipeline in the first imaging switching mode based at least on generating one or more frames of RGB image data and one or more frames of IR image data, and determining whether to perform at least one detection task of the one or more detection tasks on the one or more frames of RGB image data or on the one or more frames of IR image data based at least on a second amount of detected light intensity.
5 . The one or more processors of claim 1 , wherein the processing circuitry is further to operate the image processing pipeline in the first imaging switching mode based at least on generating and applying first image data to a first set of the plurality of supported machine learning models, and operate the image processing pipeline in the second imaging switching mode based at least on generating and applying second image data to a second set of the plurality of supported machine learning models that is different from the first set.
6 . The one or more processors of claim 1 , wherein the processing circuitry is further to operate the image processing pipeline in the second imaging switching mode based at least on generating and applying a representation of one or more RGB image frames to a first set of the plurality of supported machine learning models, and generating and applying a representation of one or more IR image frames to a second set of the plurality of supported machine learning models that is different from the first set.
7 . The one or more processors of claim 1 , wherein the processing circuitry is further to operate the image processing pipeline in the first imaging switching mode based at least on generating and applying a representation of one or more RGB image frames and one or more IR image frames to at least one individual machine learning model of the plurality of supported machine learning models.
8 . The one or more processors of claim 1 , wherein the processing circuitry is further to operate the image processing pipeline in the first imaging switching mode based at least on using a common sensor to generate one or more IR image frames and one or more RGB image frames, monitor an operator of the ego-machine based at least on executing a first detection task of the one or more detection tasks using the one or more IR image frames generated by the common sensor, and monitor a non-operator occupant of the ego-machine based at least on executing a second detection task of the one or more detection tasks using the one or more RGB image frames generated by the common sensor.
9 . The one or more processors of claim 1 , wherein the processing circuitry is further to:
based at least on the ego-machine being in a state corresponding to a rate of velocity below a designated threshold, operate the image processing pipeline in the first imaging switching mode to generate first image data and perform at least one of child presence detection, three-dimensional (3D) pose estimation, or monocular depth estimation of the one or more detection tasks on the first image data; and
based at least on the ego-machine being in motion, operate the image processing pipeline in the second imaging switching mode to generate second image data and perform at least one of driver distraction detection, driver drowsiness detection, driver hands on wheel detection, driver 3D pose estimation, occupant pose estimation, or monocular depth estimation of the one or more detection tasks on the second image data.
10 . The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
11 . A system comprising one or more hardware processors to:
switch, by an image processing pipeline, between a first switching mode that produces infrared (IR) illumination at a fixed rate, and a second switching mode that produces the IR illumination at a dynamic rate determined based at least on an amount of detected light intensity;
select, while operating in at least one individual switching mode of the first switching mode or the second switching mode, one or more machine learning models from a plurality of supported machine learning models based at least on the at least one individual switching mode being in operation; and
execute one or more detection tasks using the one or more machine learning models.
12 . The system of claim 11 , wherein the one or more hardware processors are further to switch, based at least on detecting that a speed of an ego-machine associated with the image processing pipeline is above a threshold, to operating the image processing pipeline in the first switching mode that produces the IR illumination at the fixed rate.
13 . The system of claim 11 , wherein the one or more hardware processors are further to switch, based at least on detecting that a speed of an ego-machine associated with the image processing pipeline is below a threshold, to operating the image processing pipeline in the second switching mode that produces the IR illumination at the dynamic rate.
14 . The system of claim 11 , wherein, in the first switching mode, the one or more hardware processors are further to generate one or more frames of RGB image data and one or more frames of IR image data, and determine whether to perform at least one detection task of the one or more detection tasks on the one or more frames of RGB image data or on the one or more frames of IR image data based at least on a second amount of detected light intensity.
15 . The system of claim 11 , wherein, in the first switching mode, the one or more hardware processors are further to generate and apply first image data to a first set of the plurality of supported machine learning models, wherein, in the second switching mode, the one or more hardware processors are further to generate and apply second image data to a second set of the plurality of supported machine learning models that is different from the first set.
16 . The system of claim 11 , wherein, in the second switching mode, the one or more hardware processors are further to generate and apply a representation of one or more RGB image frames to a first set of the plurality of supported machine learning models, and generate and apply a representation of one or more IR image frames to a second set of the plurality of supported machine learning models that is different from the first set.
17 . The system of claim 11 , wherein, in the first switching mode, the one or more hardware processors are further to generate and apply a representation of one or more RGB image frames and one or more IR image frames to at least one individual machine learning model of the plurality of supported machine learning models.
18 . The system of claim 11 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
19 . A method comprising:
operating an image processing pipeline of an ego-machine that switches, based at least on a speed or an active gear of the ego-machine, between a first switching mode that causes infrared (IR) illumination to be produced at a fixed rate and a second switching mode that causes the IR illumination to be produced at a dynamic rate based at least on an amount of detected light intensity; and
selecting, while operating in at least one individual switching mode of the first switching mode or the second switching mode, one or more machine learning models from a plurality of supported machine learning models based at least on the at least one individual switching mode being active.
20 . The method of claim 19 , wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system implementing one or more language models;
a system implementing one or more large language models (LLMs);
a system for generating synthetic data;
a system for generating synthetic data using AI;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.