Simulating viewpoint transformations for sensor independent scene understanding in autonomous systems
In various examples, sensor data used to train an MLM and/or used by the MLM during deployment, may be captured by sensors having different perspectives (e.g., fields of view). The sensor data may be transformed—to generate transformed sensor data—such as by altering or removing lens distortions, shifting, and/or rotating images corresponding to the sensor data to a field of view of a different physical or virtual sensor. As such, the MLM may be trained and/or deployed using sensor data captured from a same or similar field of view. As a result, the MLM may be trained and/or deployed—across any number of different vehicles with cameras and/or other sensors having different perspectives—using sensor data that is of the same perspective as the reference or ideal sensor.
1. A computer implemented method comprising:
receiving image data generated using a first camera having a first perspective view from a first mounting location of a vehicle associated with the first camera;
applying a transformation to the image data to generate transformed image data, the transformation converting at least the first mounting location of the first perspective view to a second mounting location to simulate a second camera having a second perspective view being mounted to the vehicle at the second mounting location based at least on the second mounting location being used with respect to training images used to train a machine learning model;
applying the transformed image data to the machine learning model trained using the training images corresponding to the second mounting location;
computing, using the machine learning model and based at least on the machine learning model processing the transformed image data, output data representative of one or more predictions corresponding to the transformed image data; and
transmitting the output data to cause the vehicle to perform one or more operations based at least on the one or more predictions.
2. The computer implemented method of claim 1 , wherein the training images were generated using a physical camera mounted to the vehicle or a different vehicle at the second mounting location.
3. The computer implemented method of claim 1 , wherein:
the converting includes converting a first lens distortion captured by the image data and associated with the first camera to a second lens distortion associated with the second camera; and
the applying the transformed image data includes applying the transformed image data having the second lens distortion as an input to the machine learning model.
4. The computer implemented method of claim 1 , wherein:
the converting includes:
determining boundaries that define a region of interest corresponding to the second perspective view in world space; and
extracting the region of interest from one or more images that correspond to the image data; and
the applying the transformed image data includes applying the region of interest extracted from the one or more images as an input to the machine learning model.
5. The computer implemented method of claim 1 , wherein:
the converting includes:
determining boundaries that define a region of interest corresponding to the second perspective view in one or more images that corresponds to the transformed image data;
identifying pixels that correspond to the region of interest using the boundaries; and
applying a viewpoint transformation to the region of interest using the pixels identified using the boundaries to extract the region of interest from the one or more images; and
the applying the transformed image data includes applying the region of interest extracted from the one or more images as an input to the machine learning model.
6. The computer implemented method of claim 1 , wherein the method further includes:
determining, using the machine learning model, one or more second predictions from second transformed image data generated using a third camera having a third perspective view and converted to the second perspective view of the second camera; and
fusing the one or more predictions with the one or more second predictions to determine one or more fused predictions, wherein the one or more operations are based at least on the one or more fused predictions.
7. The computer implemented method of claim 1 , wherein the one or more predictions include one or more trajectory points in world space of a trajectory for the vehicle through an environment.
8. The computer implemented method of claim 1 , wherein the second camera has a same facing direction relative to the vehicle as the first camera and the second camera has a fixed pose.
9. A system comprising:
one or more processing units to execute operations comprising:
identifying image coordinates that define a region of interest within an image generated using a first camera;
transforming the region of interest based at least on converting the first camera to a mounting location to simulate a second camera being mounted to a machine at the mounting location based at least on the mounting location being used in training images used to train a machine learning model;
determining one or more predictions based at least on applying the region of interest that corresponds to the mounting location to the machine learning model; and
transmitting data to cause the machine to perform one or more operations based at least on the one or more predictions.
10. The system of claim 9 , wherein the transforming further includes removing lens distortions in at least the region of interest caused by a lens of the first camera to concert the region of interest to a lens-independent format, and the region of interest that is applied to the machine learning model has the lens-independent format.
11. The system of claim 9 , wherein the transforming further includes converting one or more intrinsics of the first camera to one or more intrinsics of the second camera used to generate the training images, and the region of interest that is applied to the machine learning model has the one or more intrinsics of the second camera.
12. The system of claim 9 , wherein the converting includes:
selecting a set of pixels in the image that correspond to the region of interest based at least on the identifying; and
based at least on the selecting, adjusting a second field of view depicted in the set of pixels to generate an input image of the region of interest depicting the mounting location, wherein the one or more predictions are determined from the input image.
13. The system of claim 9 , wherein the transforming includes applying a viewpoint transform to a subset of the image, the subset representing the region of interest.
14. The system of claim 9 , wherein identifying the region of interest includes determining one or more boundaries of the region of interest using one or more reference lines in an environment depicted in the image.
15. A processor comprising:
one or more circuits to:
receive image data generated using a first camera associated with a machine, the first camera having a first field of view in an environment,
apply a transformation to the image data to generate transformed image data, the transformation converting a first mounting location corresponding to the first field of view to a second field of view that comprises a perspective view to simulate a second camera mounted to the machine at a second mounting location, and
update one or more parameters of a machine learning model such that the machine learning model is trained to generate one or more predictions from the transformed image data capturing the second field of view that comprises the perspective view from the second mounting location.
16. The processor of claim 15 , wherein the second camera corresponds to a physical camera and the processor is further to update the one or more parameters of the machine learning model based at least on applying at least one image generated using the physical camera, mounted at the second mounting location, to the machine learning model.
17. The processor of claim 15 , wherein the simulated second camera has a fixed pose throughout training the machine learning model.
18. The processor of claim 15 , wherein:
the converting includes:
determining boundaries that define a region of interest corresponding to the second field of view in world space; and
extracting the region of interest from one or more images that correspond to the image data;
the updating the one or more parameters includes applying the region of interest extracted from the one or more images as an input to the machine learning model.
19. The processor of claim 15 , wherein:
the converting includes:
determining boundaries that define a region of interest corresponding to the second field of view in one or more images that corresponds to the image data;
identifying pixels that correspond to the region of interest using the boundaries; and
applying a viewpoint transformation to the pixels identified using the boundaries to extract the region of interest from the one or more images; and
the updating the one or more parameters includes applying the region of interest extracted from the one or more images as an input to the machine learning model.
20. The processor of claim 15 , wherein the transformation uses one or more references points corresponding to one or more axles of the machine.