Systems and methods for utilizing a deep learning model to determine vehicle viewpoint estimations

1. A method comprising:

receiving, by a device, a first image;

processing, by the device, the first image to identify an object in the first image and a location of the object within the first image;

extracting, by the device, a second image from the first image based on the location of the object within the first image;

processing, by the device, the second image to determine at least one of a coarse-grained viewpoint estimate or a fine-grained viewpoint estimate associated with the object,

wherein the coarse-grained viewpoint estimate, of the at least one of the coarse-grained viewpoint estimate, is associated with a first range of azimuth angles, and the fine-grained viewpoint estimate, of the at least one of the fine-grained viewpoint estimate, is associated with a second range of azimuth angles that is less than the first range of azimuth angles;

determining, by the device, that the at least one of the coarse-grained viewpoint estimate or the fine-grained viewpoint estimate satisfies a threshold;

determining, by the device, an object viewpoint associated with the object based on the at least one of the coarse-grained viewpoint estimate or the fine-grained viewpoint estimate satisfying the threshold; and

performing, by the device, one or more actions based on the object viewpoint.

2. The method of claim 1 , wherein processing the second image comprises:

processing the second image to determine a series of probabilities;

grouping the series of probabilities into a first quantity of groups;

determining the coarse-grained viewpoint estimate based on grouping the series of probabilities into the first quantity of groups;

grouping the series of probabilities into a second quantity of groups,

wherein the second quantity of groups is greater than the first quantity of groups; and

determining the fine-grained viewpoint estimate based on grouping the series of probabilities into the second quantity of groups.

3. The method of claim 1 , wherein processing the second image comprises:

providing an output of an object detection model as an input to a neural network portion of a deep learning model; and

determining the coarse-grained viewpoint estimate and the fine-grained viewpoint estimate based on an output of the neural network portion of the deep learning model.

4. The method of claim 3 , wherein processing the second image further comprises:

providing the output of the neural network portion of the deep learning model to a pooling layer of the deep learning model,

wherein the coarse-grained viewpoint estimate and the fine-grained viewpoint estimate are determined based on an output of the pooling layer.

5. The method of claim 1 , wherein processing the first image comprises:

processing the first image with an object detection model to identify the object in the first image and a bounding box corresponding to the location of the object within the first image.

6. The method of claim 1 , wherein performing the one or more actions comprises:

providing information identifying the object viewpoint to an autonomous driving system of a vehicle.

7. The method of claim 3 , wherein the deep learning model includes one or more of:

a convolutional neural network model,

a Siamese neural network model, or

a combination of the convolutional neural network model and the Siamese neural network model.

8. A device, comprising:

one or more processors configured to:

normalize, prior to training a deep learning model, images of image data by subtracting a mean and dividing by a standard deviation of the image data;

train the deep learning model based on applying random horizontal flipping and applying a Siamese network to the images of the image data;

receive, from a vehicle device of a first vehicle, a first image;

process the first image to identify an object in the first image and a location of the object within the first image;

extract, from the first image, a second image based on the location of the object within the first image;

process the second image, utilizing the deep learning model, to determine a coarse-grained viewpoint estimate and a fine-grained viewpoint estimate for the object;

determine that at least one of the coarse-grained viewpoint estimate or the fine-grained viewpoint estimate satisfies a threshold confidence level;

determine an object viewpoint associated with the object based on the at least one of the coarse-grained viewpoint estimate or the fine-grained viewpoint estimate; and

perform one or more actions based on the object viewpoint.

9. The device of claim 8 , wherein the first image is processed using an object detection model, and wherein the object detection model includes one or more of:

a you only look once model,

a single-shot detector model, or

a faster region convolutional neural network model.

10. The device of claim 8 , wherein the vehicle device includes a single dash camera provided in the vehicle.

11. The device of claim 8 , wherein the one or more processors, when determining that the at least one of the coarse-grained viewpoint estimate or the fine-grained viewpoint estimate satisfies the threshold confidence level, are further configured to:

determine a first confidence level associated with the coarse-grained viewpoint estimate and a second confidence level associated with the fine-grained viewpoint estimate;

determine that the first confidence level satisfies the threshold confidence level; and

determine that the second confidence level fails to satisfy the threshold confidence level,

wherein the object viewpoint is determined based on the coarse-grained viewpoint estimate based on the first confidence level satisfying the threshold and based on the second confidence level failing to satisfy the threshold.

12. The device of claim 8 , wherein the object comprises a vehicle, and wherein the one or more processors are configured further to:

determine, based on the object viewpoint, a direction of travel associated with the vehicle.

13. The device of claim 8 , wherein:

the coarse-grained viewpoint estimate is associated with a first range of azimuth angles, and

the fine-grained viewpoint estimate is associated with a second range of azimuth angles that is less than the first range of azimuth angles.

14. The device of claim 8 , wherein the one or more one or more processors, to process the second image, are to:

provide an output of an object detection model as an input to a neural network portion of a deep learning model; and

determine the coarse-grained viewpoint estimate and the fine-grained viewpoint estimate based on an output of the neural network portion of the deep learning model.

15. A non-transitory computer-readable medium storing instructions, the instructions comprising:

one or more instructions that, when executed by one or more processors, cause the one or more processors to:

receive, from a vehicle device of a vehicle, a first image;

process the first image, with an object detection model, to identify an object in the first image and a location of the object within the first image;

extract, from the first image, a second image of the object based on the location of the object within the first image;

process the second image, with a deep learning model, to determine one or more coarse-grained viewpoint estimates and one or more fine-grained viewpoint estimates associated with the object;

determine an object viewpoint associated with the object based on the one or more coarse-grained viewpoint estimates or the one or more fine-grained viewpoint estimates;

provide information identifying the object viewpoint to the vehicle; and

determine that an accuracy associated with the one or more fine-grained viewpoint estimates fails to satisfy a threshold,

wherein the object viewpoint is determined based on a course-grained viewpoint estimate of the one or more coarse-grained viewpoint estimates and based on the accuracy associated with the one or more fine-grained viewpoint estimates failing to satisfy the threshold.

16. The non-transitory computer-readable medium of claim 15 , wherein the one or more coarse-grained viewpoint estimates include a plurality of coarse-grained viewpoint estimates and the one or more fine-grained viewpoint estimates include a plurality of fine-grained viewpoint estimates.

17. The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions, when executed by the one or more processors, further cause the one or more processors to:

determine a direction that the object is facing based on the object viewpoint.

18. The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions, when executed by the one or more processors, further cause the one or more processors to:

determine that the accuracy associated with a fine-grained viewpoint estimate, of the one or more fine-grained viewpoint estimates, satisfies a threshold; and

determine that a granularity associated with the fine-grained viewpoint estimate is a finer granularity relative to other fine-grained viewpoint estimates, of the one or more fine-grained viewpoint estimates, and a granularity of the one or more coarse-grained viewpoint estimates,

wherein the object viewpoint is determined as the fine-grained viewpoint estimate based on the accuracy associated with the fine-grained viewpoint estimate satisfying the threshold and the granularity associated with the fine-grained viewpoint estimate being the finer granularity.

19. The non-transitory computer-readable medium of claim 15 , wherein:

a coarse-grained viewpoint estimate, of the one or more coarse-grained viewpoint estimates, is associated with a first range of azimuth angles, and

a fine-grained viewpoint estimate, of the one or more fine-grained viewpoint estimates, is associated with a second range of azimuth angles that is less than the first range of azimuth angles.

20. The non-transitory computer-readable medium of claim 15 , wherein the one or more instructions, that cause the one or more processors to process the second image, cause the one or more processors to:

provide an output of an object detection model as an input to a neural network portion of a deep learning model; and

determine the coarse-grained viewpoint estimate and the fine-grained viewpoint estimate based on an output of the neural network portion of the deep learning model.