Pose detection of an object in a video frame

1. A system for determining a position of an aerial refueling boom from a video frame, the system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving the video frame;

identifying the aerial refueling boom in the video frame, the aerial refueling boom comprising a rigid part and a moveable part;

generating a segmentation mask of the aerial refueling boom;

accessing a three-dimensional (3D) model of the aerial refueling boom, the 3D model of the aerial refueling boom comprising constraints for the moveable part of the aerial refueling boom,

wherein the constraints include information on limitations of how the moveable part may move;

making a first comparison of the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom;

based on the first comparison, determining a match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is below a threshold;

based on determining the match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is below the threshold, using the constraints to adjust a 3D representation of the moveable part in the 3D model of the aerial refueling boom,

wherein the adjusting is performed using gradient descent;

upon adjusting the 3D representation of the moveable part in the 3D model of the aerial refueling boom, making a second comparison of the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom;

based on the second comparison, determining the match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is above the threshold;

based on determining the match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is above the threshold, determining the position of the aerial refueling boom; and

controlling, based on determining the position of the aerial refueling boom, the aerial refueling boom to engage a fuel receptacle.

2. The system of claim 1 , wherein the rigid part is a pole of the aerial refueling boom and the moveable part is a stabilizer or an extension of the aerial refueling boom.

3. The system of claim 1 , wherein the aerial refueling boom further comprises a second moveable part, and wherein generating the segmentation mask comprises:

generating a segmentation of the rigid part;

generating a segmentation of the moveable part; and

generating a segmentation of the second moveable part.

4. The system of claim 3 , wherein the moveable part and the second moveable part have different constraints.

5. The system of claim 1 , wherein the position of the aerial refueling boom is relative to an aircraft or a camera on the aircraft providing the video frame.

6. The system of claim 1 , wherein the instructions further cause the one or more processors to perform the following operations:

prior to making the first comparison of the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom and based at least on an expected position of the aerial refueling boom:

adjusting a location of the 3D representation of the rigid part of the 3D model; and

using the constraints to adjust the 3D representation of the moveable part.

7. The system of claim 1 , wherein the instructions further cause the one or more processors to perform the following operations:

identifying an aircraft to be refueled from within the video frame;

determining a position of the fuel receptacle on the aircraft; and

based on the determined position of the aerial refueling boom:

determining whether controlling the aerial refueling boom to engage the fuel receptacle is out of compliance with operational parameters; and

based on at least determining that controlling the aerial refueling boom to engage the fuel receptacle is out of compliance with operational parameters, generating an alert.

8. The system of claim 1 , further comprising a boom control that controls the aerial refueling boom.

9. A method of determining a position of an object in a video frame, the method comprising:

receiving the video frame;

identifying the object from the video frame, the object comprising a rigid part and a moveable part;

generating a segmentation mask of the object within the video frame;

accessing a three-dimensional (3D) model of the object, the 3D model comprising: a 3D representation of the rigid part, a 3D representation of the moveable part, and constraints corresponding to the moveable part of the object,

wherein the constraints include information on limitations of how the moveable part may move;

making a first comparison of the 3D model of the object to the segmentation mask of the object;

based on the first comparison, determining a match between the 3D model of the object to the segmentation mask of the object is below a threshold;

based on determining the match between the 3D model of the object to the segmentation mask of the object is below the threshold, adjusting at least one of the 3D representation of the rigid part and the 3D representation of the moveable part using the constraints,

wherein the adjusting is performed using gradient descent;

making a second comparison of the 3D model of the object to the segmentation mask of the object;

based on the second comparison, determining the match between the 3D model of the object to the segmentation mask of the object is above the threshold;

based on determining the match between the 3D model of the object to the segmentation mask of the object is above the threshold, determining the position of the object; and

controlling, based on determining the position of the object, the object to engage a receptacle.

10. The method of claim 9 , further comprising:

prior to making the first comparison of the 3D model of the object to the segmentation mask of the object and based at least on an expected position of the object:

adjusting a location of the 3D representation of the rigid part of the 3D model;

and using the constraints to adjust the 3D representation of the moveable part.

11. The method of claim 9 , wherein generating the segmentation mask comprises:

generating a segmentation of the rigid part; and

generating a segmentation of the moveable part.

12. The method of claim 11 , further comprising:

based on the first comparison, determining that the segmentation of the moveable part does not match the 3D representation of the moveable part; and

using the constraints to adjust the 3D representation of the moveable part to match the segmentation of the moveable part.

13. The method of claim 9 , wherein the position of the object is determined prior to receiving a second video frame.

14. The method of claim 9 , further comprising providing a confidence level based on how closely the 3D model of the object matches the segmentation mask of the object.

15. A non-transitory computer readable medium, comprising a computer-readable program code embodied therein, the computer-readable program code adapted to be executed to implement a method comprising:

receiving a segmentation mask of an aerial refueling boom identified in a video frame, the segmentation mask comprising a segmentation of a moveable part of the aerial refueling boom and a segmentation of a rigid part of the aerial refueling boom;

accessing a three-dimensional (3D) model of the aerial refueling boom, the 3D model of the aerial refueling boom comprising constraints for the moveable part of the aerial refueling boom,

wherein the constraints include information on limitations of how the moveable part may move;

making a first comparison of the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom;

based on the first comparison, determining a match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is below a threshold;

based on determining the match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is below the threshold, using the constraints to adjust a 3D representation of the moveable part in the 3D model of the aerial refueling boom,

wherein the adjustment is performed using gradient descent;

upon adjusting the 3D representation of the moveable part in the 3D model of the aerial refueling boom, making a second comparison of the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom;

based on the second comparison, determining the match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is above the threshold and based on the second comparison, determining a position of the aerial refueling boom; and

controlling, based on determining the position of the aerial refueling boom, the aerial refueling boom to engage a receptacle.

16. The non-transitory computer readable medium of claim 15 , wherein the 3D model is a composite mesh.

17. The non-transitory computer readable medium of claim 15 , wherein the method further comprises:

providing a confidence level based on how closely the 3D model of the aerial refueling boom matches the segmentation mask of the aerial refueling boom.

18. The non-transitory computer readable medium of claim 15 , wherein determining the position of the aerial refueling boom is based at least on determining the match between the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom is above the threshold or a threshold number of adjustment iterations has been exceeded.

19. The non-transitory computer readable medium of claim 15 , wherein the method further comprises:

prior to making the first comparison of the 3D model of the aerial refueling boom to the segmentation mask of the aerial refueling boom and based at least on an expected position of the aerial refueling boom:

adjusting a location of the 3D representation of the rigid part of the 3D model; and

using the constraints to adjust the 3D representation of the moveable part of the 3D model.

20. The system of claim 1 , further comprising a boom control that controls the aerial refueling boom; and

wherein the instructions further cause the one or more processors to perform the following operations:

determining boom parameters based on determining the position of the aerial refueling boom; and

providing the boom parameters to the boom control to control the aerial refueling boom.