Device and method for predicted autofocus on an object
A method and device for predictive autofocus on an object are provided. The method includes capturing an image of a target object to be tracked by a user of camera of a device, selecting a region of interest (ROI) corresponding to a target point of the captured image of the target object, initializing a target object tracker via input of the selected ROI at i-th frame, where i=1, updating the target object tracker based on data at previous (i−1)th frame for i-th frame, where i>1, determining current camera lens position for the tracked target object at current frame, when the target object is out of the camera focus, determining disparity for the selected ROI at i-th frame from previously collected data defining dependence of distance between the target object and lens surface on the camera lens position, determining delta lens position to focus the camera on the target object as a function of disparity and current lens position or target object position in the frame by a computing unit, calculating predicted target object position and delta lens position for N future frames from coordinates of current ROI and current delta lens position, and providing predicted autofocus in real time.
1. A method of predictive autofocus on an object, the method comprising:
capturing an image of a target object to be tracked by a user of camera of a predictive autofocus device;
selecting a region of interest (ROI) corresponding to a user-selected target point of the captured image of the target object;
initializing a target object tracker via input defining (x i ,y i ) coordinates of the selected ROI at an i-th frame, where i=1, and updating the target object tracker based on data received at the tracker at a previous (i−1)th frame of the i-th frame, where i>1;
determining a current camera lens position for the tracked target object at a current frame, when the target object is out of a camera focus;
determining disparity (Di) for the selected ROI at the i-th frame from previously collected data defining a dependence of distance between the target object and a lens surface on the camera lens position;
determining a delta lens position to focus the camera on the target object as a function of disparity (Di) and a current lens position or target object position in the frame by a computing unit of the predictive autofocus device; and
calculating a predicted target object position and delta lens position of the camera for N future frames from (x i ,y i ) coordinates of current ROI and current delta lens position for the target object and providing predicted autofocus in real time by the computing unit.
2. The method of claim 1 , wherein the determining of delta lens position to focus the camera on the target object as a function of disparity (Di) and current lens position or target object position in the frame is performed by a trained disparity-to-delta lens position mapping machine learning (ML) model and a trained physical distance-to-lens position mapping machine learning model, both the disparity-to-delta lens position mapping machine learning model and the trained physical distance-to-lens position mapping machine learning model being implemented in the computing unit.
3. The method of claim 2 , wherein the physical distance-to-lens position mapping machine learning model, implemented in the computing unit, is trained prior to the target object tracker.
4. The method of claim 2 , wherein the physical distance-to-lens position mapping machine learning model is trained on previously collected data defining different distances (d) from the camera to the target object, corresponding to lens position values (L d ) for d∈{a,b}, where {a,b} are values corresponding to a minimum and a maximum distance from the target object to the camera.
5. The method of claim 2 , wherein the disparity-to-delta lens position mapping machine learning model, implemented in the computing unit, is trained prior to the target object tracker.
6. The method of claim 2 , wherein the disparity-to-delta lens position mapping machine learning model, implemented in the computing unit, is trained on previously collected data defining different camera distances (d) to the target object, corresponding to lens position values (L d ) and disparity values D d for distance d, corresponding to ROI at distance d for d∈{a,b}, where {a,b} are values corresponding to a minimum and a maximum distance from the target object to the camera.
7. The method of claim 2 , wherein the delta lens position is a difference between a fixed lens position and a position obtained for the distance.
8. The method of claim 1 , wherein a disparity value (Di) for selected ROI at the i-th frame is determined from phase detection autofocus (PDAF) data obtained by a camera sensor matrix of the predictive autofocus device as a local minimum of a sum of absolute differences (SAD) curve.
9. The method of claim 8 , wherein the determining of the disparity value (Di) for the selected ROI at the i-th frame from obtained PDAF data representing right and left images of the target object, obtained by the camera sensor matrix, includes calculating a sum of absolute differences as absolute difference between pixels of left and right images, based on which SAD curves are plotted.
10. The method of claim 1 , further comprising determining reliability for correcting the lens position at which the target object is in focus.
11. The method of claim 10 , wherein reliability is determined as a signal-to-noise ratio from an expression:
R
=
1
0
0
x
Y
S
Y
N
,
where R is reliability, Y S is signal level, and Y N is noise level.
12. The method of claim 10 , wherein reliability is determined via a machine learning model from sum of absolute differences (SAD) curves and features defining lens position.
13. The method of claim 1 , further comprising generating a correlation filter for determining the object target point changing with a shape and position of the target object relative to the camera.
14. The method of claim 1 , further comprising generating a correlation filter for determining an ROI size changing with a shape and position of the target object relative to the camera.
15. The method of claim 13 , wherein the determining of current camera lens position for the tracked target object at current frame comprises using the correlation filter for determining the object target point and the correlation filter for determining a size of the ROI.
16. The method of claim 13 , wherein the correlation filter for determining the object target point is used once for N frames, and the correlation filter for determining a size of the ROI is used once for M frames, where M and N can be equal to or different from each other.
17. The method of claim 13 , further comprising providing a binary mask when generating a correlation filter for determining the object target point and a correlation filter for determining a size of the ROI.
18. The method of claim 13 , further comprising generating a histogram of oriented gradients (HoG) when calculating a correlation filter for determining the object target point and a correlation filter for determining a size of the ROI.
19. The method of claim 18 , wherein the generating of the HoG is performed based on a grayscale representation of the image.
20. A device for predictive autofocus on an object, the device comprising:
an input/output (I/O) unit including a graphics processing unit, a display, a camera, and an I/O interface; and
a computing unit including a central processor, a controller, a memory, a system memory, and storage,
wherein the computing unit is configured to:
capture an image of a target object to be tracked by a user of camera of a predictive autofocus device,
select a region of interest (ROI) corresponding to a user-selected target point of the captured image of the target object,
initialize a target object tracker via input defining (x i ,y i ) coordinates of the selected region of interest (ROI) at i-th frame, where i=1, and update the target object tracker based on data received at the tracker at previous (i−1)th frame for i-th frame, where i>1,
determine current camera lens position for the tracked target object at current frame, when the target object is out of a camera focus,
determine disparity (Di) for the selected ROI at i-th frame from previously collected data defining a dependence of distance between the target object and a lens surface on the camera lens position,
determine delta lens position to focus the camera on the target object as a function of disparity (Di) and current lens position or target object position in the frame by the computing unit of the predictive autofocus device, and
calculate predicted target object position and delta lens position of the camera for N future frames from (x i ,y i ) coordinates of current region of interest (ROI) and current delta lens position for the target object and providing predicted autofocus in real time by the computing unit,
wherein the camera comprises a matrix of sensors and is configured to capture a target object and transfer image data of the target object to the system memory of the computing unit via the I/O interface,
wherein the computing unit is further configured to calculate target object position and delta lens position of the camera predicted for N future frames, the calculated target object position being transferred to the system memory and via the I/O interface to the camera for auto-focusing,
wherein the camera is configured to transfer graphic information based on calculated data of the predicted target object position via the I/O interface and the graphics processing unit to the display of the device, and
wherein the display of the device is configured to render the target object being in focus in real time.