Image processing apparatus and control method thereof

1 . An image processing apparatus comprising:

one or more processors that execute a program stored in a memory and thereby function as:

a detecting unit configured to detect, in a target frame, one or more regions of a predetermined subject as candidate region(s);

a determining unit configured to determine a subject region to be tracked from the candidate region(s);

a difficulty computing unit configured to compute an evaluation value indicating a calculated difficulty score pertaining to difficulty of tracking, based on the subject region determined by the determining unit, wherein the difficulty score is incremented when an object class of the subject region includes objects for which shape changes significantly, and also incremented when a number of candidate regions having the same object class as the subject region is greater than a first preset value;

a tracking unit configured to execute tracking processing for estimating a position, in the target frame, of a region corresponding to the subject region to be tracked in a frame previous to the target frame, the tracking unit including a first tracking unit and a second tracking unit both operating on the same frames; and

a control unit configured to selectively determine, based on the evaluation value, operation of the first tracking unit and operation of the second tracking unit, on a subsequent frame to the target frame,

wherein the second tracking unit operates a deep-learning neural network and has a higher processing accuracy and a greater operational load than the first tracking unit that does not operate a deep-learning neural network, and

based on the evaluation value computed by the difficulty computing unit, the control unit selectively activates both the first tracking unit and the second tracking unit to operate in parallel on the same frames, or deactivates at least one of the first tracking unit and the second tracking unit.

2 . The image processing apparatus according to claim 1 , wherein

the computing unit computes the evaluation value based on at least one of a size of the subject region, a position of the subject region, a type of the subject, and a total number of subject regions pertaining to subjects of a same type.

3 . The image processing apparatus according to claim 1 , wherein

the higher the evaluation value is, the higher the difficulty of the tracking processing is, and

the control unit deactivates the first tracking unit and activates the second tracking unit when the evaluation value is at least a first threshold.

4 . The image processing apparatus according to claim 1 , wherein

the higher the evaluation value is, the higher the difficulty of the tracking processing is, and

the control unit deactivates the second tracking unit and activates the first tracking unit when the evaluation value is less than a second threshold.

5 . The image processing apparatus according to claim 1 , wherein

when activating both the first tracking unit and the second tracking unit, the control unit lowers an operational frequency of the second tracking unit than an operational frequency of the first tracking unit.

6 . The image processing apparatus according to claim 1 , wherein

when the subject region determined by the determining unit and the subject region for which the position is estimated by the tracking unit relate to the same subject, the control unit does not change the operation of the first tracking unit and the operation of the second tracking unit on the subsequent frame from the operation of the first tracking unit and the operation of the second tracking unit on the target frame.

7 . The image processing apparatus according to claim 1 , wherein

when the subject region determined by the determining unit and the subject region for which the position is estimated by the tracking unit relate to different subjects, the control unit determines, based on the evaluation value, the operation of the first tracking unit and the operation of the second tracking unit on the subsequent frame.

8 . The image processing apparatus according to claim 1 , wherein

the second tracking unit executes the tracking processing using a deep learning neural network and the first tracking unit executes tracking processing using a non-deep learning neural network.

9 . The image processing apparatus according to claim 1 , wherein the control unit activates both the first tracking unit and the second tracking unit to operate in parallel on the same frames, but with different frequencies.

10 . An image processing apparatus comprising:

one or more processors that execute a program stored in a memory and thereby function as:

a detecting unit configured to detect, in a target frame, one or more regions of a predetermined subject as candidate region(s);

a determining unit configured to determine a subject region to be tracked from the candidate region(s);

a tracking unit configured to execute tracking processing for estimating a position, in the target frame, of a region corresponding to the subject region to be tracked in a frame previous to the target frame, the tracking unit including first tracking unit and second tracking unit both operating on the same frames; and

a control unit configured to selectively determine, based on a type of the subject detected by the detecting unit, operation of the first tracking unit and operation of the second tracking unit on a subsequent frame to the target frame,

wherein the second tracking unit operates a deep-learning neural network and has a higher processing accuracy and a greater operational load than the first tracking unit that does not operate a deep-learning neural network, and

based on the type of the subject detected by the detecting unit, the control unit selectively activates both the first tracking unit and the second tracking unit to operate in parallel on the same frames, or deactivates at least one of the first tracking unit and the second tracking unit.

11 . The image processing apparatus according to claim 10 , wherein the control unit activates both the first tracking unit and the second tracking unit to operate in parallel on the same frames, but with different frequencies.

12 . A control method for an image processing apparatus, wherein

the image processing apparatus includes a tracking unit for executing tracking processing for estimating a position, in a target frame, of a region corresponding to a subject region to be tracked in a frame previous to the target frame,

the tracking unit includes a first tracking unit, and a second tracking unit operating a deep-learning neural network and having a higher processing accuracy and a greater operational load than the first tracking unit that does not operate a deep-learning neural network, wherein the first and second tracking units operate on the same frames, and

the control method comprises:

detecting, in a target frame, one or more regions of a predetermined subject as candidate region(s);

determining a subject region to be tracked from the candidate region(s);

computing an evaluation value indicating a calculated difficulty score pertaining to difficulty of tracking, based on the subject region determined in the determining, wherein the difficulty score is incremented when an object class of the subject region includes objects for which shape changes significantly, and also incremented when a number of candidate regions having the same object class as the subject region is greater than a first preset value; and

selectively controlling, based on the evaluation value, operation of the first tracking unit and operation of the second tracking unit on a subsequent frame to the target frame,

wherein the controlling, based on the evaluation value computed based on the incremented difficulty score, selectively activates both the first tracking unit and the second tracking unit to operate in parallel on the same frames, or deactivates at least one of the first tracking unit and the second tracking unit.

13 . A control method for an image processing apparatus, wherein

the image processing apparatus includes a tracking unit for executing tracking processing for estimating a position, in a target frame, of a region corresponding to a subject region to be tracked in a frame previous to the target frame,

the tracking unit includes a first tracking unit, and a second tracking unit operating a deep-learning neural network and having a higher processing accuracy and a greater operational load than the first tracking unit that does not operate a deep-learning neural network, wherein the first and second tracking units operate on the same frames, and

the control method comprises:

detecting, in a target frame, one or more regions of a predetermined subject as candidate region(s);

determining a subject region to be tracked from the candidate region(s); and

selectively controlling, based on a type of the subject detected in the detecting, operation of the first tracking unit and operation of the second tracking unit on a subsequent frame to the target frame,

wherein the controlling, based on the type of the subject detected in the detecting step, selectively activates both the first tracking unit and the second tracking unit to operate in parallel on the same frames, or deactivates at least one of the first tracking unit and the second tracking unit.

14 . A non-transitory machine-readable medium that stores a program for causing, when executed by a computer, the computer to function as an image processing apparatus comprising:

a detecting unit configured to detect, in a target frame, one or more regions of a predetermined subject as candidate region(s);

a determining unit configured to determine a subject region to be tracked from the candidate region(s);

a difficulty computing unit configured to compute an evaluation value indicating a calculated difficulty score pertaining to difficulty of tracking, based on the subject region determined by the determining unit, wherein the difficulty score is incremented when an object class of the subject region includes objects for which shape changes significantly, and also incremented when a number of candidate regions having the same object class as the subject region is greater than a first preset value;

a tracking unit configured to execute tracking processing for estimating a position, in the target frame, of a region corresponding to the subject region to be tracked in a frame previous to the target frame, the tracking unit including a first tracking unit and a second tracking unit both operating on the same frames; and

a control unit configured to selectively determine, based on the evaluation value, operation of the first tracking unit and operation of the second tracking unit, on a subsequent frame to the target frame,

wherein the second tracking unit operates a deep-learning neural network and has a higher processing accuracy and a greater operational load than the first tracking unit that does not operate a deep-learning neural network, and

based on the evaluation value computed by the difficulty computing unit, the control unit selectively activates both the first tracking unit and the second tracking unit to operate in parallel on the same frames, or deactivates at least one of the first tracking unit and the second tracking unit.

15 . An image processing apparatus comprising:

one or more processors that execute a program stored in a memory and thereby function as:

a detecting unit configured to detect, in a target frame, one or more regions of a predetermined subject as candidate region(s);

a computing unit configured to, based on a detection result for the candidate region(s), compute an evaluation value pertaining to difficulty of determination processing for determining a region to be subject to tracking processing from the candidate region(s);

a difficulty computing unit configured to, based on a detection result for the candidate region(s), compute an evaluation value indicating a calculated difficulty score pertaining to difficulty of determination processing for determining a region to be subject to tracking processing from the candidate region(s), wherein the difficulty score is incremented when an object class of the subject region includes objects for which shape changes significantly, and also incremented when a number of candidate regions having the same object class as the subject region is greater than a first preset value;

a target determining unit, including a first determining unit and a second determining unit both operating on the same frames, configured to perform the determination processing on the candidate region(s) detected by the detecting unit; and

a control unit configured to selectively control, based on the evaluation value, operation of the first determining unit and operation of the second determining unit,

wherein the second tracking unit operates a deep-learning neural network and has a higher processing accuracy and a greater operational load than the first tracking unit that does not operate a deep-learning neural network, and

based on the evaluation value computed by the difficulty computing unit, the control unit selectively activates both to operate in parallel on the same frames, deactivates one, or deactivates both of the first determining unit and the second determining unit.

16 . The image processing apparatus according to claim 15 , wherein

the computing unit computes the evaluation value based on at least one of a size, a position, and a type of each of the candidate region(s), and a total number of the candidate region(s) related to a subject of the same type.

17 . The image processing apparatus according to claim 15 , wherein

the higher the evaluation value is, the higher the difficulty of the determination processing is, and

the control unit activates the second determining unit when the evaluation value is at least a first threshold.

18 . The image processing apparatus according to claim 15 , wherein

the higher the evaluation value is, the higher the difficulty of the determination processing is, and

the control unit deactivates the second determining unit and activates the first determining unit when the evaluation value is less than a second threshold.

19 . The image processing apparatus according to claim 15 , wherein

the higher the evaluation value is, the higher the difficulty of the determination processing, and

the control unit deactivates both the first determining unit and the second determining unit when the evaluation value is less than a third threshold.

20 . The image processing apparatus according to claim 15 , wherein

the second determining unit executes the determination processing using a pre-trained deep learning neural network and the first determination unit executes determination processing using a non-deep learning neural network.

21 . An image processing apparatus comprising:

one or more processors that execute a program stored in a memory and thereby function as:

a detecting unit configured to detect, in a target frame, one or more regions of a predetermined subject as candidate region(s);

a target determining unit configured to perform determination processing for determining a region to be subject to tracking processing from the candidate region(s) detected by the detecting unit, the target determining unit including a first determining unit and a second determining unit operating a deep-learning neural network and having a higher accuracy and a greater operational load than the first determining unit that does not operate a deep-learning neural network, wherein the first and second detecting units operate on the same frames; and

a control unit configured to selectively control, based on a detection result for the candidate region(s), operation of the first determining unit and operation of the second determining unit,

wherein based on a type and number of the detected candidate region(s), the control unit selectively activates both to operate in parallel on the same frames, deactivates one, or deactivates both of the first determining unit and the second determining unit.

22 . The image processing apparatus according to claim 21 , wherein

the control unit deactivates the second determining unit and activates the first determining unit when, among the detected candidate region(s), a total number of the candidate region(s) of a predetermined type is less than a first threshold.

23 . The image processing apparatus according to claim 21 , wherein

the control unit activates the second determining unit when, among the detected candidate region(s), a total number of the candidate region(s) of a first type is at least a first threshold and a total number of the candidate region(s) of a second type is at least a second threshold.

24 . The image processing apparatus according to claim 21 , wherein

the control unit deactivates both the first determining unit and the second determining unit when, among the detected candidate region(s), a total number of the candidate region(s) of any type is less than a third threshold.

25 . The image processing apparatus according to claim 21 , further comprising:

line-of-sight detecting unit for detecting a line of sight of a user of the image processing apparatus,

wherein the first determining unit executes the determination processing based on the line of sight detected by the line-of-sight detecting unit, and

the control unit deactivates the first determining unit when the second determining unit is active.

26 . The image processing apparatus according to claim 21 , wherein

the second determining unit executes the determination processing using a pre-trained deep learning neural network and the first determination unit executes determination processing using a non-deep learning neural network.

27 . A control method for an image processing apparatus comprising:

detecting, in a target frame, one or more regions of a predetermined subject as candidate region(s);

computing, based on a detection result for the candidate region(s), an evaluation value indicating a calculated difficulty score pertaining to difficulty of determination processing for determining a region to be subject to tracking processing from the candidate region(s), wherein the difficulty score is incremented when an object class of the subject region includes objects for which shape changes significantly, and also incremented when a number of candidate region(s) having the same object class as the subject region is greater than a first preset value; and

selectively controlling, based on the evaluation value, operation of target determining unit, included in the image processing apparatus, that performs the determination processing,

wherein the target determining unit includes a first determining unit and a second determining unit operating a deep-learning neural network and having a higher accuracy and a greater operational load than the first determining unit that does not operate a deep-learning neural network, wherein the first and second determination units operate on the same frames, and

the controlling, based on the evaluation value computed by the difficulty computing unit, selectively activates both to operate in parallel on the same frames, deactivates one, or deactivates both of the first determining unit and the second determining unit.

28 . A control method for an image processing apparatus comprising:

detecting, in a target frame, one or more regions of a predetermined subject as candidate region(s); and

selectively controlling, based on a detection result for the candidate region(s), operation of target determining unit, included in the image processing apparatus, that performs determination processing for determining a region to be subject to tracking processing from the candidate region(s),

wherein the target determining unit includes a first determining unit and a second determining unit operating a deep-learning neural network and having a higher accuracy and a greater operational load than the first determining unit that does not operate a deep-learning neural network, wherein the first and second determination units operate on the same frames, and

the controlling, based on a type and number of the detected candidate region(s), selectively activates both to operate in parallel on the same frames, deactivates one, or deactivates both of the first determining unit and the second determining unit.

29 . A non-transitory machine-readable medium storing a program for causing, when executed by a computer, the computer to function as an image processing apparatus comprising:

a detecting unit configured to detect, in a target frame, one or more regions of a predetermined subject as candidate region(s);

a computing unit configured to, based on a detection result for the candidate region(s), compute an evaluation value indicating a calculated difficulty score pertaining to difficulty of determination processing for determining a region to be subject to tracking processing from the candidate region(s), wherein the difficulty score is incremented when an object class of the subject region includes objects for which shape changes significantly, and also incremented when a number of candidate region(s) having the same object class as the subject region is greater than a first preset value;

a target determining unit, including a first determining unit and a second determining unit, configured to perform the determination processing on the candidate region(s) detected by the detecting unit; and

a control unit configured to selectively control, based on the evaluation value, operation of the first determining unit and operation of the second determining unit,

wherein the second determining unit operates a deep-learning neural network and has a higher processing accuracy and a greater operational load than the first determining unit that does not operate a deep-learning neural network, and

based on the evaluation value computed by the difficulty computing unit, the control unit selectively activates both to operate in parallel on the same frames, deactivates one, or deactivates both of the first determining unit and the second determining unit.

30 . A non-transitory machine-readable medium storing a program for causing, when executed by a computer, the computer to function as an image processing apparatus comprising:

a detecting unit configured to detect, in a target frame, one or more regions of a predetermined subject as candidate region(s);

a target determining unit configured to perform determination processing for determining a region to be subject to tracking processing from the candidate region(s) detected by the detecting unit, the target determining unit including a first determining unit and a second determining unit operating a deep-learning neural network and having a higher accuracy and a greater operational load than the first determining unit that does not operate a deep-learning neural network; and

a control unit configured to selectively control, based on a detection result for the candidate region(s), operation of the first determining unit and operation of the second determining unit operating on the same frames,

wherein based on a type and number of the detected candidate region(s), the control unit selectively activates both to track in parallel on the same frames, deactivates one, or deactivates both of the first determining unit and the second determining unit.