Computationally adaptive occlusion aware hand landmark estimation
A method performed by an electronic device for estimating a landmark point of a body part of subject by electronic device is provided. The method includes generating, by the electronic device, an initial coarse estimation of the landmark point of the body part using a light-weight deep neural network, determining, by the electronic device, an occluded region of the body part based on the generated initial coarse estimation of the landmark point using a segmentation mask, estimating, by the electronic device, the occlusion probability for the landmark point in the at least one occluded region and the generated initial coarse estimation, determining, by the electronic device, a correction factor for applying on the generated initial coarse estimation as a measure of the estimated occlusion probability, and selecting, by the electronic device, a pre-defined number of neural networks by applying the determined correction factor for processing the at least one occluded region and the generated initial coarse estimation to generate final estimation of the landmark point.
1 . A method performed by an electronic device for estimating at least one landmark point of a body part of a subject, the method comprising:
generating, by the electronic device, an initial coarse estimation of the at least one landmark point of the body part using a light-weight deep neural network;
determining, by the electronic device, at least one occluded region of the body part based on the generated initial coarse estimation of the at least one landmark point using a segmentation mask;
estimating, by the electronic device, an occlusion probability for the at least one landmark point in the at least one occluded region and the generated initial coarse estimation;
determining, by the electronic device, a correction factor for applying on the generated initial coarse estimation as a measure of the estimated occlusion probability; and
selecting, by the electronic device, a pre-defined number of neural networks by applying the determined correction factor, for processing the at least one occluded region and the generated initial coarse estimation to generate final estimation of the at least one landmark point.
2 . The method of claim 1 , wherein the pre-defined number of the neural networks in each neural network sequence is inversely proportional to the determined correction factor of the at least one landmark point estimation.
3 . The method of claim 1 , wherein light-weight deep neural network predicts a confidence score associated with the generated initial coarse estimation of at least one landmark point of the body part.
4 . The method of claim 1 , further comprising:
generating, by the electronic device, the segmentation mask, wherein the generating of the segmentation mask comprises:
estimating, by the electronic device, a hand bounding box;
performing, by the electronic device, a skin segmentation;
performing, by the electronic device, a connected component analysis for the hand bounding box and the skin segmentation to join at least one component of the body part of the subject; and
generating, by the electronic device, the segmentation mask based on the at least one joined component of the body part of the subject.
5 . The method of claim 1 ,
wherein the occlusion probability for the at least one landmark point is estimated from an occlusion map, and
wherein the occlusion probability differentiates between occlusion due to at least one external object, a self-occlusion by the body part of the subject and a self-occlusion by another body part of the subject.
6 . An electronic device, comprising:
memory storing one or more computer programs;
and
one or more processors communicatively coupled to the memory,
wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
generate an initial coarse estimation of at least one landmark point of a body part of a subject using a light-weight deep neural network,
determine at least one occluded region of the body part based on the generated initial coarse estimation of the at least one landmark points using a segmentation mask,
estimate an occlusion probability for the at least one landmark point in the at least one occluded region and the generated initial coarse estimation,
determine a correction factor for applying on the generated initial coarse estimation as a measure of the estimated occlusion probability, and
select a pre-defined number of neural networks by applying the correction factor, for processing the at least one occluded region and the generated initial coarse estimation to generate final estimation of the at least one landmark point.
7 . The electronic device of claim 6 , wherein the pre-defined number of the neural networks in each neural network sequence is inversely proportional to the correction factor of the at least one landmark point estimation.
8 . The electronic device of claim 6 , wherein the light-weight deep neural network predicts a confidence score associated with the generated initial coarse estimation of at least one landmark point of the body part.
9 . The electronic device of claim 6 ,
wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to generate the segmentation mask, and
wherein, to generate the segmentation mask, the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors, cause the electronic device to:
estimate a hand bounding box,
perform a skin segmentation,
perform a connected component analysis for the hand bounding box and the skin segmentation to join at least one component of the body part of the subject, and
generate the segmentation mask based on the at least one joined component of the body part of the subject.
10 . The electronic device of claim 6 ,
wherein the occlusion probability for the at least one landmark point is estimated from an occlusion map, and
wherein the occlusion probability differentiates between occlusion due to at least one external object, a self-occlusion by the body part of the subject and a self-occlusion by another body part of the subject.
11 . The electronic device of claim 8 , wherein the segmentation mask is received together with the confidence score by an occlusion probability estimation engine.
12 . The electronic device of claim 11 , wherein the occlusion probability estimation engine is configured to receive a skin map(S) and a hand mask (H) to determine an external occlusion map (O) equal to an intersection of S and H.
13 . The electronic device of claim 12 , wherein the occlusion probability estimation engine is configured to receive confidence values (C) [1×21] vector and coarse hand landmark (HLM) estimates (Pos) [2×21] vector to determine O[Pos]==0.
14 . The electronic device of claim 13 , wherein, when O[Pos] is zero, the occlusion probability estimation engine is configured to check a confidence value.
15 . The electronic device of claim 14 , wherein, when confidence value is greater than a threshold, the occlusion probability estimation engine is configured to set the occlusion probability to 0.
16 . One or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform operations for estimating at least one landmark point of a body part of a subject, the operations comprising:
generating, by the electronic device, an initial coarse estimation of the at least one landmark point of the body part using a light-weight deep neural network;
determining, by the electronic device, at least one occluded region of the body part based on the generated initial coarse estimation of the at least one landmark point using a segmentation mask;
estimating, by the electronic device, an occlusion probability for the at least one landmark point in the at least one occluded region and the generated initial coarse estimation;
determining, by the electronic device, a correction factor for applying on the generated initial coarse estimation as a measure of the estimated occlusion probability; and
selecting, by the electronic device, a pre-defined number of neural networks by applying the determined correction factor, for processing the at least one occluded region and the generated initial coarse estimation to generate final estimation of the at least one landmark point.
17 . The one or more non-transitory computer-readable storage media of claim 16 , wherein the pre-defined number of the neural networks in each neural network sequence is inversely proportional to the correction factor of the at least one landmark point estimation.
18 . The one or more non-transitory computer-readable storage media of claim 16 , wherein light-weight deep neural network predicts a confidence score associated with the generated initial coarse estimation of at least one landmark point of the body part.
19 . The one or more non-transitory computer-readable storage media of claim 16 , the operations further comprising:
generating, by the electronic device, the segmentation mask,
wherein the generating of the segmentation mask comprises:
estimating, by the electronic device, a hand bounding box;
performing, by the electronic device, a skin segmentation;
performing, by the electronic device, a connected component analysis for the hand bounding box and the skin segmentation to join at least one component of the body part of the subject; and
generating, by the electronic device, the segmentation mask based on the at least one joined component of the body part of the subject.
20 . The one or more non-transitory computer-readable storage media of claim 16 ,
wherein the occlusion probability for the at least one landmark point is estimated from an occlusion map, and
wherein the occlusion probability differentiates between occlusion due to at least one external object, a self-occlusion by the body part of the subject and a self-occlusion by another body part of the subject.