Human pose estimation with data driven belief propagation
A statistical formulation estimates two-dimensional human pose from single images. This is based on a Markov network and on inferring pose parameters from cues such as appearance, shape, edge, and color. A data-driven belief propagation Monte Carlo algorithm performs efficient Bayesian inferencing within a rigorous statistical framework. Experimental results demonstrate the effectiveness of the method in estimating human pose from single images.
1. A method for estimating a pose of a human subject within a digital image, the method comprising:
receiving one or more training digital images representing a plurality of human subjects, each human subject having at least one training body part;
labeling each of said training body parts with a quadrangle;
automatically creating a two-dimensional (2-D) shape model of each of said training body parts based on said quadrangles, each shape model having at least one associated link point identifying a point of attachment with a paired link point associated with an adjacent shape model;
receiving a target digital image representing a target human subject having at least one target body part; and
estimating a pose parameter of said target body part using said 2-D shape models, data driven importance sampling, a Markov network and a belief propagation Monte Carlo algorithm.
2. The method of claim 1 , further comprising applying a principal component analysis to said 2-D shape models for dimensionality reduction.
3. The method of claim 1 , further comprising storing said 2-D shape models.
4. The method of claim 1 , wherein said belief propagation Monte Carlo algorithm is iterative and includes message passing based on importance samples.
5. The method of claim 1 , wherein said belief propagation Monte Carlo algorithm includes performing local inference tasks in parallel based on the data driven importance sampling.
6. A method for estimating a pose of a human subject within a digital image, the method comprising:
receiving one or more training digital images representing a plurality of human subjects, each human subject having at least one training body part;
labeling each of said training body parts with a quadrangle;
automatically creating a two-dimensional (2-D) shape model of each of said training body parts based on said quadrangles, each shape model having at least one associated link point identifying a point of attachment with a paired link point associated with an adjacent shape model;
receiving a target digital image representing a target human subject having at least one target body part; and
estimating a pose parameter of said target body part using said 2-D shape models, data driven importance sampling, a Markov network and a belief propagation Monte Carlo algorithm,
wherein said Markov network comprises:
a first set of nodes, each representing a pose parameter of one of a set of target body parts;
a second set of nodes, each representing an observation of one of the set of target body parts;
a set of undirected links, each connecting two of said first set of nodes and modeling a constraint between two adjacent body parts of the set of target body parts according to a first function; and
a set of directed links, each directed from one of said first set of nodes to one of said second set of nodes and describing a likelihood of a corresponding observation according to a second function,
wherein the second function is different than the first function.
7. The method of claim 6 , wherein said belief propagation Monte Carlo algorithm is iterative and comprises message passing based on importance samples.
8. The method of claim 7 , wherein said first function is
ψ
ij
(
x
i
,
x
j
)
∝
exp
(
-
Pt
~
ij
-
Pt
~
ji
2
2
σ
ij
2
)
,
wherein
:
x i and x j represent said two of said first set of nodes,
∥•∥ is a Euclidean distance function,
exp is a natural exponential function,
σ 2 ij is a variance learned from said labeled body parts,
{tilde over (P)}t ij is a link point of an i th to a j th body part, and
{tilde over (P)}t jt is a link point of a j th to an i th body part; and
wherein said second function is φ i (z i |x i )=max(ε α (R) (x i ), ε α (G) (x i ), ε α (B) (x i )), wherein:
x i represents said one of said first set of nodes,
z i represents said one of said second set of nodes,
max (•) represents a maximum value function, and
ε α (R) (x i ), ε α (G) (x i ) and ε α (B) (x i ) represent a set of steered edge responses corresponding to a first, a second and a third color channel, respectively.
9. The method of claim 8 , further comprising:
detecting a face region of said target human subject;
building a skin color model from said detected face region;
building an edge map of a torso region of said target human subject; and
extracting a set of substantially horizontal and substantially vertical line segments from said edge map.
10. The method of claim 9 , wherein said importance samples are drawn from importance functions corresponding to a head pose, an arm pose, a leg pose and a torso pose, wherein:
the head pose is derived using said face region;
the arm and leg poses are derived using said skin color model; and
the torso pose is derived using said line segments.
11. The method of claim 10 , wherein a joint posterior distribution of said Markov network is
P
(
X
❘
Y
)
∝
∏
(
i
,
j
)
∈
E
ψ
i
,
j
(
x
i
,
x
j
)
∏
i
∈
υ
ϕ
i
(
z
i
❘
x
i
)
,
wherein X represents the set of all x i and Z represents the set of all z i .
12. The method of claim 6 , wherein said belief propagation Monte Carlo algorithm includes performing local inference tasks in parallel based on the data driven importance sampling.
13. An apparatus for estimating a pose of a human subject within a digital image, the apparatus comprising:
an input module configured to:
receive one or more training digital images representing a plurality of human subjects, each human subject having at least one training body part, and
receive a target digital image representing a target human subject having at least one target body part; and
a processor module configured to:
label each of said training body parts with a quadrangle,
automatically create a two-dimensional (2-D) shape model of each of said training body parts based on said quadrangles, each shape model having at least one associated link point identifying a point of attachment with a paired link point associated with an adjacent shape model, and
estimate a pose parameter of said target body part using said 2-D shape models, data driven importance sampling, a Markov network and a belief propagation Monte Carlo algorithm.
14. The apparatus of claim 13 , wherein said belief propagation Monte Carlo algorithm is iterative and includes message passing based on importance samples.
15. The apparatus of claim 13 , wherein said belief propagation Monte Carlo algorithm includes performing local inference tasks in parallel based on the data driven importance sampling.
16. An apparatus for estimating a pose of a human subject within a digital image, the apparatus comprising:
means for receiving one or more training digital images representing a plurality of human subjects, each human subject having at least one training body part;
means for labeling each of said training body parts with a quadrangle;
means for automatically creating a two-dimensional (2-D) shape model of each of said training body parts based on said quadrangles, each shape model having at least one associated link point identifying a point of attachment with a paired link point associated with an adjacent shape model;
means for receiving a target digital image representing a target human subject having at least one target body part; and
means for estimating a pose parameter of said target body part using said 2-D shape models, data driven importance sampling, a Markov network and a belief propagation Monte Carlo algorithm.
17. The apparatus of claim 16 , wherein said belief propagation Monte Carlo algorithm is iterative and includes message passing based on importance samples.
18. The apparatus of claim 16 , wherein said belief propagation Monte Carlo algorithm includes performing local inference tasks in parallel based on the data driven importance sampling.
19. A computer program product, comprising a computer-readable medium having computer program instructions embodied thereon to cause a computer processor to implement a method for estimating a pose of a human subject within a digital image, the method comprising:
receiving one or more training digital images representing a plurality of human subjects, each human subject having at least one training body part;
labeling each of said training body parts with a quadrangle;
automatically creating a two-dimensional (2-D) shape model of each of said training body parts based on said quadrangles, each shape model having at least one associated link point identifying a point of attachment with a paired link point associated with an adjacent shape model;
receiving a target digital image representing a target human subject having at least one target body part; and
estimating a pose parameter of said target body part using said 2-D shape models, data driven importance sampling, a Markov network and a belief propagation Monte Carlo algorithm.
20. The computer program product of claim 19 , wherein said belief propagation Monte Carlo algorithm is iterative and includes message passing based on importance samples.
21. The computer program product of claim 19 , wherein said belief propagation Monte Carlo algorithm includes performing local inference tasks in parallel based on the data driven importance sampling.