System and method for determining a grasping hand model
Method for determining a grasping hand model suitable for grasping an object by receiving an image including at least one object; obtaining an object model estimating a pose and shape of the object from the image of the object; selecting a grasp class from a set of grasp classes by means of a neural network, with a cross entropy loss, thus, obtaining a set of parameters defining a coarse grasping hand model; refining the coarse grasping hand model, by minimizing loss functions referring to the parameters of the hand model for obtaining an operable grasping hand model while minimizing the distance between the finger of the hand model and the surface of the object and preventing interpenetration; and obtaining a mesh of the hand represented by the enhanced set of parameters.
1. A system for determining parameters of a hand model suitable for grasping an object, comprising:
a first neural network for segmenting the object in an image;
a second neural network for predicting parameters of the hand model that define a pose for grasping the segmented object; and
a third neural network for refining the predicted parameters of the hand model to fit the segmented object by maximizing a number of contact points between the object in the image and the hand model while minimizing interpenetration;
wherein the refined predicted parameters of the hand model represent how a hand may grasp the object.
2. The system according to claim 1 , wherein said first neural network segments the object in an image by estimating a pose and a 3D shape of the object from the image of the object.
3. The system according to claim 1 , wherein said second neural network predicts parameters of the hand model by predicting a grasp class from a set of grasp classes to obtain a set of parameters defining a coarse grasping hand model.
4. The system according to claim 1 , wherein said third neural network refines the predicted parameters of the hand model by minimizing loss functions referring to the parameters of the hand model for obtaining an operable grasping hand model while minimizing a distance between the fingers of the hand model and a surface of the object and preventing interpenetration.
5. The system according to claim 1 , wherein said second neural network is a Convolutional Neural Network, with a cross entropy loss L class defined as:
L class =Σ c∈K C o,c log(1− P o,c )
wherein C represents a grasp type for the particular object (o), c represents the grasp classes among the K possible grasps classes, and P represents pose predictions for the particular object (o).
6. The system according to claim 1 , wherein said third neural network obtains a representation of a hand grasping the object by using the refined predicted parameters of the hand model, the representation being a mesh of the refined hand model.
7. The system according to claim 1 , wherein said third neural network uses a MANO model, being a 51 degrees of freedom model of a possible human hand, for refining the predicted parameters of the hand model.
8. The system according to claim 1 , wherein said third neural network evaluates the grasping hand model by calculating at least one evaluating metric of an analytical grasp metric, which computes an approximation of the minimum force to be applied to break the grasp stability; an average number of contact fingers, wherein numerous contact points between hand and object favor a strong grasp; a hand-object interpenetration volume, wherein object and hand are voxelized, and the volume shared by both 3D models is computed; a simulation displacement of the object mesh subjected to gravity; and a percentage of graspable objects for which an operable grasp could be predicted, being an operable grasp the one with at least two contact points and no interpenetration.
9. The system according to claim 8 , wherein said third neural network (a) randomly rotates an object model; (b) obtains a grasping hand model for each rotated object model; (c) evaluates each rotated grasping hand model using evaluating metrics; and (d) selects the rotated grasping hand models having the highest score.
10. The system according to claim 1 , wherein said second neural network estimates a pose and shape of the object by using an object reconstruction phase for obtaining a cloud of points representing the object form the obtained image.
11. The system according to claim 1 , wherein said image includes more than one object;
said first neural network segmenting each object in said image and estimating a 3D shape of each segmented object;
said second neural network predicting parameters of the hand model that define a pose for grasping each segmented object;
said third neural network refining the predicted parameters of the hand model to fit each segmented object.
12. The system according to claim 1 , wherein said second neural network selects a grasp class by utilizing a phase of predicting an increment of translation and rotation of the hand model and a modified coarse configuration of the hand model.
13. The system according to claim 1 , wherein said third neural network refines the coarse grasping model by:
(a) selecting at least one articulation (i) of the hand model;
(b) calculating an arc (Ai) between a finger (j) of the hand model and close object vertices (O),
D θ ←min i (min k (∥ A i θ ,O k ∥ 2 ))
(c) estimating the angle the finger needs to be rotated to collide with the object, rotating the articulation for minimizing the arc, thus, reducing the distance between the hand model and the object vertices, including a hyperparameter for controlling the interpenetration of the hand model into the object,
γ j ′←arg min θ D θ +δ, ∀θs.t. D θ <t d
(d) defining the following loss functions:
L
a
r
c
=
1
❘
"\[LeftBracketingBar]"
J
❘
"\[RightBracketingBar]"
∑
j
∈
J
D
θ
j
L
γ
←
∑
j
J
❘
"\[LeftBracketingBar]"
❘
"\[LeftBracketingBar]"
γ
j
′
-
γ
j
❘
"\[RightBracketingBar]"
❘
"\[RightBracketingBar]"
2
;
and
(e) minimizing the loss functions defined.
14. The system according to claim 13 , wherein said third neural network refines the coarse grasping model for each articulation sequentially from the knuckle to the tip for each finger.
15. The system according to claim 1 , wherein said third neural network refines the coarse grasping model by minimizing a loss function from a distance between the hand vertices and the target object, wherein is considered that there is a contact when the distance is below to 2 mm, defined by
L
cont
=
1
❘
"\[LeftBracketingBar]"
V
cont
❘
"\[RightBracketingBar]"
∑
v
∈
V
cont
min
k
❘
"\[LeftBracketingBar]"
❘
"\[LeftBracketingBar]"
v
,
O
k
t
❘
"\[RightBracketingBar]"
❘
"\[RightBracketingBar]"
2
.
16. The system according to claim 1 , wherein said third neural network refines the coarse grasping model by minimizing a loss function from a distance of interpenetration between a vertex of the hand model and the object, defined by
L
int
=
1
❘
"\[LeftBracketingBar]"
V
i
❘
"\[RightBracketingBar]"
∑
j
❘
"\[LeftBracketingBar]"
O
❘
"\[RightBracketingBar]"
∑
v
∈
V
i
min
k
v
,
O
k
j
2
.
17. The system according to claim 1 , wherein said third neural network refines the coarse grasping model by minimizing a loss function from a distance below a table plane, between a vertex of the hand model and the table plane, wherein the distance is favored to be positive, defined by L p =Σ v v min (0, |(v−p p )·v p |).
18. The system according to claim 1 , wherein said third neural network refines the coarse grasping model by minimizing a loss function from an adversarial loss function, using a Wasserstein loss including a gradient penalty loss, defined by L adv =−E H,R,T˜p(H,R,T) [D(G(I))]+E H,R,T˜p(H,R,T) [D(H*, R*, T*)].
19. The system according to claim 1 , wherein the hand is a human hand.
20. A method for determining parameters of a hand model suitable for grasping an object, comprising:
(a) receiving an object segmented in an image;
(b) predicting, with a first neural network, parameters of the hand model that define a pose for grasping the segmented object; and
(c) refining, with a second neural network, the predicted parameters of the hand model to fit the segmented object by maximizing a number of contact points between the object in the image and the hand model while minimizing interpenetration;
wherein the refined predicted parameters of the hand model represent how a hand may grasp the object.
21. The method according to claim 20 , wherein said receiving is performed using a third neural network.
22. The method according to claim 20 , wherein (b) predicts parameters of the hand model by predicting a grasp class from a set of grasp classes to obtain a set of parameters defining a coarse grasping hand model.
23. The method according to claim 20 , wherein (c) refines the predicted parameters of the hand model by minimizing loss functions referring to the parameters of the hand model for obtaining an operable grasping hand model while minimizing the distance between the fingers of the hand model and the surface of the object and preventing interpenetration.
24. The method according to claim 20 , further comprising:
(d) obtaining a representation of a hand grasping the object by using the refined hand model.
25. The method according to claim 20 , wherein the first neural network is a Convolutional Neural Network, with a cross entropy loss L class defined as:
L class =Σ c∈K C o,c log (1 −P o,c );
wherein C represents a grasp type for the particular object (o), c represents the grasp classes among the K possible grasps classes, and P represents pose predictions for the particular object (o).
26. The method according to claim 24 , wherein the representation obtained in (d) is a mesh of the refined hand model.
27. The method according to claim 20 , wherein the hand model is represented by using a MANO model, being a 51 degrees of freedom model of a possible human hand.
28. The method according to claim 20 , further comprising:
(d) evaluating the grasping hand model obtained by calculating at least one evaluating metric of an analytical grasp metric, which computes an approximation of the minimum force to be applied to break the grasp stability; an average number of contact fingers, wherein numerous contact points between hand and object favor a strong grasp; a hand-object interpenetration volume, wherein object and hand are voxelized, and the volume shared by both 3D models is computed; a simulation displacement of the object mesh subjected to gravity; and a percentage of graspable objects for which an operable grasp could be predicted, being an operable grasp the one with at least two contact points and no interpenetration.
29. The method according to claim 28 , further comprising:
(e) randomly rotating an object model;
(f) obtaining a grasping hand model for each rotated object model;
(g) evaluating each rotated grasping hand model using evaluating metrics; and
(h) selecting the rotated grasping hand models having the highest score.
30. The method according to claim 21 , wherein estimating a pose and shape of the object comprises an object reconstruction phase for obtaining a cloud of points representing the object form the obtained image.
31. The method according to claim 20 , wherein the image comprises more than one object and the method further comprises the step of repeating (a) to (c) for each object in the image, wherein the objects are known.
32. The method according to claim 20 , wherein (b) selects a grasp class by utilizing a phase of predicting an increment of translation and rotation of the hand model and a modified coarse configuration of the hand model.
33. The method according to claim 20 , wherein (c) includes:
(c1) selecting at least one articulation (i) of the hand model;
(c2) calculating an arc (Ai) between a finger (j) of the hand model and close object vertices (O),
D θ ←min i (min k (∥i A i θ , O k ∥ 2 ));
(c3) estimating the angle the finger needs to be rotated to collide with the object, rotating the articulation for minimizing the arc, thus, reducing the distance between the hand model and the object vertices, including a hyperparameter for controlling the interpenetration of the hand model into the object,
γ j ′←arg min θ D θ +δ,∀θs.t. D θ <t d ;
(c 4 ) defining the following loss functions:
L
arc
=
1
❘
"\[LeftBracketingBar]"
J
❘
"\[RightBracketingBar]"
∑
j
∈
J
D
θ
j
L
γ
←
∑
j
J
γ
j
′
-
γ
j
2
;
and
(c5) minimizing the loss functions defined.
34. The method according to claim 33 , wherein (c) further includes (c6) repeating phases (c2) and (c3) for each articulation sequentially from the knuckle to the tip for each finger.
35. The method according to claim 20 , wherein said (c) further comprises minimizing a loss function from a distance between the hand vertices and the target object, wherein is considered that there is a contact when the distance is below to 2 mm, defined by
L
cont
=
1
❘
"\[LeftBracketingBar]"
V
cont
❘
"\[RightBracketingBar]"
∑
v
∈
V
cont
min
k
v
,
O
k
t
2
.
36. The method according to claim 20 , wherein said (c) further comprises minimizing a loss function from a distance of interpenetration between a vertex of the hand model and the object, defined by
L
int
=
1
❘
"\[LeftBracketingBar]"
V
i
❘
"\[RightBracketingBar]"
∑
j
❘
"\[LeftBracketingBar]"
O
❘
"\[RightBracketingBar]"
∑
v
∈
V
i
min
k
v
,
O
k
j
2
.
37. The method according to claim 20 , wherein said (c) further comprises minimizing a loss function from a distance below a table plane, between a vertex of the hand model and the table plane, wherein the distance is favored to be positive, defined by L p =Σ v v min (0, |(v−p p )·v p |).
38. The method according to claim 20 , wherein said (c) further comprises minimizing a loss function from an adversarial loss function, using a Wasserstein loss including a gradient penalty loss, defined by L adv =−E H,R,T˜p(H,R,T) [D(G(I))]+E H,R,T˜p(H,R,T) [D(H*, R*, T*)].
39. The method according to claim 20 , wherein the hand is a human hand.
40. The method according to claim 20 , wherein (a) segments an object in an image by estimating a pose and shape of the object from the image of the object.