Adversarial 3D deformations learning
A computer-implemented method of machine-learning. The method includes obtaining a dataset of 3D modeled objects representing real-world objects. The method further includes learning, based on the dataset, a generative neural network. The generative neural network is configured for generating a deformation basis of an input 3D modeled object. The learning includes an adversarial training.
1 . A computer-implemented method of machine-learning, the method comprising:
obtaining a dataset of 3D modeled objects representing real-world objects; and
learning, based on the dataset, a generative neural network configured for generating a deformation basis of an input 3D modeled object, the learning including an adversarial training, the deformation basis being a set of vectors each being a direction of deformation of the input 3D modeled object such that the vectors are linearly combined to form a linear deformation of the input 3D modeled object,
wherein the method further includes performing one or more of:
a depth frame reconstruction process by deformation of a 3D mesh to fit a depth map, the depth frame reconstruction process including:
obtaining the 3D mesh, the 3D mesh representing a first real-world object,
obtaining the depth map, the depth map being obtained from depth physical measurements on a second real-world object and representing the second real-world object, the second real-world object being close to the first real-world object,
applying the generative neural network to the 3D mesh, which yields a deformation basis of the 3D mesh,
optimizing coefficients of a linear combination of the deformation basis vector so that deforming the 3D mesh by the linear combination fits the depth map; and/or
a shape reconstruction process from a digital image by deformation of a 3D mesh to fit the digital image, the shape reconstruction process including:
obtaining the 3D mesh, the 3D mesh representing a first real-world object,
obtaining a digital image of a second real-world object, the second real-world object being close to the first real-world object,
applying the generative neural network to the 3D mesh, which yields a deformation basis of the 3D mesh,
optimizing coefficients of a linear combination of the deformation basis vector so that deforming the 3D mesh by the linear combination fits the digital image; and/or
a shape matching process by deformation of a first 3D modeled object to fit a second 3D modeled object, the shape matching process including:
obtaining the first 3D modeled object and the second 3D modeled object, the second 3D modeled object being obtained from a scan of a real-world object,
applying the generative neural network to the first 3D modeled object to compute a deformation basis of the first 3D modeled object, and
optimizing coefficients of a linear combination of the deformation basis vectors in order to deform the first 3D modeled object so that it matches the second 3D modeled object; and/or
a segmentation process of segmentation of a 3D modeled object representing a scanned real-world object, the segmentation process including:
obtaining, from a physical scan of the real-world object, the 3D modeled object, the 3D modeled object corresponding to scan data representing the real-world object,
applying the generative neural network to the 3D modeled object, thereby yielding a deformation basis of the 3D modeled object consisting of vectors each representing a local deformation of the 3D modeled object, and
determining local parts of the 3D modeled object by inferring a respective local part for each vector of the basis, by:
for each vector, deforming the object according to the vector multiplied by a unitary coefficient, and
for each unitary deformation, determining points/vertices of the 3D modeled object which have been deformed above a predefined threshold, the determined points forming a segment of the segmentation of the 3D modeled object,
wherein a generative loss is of a type:
E
2
=
∑
i
=
1
N
log
(
1
-
d
(
y
i
)
)
-
λ
m
D
C
H
(
x
i
,
y
i
)
+
δ
m
n
∑
j
=
1
n
(
g
w
(
h
,
x
i
,
1
)
j
2
⋮
g
w
(
h
,
x
i
,
m
)
j
2
)
1
,
where:
d is a discriminative neural network;
x 1 , . . . , x N are 3D modeled objects forming a mini-batch of the dataset, N being a number of objects in the mini-batch;
(
g
w
(
h
,
x
i
,
1
)
j
⋮
g
w
(
h
,
x
i
,
m
)
j
)
,
1
≤
j
≤
n
,
is the deformation basis generated by the generative neural network at object x i , n being a number of vectors of the deformation basis, x i,1 . . . x i,m being points or vertices of x i ;
m is a number of points or vertices of each 3D modeled object of the mini-batch;
y i is a linear deformation of x i spanned by the deformation basis generated by the generative neural network at object x i ;
D CH is a Chamfer distance; and
λ and δ are parameters of the generative loss.
2 . The computer-implemented method of claim 1 , wherein the method further includes training, by minimizing a discriminative loss, the discriminative neural network configured for determining whether or not a 3D modeled object resulting from application of a linear combination of a deformation basis generated by the generative neural network is realistic or synthetic.
3 . The computer-implemented method of claim 2 , wherein the learning of the generative neural network further includes training the generative neural network to fool the discriminative neural network.
4 . The computer-implemented method of claim 2 , wherein the learning of the generative neural network further includes minimizing the generative loss, the method alternating minimizations of the generative loss and minimizations of the discriminative loss.
5 . The computer-implemented method of claim 4 , wherein a minimization of the discriminative loss is based on random linear combinations of deformation basis generated by the generative neural network.
6 . The method of claim 5 , wherein the learning is carried out mini-batch by mini-batch, and wherein for each mini-batch, the learning includes, for each 3D modeled object of the mini-batch, generating a random linear combination of the deformation basis of the 3D modeled object generated by the generative neural network, by randomly sampling coefficients of the linear combination.
7 . The method of claim 6 , wherein the random sampling includes a random sampling of a number of non-zero coefficients and/or, for each non-zero coefficient, a random sampling of a coefficient value according to a pseudo-bounded probability distribution.
8 . The method of claim 4 , wherein the generative loss includes a term rewarding fooling of the discriminative neural network by a linear combination of the deformation basis generated by the generative neural network.
9 . The method of claim 8 , wherein the generative loss further includes a term rewarding a disparity between an input 3D modeled object and a linear deformation of the input 3D modeled object spanned by the deformation basis generated by the generative neural network.
10 . The method of claim 8 , wherein the generative loss further includes a term rewarding sparsity of the deformation basis generated by the generative neural network.
11 . A computer-implemented method of applying a generative neural network learned according to machine-learning, the method comprising:
applying the generative neural network including using the learned generative neural network by performing one or any combination of the following:
a depth frame reconstruction process by deformation of a 3D mesh to fit a depth map, the depth frame reconstruction process including:
obtaining the 3D mesh, the 3D mesh representing a first real-world object,
obtaining the depth map, the depth map being obtained from depth physical measurements on a second real-world object and representing the second real-world object, the second real-world object being close to the first real-world object,
applying the generative neural network to the 3D mesh, which yields a deformation basis of the 3D mesh,
optimizing coefficients of a linear combination of the deformation basis vector so that deforming the 3D mesh by the linear combination fits the depth map; and/or
a shape reconstruction process from a digital image by deformation of a 3D mesh to fit the digital image, the shape reconstruction process including:
obtaining the 3D mesh, the 3D mesh representing a first real-world object,
obtaining a digital image of a second real-world object, the second real-world object being close to the first real-world object,
applying the generative neural network to the 3D mesh, which yields a deformation basis of the 3D mesh,
optimizing coefficients of a linear combination of the deformation basis vector so that deforming the 3D mesh by the linear combination fits the digital image; and/or
a shape matching process by deformation of a first 3D modeled object to fit a second 3D modeled object, the shape matching process including:
obtaining the first 3D modeled object and the second 3D modeled object, the second 3D modeled object being obtained from a scan of a real-world object,
applying the generative neural network to the first 3D modeled object to compute a deformation basis of the first 3D modeled object, and
optimizing coefficients of a linear combination of the deformation basis vectors in order to deform the first 3D modeled object so that it matches the second 3D modeled object; and/or
a segmentation process of segmentation of a 3D modeled object representing a scanned real-world object, the segmentation process including:
obtaining, from a physical scan of the real-world object, the 3D modeled object, the 3D modeled object corresponding to scan data representing the real-world object,
applying the generative neural network to the 3D modeled object, thereby yielding a deformation basis of the 3D modeled object consisting of vectors each representing a local deformation of the 3D modeled object, and
determining local parts of the 3D modeled object by inferring a respective local part for each vector of the basis, by:
for each vector, deforming the object according to the vector multiplied by a unitary coefficient, and
for each unitary deformation, determining points/vertices of the 3D modeled object which have been deformed above a predefined threshold, the determined points forming a segment of the segmentation of the 3D modeled object,
wherein a generative loss is of a type:
E
2
=
∑
i
=
1
N
log
(
1
-
d
(
y
i
)
)
-
λ
m
D
C
H
(
x
i
,
y
i
)
+
δ
m
n
∑
j
=
1
n
g
w
(
h
,
x
i
,
1
)
j
2
⋮
g
w
(
h
,
x
i
,
m
)
j
2
1
,
where:
d is a discriminative neural network;
x 1 , . . . , x N are 3D modeled objects forming a mini-batch of the dataset, N being a number of objects in the mini-batch;
(
g
w
(
h
,
x
i
,
1
)
j
⋮
g
w
(
h
,
x
i
,
m
)
j
)
,
1≤j≤n, is the deformation basis generated by the generative neural network at object x i , n being a number of vectors of the deformation basis, x i,1 . . . x i,m being points or vertices of x i ;
m is a number of points or vertices of each 3D modeled object of the mini-batch;
y i is a linear deformation of x i spanned by the deformation basis generated by the generative neural network at object x i ;
D CH is a Chamfer distance; and
λ and δ are parameters of the generative loss, and
wherein the machine-learning includes:
obtaining a dataset of 3D modeled objects representing real-world objects; and
learning, based on the dataset, the generative neural network configured for generating a deformation basis of an input 3D modeled object, the learning includes an adversarial training.
12 . A device comprising:
a processor; and
a non-transitory data storage medium having recorded thereon a computer program comprising instructions for machine-learning that when executed with a processor causes the processor to be configured to:
obtain a dataset of 3D modeled objects representing real-world objects, and
learn, based on the dataset, a generative neural network configured for generating a deformation basis of an input 3D modeled object, the learning comprising an adversarial training, the deformation basis being a set of vectors each being a direction of deformation of the input 3D modeled object such that the vectors are linearly combined to form a linear deformation of the input 3D modeled object,
wherein the processor is further configured to perform one or more of:
a depth frame reconstruction process by deformation of a 3D mesh to fit a depth map, the depth frame reconstruction process causing the processor to be configured to:
obtain the 3D mesh, the 3D mesh representing a first real-world object,
obtain the depth map, the depth map being obtained from depth physical measurements on a second real-world object and representing the second real-world object, the second real-world object being close to the first real-world object,
apply the generative neural network to the 3D mesh, which yields a deformation basis of the 3D mesh,
optimize coefficients of a linear combination of the deformation basis vector so that deforming the 3D mesh by the linear combination fits the depth map; and/or
a shape reconstruction process from a digital image by deformation of a 3D mesh to fit the digital image, the shape reconstruction process causing the processor to be configured to:
obtain the 3D mesh, the 3D mesh representing a first real-world object,
obtain a digital image of a second real-world object, the second real-world object being close to the first real-world object,
apply the generative neural network to the 3D mesh, which yields a deformation basis of the 3D mesh,
optimize coefficients of a linear combination of the deformation basis vector so that deforming the 3D mesh by the linear combination fits the digital image; and/or
a shape matching process by deformation of a first 3D modeled object to fit a second 3D modeled object, the shape matching process causing the processor to be configured to:
obtain the first 3D modeled object and the second 3D modeled object, the second 3D modeled object being obtained from a scan of a real-world object,
apply the generative neural network to the first 3D modeled object to compute a deformation basis of the first 3D modeled object, and
optimize coefficients of a linear combination of the deformation basis vectors in order to deform the first 3D modeled object so that it matches the second 3D modeled object; and/or
a segmentation process of segmentation of a 3D modeled object representing a scanned real-world object, the segmentation process causing the processor to be configured to:
obtain, from a physical scan of the real-world object, the 3D modeled object, the 3D modeled object corresponding to scan data representing the real-world object,
apply the generative neural network to the 3D modeled object, thereby yielding a deformation basis of the 3D modeled object consisting of vectors each representing a local deformation of the 3D modeled object, and
determine local parts of the 3D modeled object by inferring a respective local part for each vector of the basis, by the processor being configured to:
for each vector, deform the object according to the vector multiplied by a unitary coefficient, and
for each unitary deformation, determine points/vertices of the 3D modeled object which have been deformed above a predefined threshold, the determined points forming a segment of the segmentation of the 3D modeled object,
wherein a generative loss is of a type:
E
2
=
∑
i
=
1
N
log
(
1
-
d
(
y
i
)
)
-
λ
m
D
C
H
(
x
i
,
y
i
)
+
δ
m
n
∑
j
=
1
n
g
w
(
h
,
x
i
,
1
)
j
2
⋮
g
w
(
h
,
x
i
,
m
)
j
2
1
,
where:
d is a discriminative neural network;
x 1 , . . . , x N are 3D modeled objects forming a mini-batch of the dataset, N being a number of objects in the mini-batch;
(
g
w
(
h
,
x
i
,
1
)
j
⋮
g
w
(
h
,
x
i
,
m
)
j
)
,
1≤j≤n, is the deformation basis generated by the generative neural network at object x i , n being a number of vectors of the deformation basis, x i,1 . . . x i,m being points or vertices of x i ;
m is a number of points or vertices of each 3D modeled object of the mini-batch;
y i is a linear deformation of x i spanned by the deformation basis generated by the generative neural network at object x i ;
D CH is a Chamfer distance; and
λ and δ are parameters of the generative loss, and/or
apply the generative neural network learned according to the machine-learning.
13 . The device of claim 12 , wherein the machine-learning further includes training, by minimizing a discriminative loss, the discriminative neural network configured for determining whether or not a 3D modeled object resulting from application of a linear combination of a deformation basis generated by the generative neural network is realistic or synthetic.
14 . The device of claim 13 , wherein the learning of the generative neural network further includes training the generative neural network to fool the discriminative neural network.
15 . The device of claim 13 , wherein the learning of the generative neural network further includes minimizing the generative loss, and the processor is further configured to alternate minimizations of the generative loss and minimizations of the discriminative loss.
16 . The computer-implemented method of claim 11 , wherein the machine-learning further includes training, by minimizing a discriminative loss, the discriminative neural network configured for determining whether or not a 3D modeled object resulting from application of a linear combination of a deformation basis generated by the generative neural network is realistic or synthetic.
17 . The computer-implemented method of claim 16 , wherein the learning of the generative neural network further includes training the generative neural network to fool the discriminative neural network.
18 . The computer-implemented method of claim 16 , wherein the learning of the generative neural network further includes minimizing the generative loss, the method alternating minimizations of the generative loss and minimizations of the discriminative loss.
19 . A non-transitory computer readable medium having stored thereon a program that when executed by a computer causes the computer to implement the method of claim 1 .