Deep deformation network for object landmark localization
A system and method are provided. The system includes a processor. The processor is configured to generate a response map for an image, using a four stage convolutional structure. The processor is further configured to generate a plurality of landmark points for the image based on the response map, using a shape basis neural network. The processor is additionally configured to generate an optimal shape for the image based on the plurality of landmark points for the image and the response map, using a point deformation neural network. A recognition system configured to identify the image based on the generated optimal shape to generate a recognition result of the image. The processor is also configured to operate a hardware-based machine based on the recognition result.
1. A method for using a convolutional neural network comprising:
generating, by a processor using a four stage convolutional structure, a response map for an image;
generating, by the processor using a shape basis neural network (SBN), a plurality of landmark points for the image based on the response map;
generating, by the processor using a point deformation neural network (PDN), an optimal shape for the image based on the plurality of landmark points for the image, synthesized landmarks generated from a plurality of landmarks, and the response map;
identifying, by a recognition system, the image based on the generated optimal shape to generate a recognition result of the image; and
operating a hardware-based machine based on the recognition result,
wherein performance of the convolutional neural network is improved by decomposing joint tasks into two less complex tasks and pre-training an SBN and PDN separately, and
wherein the pre-training includes, given a PCA shape model ( y and Q) and a set of training images (x), pre-training the SBN for an optimal embedding of x s =f(w s , x), minimizing a Euclidian distance between an SBN prediction and a ground-truth (y) by
min
w
s
F
=
y
-
(
y
_
+
Q
x
s
)
2
2
+
λ
x
s
2
2
,
and wherein a gradient of a loss F over x s is determined by
∂
F
∂
x
s
=
2
λ
x
s
-
2
Q
T
(
y
-
(
y
_
+
Q
x
s
)
)
and back-propagated to update parameters of fully connected layers and lower convolutional layers, where w s represents the fully connector layers in the SBN, and λ is a regularization factor penalizing coefficients with a large l 2 norm.
2. The method as recited in claim 1 , wherein the point deformation neural network applies a non-linear transformation to the plurality of landmark points for the image.
3. The method as recited in claim 1 , wherein the four stage convolutional structure comprises a plurality of convolutional layers and a plurality of rectified linear unit layers.
4. The method as recited in claim 1 , wherein the response map includes a 7-by-7 response map for each of 512 channels.
5. The method as recited in claim 1 , wherein the point deformation neural network deforms the plurality of landmarks for the image using a Thin-Plate-Spline (TPS) transformation.
6. The method as recited in claim 5 , wherein the deformation of the plurality of landmark points for the image is constrained by controlling a point grid regularization method.
7. The method as recited in claim 1 , wherein the shape basis neural network applies a constraint on a global geometry of the plurality of landmarks for the image.
8. The method as recited in claim 1 , wherein the shape basis neural network and the point deformation neural network are trained jointly with the same plurality of images.
9. The method as recited in claim 1 , wherein the plurality of landmark points for the image comprises a linear combination of pre-defined shape bases.
10. The method as recited in claim 1 , wherein the shape basis neural network applies a non-linear transformation to the response map.
11. The method as recited in claim 1 , wherein the recognition system is included in a security system that captured the image.
12. The method as recited in claim 1 , wherein operating the hardware-based machine based on the recognition result comprises actuating an unlock mechanism of a lock to grant, to a user depicted and recognized in the image, access to an item selected from the group consisting of a device and a facility.
13. The method as recited in claim 1 , wherein operating the hardware-based machine based on the recognition result comprises stopping the hardware-based machine to mitigate a risk posed to a user by the hardware-based machine.
14. A non-transitory article of manufacture tangibly embodying a computer readable program which when executed causes a computer to perform the steps of claim 1 .
15. A system comprising:
a processor, configured to:
generate a response map for an image, using a four stage convolutional structure;
generate a plurality of landmark points for the image based on the response map, using a shape basis neural network (SBN); and
generate an optimal shape for the image based on the plurality of landmark points for the image, synthesized landmarks generated from a plurality of landmarks, and the response map, using a point deformation neural network (PDN); and
a recognition system configured to identify the image based on the generated optimal shape to generate a recognition result of the image,
wherein the processor is further configured to operate a hardware-based machine based on the recognition result,
wherein performance of the convolutional neural network is improved by decomposing joint tasks into two less complex tasks and pre-training an SBN and PDN separately, and
wherein the pre-training includes, given a PCA shape model ( y and Q) and a set of training images (x), pre-training the SBN for an optimal embedding of x s =f(w s , x), minimizing a Euclidian distance between an SBN prediction and a ground-truth (y) by
min
w
s
F
=
y
-
(
y
_
+
Q
x
s
)
2
2
+
λ
x
s
2
2
,
and wherein a gradient of a loss F over x s is determined by
∂
F
∂
x
s
=
2
λ
x
s
-
2
Q
T
(
y
-
(
y
_
+
Q
x
s
)
)
and back-propagated to update parameters of fully connected layers and lower convolutional layers, where w s represents the fully connector layers in the SBN, and λ is a regularization factor penalizing coefficients with a large l 2 norm.
16. The system as recited in claim 15 , wherein the point deformation neural network applies a non-linear transformation to the plurality of landmark points for the image.
17. The system as recited in claim 15 , wherein the four stage convolutional structure comprises a plurality of convolutional layers and a plurality of rectified linear unit layers.
18. The system as recited in claim 15 , wherein the point deformation neural network deforms the plurality of landmarks for the image using a Thin-Plate-Spline (TPS) transformation.
19. The system as recited in claim 15 , wherein the shape basis neural network applies a constraint on a global geometry of the plurality of landmarks for the image.
20. The system as recited in claim 15 , wherein the plurality of landmark points for the image comprises a linear combination of pre-defined shape bases.
21. A processor configuration comprising:
a processor, configured to:
generate a response map for an image, using a four stage convolutional structure;
generate a plurality of landmark points for the image based on the response map, using a shape basis neural network (SBN); and
generate an optimal shape for the image based on the plurality of landmark points for the image, synthesized landmarks generated from a plurality of landmarks, and the response map, using a point deformation neural network (PDN); and
operate a hardware-based machine based on a recognition result obtained from a recognition system configured to identify the image based on the generated optimal shape to generate the recognition result of the image,
wherein performance of the convolutional neural network is improved by decomposing joint tasks into two less complex tasks and pre-training an SBN and PDN separately, and
wherein the pre-training includes, given a PCA shape model ( y and Q) and a set of training images (x), pre-training the SBN for an optimal embedding of x s =f(w s , x), minimizing a Euclidian distance between an SBN prediction and a ground-truth (y) by
min
w
s
F
=
y
-
(
y
_
+
Q
x
s
)
2
2
+
λ
x
s
2
2
,
and wherein a gradient of a loss F over x s is determined by
∂
F
∂
x
s
=
2
λ
x
s
-
2
Q
T
(
y
-
(
y
_
+
Q
x
s
)
)
and back-propagated to update parameters of fully connected layers and lower convolutional layers, where w s represents the fully connector layers in the SBN, and λ is a regularization factor penalizing coefficients with a large l 2 norm.