Methods and systems for optimal transport of non-linear transformations
Herein is presented a light-weight, high-quality texture synthesis algorithm that generalizes to other applications. We utilize an optimal transport optimization process within a bottleneck layer of an auto-encoder, achieving quality and flexibility on par with expensive back-propagation based neural texture synthesis methods, but at interactive rates. In addition to superior synthesis quality, our statistically motivated approach generalizes better to other special case texture synthesis problems such as Style Transfer, Inverse-Texture Synthesis, Texture Mixing, Multi-Scale Texture Synthesis, Structured Image Hybrids and Texture Painting. We treat the texture synthesis problem as the optimal transport between Probably Density Function of the deep neural activation vectors of the image being synthesized and the exemplar texture. We present a fast algorithm that matches random sliced 1-Dimensional histograms projected from the full N-Dimensional distribution and we propose an extension of this algorithm that reduces dimensionality of neural feature space.
1. A system comprising:
one or more computer processors;
one or more computer memories;
a set of instructions stored in the one or more computer memories, the set of instructions configuring the one or more computer processors to perform operations, the operations comprising:
accessing data describing an input source texture and an initial input texture;
transforming the input source texture and the initial input texture into respective feature representations of each input, the transforming including using a first nonlinear transformation process;
applying an optimal transport process to the respective feature representations to modify a set of first order statistics, the set of first order statistics belonging to the initial input texture feature representation, the modifying including performing a matching of the set of first order statistics to a second set of first order statistics, the second set of first order statistics belonging to the input source texture feature representation; and
passing the modified set of first order statistics to a second nonlinear transform to convert the modified set of first order statistics into an image space.
2. The system of claim 1 , wherein the first nonlinear transform and the second nonlinear transform are neural networks and form an input and output of an autoencoder neural network, respectively, and wherein the optimal transport process is applied on a bottleneck layer of the autoencoder.
3. The system of claim 1 , wherein the operations further include:
configuring an additional first nonlinear transformation process and an additional second nonlinear transformation process in a lower level of texture synthesis, wherein the additional pair of nonlinear transformation processes generate coarser feature representations as compared to the first nonlinear transformation process and the second nonlinear transformation process; and
providing the input source texture and the output image as inputs to the additional first nonlinear transformation process.
4. The system of claim 2 , wherein feature representations from the first nonlinear transform include a probability density function of deep neural network activation vectors of an input to the first nonlinear transform.
5. The system of claim 4 , wherein the optimal transport process matches random sliced 1-dimensional histograms projected from the probability density function.
6. The system of claim 5 , wherein the operations further include:
determining an additional texture synthesis is required based on a metric applied to the output;
reducing a number of the random sliced 1-dimensional histograms; and
using the output as a new initial input texture.
7. The system of claim 1 , wherein the operations include:
accessing a content image for use as an additional input source texture; and
weighting the content image prior to applying the optimal transport process to favor the content image at coarse spatial frequencies and favor the input source texture at finer spatial frequencies.
8. The system of claim 1 , wherein the operations further include:
determining a lower dimensional subspace for the respective feature representations using principal component analysis; and
applying the optimal transport process on the lower dimensional subspace.
9. A non-transitory computer-readable storage medium storing a set of instructions that, when executed by one or more computer processors, causes the one or more computer processors to perform operations, the operations comprising:
accessing data describing an input source texture and an initial input texture;
transforming the input source texture and the initial input texture into respective feature representations of each input, the transforming including using a first nonlinear transformation process;
applying an optimal transport process to the respective feature representations to modify a set of first order statistics, the set of first order statistics belonging to the initial input texture feature representation, the modifying including performing a matching of the set of first order statistics to a second set of first order statistics, the second set of first order statistics belonging to the input source texture feature representation; and
passing the modified set of first order statistics to a second nonlinear transform to convert the modified set of first order statistics into an image space.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the first nonlinear transform and the second nonlinear transform are neural networks and form an input and output of an autoencoder neural network, respectively, and wherein the optimal transport process is applied on a bottleneck layer of the autoencoder.
11. The non-transitory computer-readable storage medium of claim 9 , wherein the operations further include:
configuring an additional first nonlinear transformation process and an additional second nonlinear transformation process in a lower level of texture synthesis, wherein the additional pair of nonlinear transformation processes generate coarser feature representations as compared to the first nonlinear transformation process and the second nonlinear transformation process; and
providing the input source texture and the output image as inputs to the additional first nonlinear transformation process.
12. The non-transitory computer-readable storage medium of claim 10 , wherein feature representations from the first nonlinear transform include a probability density function of deep neural network activation vectors of an input to the first nonlinear transform.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the optimal transport process matches random sliced 1-dimensional histograms projected from the probability density function.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the operations further include:
determining an additional texture synthesis is required based on a metric applied to the output;
reducing a number of the random sliced 1-dimensional histograms; and
using the output as a new initial input texture.
15. The non-transitory computer-readable storage medium of claim 9 , wherein the operations include:
accessing a content image for use as an additional input source texture; and
weighting the content image prior to applying the optimal transport process to favor the content image at coarse spatial frequencies and favor the input source texture at finer spatial frequencies.
16. The non-transitory computer-readable storage medium of claim 9 , wherein the operations further include:
determining a lower dimensional subspace for the respective feature representations using principal component analysis; and
applying the optimal transport process on the lower dimensional subspace.
17. A method comprising:
accessing data describing an input source texture and an initial input texture;
transforming the input source texture and the initial input texture into respective feature representations of each input, the transforming including using a first nonlinear transformation process;
applying an optimal transport process to the respective feature representations to modify a set of first order statistics, the set of first order statistics belonging to the initial input texture feature representation, the modifying including performing a matching of the set of first order statistics to a second set of first order statistics, the second set of first order statistics belonging to the input source texture feature representation; and
passing the modified set of first order statistics to a second nonlinear transform to convert the modified set of first order statistics into an image space.
18. The method of claim 17 , wherein the first nonlinear transform and the second nonlinear transform are neural networks and form an input and output of an autoencoder neural network, respectively, and wherein the optimal transport process is applied on a bottleneck layer of the autoencoder.
19. The method of claim 17 , wherein the operations further include:
configuring an additional first nonlinear transformation process and an additional second nonlinear transformation process in a lower level of texture synthesis, wherein the additional pair of nonlinear transformation processes generate coarser feature representations as compared to the first nonlinear transformation process and the second nonlinear transformation process; and
providing the input source texture and the output image as inputs to the additional first nonlinear transformation process.
20. The method of claim 18 , wherein feature representations from the first nonlinear transform include a probability density function of deep neural network activation vectors of an input to the first nonlinear transform.