Image generation using one-dimensional inputs
Image-to-image translations using 1D inputs includes concatenating multiple 1D vectors forming a concatenated 1D vector. The multiplicity of 1D vectors includes 1D vectors of at least two different modalities. An encoded 1D vector is generated by encoding the concatenated 1D vector. An encoded 2D array of features is generated by reshaping an arrangement of features of the encoded 1D feature vector. An image of a virtual human is generated by decoding the encoded 2D array.
1 . A computer-implemented method, comprising:
concatenating a plurality of 1D vectors forming a concatenated 1D vector, wherein the plurality of 1D vectors include 1D vectors of at least two different modalities;
generating an encoded 1D feature vector by encoding the concatenated 1D vector;
generating an encoded 2D array of features by reshaping an arrangement of features of the encoded 1D feature vector; and
generating an image of a virtual human by decoding the encoded 2D array.
2 . The computer-implemented method of claim 1 , further comprising:
generating a video rendering of the virtual human, wherein the video rendering includes the image of the virtual human.
3 . The computer-implemented method of claim 1 , wherein each 1D vector of the plurality of 1D vectors is of a different modality.
4 . The computer-implemented method of claim 1 , wherein the plurality of 1D vectors includes a 1D vector having a first modality including head pose information.
5 . The computer-implemented method of claim 1 , wherein the plurality of 1D vectors includes a 1D vector having a second modality including features representing audio data.
6 . The computer-implemented method of claim 5 , wherein the features representing audio data includes a plurality of viseme coefficients.
7 . The computer-implemented method of claim 1 , wherein the plurality of 1D vectors includes a 1D vector with features comprising blend shape coefficients.
8 . A computer-implemented method, comprising:
generating an encoded 2D image template by encoding a 2D static template image;
generating an encoded 1D feature vector by encoding a concatenation of a plurality of 1D vectors;
generating an encoded 2D array of features by reshaping an arrangement of multiple features of the encoded 1D feature vector;
concatenating the encoded 2D image template and the encoded 2D array of features;
generating a fused tensor by fusing the encoded 2D image template with the encoded 2D array of features; and
generating an image of a virtual human by decoding the fused tensor, wherein the image of the virtual human is changeable to a different virtual human in response to encoding a different 2D static template image.
9 . The computer-implemented method of claim 8 , further comprising:
generating a video rendering of the virtual human, wherein the video rendering includes the image of the virtual human.
10 . The computer-implemented method of claim 8 , further comprising:
changing at least one visual characteristic of the image of the virtual human in response to encoding a different concatenation of 1D vectors.
11 . The computer-implemented method of claim 10 , wherein the changing at least one visual characteristic includes changing a direction of illumination of the image of the virtual human.
12 . The computer-implemented method of claim 8 , wherein the concatenating concatenates the encoded 2D image template and the encoded 2D array of features along a channel dimension.
13 . The computer-implemented method of claim 8 , wherein the generating the encoded 1D feature vector includes encoding a concatenation of a plurality of 1D vectors that each comprise multiple features corresponding to different modalities.
14 . The computer-implemented method of claim 8 , wherein the plurality of 1D vectors includes a 1D vector whose features include head pose information.
15 . The computer-implemented method of claim 8 , wherein the plurality of 1D vectors includes a 1D vector whose features comprise blend shape coefficients.
16 . The computer-implemented method of claim 8 , wherein the plurality of 1D vectors includes a 1D vector comprising a plurality of features representing audio data.
17 . The computer-implemented method of claim 16 , wherein the plurality of features representing audio data includes a plurality of viseme coefficients.
18 . A computer-implemented method, comprising:
generating an encoded 2D image template by encoding a 2D static template image;
generating an encoded 1D feature vector by encoding a concatenation of a plurality of 1D vectors;
generating an encoded 2D array of features by reshaping an arrangement of multiple features of the encoded 1D feature vector;
concatenating the encoded 2D image template and the encoded 2D array of features;
generating a fused tensor by fusing the encoded 2D image template with the encoded 2D array of features; and
generating an image of a virtual human by decoding the fused tensor, wherein at least one visual characteristic of the image of the virtual human is changeable in response to encoding a different concatenation of 1D vectors.
19 . The computer-implemented method of claim 18 , further comprising:
generating a video rendering of the virtual human, wherein the video rendering includes the image of the virtual human.
20 . The computer-implemented method of claim 18 , wherein the generating the encoded 1D feature vector includes encoding a concatenation of a plurality of 1D vectors that each comprise multiple features corresponding to different modalities.