Co-disentagled series/text multi-modal representation learning for controllable generation
A computer-implemented method for multi-model representation learning is provided. The method includes encoding, by a trained time series (TS) encoder, an input TS segment into a TS-shared latent representation and a TS-private latent representation. The method further includes generating, by a trained text generator, a natural language text that explains the input TS segment, responsive to the TS-shared latent representation, the TS-private latent representation, and a text-private latent representation.
1. A computer-implemented method for multi-model representation learning, comprising:
encoding, by a trained time series (TS) encoder, an input TS segment into a TS-shared latent representation and a TS-private latent representation; and
generating, by a trained text generator, a natural language text that explains the input TS segment, responsive to the TS-shared latent representation, the TS-private latent representation, and a text-private latent representation.
2. The computer-implemented method of claim 1 , wherein the text-private latent representation is obtaining by inputting a list of example texts with a desired style into a trained text encoder which outputs the text-private latent representation in response thereto.
3. The computer-implemented method of claim 2 , further comprising pre-processing the example texts by translating each of the example texts into a foreign language and translating back using a different translation tool to obtain resultant text with a same meaning but different words for use as the list of example texts.
4. The computer-implemented method of claim 1 , wherein the text-private latent representation is selected from a list of style latent prototypes.
5. The computer-implemented method of claim 1 , wherein the natural language text that explains the input TS segment is in a user-selected style.
6. The computer-implemented method of claim 1 , further comprising pre-processing the input TS segment by normalizing the input TS segment to have values from 0 to 1, and augmenting the normalized TS segment by adding Gaussian noise to the normalized TS segment.
7. The computer-implemented method of claim 1 , further comprising:
adding a perturbation to dimension i of the TS-shared latent representation; and
visualizing generated pairs of TS and text as a function of changing perturbation for each dimension i of the TS-shared latent variable.
8. The computer-implemented method of claim 1 , wherein a hidden state vector of a last timestep is used as the TS-shared latent representation and the TS-private latent representation of the TS segment.
9. The computer-implemented method of claim 1 , further comprising controlling an operating parameter of a motorized device to prevent an impending device failure responsive to the natural language text that explains the input TS segment.
10. A computer program product for multi-model representation learning, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
encoding, by a trained time series (TS) encoder, an input TS segment into a TS-shared latent representation and a TS-private latent representation; and
generating, by a trained text generator, a natural language text that explains the input TS segment, responsive to the TS-shared latent representation, the TS-private latent representation, and a text-private latent representation.
11. The computer program product of claim 10 , wherein the text-private latent representation is obtaining by inputting a list of example texts with a desired style into a trained text encoder which outputs the text-private latent representation in response thereto.
12. The computer program product of claim 11 , further comprising pre-processing the example texts by translating each of the example texts into a foreign language and translating back using a different translation tool to obtain resultant text with a same meaning but different words for use as the list of example texts.
13. The computer program product of claim 10 , wherein the text-private latent representation is selected from a list of style latent prototypes.
14. The computer program product of claim 10 , wherein the natural language text that explains the input TS segment is in a user-selected style.
15. The computer program product of claim 10 , further comprising pre-processing the input TS segment by normalizing the input TS segment to have values from 0 to 1, and augmenting the normalized TS segment by adding Gaussian noise to the normalized TS segment.
16. The computer program product of claim 10 , further comprising:
adding a perturbation to dimension i of the TS-shared latent representation; and
visualizing generated pairs of TS and text as a function of changing perturbation for each dimension i of the TS-shared latent variable.
17. The computer program product of claim 10 , wherein a hidden state vector of a last timestep is used as the TS-shared latent representation and the TS-private latent representation of the TS segment.
18. The computer program product of claim 10 , further comprising controlling an operating parameter of a motorized device to prevent an impending device failure responsive to the natural language text that explains the input TS segment.
19. A computer processing system for multi-model representation learning, comprising:
a memory device for storing program code; and
a hardware processor operatively coupled to the memory device for running the program code to:
encode, using a trained time series (TS) encoder, an input TS segment into a TS-shared latent representation and a TS-private latent representation; and
generate, using a trained text generator, a natural language text that explains the input TS segment, responsive to the TS-shared latent representation, the TS-private latent representation, and a text-private latent representation.
20. The computer processing system of claim 19 , wherein the text-private latent representation is obtaining by inputting a list of example texts with a desired style into a trained text encoder which outputs the text-private latent representation in response thereto.