Embedding multi-modal time series and text data
Methods and systems of training and using a neural network model include training a time series embedding model and a text embedding model with unsupervised clustering to translate time series and text, respectively, to a shared latent space. The time series embedding model and the text embedding model are further trained using semi-supervised clustering that samples training data pairs of time series information and associated text for annotation.
1. A method of querying a time series database, comprising:
transforming a query to an embedded vector in a multi-modal shared latent space that encodes time series info, illation and textual information;
identifying a feature vector in the multi-modal shared latent space, stored in a time series dataspace, that matches the embedded vector, and that is associated with a data type complementary to the query; and
returning data associated with the identified feature vector, responsive to the query.
2. The method of claim 1 , wherein the query includes a text information, and the feature vector is associated with time series information.
3. The method of claim 2 , wherein the associated text describes circumstances relating to the time series segment.
4. The method of claim 2 , wherein the query further includes a time series segment.
5. The method of claim 1 , wherein the query includes time series information, and the feature vector is associated with text information.
6. The method of claim 1 , wherein identifying the feature vector that matches the embedded vector includes identifying a nearest neighbor according to Euclidean distance within in the multi-modal shared latent space.
7. A system for training a neural network, comprising:
a hardware processor; and
a memory that stores a computer program product, which, when executed by the hardware processor, causes the hardware processor to:
train a time series embedding model and a text embedding model using unsupervised clustering to translate time series and text, respectively, to a multi-modal shared latent space;
train the time series embedding model and the text embedding model further using semi-supervised clustering that samples training data pairs of time series information and associated text for annotation;
transform a query to an embedded vector in the multi-modal shared latent space that encodes time series information and textual information;
identifying a feature vector in the multi-modal shared latent space, stored in a time series dataspace, that matches the embedded vector, and that is associated with a data type complementary to the query; and
returning data associated with the identified feature vector, responsive to the query.
8. The system of claim 7 , wherein the training data pairs each include a time series segment and an associated text.
9. The system of claim 8 , wherein the associated text of each pair describes circumstances relating to the respective time series segment.
10. The system of claim 8 , wherein the computer program product further causes the hardware processor to annotate the sampled pairs to indicate constraints during semi-supervised clustering.
11. The system of claim 10 , wherein the constraints are selected from the group consisting of a “must-link” constraint and a “cannot-link” constraint.
12. The system of claim 7 , wherein the query includes a text information, and the feature vector is associated with time series information.
13. The system of claim 11 , wherein the query further includes a time series segment.
14. The system of claim 7 , wherein the query includes time series information, and the feature vector is associated with text information.
15. The system of claim 7 , wherein the computer program product further causes the hardware processor to identify a nearest neighbor of the embedded vector according to Euclidean distance within in the multi-modal shared latent space.