Dynamic word embeddings

1. A method, comprising:

deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, wherein deriving the machine learning data model comprises applying a diffusion process to the word embedding vectors and the context embedding vectors of the plurality of skip-gram models such that the word embedding vectors and the context embedding vectors are aligned to a common frame of reference of time;

generating a smoothed model by applying a variational inference operation; and

identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.

2. The method of claim 1 , further comprising:

prior to identifying the change in the semantic use of the word, receiving a request to monitor the semantic use of the word;

monitoring the semantic use of the word based on the smoothed model and new text added to the corpus of electronic text;

identifying the change in the semantic use of the word based on at least one of: (i) a distance between two of the word embedding vectors, or (ii) a distance between two of the context embedding vectors;

generating an indication of the change in the semantic use of the word; and

outputting the indication.

3. The method of claim 1 , wherein the word embedding vectors comprise word embeddings for each word in the corpus of electronic text, wherein the context embedding vectors comprise context embeddings for each word in the corpus of electronic text, wherein the method further comprises segmenting each text element in the corpus of electronic text into a respective time step of a plurality of time steps based on a respective timestamp of each text element, wherein the plurality of time steps comprises each time step associated with the plurality of skip-gram models.

4. The method of claim 1 , wherein the corpus of electronic text includes a plurality of pairs of words, wherein deriving the machine learning data model further comprises: generating a positive count matrix, wherein the positive count matrix specifies, for each of the plurality of pairs of words in the corpus of electronic text, a respective count of observed occurrences of each respective pair of words within a predefined window of words in a text element of each time step.

5. The method of claim 1 , wherein deriving the machine learning data model further comprises: generating a negative count matrix based on a plurality of rejected pairs of words in a second training corpus of electronic text.

6. The method of claim 1 , wherein the variational inference operation comprises a filtering algorithm comprising:

initializing a plurality of variational parameters for the word embedding vectors;

initializing a plurality of variational parameters for the context embedding vectors; and

optimizing the plurality of variational parameters for the word embedding vectors and the plurality of variational parameters for the context embedding vectors using stochastic gradient descent.

7. The method of claim 1 , wherein the variational inference operation comprises a smoothing algorithm comprising:

initializing a plurality of variational parameters for the word embedding vectors;

initializing a plurality of variational parameters for the context embedding vectors;

optimizing the plurality of variational parameters for the word embedding vectors using a first bidiagonal matrix; and

optimizing the plurality of variational parameters for the context embedding vectors using a second bidiagonal matrix.

8. A non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable to perform an operation comprising:

deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, wherein deriving the machine learning data model comprises applying a diffusion process to the word embedding vectors and the context embedding vectors of the plurality of skip-gram models such that the word embedding vectors and the context embedding vectors are aligned to a common frame of reference of time;

generating a smoothed model by applying a variational inference operation over the machine learning data model; and

identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.

9. The computer-readable storage medium of claim 8 , the operation further comprising:

prior to identifying the change in the semantic use of the word, receiving a request to monitor the semantic use of the word;

monitoring the semantic use of the word based on the smoothed model and new text added to the corpus of electronic text;

identifying the change in the semantic use of the word based on at least one of: (i) a distance between two word embedding vectors of the plurality of skip-gram models, or (ii) a distance between two context embedding vectors of the plurality of skip-gram models;

generating an indication of the change in the semantic use of the word; and

outputting the indication.

10. The computer-readable storage medium of claim 8 , wherein the word embedding vectors of the plurality of skip-gram models comprise word embeddings for each word in the corpus of electronic text, wherein the context embedding vectors of the plurality of skip-gram models comprise context embeddings for each word in the corpus of electronic text, wherein the operation further comprises segmenting each text element in the corpus of electronic text into a respective time step of a plurality of time steps based on a respective timestamp of each text element, wherein the plurality of time steps comprises each time step associated with the plurality of skip-gram models.

11. The computer-readable storage medium of claim 8 , wherein the corpus of electronic text includes a plurality of pairs of words, wherein deriving the machine learning data model further comprises:

generating a positive count matrix, wherein the positive count matrix specifies, for each of the plurality of pairs of words in the corpus of electronic text, a respective count of observed occurrences of each respective pair of words within a predefined window of words in a text element of each time step.

12. The computer-readable storage medium of claim 8 , wherein deriving the machine learning data model further comprises:

generating a negative count matrix based on a plurality of rejected pairs of words in a second training corpus of text.

13. The computer-readable storage medium of claim 8 , wherein the variational inference operation comprises a filtering algorithm comprising:

initializing a plurality of variational parameters for the word embedding vectors of the plurality of skip-gram models;

initializing a plurality of variational parameters for the context embedding vectors of the plurality of skip-gram models; and

optimizing the plurality of variational parameters for the word embedding vectors and the plurality of variational parameters for the context embedding vectors using stochastic gradient descent.

14. The computer-readable storage medium of claim 8 , wherein the variational inference operation comprises a smoothing algorithm comprising:

initializing a plurality of variational parameters for the word embedding vectors of the plurality of skip-gram models;

initializing a plurality of variational parameters for the context embedding vectors of the plurality of skip-gram models;

optimizing the plurality of variational parameters for the word embedding vectors using a first bidiagonal matrix; and

optimizing the plurality of variational parameters for the context embedding vectors using a second bidiagonal matrix.

15. A system, comprising:

a computer processor; and

a memory containing a program which when executed by the computer processor performs an operation comprising:

deriving, based on a corpus of electronic text, a machine learning data model that associates words with corresponding usage contexts over a window of time, wherein the machine learning data model comprises a plurality of skip-gram models, wherein each skip-gram model comprises a word embedding vector and a context embedding vector for a respective time step associated with the respective skip-gram model, wherein deriving the machine learning data model comprises applying a diffusion process to the word embedding vectors and the context embedding vectors of the plurality of skip-gram models such that the word embedding vectors and the context embedding vectors are aligned to a common frame of reference of time;

generating a smoothed model by applying a variational inference operation over the machine learning data model; and

identifying, based on the smoothed model and the corpus of electronic text, a change in a semantic use of a word over at least a portion of the window of time.

16. The system of claim 15 , the operation further comprising:

prior to identifying the change in the semantic use of the word, receiving a request to monitor the semantic use of the word;

monitoring the semantic use of the word based on the smoothed model and new text added to the corpus of electronic text;

identifying the change in the semantic use of the word based on at least one of: (i) a distance between two word embedding vectors of the plurality of skip-gram models, or (ii) a distance between two context embedding vectors of the plurality of skip-gram models;

generating an indication of the change in the semantic use of the word; and

outputting the indication.

17. The system of claim 15 , wherein the word embedding vectors of the plurality of skip-gram models comprise word embeddings for each word in the corpus of electronic text, wherein the context embedding vectors of the plurality of skip-gram models comprise context embeddings for each word in the corpus of electronic text, wherein the operation further comprises segmenting each text element in the corpus of electronic text into a respective time step of a plurality of time steps based on a respective timestamp of each text element, wherein the plurality of time steps comprises each time step associated with the plurality of skip-gram models.

18. The system of claim 15 , wherein the corpus of electronic text includes a plurality of pairs of words and wherein deriving the machine learning data model further comprises:

generating a positive count matrix, wherein the positive count matrix specifies, for each of the plurality of pairs of words in the corpus of electronic text, a respective count of observed occurrences of each respective pair of words within a predefined window of words in a text element of each time step; and

generating a negative count matrix based on a plurality of rejected pairs of words in a second training corpus of electronic text.

19. The system of claim 15 , wherein the variational inference operation comprises a filtering algorithm comprising:

initializing a plurality of variational parameters for the word embedding vectors of the plurality of skip-gram models;

initializing a plurality of variational parameters for the context embedding vectors of the plurality of skip-gram models;

optimizing the plurality of variational parameters for the word embedding vectors and the plurality of variational parameters for the context embedding vectors using stochastic gradient descent.

20. The system of claim 15 , wherein the variational inference operation comprises a smoothing algorithm comprising:

initializing a plurality of variational parameters for the word embedding vectors of the plurality of skip-gram models;

initializing a plurality of variational parameters for the context embedding vectors of the plurality of skip-gram models;

optimizing the plurality of variational parameters for the word embedding vectors using a first bidiagonal matrix; and

optimizing the plurality of variational parameters for the context embedding vectors using a second bidiagonal matrix.