System and method for natural language processing

1. A natural language processing system, comprising:

a processor,

wherein the processor is configured to receive an input sequence c i of input words representing a first sequence of words in a natural language of a first text and generate an output sequence of output words representing a second sequence of words in a natural language of a second text and modeled by a multinominal topic model, wherein the multinominal topic model is extended by an incorporation of full contextual information around each word v i , wherein both preceding words v <i and following words v >i around each word v i are captured by using a bi-directional language modelling and a feed-forward fashion, wherein position dependent forward hidden layers {right arrow over (h)} i and backward hidden layers i for each word v i are computed, wherein the multinominal topic model is a document neural autoregressive topic model, DocNADE, and the extended multinominal topic model is an informed document neural autoregressive topic model, iDocNADE.

2. The natural language processing system of claim 1 , wherein the forward hidden layers {right arrow over (h)} i and backward hidden layers i for each word v i are computed as:

{right arrow over (h)} i ( v <i )= g ( D{right arrow over (c)}+Σ k<i W :,v k )

i ( v >i )= g ( D +Σ k>i W :,v k )

where {right arrow over (c)}∈ H and ∈ H are bias parameters in forward and backwards passes and H is the number of hidden topics.

3. The natural language processing system of claim 1 , wherein a log-likelihood DN (v) for a document v is computed by using forward and backward language models as:

log p ( v )=½Σ i=1 D log p ( v i |v <i )+log p ( v i |v >i ).

4. The natural language processing system of claim 1 , wherein the iDocNADE model is extended by the incorporation of word embeddings for generating an iDocNADE 2 model.

5. The natural language processing system of claim 4 , wherein the word embeddings is a word embedding aggregation at each representation Σ k<i E :,v k with a pre-trained embedding matrix E∈ H×K .

6. The natural language processing system of claim 5 , wherein the position dependent forward hidden layers {right arrow over (h)} i,e ( v <i ) and backward hidden layers i,e (v >i ) hidden layers for each word v i , depend now on E as:

{right arrow over (h)} i,e ( v <i )= g ( D{right arrow over (c)}+Σ k<i W :,v k +Σ k<i E :,v k )

i,e ( v <i )= g ( D +Σ k>i W :,v k +Σ k>i E :,v k )

and the forward and backward autoregressive conditionals are computed via hidden vectors {right arrow over (h)} i,e (v <i ) and i,e (v >i ).

7. A computer-implemented method for processing natural language, comprising:

receiving an input sequence c i of input words representing a first sequence of words in a natural language of a first text and generating an output sequence of output words representing a second sequence of words in a natural language of a second text and modeled by a multinominal topic model,

extending the multinominal topic model by an incorporation of full contextual information around each word v i ;

capturing both preceding words v <i and following words v >i around each word v i ;

using a bi-directional language modelling and a feed-forward fashion; and

computing position dependent forward hidden layers {right arrow over (h)} i and backward hidden layers i for each word v i ;

wherein the multinominal topic model is a document neural autoregressive topic model, DocNADE, and the extended multinominal topic model is an informed document neural autoregressive topic model, iDocNADE.

8. The method of claim 7 , wherein the forward hidden layers {right arrow over (h)} i and backward hidden layers i for each word v i are computed as:

{right arrow over (h)} i ( v <i )= g ( D{right arrow over (c)}+Σ k<i W :,v k )

i ( v >i )= g ( D +Σ k>i W :,v k )

where {right arrow over (c)}∈ H and ∈ H are bias parameters in forward and backwards passes and H is the number of hidden topics.

9. The method of claim 7 , wherein the log-likelihood DN (v) for a document v is computed by using forward and backward language models as:

log

⁢

⁢

p

⁡

(

v

)

=

1

2

⁢

∑

i

=

1

D

⁢

log

⁢

⁢

p

⁡

(

v

i

❘

v

<

i

)

+

log

⁢

⁢

p

⁡

(

v

i

❘

v

>

i

)

.

10. The method of claim 7 , wherein the iDocNADE model is extended by the incorporation of word embeddings for generating an iDocNADE 2 model.

11. The method of claim 10 , wherein the word embeddings is a word embedding aggregation at each representation Σ k<i E :,v k with a pre-trained embedding matrix E∈ H×K .

12. The method of claim 11 , wherein the position dependent forward hidden layers {right arrow over (h)} i,e ( v <i ) and backward hidden layers i,e (v >i ) hidden layers for each word v i , depend now on E as:

{right arrow over (h)} i,e ( v <i )= g ( D{right arrow over (c)}+Σ k<i W :,v k +Σ k<i E :,v k )

i,e ( v <i )= g ( D +Σ k>i W :,v k +Σ k>i E :,v k )

and the forward and backward autoregressive conditionals are computed via hidden vectors {right arrow over (h)} i,e (v <i ) and i,e (v >i ).

13. A non-transitory computer-readable data storage medium comprising executable program code configured to, when executed, perform a method for processing natural language, the method comprising:

receiving an input sequence c i of input words representing a first sequence of words in a natural language of a first text and generating an output sequence of output words representing a second sequence of words in a natural language of a second text and modeled by a multinominal topic model,

extending the multinominal topic model by an incorporation of full contextual information around each word v i ;

capturing both preceding words v <i and following words v >i around each word v i ;

using a bi-directional language modelling and a feed-forward fashion; and

computing position dependent forward hidden layers {right arrow over (h)} i and backward hidden layers i for each word v i ;

wherein the multinominal topic model is a document neural autoregressive topic model, DocNADE, and the extended multinominal topic model is an informed document neural autoregressive topic model, iDocNADE.

14. The non-transitory computer-readable data storage medium of claim 13 , wherein the forward hidden layers {right arrow over (h)} i and backward hidden layers i for each word v i are computed as:

{right arrow over (h)} i ( v <i )= g ( D{right arrow over (c)}+Σ k<i W :,v k )

i ( v >i )= g ( D +Σ k>i W :,v k )

where {right arrow over (c)}∈ H and ∈ H are bias parameters in forward and backwards passes and H is the number of hidden topics.

15. The non-transitory computer-readable data storage medium of claim 13 , wherein the log-likelihood DN (v) for a document v is computed by using forward and backward language models as:

log

⁢

⁢

p

⁡

(

v

)

=

1

2

⁢

∑

i

=

1

D

⁢

log

⁢

⁢

p

⁡

(

v

i

❘

v

<

i

)

+

log

⁢

⁢

p

⁡

(

v

i

❘

v

>

i

)

.

16. The non-transitory computer-readable data storage medium of claim 13 , wherein the iDocNADE model is extended by the incorporation of word embeddings for generating an iDocNADE 2 model.

17. The non-transitory computer-readable data storage medium of claim 16 , wherein the word embeddings is a word embedding aggregation at each representation Σ k<i E :,v k with a pre-trained embedding matrix E∈ H×K .

18. The non-transitory computer-readable data storage medium of claim 17 , wherein the position dependent forward hidden layers {right arrow over (h)} i,e ( v <i ) and backward hidden layers i,e (v >i ) hidden layers for each word v i , depend now on E as:

{right arrow over (h)} i,e ( v <i )= g ( D{right arrow over (c)}+Σ k<i W :,v k +Σ k<i E :,v k )

i,e ( v <i )= g ( D +Σ k>i W :,v k +Σ k>i E :,v k )

and the forward and backward autoregressive conditionals are computed via hidden vectors {right arrow over (h)} i,e (v <i ) and i,e (v >i ).