Model globalization for long document summarization

1 . A summarization system, comprising:

K embedding modules configured to:

receive a respective one of K blocks of text of a document to be summarized,

wherein each of the K blocks includes:

a predetermined starting token at the front of that block;

a predetermined ending token at the end of that block; and

tokens between the predetermined starting token and the predetermined ending token that represent the text of that one of the K blocks; and

generate a respective one of K first representations based on the respective one of the K blocks of text,

where K is an integer greater than 2 and the K first representations are one of matrices and vectors;

a first propagation module including first transformer layers having a transformer architecture and first bidirectional gated recurrent neural network layers, the first bidirectional gated recurrent neural network layers interleaved between the first transformer layers, the first propagation module configured to generate global and block specific second representations based on the K first representations and propagating local information from ones of the K blocks of text to the other ones of the K blocks of text,

the global second representation including information for each of the K blocks of text and being one of a matrix and a vector, and the block specific second representations including information for the respective ones of the K blocks of text and being ones of matrices and vectors;

a second propagation module including second transformer layers having a transformer architecture and second bidirectional gated recurrent neural network layers, the second bidirectional gated recurrent neural network layers interleaved between the second transformer layers, the second propagation module configured to generate global and block specific third representations based on the second representations, the third representations being ones of matrices and vectors, the global third representation including information for each of the K blocks of text, and the block specific third representations including information for the respective ones of the K blocks of text;

an output module configured to select ones of the K blocks of text based on the third representations,

wherein the output module includes a neural network and a softmax function configured to classify each of the K blocks of text, based on the third representations, as being either a first state or a second state and is configured to select the ones of the K blocks of text based on the classifications; and

a summary module configured to generate a summary of the document from text of the selected ones of the K blocks of text.

2 . The summarization system of claim 1 wherein the document to be summarized includes at least 1024 tokens.

3 . The summarization system of claim 1 wherein the neural network includes a feed forward neural network.

4 . The summarization system of claim 1 wherein:

the first transformer layers of the first propagation module include K transformer layers configured to receive the respective ones of the K first representations and to generate fourth representations based on the K first representations;

wherein the first bidirectional recurrent neural network layers of the first propagation module are configured to receive K first ones of the fourth representations and to generate K fifth representations based on the K first ones of the fourth representations;

the first propagation module further includes K feed forward neural network modules configured to generate K sixth representations based on the K fifth representations, respectively; and

the second representations include (a) second ones of the fourth representations and (b) the K sixth representations.

5 . The summarization system of claim 4 wherein the fourth representations are a group consisting of (a) the first K ones of the fourth representations and (b) the second ones of the fourth representations.

6 . The summarization system of claim 4 wherein:

the second transformer layers of the second propagation module include K second transformer layers configured to receive the second representations and to generate seventh representations based on the sixth representations;

wherein the second bidirectional recurrent neural network layers of the second propagation module are configured to receive K first ones of the seventh representations and to generate K eighth representations based on the K first ones of the seventh representations;

the second propagation module includes K second feed forward neural network modules configured to generate K ninth representations based on the K eighth representations, respectively; and

wherein the third representations include (a) second ones of the seventh representations and (b) the K eighth representations.

7 . The summarization system of claim 4 wherein the K first ones of the fourth representations are K representations of classification (CLS) tokens of the K blocks, respectively.

8 . The summarization system of claim 1 wherein the K first representations each include a token embedding, a segment embedding, and a positional encoding, the positional encoding representing a position of the token in its one of the K blocks.

9 . The summarization system of claim 1 further comprising a third propagation module configured to generate fourth representations based on the third representations,

wherein the output module is configured to select the ones of the K blocks based on the fourth representations.

10 . The summarization system of claim 1 wherein the summary module is configured to store the summary in memory in association with the document.

11 . The summarization system of claim 1 wherein the summary module is configured to transmit the summary to a computing device via a network.

12 . The summarization system of claim 1 wherein: the predetermined starting token corresponds to a classification (CLS) token; the predetermined ending token corresponds to a separation (SEP) token; and the tokens between the predetermined starting token and the predetermined ending token correspond to at least one sentence of text from the document.

13 . The summarization system of claim 1 wherein each of the first and second propagation modules includes (a) K transformer modules configured to independently determine dependencies locally within the K blocks of text and (b) a bi-directional recurrent neural network module configured to process output of the K transformer modules and to determine dependencies globally within the document.

14 . The summarization system of claim 1 wherein at least one of the first bidirectional gated recurrent neural network layers includes one of a bidirectional gate recurrent unit (BiGRU) and a bidirectional long short-term memory (BILSTM).

15 . A query system, comprising:

a search module configured to receive a query as input and obtain search results based on the query; and

the summarization system of claim 1 , wherein the summarization module is configured to produce summaries of the search results for the search module,

wherein the search module is further configured to output the search results and the summaries of the search results produced by the summarization module.

16 . A summarization system, comprising:

K means for:

receiving a respective one of K blocks of text of a document to be summarized,

wherein each of the K blocks includes:

a predetermined starting token at the front of that block;

a predetermined ending token at the end of that block; and

tokens between the predetermined starting token and the predetermined ending token that represent the text of that one of the K blocks; and

generating a respective one of K first representations based on the respective one of the K blocks of text,

where K is an integer greater than 2 and the K first representations are one of matrices and vectors;

a means including first transformer layers having a transformer architecture and first bidirectional gated recurrent neural network layers, the first bidirectional gated recurrent neural network layers interleaved between the first transformer layers, for generating global and block specific second representations based on the K first representations and propagating local information from ones of the K blocks of text to the other ones of the K blocks of text,

the global second representation including information for each of the K blocks of text and being one of a matrix and a vector, and the block specific second representations including information for the respective ones of the K blocks of text and being one of matrices and vectors;

a means including second transformer layers having a transformer architecture and second bidirectional gated recurrent neural network layers, the second bidirectional gated recurrent neural network layers interleaved between the second transformer layers, for generating global and block specific third representations based on the second representations, the third representations being one of matrices and vectors,

the global third representation including information for each of the K blocks of text, and the block specific third representations including information for the respective ones of the K blocks of text;

a means for selecting ones of the K blocks of text based on the third representations,

wherein the means for selecting ones of the K blocks of text being for classifying each of the K blocks of text, based on the third representations, as being either a first state or a second state and selecting the ones of the K blocks of text based on the classifications; and

a means for generating a summary of the document from text of the selected ones of the K blocks of text.

17 . The summarization system of claim 16 wherein: the predetermined starting token corresponds to a classification (CLS) token; the predetermined ending token corresponds to a separation (SEP) token; and the tokens between the predetermined starting token and the predetermined ending token correspond to at least one sentence of text from the document.

18 . A summarization method, comprising:

receiving K blocks of text, respectively, of a document to be summarized,

wherein each of the K blocks includes:

a predetermined starting token at the front of that block;

a predetermined ending token at the end of that block; and

tokens between the predetermined starting token and the predetermined ending token that represent the text of that one of the K blocks;

generating K first representations based on the K blocks of text, respectively, where K is an integer greater than 2 and the K first representations are one of matrices and vectors;

generating, using first transformer layers having a transformer architecture and first bidirectional gated recurrent neural network layers, the first bidirectional gated recurrent neural network layers interleaved between the first transformer layers, global and block specific second representations based on the K first representations and propagating local information from ones of the K blocks of text to the other ones of the K blocks of text,

the global second representation including information for each of the K blocks of text and being one of a matrix and a vector, and the block specific second representations including information for the respective ones of the K blocks of text and being one of matrices and vectors;

generating, using second transformer layers having a transformer architecture and second bidirectional gated recurrent neural network layers, the second bidirectional gated recurrent neural network layers interleaved between the second transformer layers, global and block specific third representations based on the second representations, the third representations being one of matrices and vectors,

the global third representation including information for each of the K blocks of text, and the block specific third representations including information for the respective ones of the K blocks of text;

selecting ones of the K blocks of text based on the third representations,

the selecting ones of the K blocks of text including:

classifying each of the K blocks of text, based on the third representations, as being either a first state or a second state; and

selecting the ones of the K blocks of text based on the classifications; and

generating a summary of the document from text of the selected ones of the K blocks of text.

19 . The summarization method of claim 18 wherein the K first representations each include a token embedding, a segment embedding, and a positional encoding, the positional encoding representing a position of the token in its one of the K blocks.

20 . The summarization method of claim 18 further comprising generating fourth representations based on the third representations,

wherein selecting ones of the K blocks includes selecting the ones of the K blocks based on the fourth representations.

21 . The summarization method of claim 18 wherein the K blocks each include at least one sentence of text from the document.

22 . The summarization method of claim 18 wherein: the predetermined starting token corresponds to a classification (CLS) token; the predetermined ending token corresponds to a separation (SEP) token; and the tokens between the predetermined starting token and the predetermined ending token correspond to at least one sentence of text from the document.

23 . The summarization method of claim 18 wherein at least one of the first bidirectional gated recurrent neural network layers includes one of a bidirectional gate recurrent unit (BiGRU) and a bidirectional long short-term memory (BILSTM).

24 . A method comprising:

receiving a query as input;

obtaining search results based on the query; and

performing the summarization method of claim 18 to produce summaries of the search results; and

outputting the search results and the summaries of the search results.