Generating structured data records using an extraction neural network
A computer-implemented method is disclosed for generating structured data representations from unstructured input text sequences through a neural network-based extraction pipeline. The method involves acquiring input text sequences and processing them with an extraction neural network to produce output text sequences. Each output text sequence includes embedded delimiter tokens that annotate specific text segments, mapping them to semantic categories defined within a structured schema. The annotated text segments encapsulate semantically relevant information extracted from the input sequence. The output text sequences are subsequently processed to generate structured data records.
1. A method performed by one or more computers, the method comprising:
obtaining an input text sequence;
processing the input text sequence using an extraction neural network, in accordance with a set of extraction neural network parameters, to generate a corresponding output text sequence, wherein for each semantic category in a predefined schema of semantic categories:
the output text sequence includes delimiters that designate a respective text string from the output text sequence as being included in the semantic category; and
the text string from the output text sequence that is designated as being included in the semantic category expresses information from the input text sequence that is relevant to the semantic category; and
processing the output text sequence to generate a structured data record that defines a structured representation of information in the input text sequence with reference to the predefined schema of semantic categories.
2. The method of claim 1 , wherein the input text sequence is extracted from a document.
3. The method of claim 2 , wherein the document is a medical paper describing a clinical trial.
4. The method of claim 3 , wherein the predefined schema of semantic categories includes respective semantic categories corresponding to one or more of: a size of a population studied in the clinical trial, an age group of the population studied in the clinical trial, a medical intervention applied to the population in the clinical trial, a variable under study in the population in the clinical trial, or a result of the clinical trial.
5. The method of claim 1 , wherein processing the output text sequence to generate a structured data record comprises, for each semantic category in the schema:
processing the output text sequence to identify the respective text string from the output text sequence that is designated as being included in the semantic category; and
populating the semantic category in the structured data record with the text string from the output text sequence that is designated as being included in the semantic category.
6. The method of claim 5 , wherein for each semantic category in the schema, processing the output text sequence to identify the respective text string from the output text sequence that is designated as being included in the semantic category comprises:
identifying a position of a delimiter in the output text sequence that defines a start or an end of the text string from the output text sequence that is designated as being included in the semantic category.
7. The method of claim 1 , wherein the delimiters included in the output text sequence define a partition of some or all of the output text sequence into respective text strings that are each designated as being included in a respective semantic category.
8. The method of claim 1 , wherein the output text sequence includes delimiters that define a partition of the output text sequence into a plurality of subsequences, wherein each subsequence corresponds to a respective structured data record.
9. The method of claim 8 , wherein processing the output text sequence to generate the structured data record comprises:
processing the output text sequence to generate a respective structure data record corresponding to each of the plurality of subsequences of the output text sequences delineated by the delimiters.
10. The method of claim 1 , wherein the schema includes a plurality of semantic categories.
11. The method of claim 1 , wherein the extraction neural network generates the output text sequence autoregressively.
12. The method of claim 1 , wherein the extraction neural network generates each token in the output text sequence based on: (i) the input text sequence, and (ii) any preceding tokens in the output text sequence.
13. The method of claim 12 , wherein to generate each token in the output text sequence, the extraction neural network performs operations comprising:
processing: (i) the input text sequence, and (ii) any preceding tokens in the output text sequence, to generate a score distribution over a set of tokens; and
selecting a token for position in the output text sequence in accordance with the score distribution over the set of tokens.
14. The method of claim 13 , wherein selecting the token for the position in the output text sequence in accordance with the score distribution over the set of tokens comprises:
sampling from a probability distribution over the set of tokens, wherein the probability distribution over the set of tokens is based on the score distribution over the set of tokens.
15. The method of claim 13 , wherein selecting the token for the position in the output text sequence in accordance with the score distribution over the set of tokens comprises:
selecting the token having a highest score, from among the set of tokens, under the score distribution over the set of tokens.
16. The method of claim 1 , wherein the extraction neural network includes one or more self-attention neural network layers.
17. The method of claim 1 , wherein the extraction neural network has been trained on a set of training examples, wherein each training example comprises: (i) a training text sequence, and (ii) a target text sequence that should be generated by the extraction neural network by processing the training text sequence.
18. The method of claim 17 , wherein for each training example, training the extraction neural network on the training example comprises:
processing an input text sequence based on the training example using the extraction neural network to generate, for each position in the target text sequence, a respective score distribution over a set of tokens;
determining gradients of an objective function that, for each position in the target text sequence, measures an error between: (i) the score distribution over the set of tokens generated by the extraction neural network for the position, and (ii) a token at the position in the target text sequence.
19. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining an input text sequence;
processing the input text sequence using an extraction neural network, in accordance with a set of extraction neural network parameters, to generate a corresponding output text sequence, wherein for each semantic category in a predefined schema of semantic categories:
the output text sequence includes delimiters that designate a respective text string from the output text sequence as being included in the semantic category; and
the text string from the output text sequence that is designated as being included in the semantic category expresses information from the input text sequence that is relevant to the semantic category; and
processing the output text sequence to generate a structured data record that defines a structured representation of information in the input text sequence with reference to the predefined schema of semantic categories.
20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining an input text sequence;
processing the input text sequence using an extraction neural network, in accordance with a set of extraction neural network parameters, to generate a corresponding output text sequence, wherein for each semantic category in a predefined schema of semantic categories:
the output text sequence includes delimiters that designate a respective text string from the output text sequence as being included in the semantic category; and
the text string from the output text sequence that is designated as being included in the semantic category expresses information from the input text sequence that is relevant to the semantic category; and
processing the output text sequence to generate a structured data record that defines a structured representation of information in the input text sequence with reference to the predefined schema of semantic categories.