Granted Patent B1

US 12,293,843 · App. 18/815,494 · Granted May 6, 2025

Inventors: Zachary Michael Ziegler (Cambridge, MA); Jonas Sebastian Wulff (Glendale, CA); Evan Hernandez (Wimauma, FL); Daniel Joseph Nadler (Nassau, BS)

Assignee: Xyla Inc.

G16H50/70G16H10/20

Abstract

Claims (79)

1. A method performed by one or more computers, the method comprising:

obtaining a set of input text sequences;

generating a collection of structured data records from the set of input text sequences using an extraction neural network, wherein each structured data record defines a structured representation of a corresponding input text sequence with reference to a predefined schema of semantic categories, and wherein generating each structured data record comprises:

processing an input text sequence using the extraction neural network to generate an output text sequence that defines a corresponding structured data record, comprising, for each position in the output text sequence:

processing a sequence of embeddings representing the input text sequence and any part of the output text sequence preceding the position in the output text sequence in accordance with trained values of a set of extraction neural network parameters to generate a score distribution over a set of tokens; and

selecting a token, in accordance with the score distribution over the set of tokens, to occupy the position in the output text sequence;

wherein the extraction neural network has been trained by a machine learning training technique to perform a natural language understanding task;

filtering the collection of structured data records to identify and remove structured data records that are predicted to be unreliable; and

processing the collection of structured data records to generate an article that is directed to a selected topic and that aggregates information from across multiple structured data records.

2. The method of claim 1 , wherein each input text sequence is extracted from a document describing a clinical trial.

3. The method of claim 1 , wherein for each structured data record:

the output text sequence generated by the extraction neural network includes delimiters that, for each semantic category in the schema of semantic categories, designate a respective text string from the output text sequence as being included in the semantic category; and

for each semantic category in the schema of semantic categories, the text string from the output text sequence that is designated as being included in the semantic category expresses information from the input text sequence that is relevant to the semantic category.

4. The method of claim 1 , wherein the predefined schema includes respective semantic categories corresponding to one or more of: a size of a population studied in a clinical trial, an age group of a population studied in a clinical trial, a medical intervention applied to a population in a clinical trial, a variable under study in a population in a clinical trial, or a result of a clinical trial.

5. The method of claim 1 , wherein filtering the collection of structured data records to identify and remove structured data records that are predicted to be unreliable comprises, for each structured data record:

processing the structured data record to evaluate whether the structured data record satisfies each of one or more reliability criteria; and

generating a reliability prediction characterizing a predicted reliability of information included in the structured data record based on a result of evaluating whether the structured data record satisfies the reliability criteria.

6. The method of claim 5 , wherein processing the structured data record to evaluate whether the structured data record satisfies each of one or more reliability criteria comprises:

selecting a semantic category from the schema of semantic categories;

determining whether the text string included in semantic category in the structured data record is found in the corresponding input text sequence;

determining whether a reliability criterion is satisfied based on whether the text string included in the semantic category in the structured data record is found in the corresponding input text sequence.

7. The method of claim 5 , wherein processing the structured data record to evaluate whether the structured data record satisfies each of one or more reliability criteria comprises:

selecting a semantic category from the schema of semantic categories; and

determining a confidence of the extraction neural network in generating the text string included in the semantic category in the structured data record; and

determining whether a reliability criterion is satisfied based on the confidence of the extraction neural network in generating the text string included in the semantic category in the structured data record.

8. The method of claim 5 , wherein processing the structured data record to evaluate whether the structured data record satisfies each of one or more reliability criteria comprises:

generating a measure of semantic consistency between the structured data record and the corresponding input text sequence; and

determining whether a reliability criterion is satisfied based on the measure of semantic consistency between the structured data record and the corresponding input text sequence.

9. The method of claim 8 , wherein generating the measure of semantic consistency between the structured data record and the corresponding input text sequence comprises:

generating a summary text sequence, based on the structured data record, that summarizes at least some of the information included in the structured data record;

generating an augmented text sequence by combining: (i) the input text sequence, and (ii) the summary text sequence based on the structured data record;

generating a likelihood value for the augmented text sequence using a natural language processing neural network; and

determining the measure of semantic consistency between the structured data record and the input text sequence based on the likelihood value for the augmented text sequence.

10. The method of claim 9 , wherein generating the likelihood value for the augmented text sequence using a natural language processing neural network comprises:

processing the augmented text sequence using the natural language processing neural network to generate a respective score distribution over a set of tokens for each position in the augmented text sequence; and

generating the likelihood value for the augmented text sequence based on, for each position in the augmented text sequence, a score for the token at the position in the augmented text sequence under the score distribution generated by the natural language processing neural network for the position.

11. The method of claim 1 , further comprising:

selecting a semantic category from the schema of semantic categories;

clustering text strings included in the semantic category across the collection of structured data records; and

updating the collection of structured data records based on the clustering.

12. The method of claim 11 , wherein clustering the text strings included in the semantic category across the collection of structured data records comprises:

generating a set of text strings that comprises, for each structured data record in the collection of structured data records, the text string included in the semantic category in the structured data record; and

clustering the set of text strings to generate a partition of the set of text strings into a plurality of clusters.

13. The method of claim 1 , wherein processing the collection of structured data records to generate the article directed to the topic comprises:

generating an article that includes natural language text, one or more graphical visualizations, or both.

14. The method of claim 1 , wherein a request to generate the article specifies a style for the article from a set of possible styles, and wherein processing the structured data records to generate the article directed to the topic comprises:

mapping the style specified in the request to a corresponding set of programmatic instructions, expressed in a template programming language, that is associated with the style; and

applying the set of programmatic instructions associated with the style to the structured data records to generate the article directed to the topic.

15. The method of claim 14 , wherein the set of possible styles includes respective styles corresponding to different education levels.

16. The method of claim 1 , wherein generating the article directed to the topic comprises:

generating a textual summary of the topic; and

including the textual summary of the topic in the article directed to the topic.

17. The method of claim 16 , wherein generating the textual summary of the topic comprises:

obtaining a text string defining the topic;

generating one or more search queries that include the text string defining the topic;

extracting a plurality of text sequences from search results obtained by querying a search engine using the search queries;

classifying one or more of the text sequences as being relevant to the topic using a classification neural network; and

generating the textual summary of the topic using the text sequences classified as being relevant to the topic.

18. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

obtaining a set of input text sequences;

generating a collection of structured data records from the set of input text sequences using an extraction neural network, wherein each structured data record defines a structured representation of a corresponding input text sequence with reference to a predefined schema of semantic categories, and wherein generating each structured data record comprises:

processing an input text sequence using the extraction neural network to generate an output text sequence that defines a corresponding structured data record, comprising, for each position in the output text sequence:

processing a sequence of embeddings representing the input text sequence and any part of the output text sequence preceding the position in the output text sequence in accordance with trained values of a set of extraction neural network parameters to generate a score distribution over a set of tokens; and

selecting a token, in accordance with the score distribution over the set of tokens, to occupy the position in the output text sequence;

wherein the extraction neural network has been trained by a machine learning training technique to perform a natural language understanding task;

filtering the collection of structured data records to identify and remove structured data records that are predicted to be unreliable; and

processing the collection of structured data records to generate an article that is directed to a selected topic and that aggregates information from across multiple structured data records.

19. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining a set of input text sequences;

generating a collection of structured data records from the set of input text sequences using an extraction neural network, wherein each structured data record defines a structured representation of a corresponding input text sequence with reference to a predefined schema of semantic categories, and wherein generating each structured data record comprises:

processing an input text sequence using the extraction neural network to generate an output text sequence that defines a corresponding structured data record, comprising, for each position in the output text sequence:

processing a sequence of embeddings representing the input text sequence and any part of the output text sequence preceding the position in the output text sequence in accordance with trained values of a set of extraction neural network parameters to generate a score distribution over a set of tokens; and

selecting a token, in accordance with the score distribution over the set of tokens, to occupy the position in the output text sequence;

wherein the extraction neural network has been trained by a machine learning training technique to perform a natural language understanding task;

filtering the collection of structured data records to identify and remove structured data records that are predicted to be unreliable; and

processing the collection of structured data records to generate an article that is directed to a selected topic and that aggregates information from across multiple structured data records.

20. The one or more non-transitory computer storage media of claim 19 , wherein each input text sequence is extracted from a document describing a clinical trial.

Assignments (2)

Continuity (6)

References Cited (28)