Database system for analysis of longitudinal data sets
A database system performs analytics on longitudinal data, such as medical histories with events occurring to patients over time. Input data is processed into streams of events. A set of indexes of event characteristics is generated. A set of patient event histories, partitioned by patient, is generated. Several copies of event data are stored, each copy being structured to support a specific analytical task. Data is partitioned and distributed over several hardware nodes to allow parallel queries. Definitions of sets of candidate patients are translated into sets of filters applied to the set of indexes. Data for these candidates are input to analytical modules. Reports from analysis are automatically generated to be compatible with standard guidelines for reporting. Workflows support one task or a set of closely related tasks by offering the user a defined sequence of query options and analytic choices specifically arranged for the task.
1. A database system, comprising:
a plurality of computing hardware nodes for analyzing longitudinal data for a plurality of entities, each computing hardware node comprises at least a memory to store longitudinal data and a processor coupled to the memory, the longitudinal data comprising events of a plurality of types and dates on which the events occurred with respect to the entities, and an index mapping to map events to the entities; wherein at least one or more of the computing hardware nodes to:
receive a query definition,
access the longitudinal data to identify entities having events that match the query definition, each of the identified matching entities comprising a cohort;
generate a transformation measure with respect to each of the identified matching entities in view of the query definition, wherein to generate the transformation measure, the computing hardware nodes further to:
identify an event value for the transformation measure associated with at least one entity of the identified matching entities, and
apply the transformation measure to related events of the longitudinal data associated with the at least one entity to normalize distribution of the event value with respect to the related events;
receive identifiers of the identified matching entities that correspond to the query definition;
access the longitudinal data to retrieve events associated with the identifiers and the generated transformation measures for the identified matching entities; and
provide results of a statistical analysis performed on the cohort using the longitudinal data.
2. The database system of claim 1 , wherein the at least one or more of the computing hardware nodes further to:
receive the results of the statistical analysis, and
provide a report describing the cohort definition, the generated transformation measures and the statistical analysis including references to scientific literature.
3. The database system of claim 1 , wherein the index comprises a bitmap index.
4. The database system of claim 1 , wherein the index comprises an inverted index.
5. The database system of claim 1 , wherein the longitudinal data comprises a first portion stored in a primary storage device and a second portion stored in a secondary storage device, the first portion including data accessed more frequently relative to data accessed in the second portion.
6. The database system of claim 1 , wherein the longitudinal data comprises a string of integers, each integer representing a field of data.
7. The database system of claim 1 , wherein the computing hardware nodes further to determine a score indicating whether a frequency of occurrence of event data associated with the identified matching entities is increasing or decreasing during a determined time period.
8. The database system of claim 1 , further comprising an interactive user interface receiving user input to define transformation measures to apply to the longitudinal data.
9. The database system of claim 1 , further comprising an interactive user interface receiving user input to define the query definition to apply to the longitudinal data.
10. The database system of claim 9 , wherein the at least one or more of the computing hardware nodes further to log activity related to defining the query definitions.
11. The database system of claim 9 , wherein the at least one or more of the computing hardware nodes further to store versions of the query definitions.
12. The database system of claim 1 , wherein the at least one or more of the computing hardware nodes further to select a choice of workflows to support one task or a set of closely related tasks by providing a user a defined sequence of query options and analytic selections arranged for the task.
13. A method, comprising:
receiving, by a processing device of a computing hardware node, a query definition associated with longitudinal data, the longitudinal data comprising events of a plurality of types and dates on which the events occurred with respect to a plurality of entities, and an index mapping to map the events to the entities;
accessing the longitudinal data to identify entities having events that match the query definition, each of the identified matching entities comprising a cohort;
generating, by the processing device, a transformation measure with respect to each of the identified matching entities in view of the query definition, wherein the generating comprises:
identifying, by the processing device, an event value for the transformation measure associated with at least one entity of the identified matching entities, and
applying, by the processing device, the transformation measure to related events of the longitudinal data associated with the at least one entity to normalize distribution of the event value with respect to the related events;
receiving identifiers of the identified matching entities that correspond to the query definition;
accessing the longitudinal data to retrieve events associated with the identifiers and the generated transformation measures for the identified matching entities; and
providing results of a statistical analysis performed on the cohort using the longitudinal data.
14. A non-transitory computer readable storage medium comprising instructions that, when executed by a processing device of a computing hardware node, cause the processing device to:
receiving, by the processing device, a query definition associated with longitudinal data, the longitudinal data comprising events of a plurality of types and dates on which the events occurred with respect to a plurality of entities, and an index mapping to map the events to the entities;
access the longitudinal data to identify entities having events that match the query definition, each of the identified matching entities comprising a cohort;
generate a transformation measure with respect to each of the identified matching entities in view of the query definition, wherein to generate the transformation measure, the processing device further to:
identify an event value for the transformation measure associated with at least one entity of the identified matching entities, and
apply the transformation measure to related events of the longitudinal data associated with the at least one entity to normalize distribution of the event value with respect to the related events;
receive identifiers of the identified matching entities that correspond to the query definition;
access the longitudinal data to retrieve events associated with the identifiers and the generated transformation measures for the identified matching entities; and
provide results of a statistical analysis performed on the cohort using the longitudinal data.