Systems and methods for cohort analysis using compressed data objects enabling fast memory lookups
Systems and methods for structuring unstructured data according to a data object structure that enables fast query look-ups across a variety of space and time dimensions. Furthermore, many embodiments optimize the storage of the data objects using a set of compression techniques that configure the data types used for the data objects based on properties of the stored data. Furthermore, many embodiments provide are able to service query look-up requests without having to deserialize data within the byte stream format as stored in memory by encoding information that provide memory locations for requested data, thereby allowing for the immediate retrieval of the data as it is stored in the persistent memory.
1. A method for computer-implemented data analysis, comprising:
receiving a search query to analyze medical data related to a patient, wherein the search query comprises a plurality of parameters related to data components for a plurality of patient demographics;
determining a data object storing a particular data component relevant to the search query, wherein the data object:
corresponds to the patient; and
comprises:
a plurality of data components, including the particular data component, wherein:
each data component of the plurality of data components corresponds to a different type of data value related to the medical data,
medical data for the patient is encoded, within the plurality of data components, in a serialized in-memory byte-stream format,
the medical data is received from a plurality of different medical information sources, and
the data object is stored, in its entirety, at a unique and continuous memory location; and
at least one header providing information regarding memory mappings of the plurality of data components within a body of the data object, the at least one header comprising:
an offset for each of the plurality of data components in the body of the data object, and
encoding information, wherein the encoding information identifies at least one data type used in storing the offset for each of the plurality of data components;
retrieving a particular data value, responding to the search query, directly from the particular data component, wherein:
retrieving the particular data value comprises using the encoding information, the memory mappings and the offset to identify a memory location of the particular data component, and
the particular data value is retrieved in a serialized in-memory byte-stream format, while the data object remains serialized; and
generating an identification of a cohort of patients for the search query, wherein the identification includes the particular data value.
2. The method of claim 1 , wherein serialized data refers to data translated into a byte-stream format for storage in memory and deserialization comprises extracting a data structure from a series of bytes.
3. The method of claim 1 , wherein the at least one header of the data object further comprises information regarding a data type of the data object.
4. The method of claim 1 , wherein:
a data type of the at least one data type is selected from the group consisting of an INTEGER data type, a BYTE data type, and a SHORT data type; and
the offset of each data component provides a number of bytes between a start of the data component in the body of the data object, and a start of the data object in memory.
5. The method of claim 1 , wherein determining the data object storing the particular data component relevant to the search query comprises: consulting a data index mapping a plurality of data objects to on-disk shards storing the plurality of data objects.
6. The method of claim 5 , wherein determining the data object storing the particular data component relevant to the search query further comprises: consulting a memory index that maps each data object in the plurality of data objects to a memory space.
7. A system for data analysis, comprising:
a processor; and
memory containing software that, when executed, directs the processor to:
receive a search query to analyze medical data related to a patient, wherein the search query comprises a plurality of parameters related to data components for a plurality of patient demographics;
determine a data object storing a particular data component relevant to the search query, wherein the data object:
corresponds to the patient; and
comprises:
a plurality of data components, including the particular data component, wherein:
each data component of the plurality of data components corresponds to a different type of data value related to the medical data,
medical data for the patient is encoded, within the plurality of data components, in a serialized in-memory byte-stream format,
the medical data is received from a plurality of different medical information sources, and
the data object is stored, in its entirety, at a unique and continuous memory location; and
at least one header providing information regarding memory mappings of the plurality of data components within a body of the data object, the at least one header comprising:
an offset for each of the plurality of data components in the body of the data object, and
encoding information, wherein the encoding information identifies at least one data type used in storing the offset for each of the plurality of data components;
retrieve a particular data value, responding to the search query, directly from the particular data component, wherein:
retrieving the particular data value comprises using the encoding information, the memory mappings, and the offset to identify a memory location of the particular data component, and
the data value is retrieved in a serialized in-memory byte-stream format, while the data object remains serialized; and
generate an identification of a cohort of patients for the search query, wherein the identification includes the particular data value.
8. The system of claim 7 , wherein serialized data refers to data translated into a byte-stream format for storage in the memory and deserialization comprises extracting a data structure from a series of bytes.
9. The system of claim 7 , wherein the at least one header of the data object further comprises information regarding a data type of the data object.
10. The system of claim 7 , wherein:
a data type of the at least one data type is selected from the group consisting of an INTEGER data type, a BYTE data type, and a SHORT data type; and
the offset of each data component provides a number of bytes between a start of the data component in the body of the data object, and a start of the data object in memory.
11. The system of claim 7 , wherein determining the data object storing the particular data component relevant to the search query comprises: consulting a data index mapping a plurality of data objects to on-disk shards storing the plurality of data objects.
12. The system of claim 11 , wherein determining the data object storing the particular data component relevant to the search query further comprises: consulting a memory index that maps each data object in the plurality of data objects to a memory space.
13. The system of claim 7 , wherein:
the plurality of data components comprise a plurality of different data types; and
a data component from the plurality of data components possesses at least one data type selected from the group consisting of a hash map, a list, a measured value list, and a computed value list.
14. The method of claim 1 , wherein:
the plurality of data components comprise a plurality of different data types; and
a data component from the plurality of data components possesses at least one data type selected from the group consisting of a hash map, a list, a measured value list, and a computed value list.
15. A non-transitory computer-readable medium comprising instructions that, when executed, are configured to cause a processor to perform a process for computer-implemented data analysis, comprising:
receiving a search query to analyze medical data related to a patient, wherein the search query comprises a plurality of parameters related to data components for a plurality of patient demographics;
determining a data object storing a particular data component relevant to the search query, wherein the data object:
corresponds to the patient; and
comprises:
a plurality of data components, including the particular data component, wherein:
each data component of the plurality of data components corresponds to a different type of data value related to the medical data;
medical data for the patient is encoded, within the plurality of data components, in a serialized in-memory byte-stream format;
the medical data is received from a plurality of different medical information sources; and
the data object is stored, in its entirety, at a unique and continuous memory location; and
at least one header providing information regarding memory mappings of the plurality of data components within a body of the data object, the at least one header comprising:
an offset for each of the plurality of data components in the body of the data object; and
encoding information, wherein the encoding information identifies at least one data type used in storing the offset for each of the plurality of data components;
retrieving a particular data value, responding to the search query, directly from the particular data component, wherein:
retrieving the particular data value comprises using the encoding information, the memory mappings, and the offset to identify a memory location of the particular data component; and
the particular data value is retrieved in a serialized in-memory byte-stream format, while the data object remains serialized; and
generating an identification of a cohort of patients for the search query, wherein the identification includes the particular data value.
16. The non-transitory computer-readable medium of claim 15 , wherein serialized data refers to data translated into a byte-stream format for storage in memory and deserialization comprises extracting a data structure from a series of bytes.
17. The non-transitory computer-readable medium of claim 15 , wherein the at least one header of the data object further comprises information regarding a data type of the data object.
18. The non-transitory computer-readable medium of claim 15 , wherein:
a data type of the at least one data type is selected from the group consisting of an INTEGER data type, a BYTE data type, and a SHORT data type; and
the offset of each data component provides a number of bytes between a start of the data component in the body of the data object, and a start of the data object in memory.
19. The non-transitory computer-readable medium of claim 15 , wherein determining the data object storing the particular data component relevant to the search query comprises: consulting a data index mapping a plurality of data objects to on-disk shards storing the plurality of data objects.
20. The non-transitory computer-readable medium of claim 19 , wherein determining the data object storing the particular data component relevant to the search query further comprises: consulting a memory index that maps each data object in the plurality of data objects to a memory space.