System and methods for object version tracking and read-time/write-time data federation
Systems and methods are provided in order to avoid ingesting the entire contents of one or more data sources. An object may be associated with one or more pieces/fragments of data that can be stored in different data sources. When a request for the object is initiated, a search for those pieces/fragments of data can be performed. Nodes associated with each data source may generate data queries appropriate for its corresponding data source, retrieve, and if needed transform the data into an object-based data structure. Any pieces/fragments of data that have been discovered and retrieved can be joined, e.g., by an application programming interface server, and forwarded to a requesting client or application. In this way, only data relevant to the object is obtained. Moreover, object versioning can be employed so that the most up-to-date data is obtained.
1. A system for analyzing data, comprising:
one or more processors, the processors comprising adapter nodes; and
a memory storing instructions that, when executed by the one or more processors, cause the system to:
receive a request, from an application, for an object;
retrieve, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
each datastore corresponds to a particular data type,
at least one datastore has a different data type compared to a data type of another datastore;
the adapter nodes are intermediaries between the datastores and application programming interface (API) servers, the adapter nodes being specifically generated to retrieve data of a particular data type; and
the retrieving comprises:
determining, by each of the subset of the adapter nodes, a query format to search a datastore according to the particular data type of the datastore;
generating, by each of the subset of the adapter nodes, a query to search the datastore based on the determined query format;
retrieving, by each of the subset of the adapter nodes based on the generated query, a data fragment in one of the datastores; and
transforming at least a portion of the data fragments, wherein the transforming is based on a script, the script being generated according to a schema of a datastore from which a particular data fragment of the portion of the data fragments is retrieved and a target ontology of the application or a platform;
upon transformation of at least the portion of the data fragments by the subset of the adapter nodes, join the data fragments by the API servers; and
transmit, by the API servers, the joined data fragments to the application.
2. The system of claim 1 , wherein the adapter nodes further comprise respective caches in which most updated versions of the respective data fragments are stored.
3. The system of claim 1 , further comprising a data analysis platform that stores a different data fragment; and wherein the instructions further cause the system to:
join the different data fragment to the joined data fragments to form composite data fragments, and wherein the transmitting of the joined data fragment comprises transmitting the composite data fragments.
4. The system of claim 1 , wherein the instructions further cause the system to:
inactivate an adapter node and activate a new adapter node to replace the inactivated adapter node.
5. The system of claim 1 , wherein, the instructions further cause the system to:
indicate that the retrieving of the data fragments has been partially completed; and
transmit a partial object view to the application.
6. The system of claim 1 , wherein the instructions further cause the system to:
receive an indication that a data fragment has updated a version; and
in response to receiving the indication, reingesting, at the application, a new version of the data fragment.
7. The system of claim 1 , wherein the instructions further cause the system to:
in response to receiving the request for the object, generate the adapter nodes.
8. The system of claim 1 , wherein the joining of the data fragments comprises converting a first data fragment of a first data type by a first adapter node and a second data fragment of a second data type by a second adapter node into a standardized data type.
9. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
receiving a request, from an application, for an object;
retrieving, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
each datastore corresponds to a particular data type,
at least one datastore has a different data type compared to a data type of another datastore;
the adapter nodes are intermediaries between the datastores and application programming interface (API) servers, the adapter nodes being specifically generated to retrieve data of a particular data type; and
the retrieving comprises:
determining, by each of the subset of the adapter nodes, a query format to search a datastore according to the particular data type of the datastore;
generating, by each of the subset of the adapter nodes, a query to search the datastore based on the determined query format;
retrieving, by each of the subset of the adapter nodes based on the generated query, a data fragment in one of the datastores; and
transforming at least a portion of the data fragments, wherein the transforming is based on a script, the script being generated according to a schema of a datastore from which a particular data fragment of the portion of the data fragments is retrieved and a target ontology of the application or a platform;
upon transformation of at least the portion of the data fragments by the subset of the adapter nodes, joining the data fragments by the API servers; and
transmitting, by the API servers, the joined data fragments to the application.
10. The method of claim 9 , further comprising storing most updated versions of the respective data fragments in respective caches of the adapter nodes.
11. The method of claim 9 , further comprising:
storing a different data fragment in a data analysis platform; and
joining the different data fragment to the joined data fragments to form composite data fragments, wherein the transmitting of the joined data fragment comprises transmitting the composite data fragments.
12. The method of claim 9 , further comprising:
inactivating an adapter node and activating a new adapter node to replace the inactivated adapter node.
13. The method of claim 9 , further comprising:
indicating that the retrieving of the data fragments has been partially completed; and
transmitting a partial object view to the application.
14. The method of claim 9 , wherein a first of the data fragments comprises a tabular format and a second of the data fragments comprises a second, non-object oriented format.
15. The method of claim 9 , further comprising:
receiving an indication that a data fragment has updated a version; and
in response to receiving the indication, reingesting, at the application, a new version of the data fragment.
16. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
receiving a request, from an application, for an object;
retrieving, from datastores using a subset of the adapter nodes, data fragments making up or associated with the requested object, wherein:
each datastore corresponds to a particular data type,
at least one datastore has a different data type compared to a data type of another datastore;
the adapter nodes are separate from the datastores and are intermediaries between the datastores and application programming interface (API) servers, the adapter nodes being specifically generated to retrieve data of a particular data type; and
the retrieving comprises:
determining, by each of the subset of the adapter nodes, a query format to search a datastore according to the particular data type of the datastore;
generating, by each of the subset of the adapter nodes, a query to search the datastore based on the determined query format;
retrieving, by each of the subset of the adapter nodes based on the generated query, a data fragment in one of the datastores; and
transforming at least a portion of the data fragments, wherein the transforming is based on a script, the script being generated according to a schema of a datastore from which a particular data fragment of the portion of the data fragments is retrieved and a target ontology of the application or a platform;
upon transformation of at least the portion of the data fragments by the subset of the adapter nodes, joining the data fragments by the API servers; and
transmitting, by the API servers, the joined data fragments to the application.
17. The non-transitory computer readable medium of claim 16 , wherein the instructions further cause the one or more processors to perform:
storing most updated versions of the respective data fragments in respective caches of the adapter nodes.
18. The non-transitory computer readable medium of claim 16 , wherein the instructions further cause the one or more processors to perform:
storing a different data fragment in a data analysis platform; and
joining the different data fragment to the joined data fragments to form composite data fragments, wherein the transmitting of the joined data fragment comprises transmitting the composite data fragments.
19. The non-transitory computer readable medium of claim 16 , wherein the instructions further cause the one or more processors to perform:
inactivating an adapter node and activating a new adapter node to replace the inactivated adapter node.