Metadata-based generic parsing
Embodiments relate to metadata-based generic parsing of an incoming data object to transform it for different purposes, such as conversion into a format more suitable for a specific storage type. The received data object comprises members organized according to a structure, each member containing data. The object's structure and the member's type are defined by the associated metadata. The metadata is referenced by parsing logic to generate a schema representing the structure of the data object in a different format (such as an ER table schema). The schema is then stored. When data is received, the metadata is referenced to parse the corresponding object's data accordingly and for ingestion into the generated schema and storage.
1. A method comprising:
receiving a data object comprising a plurality of members organized according to a first structure, each member having data and associated metadata;
referencing logic to generate a schema representing the first structure;
storing the schema in a non-transitory computer readable storage medium;
referencing the logic and the schema to parse the data into a relational database table different from the first structure; and
ingesting the data to the relational database table in the non-transitory computer readable storage medium;
wherein generating the schema comprises generating statements to generate table and column names based on member names of the object data during a runtime, and
wherein ingesting the data comprises reading the metadata to address data values from the data object to trigger writes to the relational database table during the runtime,
wherein scalar elements in the data object are stored in a master relational table and associated with an identification stamp comprising a unique identification of the data object, and wherein vector or array elements of the data object are stored in one or more secondary relational tables having a plurality of rows associated with the master relational table by the identification stamp as a foreign key containing an index as a primary key paired with an attribute identifier for each particular vector or array member, wherein different vector or array elements are stored in different secondary relational tables.
2. A method as in claim 1 wherein:
the non-transitory computer readable storage medium comprises a relational database.
3. A method as in claim 2 wherein the schema is generated by creating:
a Data Description Language (DDL) statement; and
a prepared statement.
4. A method as in claim 2 wherein the relational database table comprises a stamp.
5. A method as in claim 1 wherein the data object is in JavaScript Object Notation (JSON), XML, or Comma Separated Value (CSV) format.
6. A method as in claim 1 wherein the logic comprises tree traversal logic.
7. A method as in claim 1 wherein the ingesting is based upon a full history of the data object.
8. A method as in claim 1 wherein the ingesting is based upon a latest version of the data object.
9. A method as in claim 1 wherein:
the non-transitory computer readable storage medium comprises an in-memory database; and
the data is ingested by an in-memory database engine of the in-memory database.
10. A method as in claim 9 wherein the schema is generated by the in-memory database engine.
11. A method as in claim 1 wherein a first secondary relational table, of the one or more secondary relational tables, comprising a vector or array having at least one member comprising an inner vector or inner array comprises a reference to a second secondary relational table, and wherein each row of the second secondary relational table comprises the identification stamp, a reference to the first secondary relational table, and members of the inner vector or inner array.
12. A non-transitory computer readable storage medium embodying a computer program for performing a method, said method comprising:
receiving a data object comprising a plurality of members organized according to a structure, each member having data and associated metadata;
referencing logic and the metadata to generate a schema representing the structure;
storing the schema in a relational database;
referencing the logic and the schema to parse the data into an Entity-Relational (ER) database table; and
ingesting the data to the ER database table stored in the database;
wherein generating the schema comprises generating statements to generate table and column names based on member names of the object data during a runtime, and
wherein ingesting the data comprises reading the metadata to address data values from the data object to trigger writes to the ER database table during the runtime,
wherein scalar elements in the data object are stored in a master relational table and associated with an identification stamp comprising a unique identification of the data object, and wherein vector or array elements of the data object are stored in one or more secondary relational tables having a plurality of rows associated with the master relational table by the identification stamp as a foreign key containing an index as a primary key paired with an attribute identifier for each particular non-scalar element, wherein different vector or array elements are stored in different secondary relational tables.
13. A non-transitory computer readable storage medium as in claim 12 wherein the logic comprises tree traversal logic.
14. A non-transitory computer readable storage medium as in claim 12 wherein the ingesting is based upon a full history of the data object.
15. A non-transitory computer readable storage medium as in claim 12 wherein the ingesting is based upon a latest version of the data object.
16. A computer system comprising:
one or more processors;
a software program, executable on said computer system, the software program configured to:
receive a data object comprising a plurality of members organized according to a structure, each member having data and associated metadata;
reference logic and the metadata to generate a schema representing the structure;
store the schema in the in-memory database;
reference the logic and the schema to parse the data into an Entity-Relational (ER) database table different from the structure; and
ingest the data to the ER database table in the in-memory database;
wherein generating the schema comprises generating statements to generate table and column names based on member names of the object data during a runtime, and
wherein ingesting the data comprises reading the metadata to address data values from the data object to trigger writes to the ER database table during the runtime,
wherein scalar elements in the data object are stored in a master relational table and associated with an identification stamp comprising a unique identification of the data object, and wherein vector or array elements of the data object are stored in one or more secondary relational tables having a plurality of rows associated with the master relational table by the identification stamp as a foreign key containing an index as a primary key paired with an attribute identifier for each particular non-scalar element, wherein different vector or array elements are stored in different secondary relational tables.
17. A computer system as in claim 16 wherein the data object is in JavaScript Object Notation (JSON), XML, or Comma Separated Value (CSV) format.
18. A computer system as in claim 16 wherein the data is ingested based upon a full history of the data object.
19. A computer system as in claim 16 wherein the data is ingested based upon a latest version of the data object.