Automatic flexible schema detection and migration
A method and apparatus for automatic schema detection and migration is disclosed. In embodiments, a file including NoSQL data is received and one or more data types are detected in a hierarchical data table description. Within a record of the NoSQL data, which may be stored in a JSON format, a field name and its data type, are stored in a schema describing the data table. As additional records are parsed, the schema is updated to include additional field names and data types, and may include designations such as repeated and optional, for some fields. In embodiments, the schema is a serialized data format, such as Google Protocol Buffers (Protobuf).
1. A data storage device, comprising:
one or more memory modules; and
a controller comprising a processor configured to perform a method for data schema detection and migration, the method comprising:
receiving a portion of a file comprising schema-less data, wherein the schema-less data comprises a plurality of records;
identifying one or more fields of a first record of the plurality of records in the schema-less data, wherein each of the one or more fields corresponds to a respective data type of one or more data types;
creating a schema having one or more columns for the schema-less data based on the identifying, wherein:
each column of the one or more columns corresponds to a respective field of the one or more fields of the first record of the plurality of records;
each field of the one or more fields of the first record is a required field;
each value of each entry in the required field is required;
each field of the one or more fields is able to be changed to an optional field from the required field; and
each value of each entry in the optional field is optional;
storing the schema and the schema-less data from the file in the one or more memory modules; and
changing a column of the stored schema from being the required field to being the optional field responsive to determining that a corresponding field of one or more fields of a second record of the plurality of records of the schema-less data is absent, wherein the optional field exists in the stored schema.
2. The data storage device of claim 1 , wherein the method further comprises converting each record of the plurality of records to a table, according to the schema, and wherein storing the schema and data from the file comprises storing the schema and the table.
3. The data storage device of claim 2 , wherein the method further comprises upon identifying a mismatched field of a record, providing the field mismatch to an exception handler.
4. The data storage device of claim 3 , wherein the mismatched field comprises one of a new field, a changed field type, and a missing field.
5. The data storage device of claim 3 , wherein the schema is updated to an updated schema that has been updated based on the mismatched field.
6. The data storage device of claim 5 , wherein previously converted records of the table are converted to the updated schema.
7. The data storage device of claim 5 , wherein previously unconverted records of the file are based on the updated schema and added to the table.
8. A controller for a data storage device, comprising:
an input/output (I/O) to one or more memory devices; and
a processor configured to execute a method for data schema detection and migration, the method comprising:
receiving a portion of a file comprising a plurality of field-delimited units of document based data;
detecting a data type of a first field-delimited unit of the plurality of field-delimited units of document based data;
defining a schema for the first field-delimited unit of the plurality of field-delimited units of document based data, based on the data type, wherein:
the schema comprises one or more columns, wherein:
each of the one or more columns corresponds with a respective detected data type;
each of the one or more columns of the first field-delimited unit is a required field;
each value of each entry corresponding to a column having required entries is required;
each column is able to be changed to an optional field from the required field; and
each value of each entry in the optional field is optional;
storing the file and the schema; and
changing a column of the stored schema from being the required field to being the optional field responsive to determining that a corresponding data type of a second field-delimited unit of the plurality of field-delimited unit is absent, wherein the optional field exists in the stored schema.
9. The controller of claim 8 , wherein the method further comprises:
defining a data table comprising a plurality of records, based on the file and the schema.
10. The controller of claim 9 , wherein the method further comprises executing one of a query, a record insert, a record update, and a record deletion, on the data table.
11. The controller of claim 9 , wherein the method further comprises detecting a field mismatch comprising one of detecting a new field not present in the schema, a change of data type, or a missing field.
12. The controller of claim 11 , wherein the method further comprises generating a new schema by updating the schema based on the field mismatch, wherein updating the schema comprises one of:
updating the schema to include the new field;
updating the data type of the field; and
updating the schema to a field designation of one of required field and the optional field.
13. The controller of claim 12 , wherein the method further comprises updating the data table based on the new schema.
14. The controller of claim 12 , wherein the method further comprises converting additional data from the file to the data table, based on the new schema.
15. The controller of claim 8 , further comprising identifying one of the plurality field-delimited units of document based data as one of hierarchy, repeated, and optional.
16. A system for storing data, comprising:
one or more memory means; and
a solid state drive (SSD) controller configured to carry out a method for data schema detection and migration, the method comprising:
detecting a field hierarchy in a portion of a file comprising one of structured or semi-structured document based data, wherein the portion of the file comprises a plurality of fields, data associated with each field of the plurality of fields, and a data type associated with the data;
defining a schema based on the field hierarchy, wherein:
the schema comprises one or more columns, wherein:
each column corresponds to a respective field of the plurality of fields;
each field of the plurality of fields of the portion of the file is a required field;
each value of each entry in the required field is required;
each field of the plurality of fields is able to be changed to the optional field from the required field; and
each value of each entry in the optional field is optional;
defining a data table based on the schema and the file; and
changing a column of the schema from being the required field to being the optional field responsive to determining that a corresponding field in a next portion of the file is absent, wherein the optional field exists in the stored schema.
17. The system of claim 16 , the method further comprising identifying a field mismatch in another portion of the file, based on the schema.
18. The system of claim 17 , wherein the method further comprises generating a new schema based on modifying the schema based on the field mismatch, and updating the table based on the new schema.
19. The system claim 18 , the method further comprising executing one of a query, a record insert, a record deletion, and a record modification of the data table.
20. The system of claim 16 , wherein the schema is a data serialization structure.