Unified structured and semi-structured data types in database systems
The subject technology receives a query, the query referencing a unified representation for structured type data and semi-structured type data, the unified representation being provided in storage and in memory during query processing, the unified representation comprising a set of structured type fields that include a set of semi-structured typed fields that enables type safety and enforcement for the set of structured type fields, and flexibility for the set of semi-structured typed fields in a same column, the unified representation in storage including type information for the semi-structured type data as part of the semi-structured type data, the unified representation being utilized for structured type data and semi-structured type data. The subject technology processes the query using the unified representation stored in the memory, the unified representation providing performance parity between structured type data and semi-structured type data.
1. A system comprising:
at least one hardware processor; and
a memory storing instructions that cause the at least one hardware processor to perform operations comprising:
receiving a query, the query referencing a unified representation for structured type data and semi-structured type data, the unified representation being provided in storage and in memory during query processing, the unified representation comprising a set of structured type fields that include a set of semi-structured typed fields that enables type safety and enforcement for the set of structured type fields, and flexibility for the set of semi-structured typed fields in a same column, the unified representation in storage including type information for the semi-structured type data as part of the semi-structured type data, the unified representation being utilized for structured type data and semi-structured type data;
processing the query using the unified representation stored in the memory, the unified representation providing performance parity between structured type data and semi-structured type data;
receiving a set of code statements, the set of code statements including first code indicating a particular structured data type;
determining that the set of code statements includes a definition of content type for an array, an object, or a map;
determining that the set of code statements includes second code to perform a type conversion from the particular structured data type to a particular semi-structured data type; and
performing, using the second code, the type conversion from the particular structured data type to the particular semi-structured data type based at least in part on the definition of content type.
2. The system of claim 1 , wherein the semi-structured type data does not require a prior definition of a schema and a new attribute can be added to the semi-structured type data at a subsequent time.
3. The system of claim 1 , wherein the semi-structured type data is stored in an array data type, an object data type, or a variant data type.
4. The system of claim 1 , wherein the operations further comprise:
receiving a set of query results based on processing the query, the set of query results including a set of structured types.
5. The system of claim 4 , wherein the set of query results comprises a JSON string format.
6. The system of claim 4 , wherein the set of query results comprises an Apache Arrow format.
7. The system of claim 1 , wherein the operations further comprise:
receiving a first semi-structured object;
iterating through a list of fields specified by a target object type;
for each field, determining whether a field with a same name is present in the first semi-structured object; and
in response to the field being found in the first semi-structured object, converting a value of the field to a target field type.
8. The system of claim 1 , wherein the operations further comprise:
receiving first data from a first column of a table;
determining that the first column comprises an array structured data type; and
converting the array structured data type to an array semi-structured data type.
9. The system of claim 1 , wherein the operations further comprise:
receiving first data from a first column of a table;
determining that the first column comprises an object structured data type; and
converting the object structured data type to an object semi-structured data type.
10. A method comprising:
receiving a query, the query referencing a unified representation for structured type data and semi-structured type data, the unified representation being provided in storage and in memory during query processing, the unified representation comprising a set of structured type fields that include a set of semi-structured typed fields that enables type safety and enforcement for the set of structured type fields, and flexibility for the set of semi-structured typed fields in a same column, the unified representation in storage including type information for the semi-structured type data as part of the semi-structured type data, the unified representation being utilized for structured type data and semi-structured type data;
processing the query using the unified representation stored in the memory, the unified representation providing performance parity between structured type data and semi-structured type data;
receiving a set of code statements, the set of code statements including first code indicating a particular structured data type;
determining that the set of code statements includes a definition of content type for an array, an object, or a map;
determining that the set of code statements includes second code to perform a type conversion from the particular structured data type to a particular semi-structured data type; and
performing, using the second code, the type conversion from the particular structured data type to the particular semi-structured data type based at least in part on the definition of content type.
11. The method of claim 10 , wherein the semi-structured type data does not require a prior definition of a schema and a new attribute can be added to the semi-structured type data at a subsequent time.
12. The method of claim 10 , wherein the semi-structured type data is stored in an array data type, an object data type, or a variant data type.
13. The method of claim 10 , further comprising:
receiving a set of query results based on processing the query, the set of query results including a set of structured types.
14. The method of claim 13 , wherein the set of query results comprises a JSON string format.
15. The method of claim 13 , wherein the set of query results comprises an Apache Arrow format.
16. The method of claim 10 , further comprising:
receiving a first semi-structured object;
iterating through a list of fields specified by a target object type;
for each field, determining whether a field with a same name is present in the first semi-structured object; and
in response to the field being found in the first semi-structured object, converting a value of the field to a target field type.
17. The method of claim 10 , further comprising:
receiving first data from a first column of a table;
determining that the first column comprises an array structured data type; and
converting the array structured data type to an array semi-structured data type.
18. The method of claim 10 , further comprising:
receiving first data from a first column of a table;
determining that the first column comprises an object structured data type; and
converting the object structured data type to an object semi-structured data type.
19. A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
receiving a query, the query referencing a unified representation for structured type data and semi-structured type data, the unified representation being provided in storage and in memory during query processing, the unified representation comprising a set of structured type fields that include a set of semi-structured typed fields that enables type safety and enforcement for the set of structured type fields, and flexibility for the set of semi-structured typed fields in a same column, the unified representation in storage including type information for the semi-structured type data as part of the semi-structured type data, the unified representation being utilized for structured type data and semi-structured type data;
processing the query using the unified representation stored in the memory, the unified representation providing performance parity between structured type data and semi-structured type data;
receiving a set of code statements, the set of code statements including first code indicating a particular structured data type;
determining that the set of code statements includes a definition of content type for an array, an object, or a map;
determining that the set of code statements includes second code to perform a type conversion from the particular structured data type to a particular semi-structured data type; and
performing, using the second code, the type conversion from the particular structured data type to the particular semi-structured data type based at least in part on the definition of content type.
20. The non-transitory computer-storage medium of claim 19 , wherein the semi-structured type data does not require a prior definition of a schema and a new attribute can be added to the semi-structured type data at a subsequent time.
21. The non-transitory computer-storage medium of claim 19 , wherein the semi-structured type data is stored in an array data type, an object data type, or a variant data type.
22. The non-transitory computer-storage medium of claim 19 , wherein the operations further comprise:
receiving a set of query results based on processing the query, the set of query results including a set of structured types.
23. The non-transitory computer-storage medium of claim 22 , wherein the set of query results comprises a JSON string format.
24. The non-transitory computer-storage medium of claim 22 , wherein the set of query results comprises an Apache Arrow format.
25. The non-transitory computer-storage medium of claim 19 , wherein the operations further comprise:
receiving a first semi-structured object;
iterating through a list of fields specified by a target object type;
for each field, determining whether a field with a same name is present in the first semi-structured object; and
in response to the field being found in the first semi-structured object, converting a value of the field to a target field type.
26. The non-transitory computer-storage medium of claim 19 , wherein the operations further comprise:
receiving first data from a first column of a table;
determining that the first column comprises an array structured data type; and
converting the array structured data type to an array semi-structured data type.
27. The non-transitory computer-storage medium of claim 19 , wherein the operations further comprise:
receiving first data from a first column of a table;
determining that the first column comprises an object structured data type; and
converting the object structured data type to an object semi-structured data type.