Technique of reducing size of binary JSON with/without compression
Data structures and methods are described for converting a text format data-interchange file into size efficient binary representations. A method comprises receiving a request to convert a data-interchange file, comprising a hierarchy of nodes, into a binary file. The method further comprises generating a tree representation of the nodes that reference a plurality of leaf values. The method further comprises, in response to determining that the binary file is to be compressed, embedding relative node jump offsets when generating the tree representation. The method further comprises, in response to determining that the data-interchange file is immutable, deduplicating the plurality of leaf values in a space optimized manner. The method further comprises, in response to determining that the data-interchange file is mutable, deduplicating the plurality of leaf values in a stream optimized manner. The method further comprises storing the deduplicated plurality of leaf values in the binary file.
1. A method comprising:
receiving a request to convert a data-interchange file, comprising a hierarchy of nodes, into a binary file for use by a database management system;
generating a tree representation of the hierarchy of nodes, wherein the tree representation references a plurality of leaf values;
determining whether the binary file is to be compressed or uncompressed;
in response to determining that the binary file is to be compressed:
embedding relative node jump offsets when generating the tree representation to enable navigation of the hierarchy of nodes; and
storing the tree representation within a compressed container of the binary file; and
in response to determining that the binary file is to be uncompressed:
determining whether the data-interchange file is immutable or mutable;
in response to determining that the data-interchange file is immutable, deduplicating the plurality of leaf values in a space optimized manner such that at least a subset of the plurality of leaf values is unique; and
in response to determining that the data-interchange file is mutable, deduplicating the plurality of leaf values in a stream optimized manner such that adjacent leaf nodes with duplicate leaf values in the tree representation reference a single shared leaf value; and
storing the deduplicated plurality of leaf values in the binary file.
2. The method of claim 1 , wherein the at least a subset of the plurality of leaf values comprises each of the plurality of leaf values.
3. The method of claim 1 , wherein deduplicating the plurality of leaf values in the stream optimized manner includes reserving, in the plurality of leaf values, one or more common leaf values including at least one of: null, true, false, empty string, 0, 1, empty object, and empty array.
4. The method of claim 1 , wherein storing the deduplicated plurality of leaf values in the binary file further includes storing the tree representation in the binary file.
5. The method of claim 1 , further comprising:
deduplicating a plurality of field names from the hierarchy of nodes; and
storing the deduplicated plurality of field names in the binary file.
6. The method of claim 1 , wherein deduplicating the plurality of leaf values in the space optimized manner further includes deduplicating array objects in the tree representation in the space optimized manner, and wherein deduplicating the plurality of leaf values in the stream optimized manner further includes deduplicating array objects in the tree representation in the stream optimized manner.
7. The method of claim 1 , further comprising:
generating a lookup data structure for the plurality of leaf values; and
storing the lookup data structure in the binary file.
8. The method of claim 1 , wherein deduplicating the plurality of leaf values in the stream optimized manner includes a size filter to skip deduplicating leaf values above a size threshold.
9. The method of claim 1 , wherein deduplicating the plurality of leaf values in the stream optimized manner includes tracking a last leaf value for each parent node in the tree representation.
10. The method of claim 1 , wherein the data-interchange file comprises a JavaScript Object Notation (JSON) text file.
11. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause:
receiving a request to convert a data-interchange file, comprising a hierarchy of nodes, into a binary file for use by a database management system;
generating a tree representation of the hierarchy of nodes, wherein the tree representation references a plurality of leaf values;
determining whether the binary file is to be compressed or uncompressed;
in response to determining that the binary file is to be compressed:
embedding relative node jump offsets when generating the tree representation to enable navigation of the hierarchy of nodes; and
storing the tree representation within a compressed container of the binary file; and
in response to determining that the binary file is to be uncompressed:
determining whether the data-interchange file is immutable or mutable;
in response to determining that the data-interchange file is immutable, deduplicating the plurality of leaf values in a space optimized manner such that at least a subset of the plurality of leaf values is unique; and
in response to determining that the data-interchange file is mutable, deduplicating the plurality of leaf values in a stream optimized manner such that adjacent leaf nodes with duplicate leaf values in the tree representation reference a single shared leaf value; and
storing the deduplicated plurality of leaf values in the binary file.
12. The one or more non-transitory storage media of claim 11 , wherein the at least a subset of the plurality of leaf values comprises each of the plurality of leaf values.
13. The one or more non-transitory storage media of claim 11 , wherein deduplicating the plurality of leaf values in the stream optimized manner includes reserving, in the plurality of leaf values, one or more common leaf values including at least one of: null, true, false, empty string, 0, 1, empty object, and empty array.
14. The one or more non-transitory storage media of claim 11 , wherein storing the deduplicated plurality of leaf values in the binary file further includes storing the tree representation in the binary file.
15. The one or more non-transitory storage media of claim 11 , further comprising:
deduplicating a plurality of field names from the hierarchy of nodes; and
storing the deduplicated plurality of field names in the binary file.
16. The one or more non-transitory storage media of claim 11 , wherein deduplicating the plurality of leaf values in the space optimized manner further includes deduplicating array objects in the tree representation in the space optimized manner, and wherein deduplicating the plurality of leaf values in the stream optimized manner further includes deduplicating array objects in the tree representation in the stream optimized manner.
17. The one or more non-transitory storage media of claim 11 , further comprising:
generating a lookup data structure for the plurality of leaf values; and
storing the lookup data structure in the binary file.
18. The one or more non-transitory storage media of claim 11 , wherein deduplicating the plurality of leaf values in the stream optimized manner includes a size filter to skip deduplicating leaf values above a size threshold.
19. The one or more non-transitory storage media of claim 11 , wherein deduplicating the plurality of leaf values in the stream optimized manner includes tracking a last leaf value for each parent node in the tree representation.
20. The one or more non-transitory storage media of claim 11 , wherein the data-interchange file comprises a JavaScript Object Notation (JSON) text file.