Data structure and format for efficient storage or transmission of objects
Data structures for the transmission or storage of records and the efficient serialization and deserialization of such records are disclosed. Embodiments of such a data structure offer a large and flexible data structure and format that may include an object format and a stream format. These data structures may be a packed sequence that may include fields of varying types and encodings with an order determined by the schema of a particular type of record being serialized.
1. A system, comprising:
a processor; and
a memory comprising a data structure for serializing data from data records representing a type of real-world entity, wherein the data representing an entity in an associated data record includes geospatial information associated with a geographic location of the entity, the data structure adapted for use in providing the serialized data from the data records representing real-world entities to requestors to allow dynamic tracking of those real-world entities, the data structure including:
a header portion comprising:
a schema identifier for a schema for the data records representing the type of entity, wherein the schema defines a number of fields for the type of the entity and an encoding of each of the fields to be used for that field in the data structure, and
a version identifier identifying a version of the schema; and
a field sequence portion, comprising multiple streams defining a series of entries corresponding to each of the fields defined by the schema, each entry of the field sequence portion including serialized values from the data records for a corresponding one of the fields for the type entity and including a stream for the corresponding one of the fields that includes serialized values for the corresponding one of the fields from the data records, such that each value for the corresponding one of the fields from each data record representing the type of entity are all included in that entry and are encoded in the stream according to the encoding defined by the schema identified by the schema identifier, and such that each stream is one of a Table stream, a Null stream, a Sequence stream or a Const stream, the Table stream providing compression of data in the field sequence portion by including a first sequence of unique values, and a second sequence of indices into the unique values indicating an order of occurrence in the first sequence, the Sequence stream providing compression of data in the field sequence portion by including a pointer to a buffer of the series using homogeneous encodings, and the Const stream including a single field value, the Const stream providing compression of data in the field sequence portion by deduplication, by indicating that all data records have a same single field value for the corresponding field.
2. The system of claim 1 , wherein the streams are encoded according to different schemas.
3. The system of claim 1 , wherein the field sequence portion includes a stream flag entry identifying a type of each stream.
4. The system of claim 3 , wherein at least one of the streams includes a pointer to a relative address corresponding to the serialized values of that stream.
5. The system of claim 3 , wherein the schema identifier comprises a universally unique identifier (UUID) and a binary length of the data structure.
6. The system of claim 5 , wherein the field sequence portion includes a count of the number of data records included in the data structure.
7. A method for the serialization of data records, comprising:
obtaining data records representing a type of real-world entity, wherein the data representing an entity in an associated data record includes geospatial information associated with a geographic location of the entity;
serializing data from the data records for the type of entity, wherein serializing the data from the data records comprises generating a data structure adapted for use in providing the serialized data from the data records representing real-world entities to requestors to allow dynamic tracking of those real-world entities, the data structure including:
a header portion comprising:
a schema identifier for a schema for the data records representing the type of entity, wherein the schema defines a number of fields for the type of the entity and an encoding of each of the fields to be used for that field in the data structure, and
a version identifier identifying a version of the schema; and
a field sequence portion, comprising multiple streams defining a series of entries corresponding to each of the fields defined by the schema, each entry of the field sequence portion including serialized values from the data records for a corresponding one of the fields for the type entity and including a stream for the corresponding one of the fields that includes serialized values for the corresponding one of the fields from the data records, such that each value for the corresponding one of the fields from each data record representing the type of entity are all included in that entry and are encoded in the stream according to the encoding defined by the schema identified by the schema identifier, and such that each stream is one of a Table stream, a Null stream, a Sequence stream or a Const stream, the Table stream providing compression of data in the field sequence portion by including a first sequence of unique values, and a second sequence of indices into the unique values indicating an order of occurrence in the first sequence, the Sequence stream providing compression of data in the field sequence portion by including a pointer to a buffer of the series using homogeneous encodings, and the Const stream including a single field value, the Const stream providing compression of data in the field sequence portion by deduplication, by indicating that all data records have a same single field value for the corresponding field.
8. The method of claim 7 , wherein the streams are encoded according to different schemas.
9. The method of claim 7 , wherein the field sequence portion includes a stream flag entry identifying a type of the streams.
10. The method of claim 9 , wherein at least one of the streams includes a pointer to a relative address corresponding to the serialized values of that stream.
11. The method of claim 9 , wherein the schema identifier comprises a universally unique identifier (UUID) and a binary length of the data structure.
12. The method of claim 11 , wherein the field sequence portion includes a count of the number of data records included in the data structure.
13. A non-transitory computer readable medium comprising instructions for the serialization of data records, comprising instructions for:
obtaining data records representing a type of real-world entity, wherein the data representing an entity in an associated data record includes geospatial information associated with a geographic location of the entity;
serializing data from the data records for the type of entity, wherein serializing the data from the data records comprises generating a data structure adapted for use in providing the serialized data from the data records representing real-world entities to requestors to allow dynamic tracking of those real-world entities, the data structure including:
a header portion comprising:
a schema identifier for a schema for the data records representing the type of entity, wherein the schema defines a number of fields for the type of the entity and an encoding of each of the fields to be used for that field in the data structure, and
a version identifier identifying a version of the schema; and
a field sequence portion, comprising multiple streams defining a series of entries corresponding to each of the fields defined by the schema, each entry of the field sequence portion including serialized values from the data records for a corresponding one of the fields for the type entity and including a stream for the corresponding one of the fields that includes serialized values for the corresponding one of the fields from the data records, such that each value for the corresponding one of the fields from each data record representing the type of entity are all included in that entry and are encoded in the stream according to the encoding defined by the schema identified by the schema identifier, and such that each stream is one of a Table stream, a Null stream, a Sequence stream or a Const stream, the Table stream providing compression of data in the field sequence portion by including a first sequence of unique values, and a second sequence of indices into the unique values indicating an order of occurrence in the first sequence, the Sequence stream providing compression of data in the field sequence portion by including a pointer to a buffer of the series using homogeneous encodings, and the Const stream including a single field value, the Const stream providing compression of data in the field sequence portion by deduplication, by indicating that all data records have a same single field value for the corresponding field.
14. The non-transitory computer readable medium of claim 13 , wherein the streams are encoded according to different schemas.
15. The non-transitory computer readable medium of claim 13 , wherein the field sequence portion includes a stream flag entry identifying a type of the streams.
16. The non-transitory computer readable medium of claim 15 , wherein at least one of the streams includes a pointer to a relative address corresponding to the serialized values of that stream.
17. The non-transitory computer readable medium of claim 15 , wherein the schema identifier comprises a universally unique identifier (UUID) and a binary length of the data structure.
18. The non-transitory computer readable medium of claim 17 , wherein the field sequence portion includes a count of the number of data records included in the data structure.