Hybrid storage and processing of very large databases
A system and method of storing data in an unstructured or semi-structured database, such as a JSON database, includes defining a columnar store, or hybrid column portion having a set of values for each field, such that each set stores a sequence of values from commonly named fields of each document together, and defining a hybrid store, or hybrid row portion having, for each document in the columnar store, a hybrid entry storing fields from the same document together, such that each entry in the hybrid store includes fields of a corresponding document represented in the columnar store. The hybrid arrangement provides that both the columnar store and the hybrid store are derived from the same set of documents, and each of the columnar store and the hybrid store are configured to fulfill a query request for determining whether to access the columnar store or the hybrid store.
1. In a database having a set of documents, each document having at least one field with a value, a method of storing data comprising:
defining a hybrid column portion having a set of values for each field, each set storing a sequence of values from commonly named fields of each document together wherein the hybrid column portion stores each field in a corresponding file, the file having a sequence of values of a commonly named field for each document in a collection; and
defining a hybrid row portion having, for each document in the hybrid column portion, a hybrid entry in the hybrid row portion, each entry in the hybrid row portion including fields of a corresponding document represented in the hybrid column portion together with the other fields in the document, the hybrid column portion and the hybrid row portion derived from the same set of documents, and each of the hybrid column portion and the hybrid row portion fulfills a query request of the database; and
determining, based on a number of fields needed to fulfill the query request, whether the hybrid column portion or the hybrid row portion can satisfy the query request in the fewest retrieval operations.
2. In a database having a set of documents, each document having at least one field with a value, a method of storing data comprising:
defining a hybrid column portion having a set of values for each field, each set storing a sequence of values from commonly named fields of each document together; and defining a hybrid row portion having, for each document in the hybrid column portion, a hybrid entry in the hybrid row portion, each entry in the hybrid row portion including fields of a corresponding document represented in the hybrid column portion together with the other fields in the document,
the hybrid column portion and the hybrid row portion derived from the same set of documents, and each of the hybrid column portion and the hybrid row portion fulfills a query request of the database; and
determining whether a threshold number of fields are to be retrieved for each document, and retrieving the values from the hybrid row portion if the number of values for retrieval is above the threshold.
3. The method of claim 2 further comprising determining, based on a number of documents sought from the query request and a number of fields from each document, whether to access the hybrid column portion or the hybrid row portion.
4. The method of claim 2 further comprising storing the hybrid column portion and the hybrid row portion together for accessibility as a common database entity for access from the same query operation, wherein the hybrid column portion stores each field in a corresponding file, the file having a sequence of values of a commonly named field for each document in a collection.
5. The method of claim 2 wherein the database has one or more storage volumes, each represented by at least one storage medium, the storage medium configured to encode values in a tangible form for subsequent retrieval.
6. The method of claim 5 wherein the hybrid column portion and the hybrid row portion represent a duplication of values in an alternate storage arrangement on the storage volume.
7. The method of claim 2 further comprising:
defining each document of the set of documents as a hybrid entry; and
storing the hybrid entries as a column in the hybrid column portion, each hybrid entry including all fields of a corresponding document.
8. The method of claim 2 further comprising:
receiving a value of a first data field;
receiving a value of a second data field;
storing the value of the first data field in a first hybrid column portion;
storing the value of the second data field in a second hybrid column portion; and
storing the first data field and the second data field together as a field in a hybrid row portion.
9. The method of claim 3 wherein the set is a collection of documents in a JSON (JavaScript Object Notation) database.
10. A database system for unstructured or semi-structured data, comprising:
a computing processor;
a set of documents, each document having at least one field with a value; a hybrid column portion on a storage medium having a set of values for each field, each set storing a sequence of values from commonly named fields of each document together wherein the hybrid column portion stores each field in a corresponding file, the file having a sequence of values of a commonly named field for each document in a collection;
a hybrid row portion on the storage medium having, for each document in the hybrid column portion, a hybrid entry in the hybrid row portion, each entry in the hybrid row portion including fields of a corresponding document represented in the hybrid column portion together with the other fields in the document, the hybrid column portion and the hybrid row portion both derived from the set of documents, and each of the hybrid column portion and the hybrid row portion fulfills a query request of the database; the query request based on a determination of a number of fields needed to fulfill the query request and whether the hybrid column portion or the hybrid row portion can satisfy the query request in the fewest retrieval operations; and the hybrid column portion and the hybrid row portion coupled for accessibility as a common database entity for access from the same query request operation.
11. The database system of claim 10 further comprising a server having query logic, the database responsive to the query logic for determining, based on a number of documents sought from the query request and a number of fields from each document, whether to access the hybrid column portion or the hybrid row portion.
12. The database system of claim 11 wherein the query logic is further operable to determine whether the hybrid column portion or the hybrid row portion can satisfy the query request in the fewest retrieval operations.
13. The database system of claim 10 further comprising query logic determining whether the hybrid column portion or the hybrid row portion can satisfy the query request in the fewest retrieval operations.
14. The database system of claim 10 further comprising:
a hybrid entry in the hybrid row portion for defining each document of the set of documents, the hybrid entries stored as a column in the hybrid column portion, each hybrid entry including all fields of a corresponding document.
15. The database system of claim 10 wherein the database has one or more storage volumes, each represented by at least one storage medium, the storage medium encodes values in a tangible form for subsequent retrieval, wherein the hybrid column portion and the hybrid row portion represent a duplication of values in an alternate storage arrangement on the storage volume.
16. A computer program product on a non-transitory computer readable storage medium having instructions that, when executed by a processor, perform a method for storing data in an unstructured or semi structured database, the method comprising:
receiving data for a plurality of fields in documents, each field having a value;
storing the values of similarly named fields together as a hybrid column portion;
storing an aggregation of fields from each document together as a hybrid row portion, each of the hybrid column portion and the hybrid row portion fulfills a query request of the database;
defining a hybrid column portion having a sequential file for each commonly named field from across all documents in a collection, the hybrid column portion storing similarly named fields adjacent, and a hybrid row portion having a sequential file for storing each document in the collection, the hybrid row portion storing fields in each document adjacent; and
determining, based on a number of fields needed to fulfill the query request, whether the hybrid column portion or the hybrid row portion can satisfy the query request in the fewest retrieval operations.
17. The method of claim 1 further comprising:
determining a number of retrieval operations from the hybrid row portion to fulfill the query request;
determining a number of retrieval operations from the hybrid column portion to fulfill the query request; and
fulfilling the query request from the portion that fulfills the query request in the fewest retrieval operations.