Query usage based organization for very large databases
A query server identifies data collections of interest in a cloud store, and categorizes the collections based on an intended usage. Depending on the intended usage, the categorized data may be cataloged, indexed, or undergo a full intake into a column store. In a database of large data collections, some collections may experience sparse or indefinite usage. Cataloging or indexing position the collections for subsequent query access, but defers the computational burden. The full intake performs a columnar shredding of the collection for facilitating eminent and regular query access. Upon invocation of query activity, an instantiation of virtual machines provided by the cloud store vendor implements query logic, such that the VMs launch in conjunction with the cloud store having the collections. Collections therefore incur processing based on their expected usage-full intake for high query traffic collections, and reduced cataloging for maintaining accessibility of collections of indefinite query interest.
1. A method for defining large databases, comprising:
receiving an identification of a collection of data in a cloud store, the cloud store accessible via a public access network for storing data in a location independent manner;
determining, based on an expected usage of the collection, whether to intake the collection into an object store, the object store having a predefined format under the control of the user;
selecting, if the collection remains in the cloud store, a catalog organization for the collection, the catalog organization defining a structure for accessing the data in the manner called for by the expected usage, the expected usage defining a computational intensity and storage demand of a query directed to the collection; and
generating the catalog organization by associating elements of the collection with a created entry in the catalog organization, wherein selecting the catalog organization includes:
determining an intended query usage and:
generating a name and source, if the query usage is not foreseeable;
generating an index on a subset of fields, if a keyword lookup query is expected; and
generating a columnar organization of the data, if data analytics are expected.
2. The method of claim 1 , further comprising:
performing, if the expected usage involves a transformation of the collection from the cloud store, the intake on the collection, the intake transferring the identified collection from the cloud store to the object store, the object store having a columnar format.
3. The method of claim 1 wherein a workload defines the expected use, the workload based on a likelihood of an eminent query of the collection and whether the query includes a lookup or online analytical processing (OLAP).
4. The method of claim 1 wherein the expected usage indicates an indefinite need for use of the collection and the intake defines acceptable delays in successive query responses based on the collection, and the catalog organization includes a file name and source location of the collection.
5. The method of claim 1 wherein the expected usage is for individual field value based lookup and the catalog organization includes an index creation on the individual fields available for query.
6. The method of claim 1 wherein the expected usage is for analytical queries and the catalog organization includes a columnar arrangement of named fields in the collection.
7. The method of claim 1 wherein the catalog organization further includes a defer flag indicative of whether to defer creation of the catalog organization until receipt of a query request.
8. The method of claim 1 further comprising:
receiving a query request directed to the collection, the collection residing on the cloud store;
partitioning the collection into a plurality of partitions, each partition containing a subset of elements from the collection;
allocating a plurality of virtual machines (VMs) in a virtualization environment, each VM assigned to a partition of the plurality of partitions; and
performing at least a portion of the query on each VM until limited by dependencies between the partitions.
9. The method of claim 8 , further comprising:
receiving a query and an identifier of a collection;
determining the catalog organization of the collection; and
transmitting the query, an indication of the catalog organization, and a set of instructions for performing the query to each of the plurality of VMs for performing the query.
10. The method of claim 9 wherein the collection further comprises a plurality of collections, and performing the query includes performing a join on at least two of the collections.
11. The method of claim 8 wherein each VM has a computing speed and a bandwidth, further comprising determining the plurality of VMs based on a cost of each VM and a number of VMs for accommodating the partitions.
12. The method of claim 1 wherein the columnar format stores values based on named fields.