Large scale unstructured database systems
Systems, methods, and computer-readable media provide techniques for on-demand access to object data hosted in distributed object storage. A query is received for the distributed object storage database, wherein the distributed object storage database includes first object data stored in a first data center at a first geographical location, and second object data stored in a second data center at a second geographical location. It is determined that the query is for at least a portion of the first object data stored by the first data center. The query is transmitted to a first computing device in the first data center. Results are received from the first computing device for the query, and the results are provided in response to the received query.
1. A method comprising the steps of:
accessing a storage configuration file to configure a distributed object storage database, the distributed object storage database comprising:
first object data stored in a first data center at a first geographical location; and
second object data stored in a second data center at a second geographical location,
wherein the storage configuration file comprises data indicative of:
a first data store comprising the first object data and a second data store comprising the second object data, wherein:
the first object data comprises a first set of objects each comprising an associated object name comprising a set of ordered fields; and
the second object data comprises a second set of objects each comprising an associated object name comprising a set of ordered fields; and
a collection within the first data store comprising a subset of data from the first data store, a subset of data from the second data store, or both; and
wherein the storage configuration file specifies the first set of objects and the second set of objects using the associated object names;
receiving a query for the distributed object storage database
determining the query is for at least a portion of the first object data stored by the first data center;
transmitting the query to a first computing device in the first data center; and
receiving results from the first computing device for the query, and providing the results in response to the received query.
2. The method of claim 1 , wherein the first computing device is configured to:
distribute:
a first portion of the query to a second computing device in the first data center to query a first portion of the first object data; and
a second portion of the query to a third computing device in the first data center to query a second portion of the first object data,
such that the second computing device and the third computing device can read the respective first and second portions of the first object data and perform the respective first and second portion of the query in parallel;
receive results from the second computing device and third computing device responsive to the second computing device and third computing device performing the respective first and second portion of the query; and
combine the results.
3. The method of claim 1 ,
wherein the storage configuration file further comprises data indicative of one or more of:
a third data store comprising an online database cluster; and
a fourth data store comprising a Hypertext Transfer Protocol (HTTP) store; and
the method further comprises processing the storage configuration file to determine a collection comprising the first data store, the second data store, and one or more of the third data store and the fourth data store.
4. The method of claim 3 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the online database cluster.
5. The method of claim 3 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the HTTP store.
6. The method of claim 1 , further comprising:
processing at least a portion of the query using a partition mapping syntax to specify a partition associated with the portion of the query using a range of a field, wherein the partition comprises:
a set of partition objects comprising a subset of the first set of objects, a subset of the second set of objects, or both;
wherein each object in the set of partition objects comprises an associated object name with a field value within the range of the field.
7. The method of claim 1 , wherein:
the first set of objects are in a first collection within the first data store;
the second set of objects are in a second collection within the first data store; and
the storage configuration file:
specifies the first collection using a first keyword comprising a first set of field values that matches the object names of the objects in the first set of objects to determine the first collection;
specifies the second collection using a second keyword comprising a second set of field values that matches the object names of the objects in the second set of objects to determine the second collection.
8. The method of claim 1 , wherein receiving the results from the first computing device for the query and providing the results in response to the received query comprises writing the results of the query to the distributed object storage database.
9. The method of claim 1 , wherein receiving the results from the first computing device for the query and providing the results in response to the received query comprises writing the results of the query to an online database cluster.
10. A non-transitory computer-readable media comprising instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform:
accessing a storage configuration file to configure a distributed object storage database, the distributed object storage database comprising:
first object data stored in a first data center at a first geographical location; and
second object data stored in a second data center at a second geographical location;
wherein the storage configuration file comprises data indicative of:
a first data store comprising the first object data and a second data store comprising the second object data, wherein:
the first object data of the first data store comprises a first set of objects, wherein each object of the first set of objects comprises an associated object name comprising a set of ordered fields; and
the second object data of the second data store comprises a second set of objects, wherein each object of the second set of objects comprises an associated object name comprising a set of ordered fields; and
a collection within the first data store comprising a subset of data from the first data store, a subset of data from the second data store, or both; and
wherein the storage configuration file specifies the first set of objects and the second set of objects using the associated object names;
receiving a query for the distributed object storage database
determining the query is for at least a portion of the first object data stored by the first data center;
transmitting the query to a first computing device in the first data center;
receiving results from the first computing device for the query, and providing the results in response to the received query.
11. The non-transitory computer-readable media of claim 10 , wherein the first computing device is configured to:
distribute:
a first portion of the query to a second computing device in the first data center to query a first portion of the first object data; and
a second portion of the query to a third computing device in the first data center to query a second portion of the first object data,
such that the second computing device and the third computing device can read the respective first and second portions of the first object data and perform the respective first and second portion of the query in parallel;
receive results from the second computing device and third computing device responsive to the second computing device and third computing device performing the respective first and second portion of the query; and
combine the results.
12. The non-transitory computer-readable media of claim 10 ,
wherein the storage configuration file further comprises data indicative of one or more of:
a third data store comprising an online database cluster; and
a fourth data store comprising a Hypertext Transfer Protocol (HTTP) store; and
the method further comprises processing the storage configuration file to determine a collection comprising the first data store, the second data store, and one or more of the third data store and the fourth data store.
13. The non-transitory computer-readable media of claim 12 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the online database cluster.
14. The non-transitory computer-readable media of claim 12 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the HTTP store.
15. The non-transitory computer-readable media of claim 10 , wherein the instructions are further operable to cause the one or more processors to perform:
processing at least a portion of the query using a partition mapping syntax to specify a partition associated with the portion of the query using a range of a field, wherein the partition comprises:
a set of partition objects comprising a subset of the first set of objects, a subset of the second set of objects, or both;
wherein each object in the set of partition objects comprises an associated object name with a field value within the range of the field.
16. A system comprising a memory storing instructions, and a processor configured to execute the instructions to perform:
accessing a storage configuration file to configure a distributed object storage database, the distributed object storage database comprising:
first object data stored in a first data center at a first geographical location; and
second object data stored in a second data center at a second geographical location,
wherein the storage configuration file comprises data indicative of:
a first data store comprising the first object data and a second data store comprising the second object data, wherein:
the first object data comprises a first set of objects each comprising an associated object name comprising a set of ordered fields; and
the second object data comprises a second set of objects each comprising an associated object name comprising a set of ordered fields; and
a collection within the first data store comprising a subset of data from the first data store, a subset of data from the second data store, or both; and
wherein the storage configuration file specifies the first set of objects and the second set of objects using the associated object names;
receiving a query for the distributed object storage database
determining the query is for at least a portion of the first object data stored by the first data center;
transmitting the query to a first computing device in the first data center;
receiving results from the first computing device for the query, and providing the results in response to the received query.