Large scale unstructured database systems
Systems, methods, and computer-readable media provide techniques for on-demand access to object data hosted in distributed object storage. A query is received for the distributed object storage database, wherein the distributed object storage database includes first object data stored in a first data center at a first geographical location, and second object data stored in a second data center at a second geographical location. It is determined that the query is for at least a portion of the first object data stored by the first data center. The query is transmitted to a first computing device in the first data center. Results are received from the first computing device for the query, and the results are provided in response to the received query.
1. A method of querying data hosted in a distributed database, the method comprising the steps of:
receiving a query from a user device for the distributed database, wherein the distributed database comprises:
first object data stored in a first data center at a first geographical location, wherein the first object data comprises a first set of objects, wherein each object of the first set of objects comprises an associated object name and including a set of ordered fields; and
second object data stored in a second data center at a second geographical location, wherein the second object data comprises a second set of objects, wherein each object of the second set of objects comprises an associated object name and including a set of ordered fields;
determining the query is for at least a portion of the first object data stored by the first data center;
transmitting the query to a first computing device in the first data center;
processing at least a portion of the query using a partition mapping syntax to specify a partition associated with the portion of the query using a range of a field, wherein the partition comprises:
a set of partition objects comprising a subset of a first set of objects in the first object data, a subset of a second set of objects in the second object data, or both;
wherein responsive objects in the set of partition objects comprise an associated object name with a field value within the range of the field;
receiving results from the first computing device for the query; and
providing the results to the user device in response to the received query.
2. The method of claim 1 , wherein the first computing device is configured to:
distribute:
a first portion of the query to a second computing device in the first data center to query a first portion of the first object data; and
a second portion of the query to a third computing device in the first data center to query a second portion of the first object data,
such that the second computing device and the third computing device can read the respective first and second portions of the first object data and perform the respective first and second portion of the query in parallel;
receive results from the second computing device and third computing device responsive to the second computing device and third computing device performing the respective first and second portion of the query; and
combine the results.
3. The method of claim 1 , further comprising accessing a storage configuration file to configure the distributed database, wherein the storage configuration file comprises data indicative of:
a first data store comprising the first object data and a second data store comprising the second object data; and
a collection within the first data store comprising a subset of data from the first data store, a subset of data from the second data store, or both.
4. The method of claim 3 ,
wherein the storage configuration file further comprises data indicative of one or more of:
a third data store comprising an online database cluster; and
a fourth data store comprising a Hypertext Transfer Protocol (HTTP) store; and
the method further comprises processing the storage configuration file to determine a collection comprising the first data store, the second data store, and one or more of the third data store and the fourth data store.
5. The method of claim 4 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the online database cluster.
6. The method of claim 4 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the HTTP store.
7. The method of claim 1 , wherein:
the first set of objects are in a first collection within a first data store;
the second set of objects are in a second collection within the first data store; and
the method further comprises accessing a storage configuration file to configure the distributed database, wherein the storage configuration file:
specifies the first collection using a first keyword comprising a first set of field values that matches the object names of the objects in the first set of objects to determine the first collection; and
specifies the second collection using a second keyword comprising a second set of field values that matches the object names of the objects in the second set of objects to determine the second collection.
8. The method of claim 1 , further comprising writing the results of the query to the distributed database.
9. The method of claim 1 , further comprising writing the results of the query to an online database cluster.
10. A non-transitory computer-readable media comprising instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform:
receiving a query from a user device for the distributed database, wherein the distributed database comprises:
first object data stored in a first data center at a first geographical location, wherein the first object data comprises a first set of objects, wherein each object of the first set of objects comprises an associated object name and including a set of ordered fields; and
second object data stored in a second data center at a second geographical location wherein the second object data comprises a second set of objects, wherein each object of the second set of objects comprises an associated object name and including a set of ordered fields;
determining the query is for at least a portion of the first object data stored by the first data center;
transmitting the query to a first computing device in the first data center;
processing at least a portion of the query using a partition mapping syntax to specify a partition associated with the portion of the query using a range of a field, wherein the partition comprises:
a set of partition objects comprising a subset of a first set of objects in the first object data, a subset of a second set of objects in the second object data, or both;
wherein responsive objects in the set of partition objects comprise an associated object name with a field value within the range of the field;
receiving results from the first computing device for the query; and
providing the results to the user device in response to the received query.
11. The non-transitory computer-readable media of claim 10 , wherein the first computing device is configured to:
distribute:
a first portion of the query to a second computing device in the first data center to query a first portion of the first object data; and
a second portion of the query to a third computing device in the first data center to query a second portion of the first object data,
such that the second computing device and the third computing device can read the respective first and second portions of the first object data and perform the respective first and second portion of the query in parallel;
receive results from the second computing device and third computing device responsive to the second computing device and third computing device performing the respective first and second portion of the query; and
combine the results.
12. The non-transitory computer-readable media of claim 10 , wherein the instructions are further operable to cause the one or more processors to perform accessing a storage configuration file to configure the distributed database, wherein the storage configuration file comprises data indicative of:
a first data store comprising the first object data and a second data store comprising the second object data; and
a collection within the first data store comprising a subset of data from the first data store, a subset of data from the second data store, or both.
13. The non-transitory computer-readable media of claim 12 ,
wherein the storage configuration file further comprises data indicative of one or more of:
a third data store comprising an online database cluster; and
a fourth data store comprising a Hypertext Transfer Protocol (HTTP) store; and
the method further comprises processing the storage configuration file to determine a collection comprising the first data store, the second data store, and one or more of the third data store and the fourth data store.
14. The non-transitory computer-readable media of claim 13 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the online database cluster.
15. The non-transitory computer-readable media of claim 13 , wherein the first computing device is configured to distribute:
a first portion of the query to a second computing device in the first data center to query the first data store; and
a second portion of the query to a third computing device to query the HTTP store.
16. A system comprising a memory storing instructions, and a processor configured to execute the instructions to perform:
receiving a query from a user device for the distributed database, wherein the distributed database comprises:
first object data stored in a first data center at a first geographical location, wherein the first object data comprises a first set of objects and the first set of objects comprises an associated object name and including a set of ordered fields; and
second object data stored in a second data center at a second geographical location wherein the second object data comprises a second set of objects the second set of objects comprises an associated object name and including a set of ordered fields;
determining the query is for at least a portion of the first object data stored by the first data center;
transmitting the query to a first computing device in the first data center;
processing at least a portion of the query using a partition mapping syntax to specify a partition associated with the portion of the query using a range of a field, wherein the partition comprises:
a set of partition objects comprising a subset of a first set of objects in the first object data, a subset of a second set of objects in the second object data, or both;
wherein responsive objects in the set of partition objects comprise an associated object name with a field value within the range of the field;
receiving results from the first computing device for the query; and
providing the results to the user device in response to the received query.