Handling semi-structured and unstructured data in a sharded database environment
Techniques are described herein for leveraging and extending relational data sharding infrastructure to provide comprehensive support for sharding semi-structured and unstructured data, in addition to the relational data, in a single sharded database system. Specifically, techniques are described in which a catalog server pre-gathers all data it needs to answer fetches from a dynamically created temporary lob. In addition, techniques are provided in which a catalog server encodes shard identifiers in lob locators sent to the client. Techniques are provided for pushing down various operators to shard servers.
1. A method comprising:
receiving, at a catalog server from a requesting entity, a cross-shard query;
in response to receiving the cross-shard query, the catalog server sending remote commands to a plurality of shard servers;
in response to sending the remote commands, receiving at the catalog server a lob locator from each shard server of the plurality of shard servers;
wherein the plurality of shard servers includes a particular shard server that provided a particular lob locator to the catalog server in response to the remote commands;
the catalog server encoding the particular lob locator with a shard identifier to produce an encoded lob locator that identifies the particular shard server;
responding to the cross-shard query by providing the encoded lob locator to the requesting entity;
receiving, at the catalog server from the requesting entity, a fetch request associated with the encoded lob locator;
based on the shard identifier in the encoded lob locator, determining at the catalog server that the fetch request is to be pushed to the particular shard server; and
the catalog server pushing the fetch request to the particular shard server.
2. The method of claim 1 , wherein the lob locator comprises information used to retrieve unstructured data.
3. The method of claim 1 , further comprising, in response to receiving the fetch request, extracting the shard identifier in the encoded lob locator.
4. The method of claim 1 , wherein the cross-shard query indicates a particular results type to the cross-shard query.
5. The method of claim 4 , wherein the particular results type is a by-reference type.
6. The method of claim 4 , further comprising:
in response to receiving the cross-shard query, the catalog server sending second remote commands to a second plurality of shard servers;
in response to sending the second remote commands to the second plurality of shard servers, the catalog server receiving data from each shard server of the second plurality of shard servers;
wherein the second plurality of shard servers includes a second particular shard server that provided particular data to the catalog server in response to the second remote commands;
responding to the cross-shard query by providing the particular data to the requesting entity.
7. The method of claim 6 , wherein the particular data comprises unstructured data.
8. The method of claim 4 , further comprising:
in response to receiving the cross-shard query, the catalog server sending a remote command to a second particular shard server;
in response to sending the remote command to the second particular shard server, the catalog server receiving data from the second particular shard server;
responding to the cross-shard query by providing the particular data to the requesting entity.
9. The method of claim 8 , wherein the plurality of shard servers does not include the second particular shard server.
10. The method of claim 8 , wherein the data comprises unstructured data.
11. One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:
receiving, at a catalog server from a requesting entity, a cross-shard query;
in response to receiving the cross-shard query, the catalog server sending remote commands to a plurality of shard servers;
in response to sending the remote commands, receiving at the catalog server a lob locator from each shard server of the plurality of shard servers;
wherein the plurality of shard servers includes a particular shard server that provided a particular lob locator to the catalog server in response to the remote commands;
the catalog server encoding the particular lob locator with a shard identifier to produce and encoded lob locator that identifies the particular shard server;
responding to the cross-shard query by providing the encoded lob locator to the requesting entity;
receiving, at the catalog server from the requesting entity, a fetch request associated with the encoded lob locator;
based on the shard identifier in the encoded lob locator, determining at the catalog server that the fetch request is to be pushed to the particular shard server; and
the catalog server pushing the fetch request to the particular shard server.
12. The one or more non-transitory computer-readable media of claim 11 , wherein the lob locator comprises information used to retrieve unstructured data.
13. The one or more non-transitory computer-readable media of claim 11 , wherein the instructions comprise instructions for, in response to receiving the fetch request, extracting the shard identifier in the encoded lob locator.
14. The one or more non-transitory computer-readable media of claim 11 , wherein the cross-shard query indicates a particular results type to the cross-shard query.
15. The one or more non-transitory computer-readable media of claim 14 , wherein the particular results type is a by-reference type.
16. The one or more non-transitory computer-readable media of claim 11 , wherein the instructions comprise instructions for:
in response to receiving the cross-shard query, the catalog server sending second remote commands to a second plurality of shard servers;
in response to sending the second remote commands to the second plurality of shard servers, the catalog server receiving data from each shard server of the second plurality of shard servers;
wherein the second plurality of shard servers includes a second particular shard server that provided particular data to the catalog server in response to the second remote commands;
responding to the cross-shard query by providing the particular data to the requesting entity.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the particular data comprises unstructured data.
18. The one or more non-transitory computer-readable media of claim 11 , wherein the instructions comprise instructions for:
in response to receiving the cross-shard query, the catalog server sending a remote command to a second particular shard server;
in response to sending the remote command to the second particular shard server, the catalog server receiving data from the second particular shard server;
responding to the cross-shard query by providing the particular data to the requesting entity.
19. The one or more non-transitory computer-readable media of claim 18 , wherein the plurality of shard servers does not include the second particular shard server.
20. The one or more non-transitory computer-readable media of claim 18 , wherein the data comprises unstructured data.