Handling semi-structured and unstructured data in a sharded database environment

1. A method comprising:

receiving, at a catalog server, a database command that references items in a relational table that has been sharded into a plurality of shards;

wherein each shard of the plurality of shards is assigned to a corresponding shard server of a plurality of shard servers;

determining, at the catalog server, that execution of the database command requires a particular operation;

wherein the particular operation is one of:

an operation specified by a semi-structured SQL operator in the database command, wherein the semi-structured SQL operator is one of:

JSON_EXISTS,

XML_EXISTS,

JSON_VALUE, or

JSON_QUERY;

an operation that creates an index from data extracted from semi-structured items in the relational table, and wherein during performance of the particular operation, each shard server of the plurality of shard servers creates a shard-specific index;

an operation that specifies a relational view of semi-structured data extracted from semi-structured items in the relational table, wherein the catalog server maintains a view definition of the relational view;

an operation that specifies generation of a data guide based on semi-structured items in the relational table, wherein during performance of the particular operation, each shard server of the plurality of shard servers independently generates a shard-specific data guide;

an operation that specifies a semi-structured data view of relational data stored in the relational table, wherein the operation specifies an operator that is one of:

JSON_OBJECT( ),

JSON_ARRAY( ),

JSON_OBJECTAGG( ),

JSON_ARRAYAGG( ),

XMLElement( ),

XMLForest( ),

XMLConcat( ),

XMLAgg( ),

XMLComment( ), and

XMLPI( );

sending a respective remote command to each shard server of the plurality of shard servers that is involved in the particular operation to cause each shard server that is involved in the particular operation to perform a respective portion of the particular operation on items in its respective shard.

2. The method of claim 1 , wherein:

the particular operation is an operation specified by a semi-structured SQL operator in the database command; and

the semi-structured SQL operator is one of:

JSON_EXISTS,

XML_EXISTS,

JSON_VALUE, or

JSON_QUERY.

3. The method of claim 1 , wherein:

the particular operation is an operation that creates an index from data extracted from semi-structured items in the relational table; and

each shard server of the plurality of shard servers creates a shard-specific index in response to its respective remote command.

4. The method of claim 1 , wherein:

the particular operation is an operation that specifies a relational view of semi-structured data extracted from semi-structured items in the relational table; and

the catalog server maintains the view definition of the relational view.

5. The method of claim 1 , wherein:

the particular operation is an operation that specifies generation of a data guide based on semi-structured items in the relational table; and

each shard server of the plurality of shard servers independently generates a shard-specific data guide in response to the its respective remote command.

6. The method of claim 1 , wherein:

the particular operation is an operation, specified by an operator, to generate a semi-structured data view of relational data stored in the relational table; and

the operator is one of:

JSON_OBJECT( ),

JSON_ARRAY( ),

JSON_OBJECTAGG( ),

JSON_ARRAYAGG( ),

XMLElement( ),

XMLForest( ),

XMLConcat( ),

XMLAgg( ),

XMLComment( ), and

XMLPI( ).

7. The method of claim 1 , further comprising:

receiving, at the catalog server, a second database command that references items in the relational table;

based on (a) a predicate of the second database command that references an attribute of semi-structured data, and (b) information obtained from data guides associated with each shard of the plurality of shards, determining that only a subset of shards of the plurality of shards could possibly have items that satisfy the predicate; and

the catalog server responding to the second database command by sending respective remote commands to only the shard servers, of the plurality of shard servers, that correspond to the subset of shards.

8. The method of claim 7 , wherein determining that only a subset of shards of the plurality of shards could possibly have items that satisfy the predicate includes determining, based on the information obtained from the data guides, that one or more shards of the plurality of shards do not have the attribute referenced in the predicate of the second database command.

9. The method of claim 7 , wherein the information obtained from the data guides is obtained dynamically by sending a data guide request to each shard server, of the plurality of shard servers, in response to receiving the second database command.

10. The method of claim 7 , wherein the information obtained from the data guides is obtained from a persistent set of one of more data guides that are constantly maintained up-to-date.

11. The method of claim 10 , wherein the persistent set of one or more data guides includes a global data guide that is kept in sync with a plurality of shard-specific data guides.

12. The method of claim 10 , wherein the persistent set of one or more data guides includes a global data guide that is allowed to go stale and is updated only when certain update conditions are satisfied.

13. One or more non-transitory computer-readable media storing instructions which, when executed by one or more computing devices, cause:

receiving, at a catalog server, a database command that references items in a relational table that has been sharded into a plurality of shards;

wherein each shard of the plurality of shards is assigned to a corresponding shard server of a plurality of shard servers;

determining, at the catalog server, that execution of the database command requires a particular operation;

wherein the particular operation is one of:

an operation specified by a semi-structured SQL operator in the database command, wherein the semi-structured SQL operator is one of:

JSON_EXISTS,

XML_EXISTS,

JSON_VALUE, or

JSON_QUERY;

an operation that creates an index from data extracted from semi-structured items in the relational table, and wherein during performance of the particular operation, each shard server of the plurality of shard servers creates a shard-specific index;

an operation that specifies a relational view of semi-structured data extracted from semi-structured items in the relational table, wherein the catalog server maintains a view definition of the relational view;

an operation that specifies generation of a data guide based on semi-structured items in the relational table, wherein during performance of the particular operation, each shard server of the plurality of shard servers independently generates a shard-specific data guide;

an operation that specifies a semi-structured data view of relational data stored in the relational table, wherein the operation specifies an operator that is one of:

JSON_OBJECT( ),

JSON_ARRAY( ),

JSON_OBJECTAGG( ),

JSON_ARRAYAGG( ),

XMLElement( ),

XMLForest( ),

XMLConcat,( ),

XMLAgg( ),

XMLComment( ), and

XMLPI( ),

sending a respective remote command to each shard server of the plurality of shard servers that is involved in the particular operation to cause each shard server that is involved in the particular operation to perform a respective portion of the particular operation on items in its respective shard.

14. The one or more non-transitory computer-readable media of claim 13 , wherein the instructions comprise instructions for:

receiving, at the catalog server, a second database command that references items in the relational table;

based on (a) a predicate of the second database command that references an attribute of semi-structured data, and (b) information obtained from data guides associated with each shard of the plurality of shards, determining that only a subset of shards of the plurality of shards could possibly have items that satisfy the predicate; and

the catalog server responding to the second database command by sending remote commands to only the shard servers, of the plurality of shard servers, that correspond to the subset of shards.

15. The one or more non-transitory computer-readable media of claim 14 , wherein determining that only a subset of shards of the plurality of shards could possibly have items that satisfy the predicate includes determining, based on the information obtained from the data guides, that one or more shards of the plurality of shards do not have the attribute referenced in the predicate of the second database command.

16. The one or more non-transitory computer-readable media of claim 14 , wherein the information obtained from the data guides is obtained dynamically by sending a data guide request to each shard server, of the plurality of shard servers, in response to receiving the second database command.

17. The one or more non-transitory computer-readable media of claim 14 , wherein the information obtained from the data guides is obtained from a persistent set of one of more data guides that are constantly maintained up-to-date.

18. The one or more non-transitory computer-readable media of claim 17 , wherein the persistent set of one or more data guides includes a global data guide that is kept in sync with a plurality of shard-specific data guides.

19. The one or more non-transitory computer-readable media of claim 17 , wherein the persistent set of one or more data guides includes a global data guide that is allowed to go stale and is updated only when certain update conditions are satisfied.

20. The one or more non-transitory computer-readable media of claim 13 , wherein the instructions comprise instructions for:

in response to sending the respective remote command to each shard server of the plurality of shard servers, receiving data from each shard server of the plurality of shard servers and transforming all data received from the plurality of shard servers.